2024-05-09T08:04:57.545 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-09T08:04:57.545 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/orig.config.yaml 2024-05-09T08:04:57.609 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T08:04:57.703 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7698662' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: be85c202969d25af6e130678e7d73a20e9c4ca11 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2334 sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc sleep_before_teardown: 0 subset: 359/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: be85c202969d25af6e130678e7d73a20e9c4ca11 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMHBIq3GGshNY65sHkAumqYoJbrsfwbEG/PpJI39TKdBB57T+2C8fuvrUiMY00zvBV0r+7ih4ruPVkXX/XM0GKg= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLz4MksoomatCa3JC/zdLKZkqG/7hq+3vKxPXA7GPD3W6Dun7A2J9yM9Jw0rPExTwHSz7kQDpN3SB/4XhxHbr7k= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpjG8knIJPj/tgexreKisoANPHcYa8Xpu229akHay7Z5wPPa9xwlwivgLqePebWBYqHxev4I2pfLemqRCDMQys= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_21:24:04 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-09T08:04:57.703 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa; will attempt to use it 2024-05-09T08:04:57.704 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks 2024-05-09T08:04:57.704 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-09T08:04:57.705 INFO:teuthology.task.internal:Checking packages... 2024-05-09T08:04:57.728 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc' 2024-05-09T08:04:57.729 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-09T08:04:57.729 INFO:teuthology.packaging:ref: None 2024-05-09T08:04:57.729 INFO:teuthology.packaging:tag: None 2024-05-09T08:04:57.729 INFO:teuthology.packaging:branch: squid 2024-05-09T08:04:57.729 INFO:teuthology.packaging:sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:04:57.729 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-09T08:04:58.046 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2529.gbf77c77e 2024-05-09T08:04:58.047 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-09T08:04:58.063 INFO:teuthology.task.internal:no buildpackages task found 2024-05-09T08:04:58.063 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-09T08:04:58.108 INFO:teuthology.task.internal:Saving configuration 2024-05-09T08:04:58.130 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-09T08:04:58.141 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T08:04:58.172 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 07:59:54.855068', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMHBIq3GGshNY65sHkAumqYoJbrsfwbEG/PpJI39TKdBB57T+2C8fuvrUiMY00zvBV0r+7ih4ruPVkXX/XM0GKg='} 2024-05-09T08:04:58.194 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 07:59:54.856495', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLz4MksoomatCa3JC/zdLKZkqG/7hq+3vKxPXA7GPD3W6Dun7A2J9yM9Jw0rPExTwHSz7kQDpN3SB/4XhxHbr7k='} 2024-05-09T08:04:58.217 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 07:59:54.853376', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpjG8knIJPj/tgexreKisoANPHcYa8Xpu229akHay7Z5wPPa9xwlwivgLqePebWBYqHxev4I2pfLemqRCDMQys='} 2024-05-09T08:04:58.217 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-09T08:04:58.224 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-09T08:04:58.224 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-09T08:04:58.224 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-09T08:04:58.224 INFO:teuthology.run_tasks:Running task console_log... 2024-05-09T08:04:58.395 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1b338b2160>, signals=[15]) 2024-05-09T08:04:58.395 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-09T08:04:58.401 INFO:teuthology.task.internal:Opening connections... 2024-05-09T08:04:58.401 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:04:58.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:04:58.488 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-05-09T08:04:58.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:04:58.562 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-05-09T08:04:58.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:04:58.639 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-09T08:04:58.648 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-05-09T08:04:58.667 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-05-09T08:04:58.668 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-05-09T08:04:58.725 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-05-09T08:04:58.726 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-05-09T08:04:58.726 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-05-09T08:04:58.726 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:04:58.726 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-05-09T08:04:58.726 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:04:58.726 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:04:58.726 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-05-09T08:04:58.726 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:04:58.727 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:04:58.727 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:04:58.727 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:04:58.727 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:04:58.727 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:04:58.727 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-05-09T08:04:58.754 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-05-09T08:04:58.772 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-05-09T08:04:58.772 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-05-09T08:04:58.829 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:04:58.830 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:04:58.831 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:04:58.831 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:04:58.831 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:04:58.831 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-05-09T08:04:58.854 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-05-09T08:04:58.872 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-05-09T08:04:58.873 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-05-09T08:04:58.930 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-05-09T08:04:58.931 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2024-05-09T08:04:58.931 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-05-09T08:04:58.931 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:04:58.931 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2024-05-09T08:04:58.931 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:04:58.931 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:04:58.931 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-05-09T08:04:58.932 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:04:58.932 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:04:58.932 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:04:58.932 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:04:58.932 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:04:58.932 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:04:58.932 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-05-09T08:04:58.953 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-09T08:04:58.964 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-09T08:04:58.991 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-09T08:04:58.991 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:04:58.994 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:04:58.996 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:04:59.012 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-09T08:04:59.033 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-09T08:04:59.033 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:04:59.052 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:04:59.054 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:04:59.152 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-09T08:04:59.227 INFO:teuthology.run_tasks:Running task kernel... 2024-05-09T08:04:59.244 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-05-09T08:04:59.244 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-05-09T08:04:59.244 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-05-09T08:04:59.245 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:04:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:04:59.263 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-05-09T08:04:59.322 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-05-09T08:04:59.323 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-437.el9.x86_64 2024-05-09T08:04:59.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-05-09T08:05:02.495 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.1 MB 00:01 2024-05-09T08:05:17.294 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-05-09T08:05:21.786 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.2 MB 00:00 2024-05-09T08:05:23.204 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-05-09T08:05:25.388 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-05-09T08:05:30.796 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-05-09T08:05:32.928 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T08:05:32.928 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T08:05:33.018 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-05-09T08:05:33.023 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-05-09T08:05:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:05:33.025 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 78 M 2024-05-09T08:05:33.025 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 123 M 2024-05-09T08:05:33.025 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-05-09T08:05:33.298 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 2.1 MB/s | 484 kB 00:00 2024-05-09T08:05:35.356 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 6.7 MB/s | 15 MB 00:02 2024-05-09T08:05:37.456 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 7.7 MB/s | 34 MB 00:04 2024-05-09T08:05:40.165 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 4.1 MB/s | 28 MB 00:06 2024-05-09T08:05:40.165 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:05:40.165 INFO:teuthology.orchestra.run.smithi028.stdout:Total 11 MB/s | 78 MB 00:07 2024-05-09T08:05:40.876 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-05-09T08:05:41.280 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-05-09T08:05:41.280 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-05-09T08:05:42.231 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-05-09T08:05:42.232 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-05-09T08:05:44.786 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-05-09T08:05:45.422 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:05:45.506 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:05:48.321 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:05:48.468 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:05:53.549 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:05:53.588 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:05:58.679 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:06:29.861 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:06:29.901 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:06:30.269 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:06:30.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:06:30.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:06:30.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:06:31.074 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:06:31.075 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:06:31.075 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-05-09T08:06:31.075 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T08:06:31.075 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T08:06:31.075 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T08:06:31.075 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T08:06:31.075 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:06:31.075 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-05-09T08:06:31.538 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:06:31.610 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:06:31.610 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:06:31.610 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T08:06:31.611 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T08:06:31.611 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T08:06:31.611 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-05-09T08:06:31.611 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T08:06:31.611 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:06:31.663 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:06:31.663 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:06:31.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-05-09T08:06:31.695 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-05-09T08:06:31.695 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-05-09T08:06:31.695 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-05-09T08:06:31.695 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-05-09T08:06:31.695 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-05-09T08:06:31.695 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T08:06:31.695 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-05-09T08:06:31.695 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-05-09T08:06:31.695 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T08:06:31.696 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T08:06:31.697 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-05-09T08:06:31.697 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:06:31.697 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T08:06:31.698 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T08:06:31.698 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-444.el9.x86_64 2024-05-09T08:06:31.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T08:06:32.373 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-05-09T08:06:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T08:06:33.759 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-05-09T08:06:33.761 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-05-09T08:06:33.785 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.0NwEo4HXWS 2024-05-09T08:06:33.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0NwEo4HXWS 2024-05-09T08:06:33.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.0NwEo4HXWS 2024-05-09T08:06:33.998 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.0NwEo4HXWS is 6KB 2024-05-09T08:06:34.018 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.0NwEo4HXWS 2024-05-09T08:06:34.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T08:06:34.101 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T08:06:34.101 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T08:06:34.101 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T08:06:34.101 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T08:06:34.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T08:06:34.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-05-09T08:06:34.338 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-05-09T08:07:04.254 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-09T08:07:04.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-09T08:07:04.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:07:22.659 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-05-09T08:07:31.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-09T08:07:31.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:07:50.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-05-09T08:08:02.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-09T08:08:02.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:08:02.771 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-05-09T08:08:03.151 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-09T08:08:03.151 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-444.el9.x86_64"... 2024-05-09T08:08:03.151 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-05-09T08:08:03.167 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-444.el9.x86_64 2024-05-09T08:08:03.167 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T08:08:03.167 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T08:08:03.167 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T08:08:04.169 DEBUG:teuthology.parallel:result is None 2024-05-09T08:08:04.169 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-09T08:08:04.176 INFO:teuthology.task.internal:Creating test directory... 2024-05-09T08:08:04.177 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T08:08:04.179 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T08:08:04.182 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T08:08:04.200 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-09T08:08:04.214 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-09T08:08:04.264 INFO:teuthology.task.internal:Creating archive directory... 2024-05-09T08:08:04.264 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T08:08:04.267 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T08:08:04.269 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T08:08:04.374 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-09T08:08:04.380 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-09T08:08:04.381 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:08:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:08:04.397 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:08:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:08:04.415 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T08:08:04.417 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T08:08:04.440 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T08:08:04.469 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:08:04.488 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:08:04.502 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:08:04.505 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:08:04.520 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:08:04.522 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:08:04.524 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-09T08:08:04.531 INFO:teuthology.task.internal:Configuring sudo... 2024-05-09T08:08:04.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T08:08:04.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T08:08:04.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T08:08:04.575 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-09T08:08:04.584 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-09T08:08:04.584 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T08:08:04.600 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T08:08:04.615 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T08:08:04.632 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:08:04.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:08:04.746 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:08:04.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:08:04.871 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:08:04.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T08:08:04.963 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:08:05.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:08:05.058 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:08:05.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:08:05.184 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:08:05.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T08:08:05.249 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:08:05.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:08:05.356 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:08:05.400 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:08:05.440 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:08:05.440 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T08:08:05.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-05-09T08:08:05.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-05-09T08:08:05.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-05-09T08:08:05.542 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T08:08:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T08:08:05.584 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T08:08:05.897 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-09T08:08:05.905 INFO:teuthology.task.internal:Starting timer... 2024-05-09T08:08:05.905 INFO:teuthology.run_tasks:Running task pcp... 2024-05-09T08:08:05.950 INFO:teuthology.run_tasks:Running task selinux... 2024-05-09T08:08:06.000 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-09T08:08:06.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-05-09T08:08:06.081 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-05-09T08:08:06.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-05-09T08:08:06.162 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-05-09T08:08:06.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2024-05-09T08:08:06.240 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2024-05-09T08:08:06.242 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-09T08:08:06.242 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-05-09T08:08:06.282 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-05-09T08:08:06.283 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-05-09T08:08:06.314 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-05-09T08:08:06.314 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2024-05-09T08:08:06.415 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2024-05-09T08:08:06.416 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi184.front.sepia.ceph.com': 'permissive'} 2024-05-09T08:08:06.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T08:08:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:08:06.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T08:08:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:08:06.471 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T08:08:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:08:06.496 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-09T08:08:06.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-05-09T08:08:06.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-05-09T08:08:06.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2024-05-09T08:08:06.695 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-09T08:08:06.705 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-09T08:08:06.713 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-09T08:08:06.839 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-09T08:08:06.854 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-09T08:08:06.856 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi165.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-09T08:12:37.167 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2024-05-09T08:12:37.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-09T08:12:37.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:12:37.249 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-05-09T08:12:37.331 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-09T08:12:37.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-05-09T08:12:37.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:12:37.406 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-05-09T08:12:37.481 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-05-09T08:12:37.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-05-09T08:12:37.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:12:37.555 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-05-09T08:12:37.598 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-05-09T08:12:37.598 INFO:teuthology.run_tasks:Running task clock... 2024-05-09T08:12:37.610 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-09T08:12:37.611 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T08:12:37.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T08:12:37.614 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T08:12:37.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T08:12:37.616 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T08:12:37.616 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T08:12:37.641 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T08:12:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T08:12:37.654 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T08:12:37.663 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T08:12:37.667 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T08:12:37.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-05-09T08:12:37.681 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T08:12:37.687 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-05-09T08:12:37.690 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-05-09T08:12:37.700 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T08:12:37.702 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-05-09T08:12:37.704 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2024-05-09T08:12:37.713 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T08:12:37.715 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2024-05-09T08:12:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T08:12:37.728 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T08:12:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T08:12:37.741 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T08:12:37.769 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-05-09T08:12:37.773 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T08:12:37.773 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-05-09T08:12:37.773 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.774 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.774 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.774 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.792 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-05-09T08:12:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-05-09T08:12:37.795 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T08:12:37.795 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-05-09T08:12:37.795 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.795 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.795 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.795 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.797 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T08:12:37.797 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-05-09T08:12:37.797 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.797 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.797 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.797 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:12:37.797 INFO:teuthology.run_tasks:Running task install... 2024-05-09T08:12:37.806 DEBUG:teuthology.task.install:project ceph 2024-05-09T08:12:37.806 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc'}} 2024-05-09T08:12:37.806 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc'} 2024-05-09T08:12:37.806 INFO:teuthology.task.install:Using flavor: default 2024-05-09T08:12:37.812 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-09T08:12:37.812 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-09T08:12:37.812 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T08:12:37.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:12:37.815 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T08:12:37.815 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:12:37.816 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T08:12:37.816 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:12:37.968 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T08:12:37.969 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T08:12:37.974 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T08:12:37.975 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T08:12:38.045 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T08:12:38.045 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T08:12:38.148 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T08:12:38.148 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:12:38.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T08:12:38.150 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T08:12:38.151 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:12:38.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T08:12:38.175 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T08:12:38.175 DEBUG:teuthology.orchestra.run.smithi165:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T08:12:38.177 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T08:12:38.177 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T08:12:38.218 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T08:12:38.218 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:12:38.218 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T08:12:38.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T08:12:38.244 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T08:12:38.245 DEBUG:teuthology.orchestra.run.smithi184:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T08:12:38.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T08:12:38.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T08:12:38.316 DEBUG:teuthology.orchestra.run.smithi165:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T08:12:38.322 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T08:12:38.378 INFO:teuthology.orchestra.run.smithi165.stdout:check_obsoletes = 1 2024-05-09T08:12:38.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum clean all 2024-05-09T08:12:38.384 DEBUG:teuthology.orchestra.run.smithi184:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T08:12:38.387 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-05-09T08:12:38.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-05-09T08:12:38.446 INFO:teuthology.orchestra.run.smithi184.stdout:check_obsoletes = 1 2024-05-09T08:12:38.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum clean all 2024-05-09T08:12:38.606 INFO:teuthology.orchestra.run.smithi165.stdout:54 files removed 2024-05-09T08:12:38.630 INFO:teuthology.orchestra.run.smithi028.stdout:54 files removed 2024-05-09T08:12:38.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T08:12:38.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T08:12:38.678 INFO:teuthology.orchestra.run.smithi184.stdout:54 files removed 2024-05-09T08:12:38.705 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T08:12:39.302 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 234 kB/s | 83 kB 00:00 2024-05-09T08:12:39.381 INFO:teuthology.orchestra.run.smithi184.stdout:ceph packages for x86_64 203 kB/s | 83 kB 00:00 2024-05-09T08:12:39.382 INFO:teuthology.orchestra.run.smithi165.stdout:ceph packages for x86_64 171 kB/s | 83 kB 00:00 2024-05-09T08:12:39.705 INFO:teuthology.orchestra.run.smithi184.stdout:ceph noarch packages 41 kB/s | 11 kB 00:00 2024-05-09T08:12:39.720 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-05-09T08:12:39.741 INFO:teuthology.orchestra.run.smithi165.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-05-09T08:12:40.108 INFO:teuthology.orchestra.run.smithi165.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-05-09T08:12:40.129 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-05-09T08:12:40.131 INFO:teuthology.orchestra.run.smithi184.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-05-09T08:12:40.604 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.1 MB 00:00 2024-05-09T08:12:40.683 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.1 MB 00:00 2024-05-09T08:12:40.800 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.1 MB 00:00 2024-05-09T08:12:43.067 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-05-09T08:12:43.188 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-05-09T08:12:47.301 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.2 MB 00:00 2024-05-09T08:12:47.381 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-05-09T08:12:48.891 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-05-09T08:12:48.952 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-05-09T08:12:50.596 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-09T08:12:50.867 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-09T08:12:55.720 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-05-09T08:12:56.225 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-09T08:12:58.013 INFO:teuthology.orchestra.run.smithi165.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:12:58.014 INFO:teuthology.orchestra.run.smithi165.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:12:58.024 INFO:teuthology.orchestra.run.smithi165.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T08:12:58.025 INFO:teuthology.orchestra.run.smithi165.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T08:12:58.026 INFO:teuthology.orchestra.run.smithi165.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T08:12:58.026 INFO:teuthology.orchestra.run.smithi165.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T08:12:58.030 INFO:teuthology.orchestra.run.smithi165.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T08:12:58.030 INFO:teuthology.orchestra.run.smithi165.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T08:12:58.031 INFO:teuthology.orchestra.run.smithi165.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T08:12:58.091 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-05-09T08:12:58.106 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-05-09T08:12:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-05-09T08:12:58.106 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-05-09T08:12:58.106 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-05-09T08:12:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T08:12:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T08:12:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T08:12:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T08:12:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T08:12:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T08:12:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T08:12:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T08:12:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T08:12:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T08:12:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T08:12:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T08:12:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T08:12:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout:Installing weak dependencies: 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout:Install 148 Packages 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 8 Packages 2024-05-09T08:12:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:12:58.119 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 225 M 2024-05-09T08:12:58.119 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-05-09T08:12:58.655 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-05-09T08:12:58.670 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:12:58.670 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:12:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T08:12:58.683 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T08:12:58.684 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T08:12:58.685 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T08:12:58.686 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T08:12:58.689 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T08:12:58.690 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T08:12:58.757 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-05-09T08:12:58.778 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-05-09T08:12:58.778 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-05-09T08:12:58.778 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-05-09T08:12:58.778 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-05-09T08:12:58.778 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T08:12:58.778 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T08:12:58.778 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T08:12:58.779 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T08:12:58.780 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:12:58.781 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T08:12:58.782 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T08:12:58.782 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-05-09T08:12:58.782 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T08:12:58.782 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T08:12:58.782 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T08:12:58.782 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T08:12:58.782 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T08:12:58.782 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T08:12:58.783 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T08:12:58.784 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T08:12:58.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T08:12:58.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T08:12:58.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T08:12:58.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T08:12:58.789 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 8 Packages 2024-05-09T08:12:58.790 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:12:58.791 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 225 M 2024-05-09T08:12:58.791 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-05-09T08:12:59.497 INFO:teuthology.orchestra.run.smithi165.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 9.4 kB/s | 6.6 kB 00:00 2024-05-09T08:13:00.052 INFO:teuthology.orchestra.run.smithi028.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-05-09T08:13:00.446 INFO:teuthology.orchestra.run.smithi165.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 937 kB/s | 889 kB 00:00 2024-05-09T08:13:00.569 INFO:teuthology.orchestra.run.smithi028.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 1.7 MB/s | 889 kB 00:00 2024-05-09T08:13:00.597 INFO:teuthology.orchestra.run.smithi165.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 968 kB/s | 145 kB 00:00 2024-05-09T08:13:00.636 INFO:teuthology.orchestra.run.smithi028.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 2.1 MB/s | 145 kB 00:00 2024-05-09T08:13:00.961 INFO:teuthology.orchestra.run.smithi028.stdout:(4/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 4.1 MB/s | 5.4 MB 00:01 2024-05-09T08:13:01.244 INFO:teuthology.orchestra.run.smithi028.stdout:(5/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 5.3 MB/s | 1.5 MB 00:00 2024-05-09T08:13:01.578 INFO:teuthology.orchestra.run.smithi028.stdout:(6/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 2.5 MB/s | 2.4 MB 00:00 2024-05-09T08:13:01.972 INFO:teuthology.orchestra.run.smithi165.stdout:(4/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 1.7 MB/s | 2.4 MB 00:01 2024-05-09T08:13:02.012 INFO:teuthology.orchestra.run.smithi028.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 6.1 MB/s | 4.7 MB 00:00 2024-05-09T08:13:02.688 INFO:teuthology.orchestra.run.smithi165.stdout:(5/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 2.1 MB/s | 1.5 MB 00:00 2024-05-09T08:13:02.697 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-05-09T08:13:03.404 INFO:teuthology.orchestra.run.smithi028.stdout:(8/156): ceph-radosgw-19.0.0-2515.g96805dcf.el9 7.4 MB/s | 10 MB 00:01 2024-05-09T08:13:03.471 INFO:teuthology.orchestra.run.smithi028.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 379 kB/s | 25 kB 00:00 2024-05-09T08:13:04.180 INFO:teuthology.orchestra.run.smithi165.stdout:(6/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 1.0 MB/s | 5.4 MB 00:05 2024-05-09T08:13:04.248 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-05-09T08:13:04.697 INFO:teuthology.orchestra.run.smithi165.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-05-09T08:13:07.387 INFO:teuthology.orchestra.run.smithi028.stdout:(10/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 2.9 MB/s | 17 MB 00:05 2024-05-09T08:13:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 1.5 MB/s | 222 kB 00:00 2024-05-09T08:13:07.596 INFO:teuthology.orchestra.run.smithi028.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 407 kB/s | 27 kB 00:00 2024-05-09T08:13:07.671 INFO:teuthology.orchestra.run.smithi028.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 424 kB/s | 32 kB 00:00 2024-05-09T08:13:07.947 INFO:teuthology.orchestra.run.smithi028.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 2.7 MB/s | 746 kB 00:00 2024-05-09T08:13:08.030 INFO:teuthology.orchestra.run.smithi028.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 1.9 MB/s | 163 kB 00:00 2024-05-09T08:13:08.097 INFO:teuthology.orchestra.run.smithi028.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 1.9 MB/s | 127 kB 00:00 2024-05-09T08:13:08.222 INFO:teuthology.orchestra.run.smithi165.stdout:(8/156): ceph-radosgw-19.0.0-2515.g96805dcf.el9 2.9 MB/s | 10 MB 00:03 2024-05-09T08:13:08.272 INFO:teuthology.orchestra.run.smithi165.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 507 kB/s | 25 kB 00:00 2024-05-09T08:13:08.556 INFO:teuthology.orchestra.run.smithi028.stdout:(17/156): ceph-test-19.0.0-2515.g96805dcf.el9.x 9.5 MB/s | 48 MB 00:05 2024-05-09T08:13:08.589 INFO:teuthology.orchestra.run.smithi028.stdout:(18/156): libradosstriper1-19.0.0-2515.g96805dc 1.0 MB/s | 500 kB 00:00 2024-05-09T08:13:08.648 INFO:teuthology.orchestra.run.smithi028.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 775 kB/s | 45 kB 00:00 2024-05-09T08:13:08.757 INFO:teuthology.orchestra.run.smithi028.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 1.2 MB/s | 135 kB 00:00 2024-05-09T08:13:08.832 INFO:teuthology.orchestra.run.smithi028.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 2.1 MB/s | 162 kB 00:00 2024-05-09T08:13:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:(22/156): ceph-common-19.0.0-2515.g96805dcf.el9 2.2 MB/s | 21 MB 00:09 2024-05-09T08:13:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:(23/156): python3-rados-19.0.0-2515.g96805dcf.e 1.2 MB/s | 321 kB 00:00 2024-05-09T08:13:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:(24/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 8.6 MB/s | 5.3 MB 00:00 2024-05-09T08:13:09.200 INFO:teuthology.orchestra.run.smithi028.stdout:(25/156): python3-rgw-19.0.0-2515.g96805dcf.el9 915 kB/s | 99 kB 00:00 2024-05-09T08:13:09.225 INFO:teuthology.orchestra.run.smithi028.stdout:(26/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 1.7 MB/s | 85 kB 00:00 2024-05-09T08:13:09.259 INFO:teuthology.orchestra.run.smithi028.stdout:(27/156): python3-rbd-19.0.0-2515.g96805dcf.el9 1.5 MB/s | 298 kB 00:00 2024-05-09T08:13:09.284 INFO:teuthology.orchestra.run.smithi028.stdout:(28/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 2.9 MB/s | 171 kB 00:00 2024-05-09T08:13:09.333 INFO:teuthology.orchestra.run.smithi028.stdout:(29/156): ceph-grafana-dashboards-19.0.0-2515.g 427 kB/s | 32 kB 00:00 2024-05-09T08:13:09.360 INFO:teuthology.orchestra.run.smithi028.stdout:(30/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 1.8 MB/s | 142 kB 00:00 2024-05-09T08:13:09.868 INFO:teuthology.orchestra.run.smithi028.stdout:(31/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 4.6 MB/s | 3.1 MB 00:00 2024-05-09T08:13:10.052 INFO:teuthology.orchestra.run.smithi028.stdout:(32/156): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-05-09T08:13:10.077 INFO:teuthology.orchestra.run.smithi028.stdout:(33/156): ceph-mgr-modules-core-19.0.0-2515.g96 1.2 MB/s | 249 kB 00:00 2024-05-09T08:13:10.102 INFO:teuthology.orchestra.run.smithi028.stdout:(34/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 987 kB/s | 49 kB 00:00 2024-05-09T08:13:10.119 INFO:teuthology.orchestra.run.smithi028.stdout:(35/156): ceph-prometheus-alerts-19.0.0-2515.g9 392 kB/s | 16 kB 00:00 2024-05-09T08:13:10.153 INFO:teuthology.orchestra.run.smithi028.stdout:(36/156): ceph-volume-19.0.0-2515.g96805dcf.el9 5.1 MB/s | 264 kB 00:00 2024-05-09T08:13:10.203 INFO:teuthology.orchestra.run.smithi028.stdout:(37/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 366 kB/s | 18 kB 00:00 2024-05-09T08:13:10.253 INFO:teuthology.orchestra.run.smithi028.stdout:(38/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 5.6 MB/s | 763 kB 00:00 2024-05-09T08:13:10.321 INFO:teuthology.orchestra.run.smithi028.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 685 kB/s | 80 kB 00:00 2024-05-09T08:13:10.387 INFO:teuthology.orchestra.run.smithi028.stdout:(40/156): ceph-mgr-dashboard-19.0.0-2515.g96805 3.3 MB/s | 3.5 MB 00:01 2024-05-09T08:13:10.422 INFO:teuthology.orchestra.run.smithi165.stdout:(10/156): ceph-common-19.0.0-2515.g96805dcf.el9 1.8 MB/s | 21 MB 00:11 2024-05-09T08:13:10.429 INFO:teuthology.orchestra.run.smithi028.stdout:(41/156): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-05-09T08:13:10.538 INFO:teuthology.orchestra.run.smithi028.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-05-09T08:13:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:(43/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-05-09T08:13:10.597 INFO:teuthology.orchestra.run.smithi165.stdout:(11/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 2.6 MB/s | 17 MB 00:06 2024-05-09T08:13:10.605 INFO:teuthology.orchestra.run.smithi028.stdout:(44/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-05-09T08:13:10.622 INFO:teuthology.orchestra.run.smithi165.stdout:(12/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 1.1 MB/s | 222 kB 00:00 2024-05-09T08:13:10.631 INFO:teuthology.orchestra.run.smithi028.stdout:(45/156): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-05-09T08:13:10.639 INFO:teuthology.orchestra.run.smithi165.stdout:(13/156): cephfs-shell-19.0.0-2515.g96805dcf.el 651 kB/s | 27 kB 00:00 2024-05-09T08:13:10.656 INFO:teuthology.orchestra.run.smithi028.stdout:(46/156): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-05-09T08:13:10.673 INFO:teuthology.orchestra.run.smithi028.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-09T08:13:10.681 INFO:teuthology.orchestra.run.smithi165.stdout:(14/156): libcephfs-devel-19.0.0-2515.g96805dcf 542 kB/s | 32 kB 00:00 2024-05-09T08:13:10.698 INFO:teuthology.orchestra.run.smithi028.stdout:(48/156): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-05-09T08:13:10.750 INFO:teuthology.orchestra.run.smithi165.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 2.4 MB/s | 163 kB 00:00 2024-05-09T08:13:10.807 INFO:teuthology.orchestra.run.smithi165.stdout:(16/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 4.4 MB/s | 746 kB 00:00 2024-05-09T08:13:10.832 INFO:teuthology.orchestra.run.smithi165.stdout:(17/156): librados-devel-19.0.0-2515.g96805dcf. 1.5 MB/s | 127 kB 00:00 2024-05-09T08:13:10.899 INFO:teuthology.orchestra.run.smithi165.stdout:(18/156): libradosstriper1-19.0.0-2515.g96805dc 5.3 MB/s | 500 kB 00:00 2024-05-09T08:13:10.933 INFO:teuthology.orchestra.run.smithi165.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 1.3 MB/s | 45 kB 00:00 2024-05-09T08:13:10.983 INFO:teuthology.orchestra.run.smithi165.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 2.6 MB/s | 135 kB 00:00 2024-05-09T08:13:11.042 INFO:teuthology.orchestra.run.smithi165.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 2.7 MB/s | 162 kB 00:00 2024-05-09T08:13:11.132 INFO:teuthology.orchestra.run.smithi028.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 247 kB/s | 107 kB 00:00 2024-05-09T08:13:11.133 INFO:teuthology.orchestra.run.smithi165.stdout:(22/156): python3-rados-19.0.0-2515.g96805dcf.e 3.4 MB/s | 321 kB 00:00 2024-05-09T08:13:11.217 INFO:teuthology.orchestra.run.smithi165.stdout:(23/156): python3-rbd-19.0.0-2515.g96805dcf.el9 3.5 MB/s | 298 kB 00:00 2024-05-09T08:13:11.241 INFO:teuthology.orchestra.run.smithi028.stdout:(50/156): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-05-09T08:13:11.276 INFO:teuthology.orchestra.run.smithi165.stdout:(24/156): python3-rgw-19.0.0-2515.g96805dcf.el9 1.7 MB/s | 99 kB 00:00 2024-05-09T08:13:11.342 INFO:teuthology.orchestra.run.smithi165.stdout:(25/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 1.3 MB/s | 85 kB 00:00 2024-05-09T08:13:11.399 INFO:teuthology.orchestra.run.smithi028.stdout:(51/156): automake-1.16.2-8.el9.noarch.rpm 928 kB/s | 689 kB 00:00 2024-05-09T08:13:11.516 INFO:teuthology.orchestra.run.smithi028.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-05-09T08:13:11.541 INFO:teuthology.orchestra.run.smithi028.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 1.0 MB/s | 313 kB 00:00 2024-05-09T08:13:11.566 INFO:teuthology.orchestra.run.smithi028.stdout:(54/156): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-05-09T08:13:11.667 INFO:teuthology.orchestra.run.smithi028.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 145 kB/s | 18 kB 00:00 2024-05-09T08:13:11.692 INFO:teuthology.orchestra.run.smithi028.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 89 kB/s | 11 kB 00:00 2024-05-09T08:13:11.792 INFO:teuthology.orchestra.run.smithi028.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-05-09T08:13:11.802 INFO:teuthology.orchestra.run.smithi165.stdout:(26/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 6.7 MB/s | 3.1 MB 00:00 2024-05-09T08:13:11.817 INFO:teuthology.orchestra.run.smithi028.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 361 kB/s | 45 kB 00:00 2024-05-09T08:13:11.851 INFO:teuthology.orchestra.run.smithi165.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 3.4 MB/s | 171 kB 00:00 2024-05-09T08:13:11.894 INFO:teuthology.orchestra.run.smithi165.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 763 kB/s | 32 kB 00:00 2024-05-09T08:13:11.944 INFO:teuthology.orchestra.run.smithi165.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 2.8 MB/s | 142 kB 00:00 2024-05-09T08:13:12.035 INFO:teuthology.orchestra.run.smithi028.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 662 kB 00:00 2024-05-09T08:13:12.168 INFO:teuthology.orchestra.run.smithi028.stdout:(60/156): libtool-2.4.6-45.el9.x86_64.rpm 1.6 MB/s | 582 kB 00:00 2024-05-09T08:13:12.193 INFO:teuthology.orchestra.run.smithi028.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 160 kB/s | 25 kB 00:00 2024-05-09T08:13:12.285 INFO:teuthology.orchestra.run.smithi028.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 406 kB/s | 47 kB 00:00 2024-05-09T08:13:12.310 INFO:teuthology.orchestra.run.smithi028.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-05-09T08:13:12.311 INFO:teuthology.orchestra.run.smithi165.stdout:(30/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 3.6 MB/s | 5.3 MB 00:01 2024-05-09T08:13:12.402 INFO:teuthology.orchestra.run.smithi028.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-05-09T08:13:12.428 INFO:teuthology.orchestra.run.smithi028.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 301 kB/s | 35 kB 00:00 2024-05-09T08:13:12.461 INFO:teuthology.orchestra.run.smithi165.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2515.g96805 6.8 MB/s | 3.5 MB 00:00 2024-05-09T08:13:12.528 INFO:teuthology.orchestra.run.smithi165.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 3.7 MB/s | 249 kB 00:00 2024-05-09T08:13:12.578 INFO:teuthology.orchestra.run.smithi165.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 992 kB/s | 49 kB 00:00 2024-05-09T08:13:12.628 INFO:teuthology.orchestra.run.smithi165.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2515.g9 329 kB/s | 16 kB 00:00 2024-05-09T08:13:12.695 INFO:teuthology.orchestra.run.smithi165.stdout:(35/156): ceph-volume-19.0.0-2515.g96805dcf.el9 3.9 MB/s | 264 kB 00:00 2024-05-09T08:13:12.812 INFO:teuthology.orchestra.run.smithi165.stdout:(36/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 6.4 MB/s | 763 kB 00:00 2024-05-09T08:13:12.862 INFO:teuthology.orchestra.run.smithi165.stdout:(37/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 367 kB/s | 18 kB 00:00 2024-05-09T08:13:12.962 INFO:teuthology.orchestra.run.smithi165.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 805 kB/s | 80 kB 00:00 2024-05-09T08:13:13.003 INFO:teuthology.orchestra.run.smithi028.stdout:(66/156): openblas-openmp-0.3.21-2.el9.x86_64.r 7.9 MB/s | 4.7 MB 00:00 2024-05-09T08:13:13.071 INFO:teuthology.orchestra.run.smithi165.stdout:(39/156): libgfortran-11.4.1-3.el9.x86_64.rpm 7.2 MB/s | 803 kB 00:00 2024-05-09T08:13:13.122 INFO:teuthology.orchestra.run.smithi165.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-05-09T08:13:13.136 INFO:teuthology.orchestra.run.smithi028.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-09T08:13:13.172 INFO:teuthology.orchestra.run.smithi165.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-05-09T08:13:13.239 INFO:teuthology.orchestra.run.smithi165.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-05-09T08:13:13.281 INFO:teuthology.orchestra.run.smithi165.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-05-09T08:13:13.323 INFO:teuthology.orchestra.run.smithi165.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-05-09T08:13:13.365 INFO:teuthology.orchestra.run.smithi165.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-05-09T08:13:13.398 INFO:teuthology.orchestra.run.smithi165.stdout:(46/156): python3-urllib3-1.26.5-5.el9.noarch.r 6.3 MB/s | 215 kB 00:00 2024-05-09T08:13:13.657 INFO:teuthology.orchestra.run.smithi165.stdout:(47/156): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 689 kB 00:00 2024-05-09T08:13:13.866 INFO:teuthology.orchestra.run.smithi165.stdout:(48/156): bison-3.7.4-5.el9.x86_64.rpm 4.4 MB/s | 943 kB 00:00 2024-05-09T08:13:13.887 INFO:teuthology.orchestra.run.smithi028.stdout:(68/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 1.3 MB/s | 3.0 MB 00:02 2024-05-09T08:13:13.916 INFO:teuthology.orchestra.run.smithi165.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-05-09T08:13:13.996 INFO:teuthology.orchestra.run.smithi028.stdout:(69/156): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-05-09T08:13:14.000 INFO:teuthology.orchestra.run.smithi165.stdout:(50/156): ceph-mgr-diskprediction-local-19.0.0- 4.4 MB/s | 7.4 MB 00:01 2024-05-09T08:13:14.150 INFO:teuthology.orchestra.run.smithi165.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 92 kB/s | 21 kB 00:00 2024-05-09T08:13:14.425 INFO:teuthology.orchestra.run.smithi165.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 737 kB/s | 313 kB 00:00 2024-05-09T08:13:14.692 INFO:teuthology.orchestra.run.smithi165.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 60 kB/s | 33 kB 00:00 2024-05-09T08:13:14.801 INFO:teuthology.orchestra.run.smithi165.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 168 kB/s | 18 kB 00:00 2024-05-09T08:13:14.851 INFO:teuthology.orchestra.run.smithi165.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-05-09T08:13:15.013 INFO:teuthology.orchestra.run.smithi028.stdout:(70/156): postgresql-server-13.14-1.el9.x86_64. 5.7 MB/s | 5.8 MB 00:01 2024-05-09T08:13:15.035 INFO:teuthology.orchestra.run.smithi165.stdout:(56/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 874 kB/s | 160 kB 00:00 2024-05-09T08:13:15.072 INFO:teuthology.orchestra.run.smithi028.stdout:(71/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.7 MB/s | 4.4 MB 00:02 2024-05-09T08:13:15.118 INFO:teuthology.orchestra.run.smithi165.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 544 kB/s | 45 kB 00:00 2024-05-09T08:13:15.122 INFO:teuthology.orchestra.run.smithi028.stdout:(72/156): postgresql-contrib-13.14-1.el9.x86_64 459 kB/s | 911 kB 00:01 2024-05-09T08:13:15.164 INFO:teuthology.orchestra.run.smithi028.stdout:(73/156): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-05-09T08:13:15.306 INFO:teuthology.orchestra.run.smithi028.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 1.7 MB/s | 244 kB 00:00 2024-05-09T08:13:15.331 INFO:teuthology.orchestra.run.smithi028.stdout:(75/156): python3-attrs-20.3.0-7.el9.noarch.rpm 337 kB/s | 87 kB 00:00 2024-05-09T08:13:15.414 INFO:teuthology.orchestra.run.smithi028.stdout:(76/156): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-05-09T08:13:15.440 INFO:teuthology.orchestra.run.smithi028.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 236 kB/s | 26 kB 00:00 2024-05-09T08:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 208 kB/s | 19 kB 00:00 2024-05-09T08:13:15.593 INFO:teuthology.orchestra.run.smithi165.stdout:(58/156): librdkafka-1.6.1-102.el9.x86_64.rpm 1.4 MB/s | 662 kB 00:00 2024-05-09T08:13:15.599 INFO:teuthology.orchestra.run.smithi028.stdout:(79/156): python3-markupsafe-1.1.1-12.el9.x86_6 379 kB/s | 35 kB 00:00 2024-05-09T08:13:15.702 INFO:teuthology.orchestra.run.smithi165.stdout:(59/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.3 MB/s | 3.0 MB 00:01 2024-05-09T08:13:15.716 INFO:teuthology.orchestra.run.smithi028.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 625 kB/s | 172 kB 00:00 2024-05-09T08:13:15.744 INFO:teuthology.orchestra.run.smithi165.stdout:(60/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 612 kB/s | 25 kB 00:00 2024-05-09T08:13:15.786 INFO:teuthology.orchestra.run.smithi165.stdout:(61/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-09T08:13:15.791 INFO:teuthology.orchestra.run.smithi028.stdout:(81/156): python3-babel-2.9.1-2.el9.noarch.rpm 8.9 MB/s | 6.0 MB 00:00 2024-05-09T08:13:15.900 INFO:teuthology.orchestra.run.smithi028.stdout:(82/156): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-05-09T08:13:15.919 INFO:teuthology.orchestra.run.smithi165.stdout:(62/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-05-09T08:13:15.953 INFO:teuthology.orchestra.run.smithi165.stdout:(63/156): libtool-2.4.6-45.el9.x86_64.rpm 1.6 MB/s | 582 kB 00:00 2024-05-09T08:13:15.995 INFO:teuthology.orchestra.run.smithi165.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-05-09T08:13:16.000 INFO:teuthology.orchestra.run.smithi028.stdout:(83/156): python3-packaging-20.9-5.el9.noarch.r 773 kB/s | 77 kB 00:00 2024-05-09T08:13:16.020 INFO:teuthology.orchestra.run.smithi165.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 527 kB/s | 35 kB 00:00 2024-05-09T08:13:16.100 INFO:teuthology.orchestra.run.smithi028.stdout:(84/156): python3-prettytable-0.7.2-27.el9.noar 423 kB/s | 42 kB 00:00 2024-05-09T08:13:16.125 INFO:teuthology.orchestra.run.smithi028.stdout:(85/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 612 kB/s | 250 kB 00:00 2024-05-09T08:13:16.244 INFO:teuthology.orchestra.run.smithi028.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 1.8 MB/s | 269 kB 00:00 2024-05-09T08:13:16.393 INFO:teuthology.orchestra.run.smithi028.stdout:(87/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.4 MB/s | 5.1 MB 00:00 2024-05-09T08:13:16.418 INFO:teuthology.orchestra.run.smithi028.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 1.6 MB/s | 279 kB 00:00 2024-05-09T08:13:16.476 INFO:teuthology.orchestra.run.smithi028.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 452 kB/s | 159 kB 00:00 2024-05-09T08:13:16.502 INFO:teuthology.orchestra.run.smithi028.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 468 kB/s | 51 kB 00:00 2024-05-09T08:13:16.527 INFO:teuthology.orchestra.run.smithi028.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 493 kB/s | 54 kB 00:00 2024-05-09T08:13:16.602 INFO:teuthology.orchestra.run.smithi028.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-05-09T08:13:16.628 INFO:teuthology.orchestra.run.smithi028.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 440 kB/s | 44 kB 00:00 2024-05-09T08:13:16.711 INFO:teuthology.orchestra.run.smithi028.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-05-09T08:13:16.736 INFO:teuthology.orchestra.run.smithi028.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 330 kB/s | 36 kB 00:00 2024-05-09T08:13:16.820 INFO:teuthology.orchestra.run.smithi028.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 436 kB/s | 47 kB 00:00 2024-05-09T08:13:16.846 INFO:teuthology.orchestra.run.smithi028.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 305 kB 00:00 2024-05-09T08:13:16.929 INFO:teuthology.orchestra.run.smithi028.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 532 kB/s | 58 kB 00:00 2024-05-09T08:13:16.954 INFO:teuthology.orchestra.run.smithi028.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 862 kB/s | 94 kB 00:00 2024-05-09T08:13:16.995 INFO:teuthology.orchestra.run.smithi165.stdout:(66/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-05-09T08:13:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 586 kB/s | 64 kB 00:00 2024-05-09T08:13:17.063 INFO:teuthology.orchestra.run.smithi028.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 413 kB/s | 45 kB 00:00 2024-05-09T08:13:17.139 INFO:teuthology.orchestra.run.smithi028.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-05-09T08:13:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 7.2 MB/s | 863 kB 00:00 2024-05-09T08:13:17.237 INFO:teuthology.orchestra.run.smithi165.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 3.8 MB/s | 4.7 MB 00:01 2024-05-09T08:13:17.271 INFO:teuthology.orchestra.run.smithi165.stdout:(68/156): postgresql-13.14-1.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-05-09T08:13:17.330 INFO:teuthology.orchestra.run.smithi165.stdout:(69/156): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-09T08:13:17.480 INFO:teuthology.orchestra.run.smithi165.stdout:(70/156): postgresql-contrib-13.14-1.el9.x86_64 3.7 MB/s | 911 kB 00:00 2024-05-09T08:13:17.722 INFO:teuthology.orchestra.run.smithi165.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-05-09T08:13:17.781 INFO:teuthology.orchestra.run.smithi165.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-05-09T08:13:18.156 INFO:teuthology.orchestra.run.smithi165.stdout:(73/156): postgresql-server-13.14-1.el9.x86_64. 7.0 MB/s | 5.8 MB 00:00 2024-05-09T08:13:18.239 INFO:teuthology.orchestra.run.smithi165.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-05-09T08:13:18.315 INFO:teuthology.orchestra.run.smithi165.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-05-09T08:13:18.322 INFO:teuthology.orchestra.run.smithi028.stdout:(104/156): gperftools-libs-2.9.1-2.el9.x86_64.r 270 kB/s | 309 kB 00:01 2024-05-09T08:13:18.382 INFO:teuthology.orchestra.run.smithi165.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 385 kB/s | 26 kB 00:00 2024-05-09T08:13:18.457 INFO:teuthology.orchestra.run.smithi165.stdout:(77/156): python3-jsonpointer-2.0-4.el9.noarch. 256 kB/s | 19 kB 00:00 2024-05-09T08:13:18.524 INFO:teuthology.orchestra.run.smithi165.stdout:(78/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-05-09T08:13:18.591 INFO:teuthology.orchestra.run.smithi165.stdout:(79/156): python3-markupsafe-1.1.1-12.el9.x86_6 521 kB/s | 35 kB 00:00 2024-05-09T08:13:18.790 INFO:teuthology.orchestra.run.smithi028.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 42 kB/s | 19 kB 00:00 2024-05-09T08:13:18.889 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 1.5 MB/s | 21 MB 00:14 2024-05-09T08:13:19.349 INFO:teuthology.orchestra.run.smithi165.stdout:(80/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.7 MB/s | 5.1 MB 00:00 2024-05-09T08:13:19.424 INFO:teuthology.orchestra.run.smithi165.stdout:(81/156): python3-babel-2.9.1-2.el9.noarch.rpm 3.6 MB/s | 6.0 MB 00:01 2024-05-09T08:13:19.450 INFO:teuthology.orchestra.run.smithi165.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-05-09T08:13:19.492 INFO:teuthology.orchestra.run.smithi165.stdout:(83/156): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-05-09T08:13:19.517 INFO:teuthology.orchestra.run.smithi165.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-05-09T08:13:19.542 INFO:teuthology.orchestra.run.smithi165.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-05-09T08:13:19.601 INFO:teuthology.orchestra.run.smithi165.stdout:(86/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-05-09T08:13:19.626 INFO:teuthology.orchestra.run.smithi165.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-05-09T08:13:19.660 INFO:teuthology.orchestra.run.smithi165.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-05-09T08:13:19.677 INFO:teuthology.orchestra.run.smithi165.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-09T08:13:19.710 INFO:teuthology.orchestra.run.smithi165.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-09T08:13:19.757 INFO:teuthology.orchestra.run.smithi028.stdout:(106/156): abseil-cpp-20211102.0-3.el9.x86_64.r 209 kB/s | 548 kB 00:02 2024-05-09T08:13:19.761 INFO:teuthology.orchestra.run.smithi165.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 834 kB/s | 42 kB 00:00 2024-05-09T08:13:19.811 INFO:teuthology.orchestra.run.smithi165.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 885 kB/s | 44 kB 00:00 2024-05-09T08:13:19.869 INFO:teuthology.orchestra.run.smithi165.stdout:(93/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-05-09T08:13:19.920 INFO:teuthology.orchestra.run.smithi165.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 718 kB/s | 36 kB 00:00 2024-05-09T08:13:19.970 INFO:teuthology.orchestra.run.smithi165.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 954 kB/s | 47 kB 00:00 2024-05-09T08:13:20.037 INFO:teuthology.orchestra.run.smithi165.stdout:(96/156): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-05-09T08:13:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 89 kB/s | 26 kB 00:00 2024-05-09T08:13:20.087 INFO:teuthology.orchestra.run.smithi165.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-09T08:13:20.155 INFO:teuthology.orchestra.run.smithi165.stdout:(98/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-05-09T08:13:20.205 INFO:teuthology.orchestra.run.smithi165.stdout:(99/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-09T08:13:20.247 INFO:teuthology.orchestra.run.smithi165.stdout:(100/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-05-09T08:13:20.315 INFO:teuthology.orchestra.run.smithi028.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 182 kB/s | 49 kB 00:00 2024-05-09T08:13:20.530 INFO:teuthology.orchestra.run.smithi165.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2024-05-09T08:13:20.740 INFO:teuthology.orchestra.run.smithi028.stdout:(109/156): libunwind-1.6.2-1.el9.x86_64.rpm 158 kB/s | 67 kB 00:00 2024-05-09T08:13:21.256 INFO:teuthology.orchestra.run.smithi165.stdout:(102/156): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-05-09T08:13:21.689 INFO:teuthology.orchestra.run.smithi165.stdout:(103/156): ceph-test-19.0.0-2515.g96805dcf.el9. 3.6 MB/s | 48 MB 00:13 2024-05-09T08:13:21.724 INFO:teuthology.orchestra.run.smithi028.stdout:(110/156): luarocks-3.9.2-1.el9.noarch.rpm 154 kB/s | 151 kB 00:00 2024-05-09T08:13:22.173 INFO:teuthology.orchestra.run.smithi165.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 598 kB/s | 548 kB 00:00 2024-05-09T08:13:22.365 INFO:teuthology.orchestra.run.smithi165.stdout:(105/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.0 MB/s | 16 MB 00:02 2024-05-09T08:13:22.390 INFO:teuthology.orchestra.run.smithi165.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 90 kB/s | 19 kB 00:00 2024-05-09T08:13:22.423 INFO:teuthology.orchestra.run.smithi165.stdout:(107/156): gperftools-libs-2.9.1-2.el9.x86_64.r 421 kB/s | 309 kB 00:00 2024-05-09T08:13:22.490 INFO:teuthology.orchestra.run.smithi165.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-05-09T08:13:22.541 INFO:teuthology.orchestra.run.smithi165.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-05-09T08:13:22.591 INFO:teuthology.orchestra.run.smithi165.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-05-09T08:13:22.658 INFO:teuthology.orchestra.run.smithi165.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-05-09T08:13:22.791 INFO:teuthology.orchestra.run.smithi165.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.1 MB/s | 839 kB 00:00 2024-05-09T08:13:22.875 INFO:teuthology.orchestra.run.smithi165.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-05-09T08:13:22.900 INFO:teuthology.orchestra.run.smithi165.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-05-09T08:13:22.991 INFO:teuthology.orchestra.run.smithi028.stdout:(111/156): libarrow-9.0.0-11.el9.x86_64.rpm 1.1 MB/s | 4.4 MB 00:04 2024-05-09T08:13:22.992 INFO:teuthology.orchestra.run.smithi165.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-05-09T08:13:23.017 INFO:teuthology.orchestra.run.smithi165.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-05-09T08:13:23.083 INFO:teuthology.orchestra.run.smithi028.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 617 kB/s | 839 kB 00:01 2024-05-09T08:13:23.109 INFO:teuthology.orchestra.run.smithi165.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-05-09T08:13:23.218 INFO:teuthology.orchestra.run.smithi165.stdout:(118/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-05-09T08:13:23.243 INFO:teuthology.orchestra.run.smithi165.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 1.6 MB/s | 358 kB 00:00 2024-05-09T08:13:23.266 INFO:teuthology.orchestra.run.smithi028.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 2.0 MB/s | 548 kB 00:00 2024-05-09T08:13:23.283 INFO:teuthology.orchestra.run.smithi028.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 217 kB/s | 43 kB 00:00 2024-05-09T08:13:23.302 INFO:teuthology.orchestra.run.smithi165.stdout:(120/156): libarrow-9.0.0-11.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-05-09T08:13:23.327 INFO:teuthology.orchestra.run.smithi165.stdout:(121/156): python3-colorama-0.4.4-12.el9.noarch 324 kB/s | 35 kB 00:00 2024-05-09T08:13:23.369 INFO:teuthology.orchestra.run.smithi165.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 1.6 MB/s | 206 kB 00:00 2024-05-09T08:13:23.442 INFO:teuthology.orchestra.run.smithi028.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 184 kB/s | 32 kB 00:00 2024-05-09T08:13:23.444 INFO:teuthology.orchestra.run.smithi165.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-05-09T08:13:23.459 INFO:teuthology.orchestra.run.smithi028.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 85 kB/s | 15 kB 00:00 2024-05-09T08:13:23.486 INFO:teuthology.orchestra.run.smithi165.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-05-09T08:13:23.553 INFO:teuthology.orchestra.run.smithi165.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-05-09T08:13:23.567 INFO:teuthology.orchestra.run.smithi028.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-05-09T08:13:23.603 INFO:teuthology.orchestra.run.smithi165.stdout:(126/156): python3-grpcio-1.46.7-10.el9.x86_64. 6.8 MB/s | 2.0 MB 00:00 2024-05-09T08:13:23.637 INFO:teuthology.orchestra.run.smithi165.stdout:(127/156): python3-jaraco-collections-3.0.0-8.e 153 kB/s | 23 kB 00:00 2024-05-09T08:13:23.659 INFO:teuthology.orchestra.run.smithi028.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 1.7 MB/s | 358 kB 00:00 2024-05-09T08:13:23.671 INFO:teuthology.orchestra.run.smithi165.stdout:(128/156): python3-jaraco-functools-3.5.0-2.el9 165 kB/s | 19 kB 00:00 2024-05-09T08:13:23.713 INFO:teuthology.orchestra.run.smithi165.stdout:(129/156): python3-jaraco-text-3.2.0-6.el9.noar 180 kB/s | 20 kB 00:00 2024-05-09T08:13:23.751 INFO:teuthology.orchestra.run.smithi028.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-05-09T08:13:23.780 INFO:teuthology.orchestra.run.smithi165.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 427 kB/s | 46 kB 00:00 2024-05-09T08:13:23.830 INFO:teuthology.orchestra.run.smithi165.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 675 kB/s | 79 kB 00:00 2024-05-09T08:13:23.897 INFO:teuthology.orchestra.run.smithi165.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 494 kB/s | 58 kB 00:00 2024-05-09T08:13:23.918 INFO:teuthology.orchestra.run.smithi028.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 136 kB/s | 35 kB 00:00 2024-05-09T08:13:23.943 INFO:teuthology.orchestra.run.smithi028.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 1.0 MB/s | 206 kB 00:00 2024-05-09T08:13:23.964 INFO:teuthology.orchestra.run.smithi165.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-05-09T08:13:23.997 INFO:teuthology.orchestra.run.smithi165.stdout:(134/156): python3-kubernetes-26.1.0-2.el9.noar 2.8 MB/s | 1.0 MB 00:00 2024-05-09T08:13:24.014 INFO:teuthology.orchestra.run.smithi165.stdout:(135/156): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-05-09T08:13:24.081 INFO:teuthology.orchestra.run.smithi165.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 768 kB/s | 90 kB 00:00 2024-05-09T08:13:24.123 INFO:teuthology.orchestra.run.smithi165.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-05-09T08:13:24.139 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-05-09T08:13:24.140 INFO:teuthology.orchestra.run.smithi165.stdout:(138/156): python3-pyperclip-1.8.2-2.el9.noarch 198 kB/s | 28 kB 00:00 2024-05-09T08:13:24.190 INFO:teuthology.orchestra.run.smithi165.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-05-09T08:13:24.224 INFO:teuthology.orchestra.run.smithi165.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-05-09T08:13:24.274 INFO:teuthology.orchestra.run.smithi165.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 267 kB/s | 36 kB 00:00 2024-05-09T08:13:24.299 INFO:teuthology.orchestra.run.smithi165.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 468 kB/s | 51 kB 00:00 2024-05-09T08:13:24.333 INFO:teuthology.orchestra.run.smithi165.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-05-09T08:13:24.392 INFO:teuthology.orchestra.run.smithi165.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 766 kB/s | 90 kB 00:00 2024-05-09T08:13:24.417 INFO:teuthology.orchestra.run.smithi165.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-05-09T08:13:24.451 INFO:teuthology.orchestra.run.smithi165.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-05-09T08:13:24.534 INFO:teuthology.orchestra.run.smithi165.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-05-09T08:13:24.594 INFO:teuthology.orchestra.run.smithi028.stdout:(122/156): python3-grpcio-1.46.7-10.el9.x86_64. 3.0 MB/s | 2.0 MB 00:00 2024-05-09T08:13:24.620 INFO:teuthology.orchestra.run.smithi028.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 214 kB/s | 144 kB 00:00 2024-05-09T08:13:24.651 INFO:teuthology.orchestra.run.smithi165.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-05-09T08:13:24.711 INFO:teuthology.orchestra.run.smithi028.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-05-09T08:13:24.737 INFO:teuthology.orchestra.run.smithi028.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-05-09T08:13:24.820 INFO:teuthology.orchestra.run.smithi028.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-05-09T08:13:24.845 INFO:teuthology.orchestra.run.smithi028.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-05-09T08:13:25.084 INFO:teuthology.orchestra.run.smithi165.stdout:(149/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 5.7 MB/s | 3.1 MB 00:00 2024-05-09T08:13:25.121 INFO:teuthology.orchestra.run.smithi028.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 65 kB/s | 20 kB 00:00 2024-05-09T08:13:25.219 INFO:teuthology.orchestra.run.smithi165.stdout:(150/156): NetworkManager-1.47.5-1.el9.x86_64.r 4.1 MB/s | 2.3 MB 00:00 2024-05-09T08:13:25.268 INFO:teuthology.orchestra.run.smithi165.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 839 kB/s | 42 kB 00:00 2024-05-09T08:13:25.271 INFO:teuthology.orchestra.run.smithi028.stdout:(129/156): python3-kubernetes-26.1.0-2.el9.noar 2.4 MB/s | 1.0 MB 00:00 2024-05-09T08:13:25.344 INFO:teuthology.orchestra.run.smithi165.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 3.2 MB/s | 248 kB 00:00 2024-05-09T08:13:25.363 INFO:teuthology.orchestra.run.smithi028.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 192 kB/s | 46 kB 00:00 2024-05-09T08:13:25.385 INFO:teuthology.orchestra.run.smithi165.stdout:(153/156): NetworkManager-libnm-1.47.5-1.el9.x8 6.1 MB/s | 1.8 MB 00:00 2024-05-09T08:13:25.413 INFO:teuthology.orchestra.run.smithi028.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 556 kB/s | 79 kB 00:00 2024-05-09T08:13:25.514 INFO:teuthology.orchestra.run.smithi028.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 384 kB/s | 58 kB 00:00 2024-05-09T08:13:25.539 INFO:teuthology.orchestra.run.smithi028.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-05-09T08:13:25.611 INFO:teuthology.orchestra.run.smithi165.stdout:(154/156): librados2-19.0.0-2515.g96805dcf.el9. 2.9 MB/s | 3.4 MB 00:01 2024-05-09T08:13:25.661 INFO:teuthology.orchestra.run.smithi165.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 8.7 MB/s | 2.4 MB 00:00 2024-05-09T08:13:25.672 INFO:teuthology.orchestra.run.smithi028.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 104 kB/s | 16 kB 00:00 2024-05-09T08:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 567 kB/s | 90 kB 00:00 2024-05-09T08:13:25.786 INFO:teuthology.orchestra.run.smithi165.stdout:(156/156): openssl-3.2.1-1.el9.x86_64.rpm 3.1 MB/s | 1.3 MB 00:00 2024-05-09T08:13:25.789 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:13:25.789 INFO:teuthology.orchestra.run.smithi165.stdout:Total 8.1 MB/s | 225 MB 00:27 2024-05-09T08:13:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 199 kB/s | 28 kB 00:00 2024-05-09T08:13:26.057 INFO:teuthology.orchestra.run.smithi028.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 86 kB/s | 31 kB 00:00 2024-05-09T08:13:26.082 INFO:teuthology.orchestra.run.smithi028.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 707 kB/s | 188 kB 00:00 2024-05-09T08:13:26.232 INFO:teuthology.orchestra.run.smithi028.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-05-09T08:13:26.257 INFO:teuthology.orchestra.run.smithi028.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 204 kB/s | 36 kB 00:00 2024-05-09T08:13:26.349 INFO:teuthology.orchestra.run.smithi028.stdout:(141/156): python3-typing-extensions-4.4.0-2.el 436 kB/s | 51 kB 00:00 2024-05-09T08:13:26.375 INFO:teuthology.orchestra.run.smithi028.stdout:(142/156): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-05-09T08:13:26.407 INFO:teuthology.orchestra.run.smithi184.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:13:26.408 INFO:teuthology.orchestra.run.smithi184.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:13:26.419 INFO:teuthology.orchestra.run.smithi184.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T08:13:26.420 INFO:teuthology.orchestra.run.smithi184.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T08:13:26.420 INFO:teuthology.orchestra.run.smithi184.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T08:13:26.421 INFO:teuthology.orchestra.run.smithi184.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T08:13:26.423 INFO:teuthology.orchestra.run.smithi184.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T08:13:26.425 INFO:teuthology.orchestra.run.smithi184.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T08:13:26.426 INFO:teuthology.orchestra.run.smithi184.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T08:13:26.483 INFO:teuthology.orchestra.run.smithi028.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 673 kB/s | 90 kB 00:00 2024-05-09T08:13:26.486 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-05-09T08:13:26.511 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2024-05-09T08:13:26.511 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T08:13:26.512 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T08:13:26.513 INFO:teuthology.orchestra.run.smithi184.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T08:13:26.514 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrading: 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T08:13:26.515 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T08:13:26.516 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T08:13:26.517 INFO:teuthology.orchestra.run.smithi184.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T08:13:26.518 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T08:13:26.519 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T08:13:26.520 INFO:teuthology.orchestra.run.smithi184.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T08:13:26.521 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T08:13:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T08:13:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T08:13:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T08:13:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T08:13:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T08:13:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T08:13:26.522 INFO:teuthology.orchestra.run.smithi184.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout:Installing weak dependencies: 2024-05-09T08:13:26.523 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T08:13:26.524 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:13:26.524 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-05-09T08:13:26.524 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2024-05-09T08:13:26.524 INFO:teuthology.orchestra.run.smithi184.stdout:Install 148 Packages 2024-05-09T08:13:26.524 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrade 8 Packages 2024-05-09T08:13:26.524 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:13:26.524 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 225 M 2024-05-09T08:13:26.524 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-05-09T08:13:26.633 INFO:teuthology.orchestra.run.smithi028.stdout:(144/156): python3-werkzeug-2.0.3-3.el9.1.noarc 1.6 MB/s | 427 kB 00:00 2024-05-09T08:13:26.684 INFO:teuthology.orchestra.run.smithi028.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 100 kB/s | 20 kB 00:00 2024-05-09T08:13:26.909 INFO:teuthology.orchestra.run.smithi028.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 696 kB/s | 192 kB 00:00 2024-05-09T08:13:26.926 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-05-09T08:13:27.074 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-05-09T08:13:27.075 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-05-09T08:13:27.259 INFO:teuthology.orchestra.run.smithi028.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 2.7 MB/s | 1.6 MB 00:00 2024-05-09T08:13:27.476 INFO:teuthology.orchestra.run.smithi028.stdout:(148/156): librados2-19.0.0-2515.g96805dcf.el9. 6.0 MB/s | 3.4 MB 00:00 2024-05-09T08:13:27.508 INFO:teuthology.orchestra.run.smithi184.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-05-09T08:13:27.627 INFO:teuthology.orchestra.run.smithi028.stdout:(149/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 8.5 MB/s | 3.1 MB 00:00 2024-05-09T08:13:27.925 INFO:teuthology.orchestra.run.smithi184.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 2.1 MB/s | 889 kB 00:00 2024-05-09T08:13:28.000 INFO:teuthology.orchestra.run.smithi184.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 1.9 MB/s | 145 kB 00:00 2024-05-09T08:13:28.069 INFO:teuthology.orchestra.run.smithi028.stdout:(150/156): NetworkManager-1.47.5-1.el9.x86_64.r 3.9 MB/s | 2.3 MB 00:00 2024-05-09T08:13:28.119 INFO:teuthology.orchestra.run.smithi028.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 840 kB/s | 42 kB 00:00 2024-05-09T08:13:28.133 INFO:teuthology.orchestra.run.smithi184.stdout:(4/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 6.0 MB/s | 5.4 MB 00:00 2024-05-09T08:13:28.211 INFO:teuthology.orchestra.run.smithi028.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.6 MB/s | 248 kB 00:00 2024-05-09T08:13:28.284 INFO:teuthology.orchestra.run.smithi184.stdout:(5/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 10 MB/s | 1.5 MB 00:00 2024-05-09T08:13:28.403 INFO:teuthology.orchestra.run.smithi028.stdout:(153/156): NetworkManager-libnm-1.47.5-1.el9.x8 2.4 MB/s | 1.8 MB 00:00 2024-05-09T08:13:28.445 INFO:teuthology.orchestra.run.smithi028.stdout:(154/156): openssl-3.2.1-1.el9.x86_64.rpm 5.8 MB/s | 1.3 MB 00:00 2024-05-09T08:13:28.584 INFO:teuthology.orchestra.run.smithi184.stdout:(6/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 4.0 MB/s | 2.4 MB 00:00 2024-05-09T08:13:28.651 INFO:teuthology.orchestra.run.smithi184.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 13 MB/s | 4.7 MB 00:00 2024-05-09T08:13:28.803 INFO:teuthology.orchestra.run.smithi028.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.0 MB/s | 2.4 MB 00:00 2024-05-09T08:13:28.863 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-05-09T08:13:28.864 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-05-09T08:13:29.435 INFO:teuthology.orchestra.run.smithi184.stdout:(8/156): ceph-radosgw-19.0.0-2515.g96805dcf.el9 13 MB/s | 10 MB 00:00 2024-05-09T08:13:29.493 INFO:teuthology.orchestra.run.smithi184.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 433 kB/s | 25 kB 00:00 2024-05-09T08:13:30.693 INFO:teuthology.orchestra.run.smithi184.stdout:(10/156): ceph-common-19.0.0-2515.g96805dcf.el9 6.1 MB/s | 21 MB 00:03 2024-05-09T08:13:30.777 INFO:teuthology.orchestra.run.smithi184.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 2.6 MB/s | 222 kB 00:00 2024-05-09T08:13:30.844 INFO:teuthology.orchestra.run.smithi184.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 406 kB/s | 27 kB 00:00 2024-05-09T08:13:30.927 INFO:teuthology.orchestra.run.smithi184.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 380 kB/s | 32 kB 00:00 2024-05-09T08:13:31.061 INFO:teuthology.orchestra.run.smithi184.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 5.5 MB/s | 746 kB 00:00 2024-05-09T08:13:31.161 INFO:teuthology.orchestra.run.smithi184.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 1.6 MB/s | 163 kB 00:00 2024-05-09T08:13:31.262 INFO:teuthology.orchestra.run.smithi184.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 1.2 MB/s | 127 kB 00:00 2024-05-09T08:13:31.378 INFO:teuthology.orchestra.run.smithi184.stdout:(17/156): libradosstriper1-19.0.0-2515.g96805dc 4.2 MB/s | 500 kB 00:00 2024-05-09T08:13:31.770 INFO:teuthology.orchestra.run.smithi184.stdout:(18/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 5.3 MB/s | 17 MB 00:03 2024-05-09T08:13:31.871 INFO:teuthology.orchestra.run.smithi184.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 450 kB/s | 45 kB 00:00 2024-05-09T08:13:31.971 INFO:teuthology.orchestra.run.smithi184.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 1.3 MB/s | 135 kB 00:00 2024-05-09T08:13:32.038 INFO:teuthology.orchestra.run.smithi184.stdout:(21/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 8.1 MB/s | 5.3 MB 00:00 2024-05-09T08:13:32.063 INFO:teuthology.orchestra.run.smithi184.stdout:(22/156): python3-cephfs-19.0.0-2515.g96805dcf. 1.7 MB/s | 162 kB 00:00 2024-05-09T08:13:32.139 INFO:teuthology.orchestra.run.smithi184.stdout:(23/156): python3-rados-19.0.0-2515.g96805dcf.e 3.1 MB/s | 321 kB 00:00 2024-05-09T08:13:32.165 INFO:teuthology.orchestra.run.smithi184.stdout:(24/156): python3-rbd-19.0.0-2515.g96805dcf.el9 2.9 MB/s | 298 kB 00:00 2024-05-09T08:13:32.223 INFO:teuthology.orchestra.run.smithi184.stdout:(25/156): python3-rgw-19.0.0-2515.g96805dcf.el9 1.2 MB/s | 99 kB 00:00 2024-05-09T08:13:32.241 INFO:teuthology.orchestra.run.smithi184.stdout:(26/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 1.1 MB/s | 85 kB 00:00 2024-05-09T08:13:32.340 INFO:teuthology.orchestra.run.smithi184.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 1.7 MB/s | 171 kB 00:00 2024-05-09T08:13:32.435 INFO:teuthology.orchestra.run.smithi184.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 346 kB/s | 32 kB 00:00 2024-05-09T08:13:32.549 INFO:teuthology.orchestra.run.smithi184.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 1.2 MB/s | 142 kB 00:00 2024-05-09T08:13:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:(30/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 6.0 MB/s | 3.1 MB 00:00 2024-05-09T08:13:33.241 INFO:teuthology.orchestra.run.smithi184.stdout:(31/156): ceph-test-19.0.0-2515.g96805dcf.el9.x 13 MB/s | 48 MB 00:03 2024-05-09T08:13:33.287 INFO:teuthology.orchestra.run.smithi028.stdout:(156/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 990 kB/s | 16 MB 00:16 2024-05-09T08:13:33.292 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:13:33.292 INFO:teuthology.orchestra.run.smithi028.stdout:Total 6.5 MB/s | 225 MB 00:34 2024-05-09T08:13:33.309 INFO:teuthology.orchestra.run.smithi184.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 3.7 MB/s | 249 kB 00:00 2024-05-09T08:13:33.358 INFO:teuthology.orchestra.run.smithi184.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 993 kB/s | 49 kB 00:00 2024-05-09T08:13:33.409 INFO:teuthology.orchestra.run.smithi184.stdout:(34/156): ceph-mgr-dashboard-19.0.0-2515.g96805 4.1 MB/s | 3.5 MB 00:00 2024-05-09T08:13:33.434 INFO:teuthology.orchestra.run.smithi184.stdout:(35/156): ceph-prometheus-alerts-19.0.0-2515.g9 218 kB/s | 16 kB 00:00 2024-05-09T08:13:33.484 INFO:teuthology.orchestra.run.smithi184.stdout:(36/156): ceph-volume-19.0.0-2515.g96805dcf.el9 3.4 MB/s | 264 kB 00:00 2024-05-09T08:13:33.543 INFO:teuthology.orchestra.run.smithi184.stdout:(37/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 6.9 MB/s | 763 kB 00:00 2024-05-09T08:13:33.568 INFO:teuthology.orchestra.run.smithi184.stdout:(38/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 218 kB/s | 18 kB 00:00 2024-05-09T08:13:33.722 INFO:teuthology.orchestra.run.smithi184.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 449 kB/s | 80 kB 00:00 2024-05-09T08:13:33.810 INFO:teuthology.orchestra.run.smithi184.stdout:(40/156): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-05-09T08:13:33.836 INFO:teuthology.orchestra.run.smithi184.stdout:(41/156): libquadmath-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 195 kB 00:00 2024-05-09T08:13:33.870 INFO:teuthology.orchestra.run.smithi184.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-05-09T08:13:34.003 INFO:teuthology.orchestra.run.smithi184.stdout:(43/156): python3-cryptography-36.0.1-4.el9.x86 7.3 MB/s | 1.2 MB 00:00 2024-05-09T08:13:34.028 INFO:teuthology.orchestra.run.smithi184.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 672 kB/s | 106 kB 00:00 2024-05-09T08:13:34.062 INFO:teuthology.orchestra.run.smithi184.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-09T08:13:34.138 INFO:teuthology.orchestra.run.smithi184.stdout:(46/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-05-09T08:13:34.163 INFO:teuthology.orchestra.run.smithi184.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 930 kB/s | 125 kB 00:00 2024-05-09T08:13:34.534 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-05-09T08:13:34.680 INFO:teuthology.orchestra.run.smithi184.stdout:(48/156): ceph-mgr-diskprediction-local-19.0.0- 3.8 MB/s | 7.4 MB 00:01 2024-05-09T08:13:34.700 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-05-09T08:13:34.700 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-05-09T08:13:34.936 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-05-09T08:13:35.023 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T08:13:35.061 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T08:13:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T08:13:35.147 INFO:teuthology.orchestra.run.smithi184.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 683 kB/s | 689 kB 00:01 2024-05-09T08:13:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T08:13:35.183 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T08:13:35.197 INFO:teuthology.orchestra.run.smithi184.stdout:(50/156): bison-3.7.4-5.el9.x86_64.rpm 912 kB/s | 943 kB 00:01 2024-05-09T08:13:35.225 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T08:13:35.531 INFO:teuthology.orchestra.run.smithi184.stdout:(51/156): boost-program-options-1.75.0-8.el9.x8 126 kB/s | 107 kB 00:00 2024-05-09T08:13:35.543 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T08:13:35.658 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:13:35.779 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:13:35.801 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:13:35.831 INFO:teuthology.orchestra.run.smithi184.stdout:(52/156): elfutils-libelf-devel-0.190-2.el9.x86 31 kB/s | 21 kB 00:00 2024-05-09T08:13:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:13:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:13:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T08:13:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T08:13:36.155 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:13:36.172 INFO:teuthology.orchestra.run.smithi184.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 321 kB/s | 313 kB 00:00 2024-05-09T08:13:36.198 INFO:teuthology.orchestra.run.smithi184.stdout:(54/156): flexiblas-3.0.4-8.el9.x86_64.rpm 49 kB/s | 33 kB 00:00 2024-05-09T08:13:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:13:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:13:36.343 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:13:36.365 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:13:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:13:36.591 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:13:36.717 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:13:36.782 INFO:teuthology.orchestra.run.smithi184.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 30 kB/s | 18 kB 00:00 2024-05-09T08:13:36.807 INFO:teuthology.orchestra.run.smithi184.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 18 kB/s | 11 kB 00:00 2024-05-09T08:13:36.829 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:13:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:13:36.962 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-05-09T08:13:36.963 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-05-09T08:13:36.996 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:13:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:13:37.268 INFO:teuthology.orchestra.run.smithi184.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.1 MB/s | 3.0 MB 00:01 2024-05-09T08:13:37.293 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:13:37.432 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T08:13:37.527 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T08:13:37.563 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T08:13:37.598 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T08:13:37.649 INFO:teuthology.orchestra.run.smithi184.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 54 kB/s | 45 kB 00:00 2024-05-09T08:13:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T08:13:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T08:13:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T08:13:37.781 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T08:13:37.809 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T08:13:37.824 INFO:teuthology.orchestra.run.smithi184.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 154 kB/s | 160 kB 00:01 2024-05-09T08:13:37.879 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T08:13:37.931 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T08:13:37.970 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T08:13:38.006 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T08:13:38.081 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T08:13:38.116 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T08:13:38.158 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:13:38.188 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T08:13:38.257 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T08:13:38.302 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T08:13:38.377 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T08:13:38.416 INFO:teuthology.orchestra.run.smithi184.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 578 kB/s | 662 kB 00:01 2024-05-09T08:13:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T08:13:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T08:13:38.591 INFO:teuthology.orchestra.run.smithi184.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 33 kB/s | 25 kB 00:00 2024-05-09T08:13:38.942 INFO:teuthology.orchestra.run.smithi184.stdout:(62/156): libtool-2.4.6-45.el9.x86_64.rpm 451 kB/s | 582 kB 00:01 2024-05-09T08:13:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T08:13:39.175 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T08:13:39.204 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T08:13:39.230 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T08:13:39.268 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T08:13:39.310 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T08:13:39.325 INFO:teuthology.orchestra.run.smithi184.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 52 kB/s | 47 kB 00:00 2024-05-09T08:13:39.464 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T08:13:39.513 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T08:13:39.568 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T08:13:39.642 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T08:13:39.671 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T08:13:39.708 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T08:13:39.755 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T08:13:39.799 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T08:13:39.842 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T08:13:39.842 INFO:teuthology.orchestra.run.smithi184.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 234 kB/s | 292 kB 00:01 2024-05-09T08:13:39.884 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T08:13:40.021 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T08:13:40.042 INFO:teuthology.orchestra.run.smithi184.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 175 kB/s | 188 kB 00:01 2024-05-09T08:13:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T08:13:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T08:13:40.126 INFO:teuthology.orchestra.run.smithi184.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 45 kB/s | 35 kB 00:00 2024-05-09T08:13:40.144 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T08:13:40.240 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T08:13:40.278 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T08:13:40.317 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T08:13:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T08:13:40.393 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T08:13:40.469 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T08:13:40.604 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T08:13:40.642 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T08:13:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T08:13:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T08:13:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T08:13:40.782 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T08:13:40.812 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T08:13:40.847 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T08:13:40.871 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T08:13:40.913 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T08:13:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T08:13:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T08:13:41.086 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T08:13:41.117 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T08:13:41.173 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T08:13:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T08:13:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T08:13:41.659 INFO:teuthology.orchestra.run.smithi184.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 2.6 MB/s | 4.7 MB 00:01 2024-05-09T08:13:41.701 INFO:teuthology.orchestra.run.smithi184.stdout:(68/156): postgresql-13.14-1.el9.x86_64.rpm 1.0 MB/s | 1.6 MB 00:01 2024-05-09T08:13:41.777 INFO:teuthology.orchestra.run.smithi184.stdout:(69/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 2.6 MB/s | 4.4 MB 00:01 2024-05-09T08:13:42.064 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T08:13:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T08:13:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T08:13:42.251 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T08:13:42.299 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T08:13:42.388 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T08:13:42.426 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T08:13:42.603 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T08:13:42.632 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T08:13:42.709 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T08:13:42.727 INFO:teuthology.orchestra.run.smithi184.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 130 kB/s | 133 kB 00:01 2024-05-09T08:13:43.002 INFO:teuthology.orchestra.run.smithi184.stdout:(71/156): postgresql-contrib-13.14-1.el9.x86_64 679 kB/s | 911 kB 00:01 2024-05-09T08:13:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-05-09T08:13:43.368 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T08:13:43.425 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T08:13:43.436 INFO:teuthology.orchestra.run.smithi184.stdout:(72/156): postgresql-server-13.14-1.el9.x86_64. 3.5 MB/s | 5.8 MB 00:01 2024-05-09T08:13:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T08:13:43.488 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T08:13:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T08:13:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T08:13:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T08:13:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T08:13:43.928 INFO:teuthology.orchestra.run.smithi184.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 94 kB/s | 87 kB 00:00 2024-05-09T08:13:43.998 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T08:13:44.053 INFO:teuthology.orchestra.run.smithi184.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 779 kB/s | 1.0 MB 00:01 2024-05-09T08:13:44.110 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:13:44.235 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:13:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:13:44.337 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:13:44.366 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:13:44.561 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T08:13:44.613 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T08:13:44.640 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:13:44.723 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T08:13:44.745 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:13:44.774 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:13:44.811 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T08:13:44.837 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:13:44.839 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T08:13:44.867 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:13:44.872 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T08:13:44.890 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:13:45.095 INFO:teuthology.orchestra.run.smithi184.stdout:(75/156): python3-babel-2.9.1-2.el9.noarch.rpm 3.6 MB/s | 6.0 MB 00:01 2024-05-09T08:13:45.120 INFO:teuthology.orchestra.run.smithi184.stdout:(76/156): python3-devel-3.9.18-3.el9.x86_64.rpm 205 kB/s | 244 kB 00:01 2024-05-09T08:13:45.170 INFO:teuthology.orchestra.run.smithi184.stdout:(77/156): python3-jinja2-2.11.3-5.el9.noarch.rp 222 kB/s | 248 kB 00:01 2024-05-09T08:13:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T08:13:45.219 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:13:45.247 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:13:45.301 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:13:45.332 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T08:13:45.368 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T08:13:45.416 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:13:45.573 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:13:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:13:45.721 INFO:teuthology.orchestra.run.smithi184.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 32 kB/s | 19 kB 00:00 2024-05-09T08:13:45.746 INFO:teuthology.orchestra.run.smithi184.stdout:(79/156): python3-jsonpatch-1.21-16.el9.noarch. 39 kB/s | 26 kB 00:00 2024-05-09T08:13:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:13:45.809 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:13:45.836 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T08:13:45.872 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:13:45.943 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:13:45.973 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T08:13:46.017 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:13:46.135 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T08:13:46.138 INFO:teuthology.orchestra.run.smithi184.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 178 kB/s | 172 kB 00:00 2024-05-09T08:13:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T08:13:46.265 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T08:13:46.300 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T08:13:46.375 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T08:13:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T08:13:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T08:13:46.488 INFO:teuthology.orchestra.run.smithi184.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 45 kB/s | 35 kB 00:00 2024-05-09T08:13:46.492 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T08:13:46.519 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T08:13:46.591 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T08:13:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T08:13:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T08:13:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T08:13:46.803 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T08:13:46.844 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T08:13:46.888 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:13:46.915 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T08:13:46.987 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T08:13:47.031 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T08:13:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T08:13:47.172 INFO:teuthology.orchestra.run.smithi184.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 242 kB/s | 250 kB 00:01 2024-05-09T08:13:47.247 INFO:teuthology.orchestra.run.smithi184.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.4 MB/s | 5.1 MB 00:01 2024-05-09T08:13:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T08:13:47.316 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T08:13:47.480 INFO:teuthology.orchestra.run.smithi184.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 224 kB/s | 222 kB 00:00 2024-05-09T08:13:47.898 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T08:13:47.931 INFO:teuthology.orchestra.run.smithi184.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 62 kB/s | 42 kB 00:00 2024-05-09T08:13:47.956 INFO:teuthology.orchestra.run.smithi184.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 98 kB/s | 77 kB 00:00 2024-05-09T08:13:47.989 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T08:13:48.024 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T08:13:48.067 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T08:13:48.105 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T08:13:48.147 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T08:13:48.264 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:13:48.293 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T08:13:48.295 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:13:48.349 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T08:13:48.397 INFO:teuthology.orchestra.run.smithi184.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 294 kB/s | 269 kB 00:00 2024-05-09T08:13:48.406 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T08:13:48.487 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T08:13:48.525 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T08:13:48.562 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T08:13:48.609 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T08:13:48.644 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T08:13:48.688 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T08:13:48.730 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T08:13:48.831 INFO:teuthology.orchestra.run.smithi184.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 177 kB/s | 159 kB 00:00 2024-05-09T08:13:48.838 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:13:48.871 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T08:13:48.881 INFO:teuthology.orchestra.run.smithi184.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 301 kB/s | 279 kB 00:00 2024-05-09T08:13:48.882 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:13:48.918 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T08:13:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T08:13:48.966 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:13:48.966 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T08:13:48.966 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T08:13:48.966 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:13:48.990 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T08:13:48.999 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:13:49.088 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T08:13:49.124 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T08:13:49.140 INFO:teuthology.orchestra.run.smithi184.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 69 kB/s | 51 kB 00:00 2024-05-09T08:13:49.173 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T08:13:49.205 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T08:13:49.230 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T08:13:49.325 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T08:13:49.463 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T08:13:49.496 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T08:13:49.531 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T08:13:49.557 INFO:teuthology.orchestra.run.smithi184.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 74 kB/s | 54 kB 00:00 2024-05-09T08:13:49.575 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T08:13:49.614 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T08:13:49.654 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T08:13:49.684 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T08:13:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T08:13:49.752 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T08:13:49.785 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T08:13:49.819 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T08:13:49.857 INFO:teuthology.orchestra.run.smithi184.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 60 kB/s | 42 kB 00:00 2024-05-09T08:13:49.908 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T08:13:49.959 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T08:13:49.989 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T08:13:50.036 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T08:13:50.139 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T08:13:50.191 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T08:13:50.258 INFO:teuthology.orchestra.run.smithi184.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 63 kB/s | 44 kB 00:00 2024-05-09T08:13:50.658 INFO:teuthology.orchestra.run.smithi184.stdout:(94/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.2 MB/s | 16 MB 00:01 2024-05-09T08:13:50.816 INFO:teuthology.orchestra.run.smithi184.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 230 kB/s | 221 kB 00:00 2024-05-09T08:13:50.956 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T08:13:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T08:13:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T08:13:51.116 INFO:teuthology.orchestra.run.smithi184.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 42 kB/s | 36 kB 00:00 2024-05-09T08:13:51.140 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T08:13:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T08:13:51.279 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T08:13:51.323 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T08:13:51.375 INFO:teuthology.orchestra.run.smithi184.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 68 kB/s | 47 kB 00:00 2024-05-09T08:13:51.508 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T08:13:51.537 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T08:13:51.619 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T08:13:51.725 INFO:teuthology.orchestra.run.smithi184.stdout:(98/156): socat-1.7.4.1-5.el9.x86_64.rpm 335 kB/s | 305 kB 00:00 2024-05-09T08:13:51.851 INFO:teuthology.orchestra.run.smithi184.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 79 kB/s | 58 kB 00:00 2024-05-09T08:13:52.201 INFO:teuthology.orchestra.run.smithi184.stdout:(100/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 117 kB/s | 94 kB 00:00 2024-05-09T08:13:52.307 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T08:13:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T08:13:52.459 INFO:teuthology.orchestra.run.smithi184.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 87 kB/s | 64 kB 00:00 2024-05-09T08:13:52.560 INFO:teuthology.orchestra.run.smithi184.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 63 kB/s | 45 kB 00:00 2024-05-09T08:13:52.843 INFO:teuthology.orchestra.run.smithi184.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-05-09T08:13:53.110 INFO:teuthology.orchestra.run.smithi184.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 996 kB/s | 548 kB 00:00 2024-05-09T08:13:53.211 INFO:teuthology.orchestra.run.smithi184.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-05-09T08:13:53.328 INFO:teuthology.orchestra.run.smithi184.stdout:(106/156): gperftools-libs-2.9.1-2.el9.x86_64.r 639 kB/s | 309 kB 00:00 2024-05-09T08:13:53.428 INFO:teuthology.orchestra.run.smithi184.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-05-09T08:13:53.537 INFO:teuthology.orchestra.run.smithi184.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-05-09T08:13:53.578 INFO:teuthology.orchestra.run.smithi184.stdout:(109/156): protobuf-compiler-3.14.0-13.el9.x86_ 790 kB/s | 863 kB 00:01 2024-05-09T08:13:53.645 INFO:teuthology.orchestra.run.smithi184.stdout:(110/156): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-05-09T08:13:53.671 INFO:teuthology.orchestra.run.smithi184.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-05-09T08:13:53.754 INFO:teuthology.orchestra.run.smithi184.stdout:(112/156): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-05-09T08:13:53.762 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T08:13:53.841 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T08:13:53.846 INFO:teuthology.orchestra.run.smithi184.stdout:(113/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-05-09T08:13:53.869 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T08:13:53.871 INFO:teuthology.orchestra.run.smithi184.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-05-09T08:13:53.903 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T08:13:53.938 INFO:teuthology.orchestra.run.smithi184.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-05-09T08:13:53.964 INFO:teuthology.orchestra.run.smithi184.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-05-09T08:13:54.039 INFO:teuthology.orchestra.run.smithi184.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-05-09T08:13:54.064 INFO:teuthology.orchestra.run.smithi184.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-05-09T08:13:54.131 INFO:teuthology.orchestra.run.smithi184.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-05-09T08:13:54.156 INFO:teuthology.orchestra.run.smithi184.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-05-09T08:13:54.223 INFO:teuthology.orchestra.run.smithi184.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-05-09T08:13:54.234 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T08:13:54.257 INFO:teuthology.orchestra.run.smithi184.stdout:(122/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.3 MB/s | 839 kB 00:00 2024-05-09T08:13:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:13:54.316 INFO:teuthology.orchestra.run.smithi184.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-05-09T08:13:54.349 INFO:teuthology.orchestra.run.smithi184.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-05-09T08:13:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:13:54.391 INFO:teuthology.orchestra.run.smithi184.stdout:(125/156): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-05-09T08:13:54.412 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T08:13:54.417 INFO:teuthology.orchestra.run.smithi184.stdout:(126/156): python3-jaraco-classes-3.2.1-5.el9.n 175 kB/s | 18 kB 00:00 2024-05-09T08:13:54.442 INFO:teuthology.orchestra.run.smithi184.stdout:(127/156): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-05-09T08:13:54.448 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T08:13:54.476 INFO:teuthology.orchestra.run.smithi184.stdout:(128/156): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-05-09T08:13:54.501 INFO:teuthology.orchestra.run.smithi184.stdout:(129/156): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-05-09T08:13:54.559 INFO:teuthology.orchestra.run.smithi184.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 553 kB/s | 46 kB 00:00 2024-05-09T08:13:54.593 INFO:teuthology.orchestra.run.smithi184.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 853 kB/s | 79 kB 00:00 2024-05-09T08:13:54.627 INFO:teuthology.orchestra.run.smithi184.stdout:(132/156): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-05-09T08:13:54.644 INFO:teuthology.orchestra.run.smithi184.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 687 kB/s | 58 kB 00:00 2024-05-09T08:13:54.686 INFO:teuthology.orchestra.run.smithi184.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-05-09T08:13:54.711 INFO:teuthology.orchestra.run.smithi184.stdout:(135/156): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-05-09T08:13:54.737 INFO:teuthology.orchestra.run.smithi184.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 974 kB/s | 90 kB 00:00 2024-05-09T08:13:54.771 INFO:teuthology.orchestra.run.smithi184.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 336 kB/s | 28 kB 00:00 2024-05-09T08:13:54.795 INFO:teuthology.orchestra.run.smithi184.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-05-09T08:13:54.829 INFO:teuthology.orchestra.run.smithi184.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-05-09T08:13:54.854 INFO:teuthology.orchestra.run.smithi184.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 705 kB/s | 59 kB 00:00 2024-05-09T08:13:54.879 INFO:teuthology.orchestra.run.smithi184.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 426 kB/s | 36 kB 00:00 2024-05-09T08:13:54.921 INFO:teuthology.orchestra.run.smithi184.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-05-09T08:13:54.947 INFO:teuthology.orchestra.run.smithi184.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-05-09T08:13:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T08:13:54.964 INFO:teuthology.orchestra.run.smithi184.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-05-09T08:13:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:13:55.006 INFO:teuthology.orchestra.run.smithi184.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-05-09T08:13:55.031 INFO:teuthology.orchestra.run.smithi184.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-05-09T08:13:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:13:55.065 INFO:teuthology.orchestra.run.smithi184.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-05-09T08:13:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T08:13:55.140 INFO:teuthology.orchestra.run.smithi184.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-09T08:13:55.457 INFO:teuthology.orchestra.run.smithi184.stdout:(149/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 8.0 MB/s | 3.1 MB 00:00 2024-05-09T08:13:55.524 INFO:teuthology.orchestra.run.smithi184.stdout:(150/156): NetworkManager-1.47.5-1.el9.x86_64.r 6.0 MB/s | 2.3 MB 00:00 2024-05-09T08:13:55.574 INFO:teuthology.orchestra.run.smithi184.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 839 kB/s | 42 kB 00:00 2024-05-09T08:13:55.666 INFO:teuthology.orchestra.run.smithi184.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.7 MB/s | 248 kB 00:00 2024-05-09T08:13:55.733 INFO:teuthology.orchestra.run.smithi184.stdout:(153/156): NetworkManager-libnm-1.47.5-1.el9.x8 6.6 MB/s | 1.8 MB 00:00 2024-05-09T08:13:55.783 INFO:teuthology.orchestra.run.smithi184.stdout:(154/156): librados2-19.0.0-2515.g96805dcf.el9. 4.5 MB/s | 3.4 MB 00:00 2024-05-09T08:13:55.883 INFO:teuthology.orchestra.run.smithi184.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-05-09T08:13:55.917 INFO:teuthology.orchestra.run.smithi184.stdout:(156/156): openssl-3.2.1-1.el9.x86_64.rpm 5.4 MB/s | 1.3 MB 00:00 2024-05-09T08:13:55.924 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:13:55.924 INFO:teuthology.orchestra.run.smithi184.stdout:Total 7.6 MB/s | 225 MB 00:29 2024-05-09T08:13:57.104 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-05-09T08:13:57.255 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-05-09T08:13:57.255 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-05-09T08:13:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:13:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:13:58.043 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:13:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:13:58.180 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:13:58.180 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T08:13:58.180 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T08:13:58.180 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:13:58.221 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:13:59.131 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-05-09T08:13:59.132 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-05-09T08:14:00.595 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:14:00.595 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /sys 2024-05-09T08:14:00.595 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /proc 2024-05-09T08:14:00.595 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /mnt 2024-05-09T08:14:00.595 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /var/tmp 2024-05-09T08:14:00.595 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /home 2024-05-09T08:14:00.595 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /root 2024-05-09T08:14:00.596 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /tmp 2024-05-09T08:14:00.596 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:00.768 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:14:00.783 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:14:00.783 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:00.783 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T08:14:00.783 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:02.272 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:14:02.298 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:14:02.298 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:02.298 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T08:14:02.298 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:14:02.298 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:14:02.298 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:02.531 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:14:02.553 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:14:02.553 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:02.553 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T08:14:02.553 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:14:02.553 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:14:02.554 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:14:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:14:03.021 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:03.021 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T08:14:03.021 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:14:03.021 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:14:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:03.120 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T08:14:03.177 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T08:14:03.233 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T08:14:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T08:14:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T08:14:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T08:14:03.641 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T08:14:03.782 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T08:14:03.823 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T08:14:03.864 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T08:14:03.891 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T08:14:04.299 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:14:04.331 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:14:04.363 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T08:14:04.426 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T08:14:04.454 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:14:05.196 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-05-09T08:14:05.279 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T08:14:05.317 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T08:14:05.370 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T08:14:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T08:14:05.431 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T08:14:05.473 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T08:14:05.494 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:14:05.531 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:14:05.795 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T08:14:05.907 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:14:06.028 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:14:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:14:06.150 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:14:06.179 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:14:06.326 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T08:14:06.376 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T08:14:06.404 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:14:06.496 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:14:06.508 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:14:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:14:06.530 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:14:06.592 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:14:06.614 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:14:06.626 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:14:06.636 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:14:06.786 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T08:14:06.817 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:14:06.831 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:14:06.839 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:14:06.839 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:06.839 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T08:14:06.839 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:14:06.839 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:14:06.839 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:06.892 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:14:06.903 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:14:06.974 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:14:07.001 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T08:14:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T08:14:07.104 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:14:07.156 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:14:07.261 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:14:07.332 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:14:07.554 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:14:07.667 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T08:14:07.761 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T08:14:07.805 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T08:14:07.839 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T08:14:07.905 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T08:14:07.939 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T08:14:07.960 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T08:14:07.983 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T08:14:08.003 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:14:08.015 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T08:14:08.026 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:14:08.026 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:08.026 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T08:14:08.026 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:14:08.026 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:14:08.027 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T08:14:08.070 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:14:08.092 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:14:08.092 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:08.092 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T08:14:08.092 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:08.112 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T08:14:08.137 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:14:08.159 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:14:08.159 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:08.159 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T08:14:08.160 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:14:08.160 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:14:08.160 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:08.165 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T08:14:08.204 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T08:14:08.239 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T08:14:08.317 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T08:14:08.350 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T08:14:08.393 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:14:08.422 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T08:14:08.463 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:14:08.485 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:14:08.486 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:08.486 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T08:14:08.486 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:14:08.486 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:14:08.486 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:08.515 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T08:14:08.562 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T08:14:08.635 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T08:14:08.768 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T08:14:08.823 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T08:14:09.342 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T08:14:09.426 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T08:14:09.456 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T08:14:09.490 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T08:14:09.528 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T08:14:09.570 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T08:14:09.721 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T08:14:09.781 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T08:14:09.836 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T08:14:09.910 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T08:14:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T08:14:09.977 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T08:14:10.023 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T08:14:10.060 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T08:14:10.094 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T08:14:10.136 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T08:14:10.270 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T08:14:10.311 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:14:10.311 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-05-09T08:14:10.311 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-05-09T08:14:10.311 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-05-09T08:14:10.311 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-05-09T08:14:10.311 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-05-09T08:14:10.311 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-05-09T08:14:10.311 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-05-09T08:14:10.311 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T08:14:10.356 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T08:14:10.397 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T08:14:10.486 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:14:10.490 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T08:14:10.503 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:14:10.503 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:10.503 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T08:14:10.503 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:10.531 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T08:14:10.570 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T08:14:10.596 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T08:14:10.621 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T08:14:10.694 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T08:14:10.821 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T08:14:10.853 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T08:14:10.881 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T08:14:10.916 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T08:14:10.955 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T08:14:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T08:14:11.016 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T08:14:11.051 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T08:14:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T08:14:11.135 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T08:14:11.169 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T08:14:11.256 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T08:14:11.307 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T08:14:11.338 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T08:14:11.386 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T08:14:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T08:14:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T08:14:12.072 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:14:12.101 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:14:12.101 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:12.101 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T08:14:12.101 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:14:12.101 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:14:12.101 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:12.242 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T08:14:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T08:14:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T08:14:12.339 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:14:12.364 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:14:12.364 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T08:14:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:14:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:14:12.365 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:12.434 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T08:14:12.487 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T08:14:12.574 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T08:14:12.645 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T08:14:12.840 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:14:12.865 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:14:12.865 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:12.865 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T08:14:12.865 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:14:12.866 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:14:12.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:12.941 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T08:14:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T08:14:13.000 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T08:14:13.029 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T08:14:13.099 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T08:14:13.099 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T08:14:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T08:14:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T08:14:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T08:14:13.123 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T08:14:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T08:14:13.294 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T08:14:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T08:14:13.358 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T08:14:13.378 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T08:14:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T08:14:13.437 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T08:14:13.438 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T08:14:13.476 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T08:14:13.527 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T08:14:13.677 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T08:14:13.724 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T08:14:13.760 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T08:14:13.764 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T08:14:13.791 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T08:14:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T08:14:14.110 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:14:14.148 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:14:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:14:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:14:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:14:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T08:14:14.330 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T08:14:14.357 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:14:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T08:14:15.093 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T08:14:15.094 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T08:14:15.135 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T08:14:15.177 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T08:14:15.205 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T08:14:15.219 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T08:14:15.238 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T08:14:15.280 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T08:14:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T08:14:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:14:15.399 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:14:15.435 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T08:14:15.435 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:14:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:14:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:14:15.516 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:14:15.541 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T08:14:15.546 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:14:15.546 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:14:15.549 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T08:14:15.585 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:14:15.676 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:14:15.706 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T08:14:15.726 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:14:15.726 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:14:15.742 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T08:14:15.819 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:14:15.883 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:14:16.214 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T08:14:16.238 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:14:16.326 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:14:16.356 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T08:14:16.500 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:14:16.517 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:14:16.628 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:14:16.796 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T08:14:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:14:16.853 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:14:16.853 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:16.853 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T08:14:16.853 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:14:16.853 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:14:16.853 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:16.903 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:14:16.915 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:14:17.024 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T08:14:17.082 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T08:14:18.034 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T08:14:18.064 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:14:18.089 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:14:18.089 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:18.090 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T08:14:18.090 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:14:18.090 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:14:18.090 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:18.131 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:14:18.156 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:14:18.156 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:18.156 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T08:14:18.157 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:14:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:14:18.240 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:18.240 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T08:14:18.240 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:14:18.241 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:14:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:18.557 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:14:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:14:18.583 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:18.583 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T08:14:18.583 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:14:18.583 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:14:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:18.642 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:14:18.686 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:14:19.225 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:14:19.273 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:14:19.366 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:14:19.366 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T08:14:19.366 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T08:14:19.366 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:19.399 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:14:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:14:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T08:14:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T08:14:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T08:14:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T08:14:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T08:14:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T08:14:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T08:14:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T08:14:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:14:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T08:14:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T08:14:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T08:14:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T08:14:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T08:14:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T08:14:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T08:14:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T08:14:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T08:14:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T08:14:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T08:14:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T08:14:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T08:14:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T08:14:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T08:14:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T08:14:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T08:14:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T08:14:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T08:14:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T08:14:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T08:14:23.378 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T08:14:23.420 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T08:14:23.461 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T08:14:23.492 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T08:14:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T08:14:23.724 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T08:14:23.773 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T08:14:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T08:14:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T08:14:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T08:14:24.062 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:14:24.062 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:24.062 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-05-09T08:14:24.062 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:24.062 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:24.062 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:24.062 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.063 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T08:14:24.065 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T08:14:24.066 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T08:14:24.067 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T08:14:24.069 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T08:14:24.070 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.071 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T08:14:24.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T08:14:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T08:14:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T08:14:24.074 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-05-09T08:14:24.074 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-05-09T08:14:24.482 DEBUG:teuthology.parallel:result is None 2024-05-09T08:14:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:14:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:14:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:14:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T08:14:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T08:14:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T08:14:25.709 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T08:14:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T08:14:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T08:14:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:14:25.869 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:14:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T08:14:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:14:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:14:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T08:14:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:14:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:14:26.212 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:14:26.213 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:14:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:14:26.357 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:14:30.886 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:14:30.886 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /sys 2024-05-09T08:14:30.886 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /proc 2024-05-09T08:14:30.886 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /mnt 2024-05-09T08:14:30.886 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /var/tmp 2024-05-09T08:14:30.887 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /home 2024-05-09T08:14:30.887 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /root 2024-05-09T08:14:30.887 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /tmp 2024-05-09T08:14:30.887 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:31.047 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:14:31.062 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:14:31.062 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:31.062 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T08:14:31.062 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:31.795 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T08:14:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T08:14:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T08:14:31.812 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T08:14:31.812 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T08:14:31.812 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T08:14:31.812 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T08:14:31.813 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T08:14:31.813 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T08:14:31.813 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:14:31.813 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T08:14:31.813 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T08:14:31.813 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T08:14:31.814 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T08:14:31.814 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T08:14:31.814 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T08:14:31.814 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T08:14:31.814 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T08:14:31.815 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T08:14:31.815 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T08:14:31.815 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T08:14:31.815 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T08:14:31.815 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T08:14:31.816 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T08:14:31.816 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T08:14:31.816 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T08:14:31.816 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T08:14:31.816 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T08:14:31.816 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T08:14:31.817 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T08:14:31.817 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T08:14:31.817 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T08:14:31.817 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T08:14:31.817 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T08:14:31.818 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T08:14:31.818 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T08:14:31.818 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T08:14:31.818 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T08:14:31.818 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T08:14:31.818 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T08:14:31.819 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T08:14:31.819 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T08:14:31.819 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T08:14:31.819 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T08:14:31.819 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T08:14:31.819 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T08:14:31.820 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T08:14:31.820 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T08:14:31.820 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T08:14:31.820 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T08:14:31.820 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T08:14:31.820 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T08:14:31.820 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T08:14:31.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T08:14:31.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T08:14:31.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T08:14:31.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T08:14:31.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T08:14:31.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T08:14:31.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T08:14:31.822 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T08:14:31.822 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T08:14:31.822 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T08:14:31.822 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T08:14:31.822 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T08:14:31.822 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T08:14:31.822 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T08:14:31.823 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T08:14:31.823 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T08:14:31.823 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T08:14:31.823 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T08:14:31.823 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T08:14:31.823 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T08:14:31.823 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T08:14:31.824 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T08:14:31.824 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T08:14:31.824 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T08:14:31.824 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T08:14:31.824 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T08:14:31.824 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T08:14:31.824 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T08:14:31.825 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T08:14:31.825 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T08:14:31.825 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T08:14:31.825 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T08:14:31.825 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T08:14:31.825 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T08:14:31.826 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T08:14:31.826 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T08:14:31.826 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T08:14:31.826 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T08:14:31.826 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T08:14:31.826 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T08:14:31.827 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T08:14:31.828 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T08:14:31.829 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T08:14:31.830 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T08:14:31.830 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T08:14:31.830 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T08:14:31.830 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T08:14:31.830 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T08:14:31.830 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T08:14:32.533 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:14:32.559 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:14:32.559 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:32.559 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T08:14:32.559 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:14:32.559 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:14:32.559 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:32.793 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:14:32.816 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:14:32.816 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:32.816 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T08:14:32.816 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:14:32.816 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:14:32.817 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:33.278 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:14:33.301 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:14:33.301 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:33.301 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T08:14:33.301 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:14:33.302 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:14:33.302 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:33.399 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T08:14:33.456 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T08:14:33.511 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T08:14:33.696 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T08:14:33.745 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T08:14:33.823 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T08:14:33.911 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T08:14:34.051 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T08:14:34.103 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T08:14:34.136 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T08:14:34.163 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T08:14:34.546 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:14:34.578 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:14:34.597 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:14:34.597 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:34.597 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-05-09T08:14:34.598 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:34.598 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:34.598 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:34.598 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:34.598 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.598 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.598 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T08:14:34.599 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T08:14:34.599 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-05-09T08:14:34.599 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T08:14:34.599 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T08:14:34.599 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T08:14:34.599 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T08:14:34.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.601 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.601 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.601 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.601 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.601 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.601 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.601 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.602 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.602 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.602 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.602 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.602 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.602 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.603 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.603 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.603 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.603 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:34.603 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T08:14:34.603 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T08:14:34.603 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T08:14:34.603 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T08:14:34.604 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T08:14:34.604 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T08:14:34.604 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T08:14:34.604 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T08:14:34.604 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T08:14:34.604 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T08:14:34.604 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T08:14:34.605 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.605 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.605 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.605 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T08:14:34.605 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T08:14:34.605 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T08:14:34.606 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T08:14:34.606 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T08:14:34.606 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.606 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.606 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T08:14:34.606 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.607 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T08:14:34.607 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T08:14:34.607 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T08:14:34.607 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T08:14:34.607 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T08:14:34.607 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T08:14:34.607 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T08:14:34.608 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T08:14:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T08:14:34.610 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T08:14:34.610 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T08:14:34.610 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T08:14:34.610 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T08:14:34.610 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T08:14:34.610 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T08:14:34.610 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T08:14:34.610 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T08:14:34.611 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T08:14:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T08:14:34.613 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T08:14:34.614 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T08:14:34.615 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T08:14:34.615 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T08:14:34.615 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T08:14:34.615 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:14:34.615 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-05-09T08:14:34.671 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T08:14:34.701 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:14:35.026 DEBUG:teuthology.parallel:result is None 2024-05-09T08:14:35.731 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:14:35.787 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:14:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:14:36.787 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:14:36.889 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:14:37.050 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T08:14:37.089 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:14:37.112 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:14:37.113 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:37.113 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T08:14:37.113 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:14:37.113 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:14:37.113 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:37.156 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:14:37.168 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:14:37.265 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T08:14:37.312 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T08:14:38.223 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T08:14:38.251 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:14:38.274 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:14:38.274 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:38.274 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T08:14:38.274 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:14:38.274 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:14:38.274 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:38.318 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:14:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:14:38.341 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:38.341 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T08:14:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:38.386 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:14:38.408 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:14:38.408 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:38.408 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T08:14:38.408 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:14:38.408 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:14:38.408 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:14:38.734 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:14:38.734 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:14:38.734 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T08:14:38.734 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:14:38.734 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:14:38.735 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:43.215 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T08:14:43.256 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T08:14:43.296 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T08:14:43.354 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T08:14:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T08:14:43.561 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T08:14:43.619 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T08:14:43.650 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T08:14:43.679 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T08:14:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T08:14:44.330 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:14:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:14:44.385 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:14:45.206 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T08:14:45.296 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T08:14:45.337 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T08:14:45.417 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T08:14:45.465 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T08:14:45.491 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T08:14:45.491 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:14:45.576 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:14:45.609 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T08:14:45.609 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:14:45.693 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:14:45.718 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T08:14:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:14:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:14:45.912 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:14:45.912 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:14:45.996 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:14:46.060 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:14:51.700 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:14:51.700 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T08:14:51.700 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T08:14:51.700 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T08:14:51.700 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T08:14:51.701 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T08:14:51.702 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T08:14:51.702 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T08:14:51.702 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T08:14:51.702 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T08:14:51.702 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T08:14:51.702 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T08:14:51.702 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T08:14:51.702 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T08:14:51.715 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T08:14:51.715 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T08:14:51.715 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T08:14:51.715 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T08:14:51.715 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T08:14:51.715 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T08:14:51.715 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T08:14:51.716 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T08:14:51.717 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T08:14:51.718 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T08:14:51.719 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T08:14:51.720 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T08:14:51.721 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T08:14:51.722 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T08:14:51.723 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T08:14:51.724 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T08:14:51.725 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T08:14:51.725 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T08:14:51.725 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T08:14:51.725 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T08:14:51.725 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T08:14:51.725 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T08:14:51.725 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T08:14:51.725 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T08:14:51.725 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T08:14:54.314 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:14:54.314 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:54.314 INFO:teuthology.orchestra.run.smithi184.stdout:Upgraded: 2024-05-09T08:14:54.314 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:54.314 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:54.314 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:54.314 INFO:teuthology.orchestra.run.smithi184.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T08:14:54.314 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.315 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.316 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T08:14:54.317 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T08:14:54.318 INFO:teuthology.orchestra.run.smithi184.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T08:14:54.319 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T08:14:54.320 INFO:teuthology.orchestra.run.smithi184.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T08:14:54.321 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T08:14:54.322 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T08:14:54.323 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T08:14:54.324 INFO:teuthology.orchestra.run.smithi184.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T08:14:54.325 INFO:teuthology.orchestra.run.smithi184.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T08:14:54.325 INFO:teuthology.orchestra.run.smithi184.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T08:14:54.325 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T08:14:54.325 INFO:teuthology.orchestra.run.smithi184.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T08:14:54.325 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:14:54.325 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-05-09T08:14:54.702 DEBUG:teuthology.parallel:result is None 2024-05-09T08:14:54.702 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:14:54.918 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T08:14:54.946 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T08:14:54.947 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T08:14:54.947 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T08:14:54.949 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:14:55.170 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T08:14:55.196 INFO:teuthology.orchestra.run.smithi165.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T08:14:55.196 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T08:14:55.196 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T08:14:55.198 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:14:55.415 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T08:14:55.439 INFO:teuthology.orchestra.run.smithi184.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T08:14:55.439 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T08:14:55.440 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T08:14:55.441 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-09T08:14:55.441 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:14:55.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T08:14:55.476 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:14:55.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T08:14:55.507 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:14:55.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T08:14:55.536 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-09T08:14:55.536 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:14:55.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T08:14:55.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T08:14:55.638 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:14:55.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T08:14:55.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T08:14:55.735 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:14:55.735 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T08:14:55.762 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T08:14:55.828 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-09T08:14:55.828 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:14:55.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T08:14:55.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T08:14:55.930 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:14:55.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T08:14:55.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T08:14:56.018 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:14:56.019 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T08:14:56.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T08:14:56.110 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-09T08:14:56.110 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:14:56.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T08:14:56.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T08:14:56.206 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:14:56.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T08:14:56.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T08:14:56.294 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:14:56.294 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T08:14:56.320 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T08:14:56.386 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-09T08:14:56.480 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-05-09T08:14:56.481 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:14:56.481 INFO:tasks.cephadm:Cluster fsid is 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:14:56.481 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-09T08:14:56.481 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.165', 'mon.c': '172.21.15.184'} 2024-05-09T08:14:56.481 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-05-09T08:14:56.481 INFO:tasks.cephadm:First mgr is x 2024-05-09T08:14:56.482 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-09T08:14:56.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-05-09T08:14:56.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-05-09T08:14:56.534 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname $(hostname -s) 2024-05-09T08:14:56.560 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-09T08:14:56.560 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:14:56.789 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55097/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2515-g96805dcf', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2515.g96805dcf'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-07 19:00:25.721799', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-09T08:14:57.008 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:14:57.010 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm 2024-05-09T08:14:57.010 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T08:14:57.380 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 08:14 /home/ubuntu/cephtest/cephadm 2024-05-09T08:14:57.380 DEBUG:teuthology.orchestra.run.smithi165:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T08:14:57.733 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 08:14 /home/ubuntu/cephtest/cephadm 2024-05-09T08:14:57.734 DEBUG:teuthology.orchestra.run.smithi184:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T08:14:58.080 INFO:teuthology.orchestra.run.smithi184.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 08:14 /home/ubuntu/cephtest/cephadm 2024-05-09T08:14:58.081 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T08:14:58.101 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T08:14:58.119 DEBUG:teuthology.orchestra.run.smithi184:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T08:14:58.152 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc on all hosts... 2024-05-09T08:14:58.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T08:14:58.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T08:14:58.162 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T08:14:59.687 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:14:59.786 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:14:59.879 INFO:teuthology.orchestra.run.smithi184.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:15:28.304 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:15:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:15:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T08:15:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-05-09T08:15:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T08:15:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:15:28.305 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:15:31.477 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-05-09T08:15:31.477 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:15:31.477 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T08:15:31.477 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-05-09T08:15:31.477 INFO:teuthology.orchestra.run.smithi165.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T08:15:31.477 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-05-09T08:15:31.477 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-05-09T08:15:34.742 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-05-09T08:15:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:15:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T08:15:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "repo_digests": [ 2024-05-09T08:15:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T08:15:34.742 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-05-09T08:15:34.742 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-05-09T08:15:34.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-05-09T08:15:34.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-05-09T08:15:34.834 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph 2024-05-09T08:15:34.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-05-09T08:15:34.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-05-09T08:15:34.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /etc/ceph 2024-05-09T08:15:34.942 INFO:tasks.cephadm:Writing seed config... 2024-05-09T08:15:34.945 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-09T08:15:34.945 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-09T08:15:34.945 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-09T08:15:34.945 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-09T08:15:34.945 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-09T08:15:34.945 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-09T08:15:34.946 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-09T08:15:34.947 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-09T08:15:34.948 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-09T08:15:34.949 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-05-09T08:15:34.950 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:15:34.950 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-09T08:15:34.971 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3891b212-0ddc-11ef-bc98-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-09T08:15:34.971 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.a.service 2024-05-09T08:15:35.014 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.x.service 2024-05-09T08:15:35.057 INFO:tasks.cephadm:Bootstrapping... 2024-05-09T08:15:35.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc -v bootstrap --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:15:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:15:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', '-v', 'bootstrap', '--fsid', '3891b212-0ddc-11ef-bc98-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-05-09T08:15:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-09T08:15:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-05-09T08:15:35.288 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-09T08:15:35.288 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-05-09T08:15:35.288 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-05-09T08:15:35.296 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-09T08:15:35.296 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-09T08:15:35.303 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-09T08:15:35.303 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-09T08:15:35.310 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-05-09T08:15:35.317 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-05-09T08:15:35.317 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-05-09T08:15:35.317 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-05-09T08:15:35.341 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-09T08:15:35.341 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-09T08:15:35.342 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-05-09T08:15:35.342 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-05-09T08:15:35.348 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-09T08:15:35.348 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-09T08:15:35.354 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-09T08:15:35.354 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-09T08:15:35.361 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-05-09T08:15:35.367 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-05-09T08:15:35.368 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-05-09T08:15:35.368 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-05-09T08:15:35.368 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:15:35.368 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 139750773108688 on /run/cephadm/3891b212-0ddc-11ef-bc98-c7b262605968.lock 2024-05-09T08:15:35.368 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 139750773108688 acquired on /run/cephadm/3891b212-0ddc-11ef-bc98-c7b262605968.lock 2024-05-09T08:15:35.368 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-05-09T08:15:35.369 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-05-09T08:15:35.369 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-05-09T08:15:35.372 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-05-09T08:15:35.372 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-05-09T08:15:35.375 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-09T08:15:35.375 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-05-09T08:15:35.375 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-05-09T08:15:35.378 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-09T08:15:35.378 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-09T08:15:35.378 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-09T08:15:35.378 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-09T08:15:35.378 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-05-09T08:15:35.378 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-09T08:15:35.379 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-05-09T08:15:35.380 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-05-09T08:15:35.380 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-09T08:15:35.380 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-09T08:15:35.380 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:15:35.865 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 2024-05-09T08:15:35.865 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:15:35.865 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-05-09T08:15:35.865 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:bf5b95bd68251da39c6cd2eb96c6ba4138bb37a62bdc39287908b25e92047d7e 2024-05-09T08:15:35.865 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:9852312f87792557815c3e860a5c4f8cbf2b59bbbde9b5f86ab07b438f905b2d 2024-05-09T08:15:35.865 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 2024-05-09T08:15:35.865 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-09T08:15:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:15:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:15:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-05-09T08:15:37.795 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-05-09T08:15:37.795 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-05-09T08:15:38.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQAqhjxm7hirBhAAeo8xnyGIrB1BzH/KcqH21A== 2024-05-09T08:15:39.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQArhjxmBDG7ABAAFJVTRsFxvbfp1o16WPcIBg== 2024-05-09T08:15:40.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQAshjxmj/AQARAA7CGCABeagAYZBwpeibZmoQ== 2024-05-09T08:15:40.715 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-05-09T08:15:41.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-09T08:15:41.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-09T08:15:41.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:15:41.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-09T08:15:41.613 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-09T08:15:41.613 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-05-09T08:15:41.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:15:41.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-09T08:15:41.613 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:15:41.613 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-05-09T08:15:42.626 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-05-09T08:15:42.838 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-09T08:15:42.984 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-09T08:15:43.160 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3891b212-0ddc-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-3891b212-0ddc-11ef-bc98-c7b262605968.target. 2024-05-09T08:15:43.161 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3891b212-0ddc-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-3891b212-0ddc-11ef-bc98-c7b262605968.target. 2024-05-09T08:15:43.441 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.a 2024-05-09T08:15:43.441 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.a.service: Unit ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.a.service not loaded. 2024-05-09T08:15:43.637 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3891b212-0ddc-11ef-bc98-c7b262605968.target.wants/ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.a.service → /etc/systemd/system/ceph-3891b212-0ddc-11ef-bc98-c7b262605968@.service. 2024-05-09T08:15:44.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:43 smithi028 podman[29716]: 2024-05-09 08:15:43.785400377 +0000 UTC m=+0.019108852 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:15:44.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:44 smithi028 podman[29716]: 2024-05-09 08:15:44.701454994 +0000 UTC m=+0.935163470 container create 15e512c044a53c703cd99005a844bcaf46c59cfefe4407b248440d1bd55368ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-09T08:15:46.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:46 smithi028 podman[29716]: 2024-05-09 08:15:46.613034493 +0000 UTC m=+2.846742978 container init 15e512c044a53c703cd99005a844bcaf46c59cfefe4407b248440d1bd55368ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:15:46.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:46 smithi028 podman[29716]: 2024-05-09 08:15:46.617710786 +0000 UTC m=+2.851419262 container start 15e512c044a53c703cd99005a844bcaf46c59cfefe4407b248440d1bd55368ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, ceph=True, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-09T08:15:46.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:46 smithi028 ceph-mon[29738]: mkfs 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:15:46.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:46 smithi028 ceph-mon[29738]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T08:15:47.593 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:15:47.593 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-05-09T08:15:47.601 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:15:47.602 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-09T08:15:47.602 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-05-09T08:15:47.602 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-05-09T08:15:47.602 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-05-09T08:15:47.602 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-05-09T08:15:47.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 bash[29716]: 15e512c044a53c703cd99005a844bcaf46c59cfefe4407b248440d1bd55368ff 2024-05-09T08:15:47.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 systemd[1]: Started Ceph mon.a for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:15:47.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T08:15:47.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: monmap epoch 1 2024-05-09T08:15:47.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:15:47.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: last_changed 2024-05-09T08:15:41.070448+0000 2024-05-09T08:15:47.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: created 2024-05-09T08:15:41.070448+0000 2024-05-09T08:15:47.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: min_mon_release 19 (squid) 2024-05-09T08:15:47.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: election_strategy: 1 2024-05-09T08:15:47.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-09T08:15:47.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: fsmap 2024-05-09T08:15:47.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: osdmap e1: 0 total, 0 up, 0 in 2024-05-09T08:15:47.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:47 smithi028 ceph-mon[29738]: mgrmap e1: no daemons active 2024-05-09T08:15:50.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:50 smithi028 ceph-mon[29738]: from='client.? 172.21.15.28:0/4071309001' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-09T08:15:56.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-05-09T08:15:56.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 3s) 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.230+0000 7f0ed494d700 1 Processor -- start 2024-05-09T08:15:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.230+0000 7f0ed494d700 1 -- start start 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.230+0000 7f0ed494d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed007c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.230+0000 7f0ed494d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed007c840 con 0x7f0ed007de50 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.231+0000 7f0ece59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed007c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.231+0000 7f0ece59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed007c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35050/0 (socket says 172.21.15.28:35050) 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.231+0000 7f0ece59c700 1 -- 172.21.15.28:0/158984377 learned_addr learned my addr 172.21.15.28:0/158984377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.231+0000 7f0ece59c700 1 -- 172.21.15.28:0/158984377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed007c9c0 con 0x7f0ed007de50 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.231+0000 7f0ece59c700 1 --2- 172.21.15.28:0/158984377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed007c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00098a0 tx=0x7f0ec0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e480b7c86b443455 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.231+0000 7f0ecd59a700 1 -- 172.21.15.28:0/158984377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ec0004160 con 0x7f0ed007de50 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.231+0000 7f0ecd59a700 1 -- 172.21.15.28:0/158984377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec0004300 con 0x7f0ed007de50 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.231+0000 7f0ecd59a700 1 -- 172.21.15.28:0/158984377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ec00036a0 con 0x7f0ed007de50 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.232+0000 7f0ed494d700 1 -- 172.21.15.28:0/158984377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 msgr2=0x7f0ed007c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.232+0000 7f0ed494d700 1 --2- 172.21.15.28:0/158984377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed007c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00098a0 tx=0x7f0ec0009bb0 comp rx=0 tx=0).stop 2024-05-09T08:15:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.232+0000 7f0ed494d700 1 -- 172.21.15.28:0/158984377 shutdown_connections 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.232+0000 7f0ed494d700 1 --2- 172.21.15.28:0/158984377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed007c300 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.232+0000 7f0ed494d700 1 -- 172.21.15.28:0/158984377 >> 172.21.15.28:0/158984377 conn(0x7f0ed0103400 msgr2=0x7f0ed0105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.232+0000 7f0ed494d700 1 -- 172.21.15.28:0/158984377 shutdown_connections 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.232+0000 7f0ed494d700 1 -- 172.21.15.28:0/158984377 wait complete. 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.233+0000 7f0ed494d700 1 Processor -- start 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.233+0000 7f0ed494d700 1 -- start start 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.233+0000 7f0ed494d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.233+0000 7f0ed494d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed007cef0 con 0x7f0ed007de50 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.233+0000 7f0ece59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.233+0000 7f0ece59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed019d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35054/0 (socket says 172.21.15.28:35054) 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.233+0000 7f0ece59c700 1 -- 172.21.15.28:0/4071309001 learned_addr learned my addr 172.21.15.28:0/4071309001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.234+0000 7f0ece59c700 1 -- 172.21.15.28:0/4071309001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed019db80 con 0x7f0ed007de50 2024-05-09T08:15:56.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.234+0000 7f0ece59c700 1 --2- 172.21.15.28:0/4071309001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed019d640 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0000c00 tx=0x7f0ec0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.234+0000 7f0ebf7fe700 1 -- 172.21.15.28:0/4071309001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ec0003db0 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.234+0000 7f0ebf7fe700 1 -- 172.21.15.28:0/4071309001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec0011630 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.234+0000 7f0ed494d700 1 -- 172.21.15.28:0/4071309001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed019de10 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.234+0000 7f0ebf7fe700 1 -- 172.21.15.28:0/4071309001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ec0011950 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.234+0000 7f0ed494d700 1 -- 172.21.15.28:0/4071309001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed01a0b00 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.234+0000 7f0ebf7fe700 1 -- 172.21.15.28:0/4071309001 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0ec001a460 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.234+0000 7f0ebf7fe700 1 -- 172.21.15.28:0/4071309001 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0ec001b780 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.235+0000 7f0ed494d700 1 -- 172.21.15.28:0/4071309001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ed0063980 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.237+0000 7f0ebf7fe700 1 -- 172.21.15.28:0/4071309001 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0ec001ba20 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.311+0000 7f0ed494d700 1 -- 172.21.15.28:0/4071309001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f0ed01a0cd0 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.311+0000 7f0ebf7fe700 1 -- 172.21.15.28:0/4071309001 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+310 (secure 0 0 0) 0x7f0ec001bc00 con 0x7f0ed007de50 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.312+0000 7f0ed494d700 1 -- 172.21.15.28:0/4071309001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 msgr2=0x7f0ed019d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:15:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.312+0000 7f0ed494d700 1 --2- 172.21.15.28:0/4071309001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed019d640 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0000c00 tx=0x7f0ec0004750 comp rx=0 tx=0).stop 2024-05-09T08:15:56.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.312+0000 7f0ed494d700 1 -- 172.21.15.28:0/4071309001 shutdown_connections 2024-05-09T08:15:56.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.312+0000 7f0ed494d700 1 --2- 172.21.15.28:0/4071309001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed007de50 0x7f0ed019d640 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:15:56.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.312+0000 7f0ed494d700 1 -- 172.21.15.28:0/4071309001 >> 172.21.15.28:0/4071309001 conn(0x7f0ed0103400 msgr2=0x7f0ed01041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:15:56.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.312+0000 7f0ed494d700 1 -- 172.21.15.28:0/4071309001 shutdown_connections 2024-05-09T08:15:56.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:50.312+0000 7f0ed494d700 1 -- 172.21.15.28:0/4071309001 wait complete. 2024-05-09T08:15:56.183 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-05-09T08:15:56.183 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-05-09T08:15:57.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:57 smithi028 ceph-mon[29738]: from='client.? 172.21.15.28:0/1954799314' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-09T08:15:57.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:57 smithi028 ceph-mon[29738]: from='client.? 172.21.15.28:0/1954799314' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-09T08:15:57.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:15:57.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-05-09T08:15:57.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:15:57.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-05-09T08:15:57.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-09T08:15:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.921+0000 7fa596d38700 1 Processor -- start 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.921+0000 7fa596d38700 1 -- start start 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.921+0000 7fa596d38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa590107100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.921+0000 7fa596d38700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5901076d0 con 0x7fa590106ce0 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.921+0000 7fa594ad4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa590107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.921+0000 7fa594ad4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa590107100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57392/0 (socket says 172.21.15.28:57392) 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.921+0000 7fa594ad4700 1 -- 172.21.15.28:0/3682629302 learned_addr learned my addr 172.21.15.28:0/3682629302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.922+0000 7fa594ad4700 1 -- 172.21.15.28:0/3682629302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa590107ee0 con 0x7fa590106ce0 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.922+0000 7fa594ad4700 1 --2- 172.21.15.28:0/3682629302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa590107100 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa5840098a0 tx=0x7fa584009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f411caffaaba05 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.922+0000 7fa58f7fe700 1 -- 172.21.15.28:0/3682629302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa584004160 con 0x7fa590106ce0 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.922+0000 7fa58f7fe700 1 -- 172.21.15.28:0/3682629302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa584004300 con 0x7fa590106ce0 2024-05-09T08:15:57.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.922+0000 7fa58f7fe700 1 -- 172.21.15.28:0/3682629302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5840036a0 con 0x7fa590106ce0 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.923+0000 7fa596d38700 1 -- 172.21.15.28:0/3682629302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 msgr2=0x7fa590107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.923+0000 7fa596d38700 1 --2- 172.21.15.28:0/3682629302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa590107100 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa5840098a0 tx=0x7fa584009bb0 comp rx=0 tx=0).stop 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.923+0000 7fa596d38700 1 -- 172.21.15.28:0/3682629302 shutdown_connections 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.923+0000 7fa596d38700 1 --2- 172.21.15.28:0/3682629302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa590107100 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.923+0000 7fa596d38700 1 -- 172.21.15.28:0/3682629302 >> 172.21.15.28:0/3682629302 conn(0x7fa590102260 msgr2=0x7fa5901046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.923+0000 7fa596d38700 1 -- 172.21.15.28:0/3682629302 shutdown_connections 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.923+0000 7fa596d38700 1 -- 172.21.15.28:0/3682629302 wait complete. 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.923+0000 7fa596d38700 1 Processor -- start 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa596d38700 1 -- start start 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa596d38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa59019b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa596d38700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa590108410 con 0x7fa590106ce0 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa594ad4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa59019b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa594ad4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa59019b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57396/0 (socket says 172.21.15.28:57396) 2024-05-09T08:15:57.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa594ad4700 1 -- 172.21.15.28:0/1954799314 learned_addr learned my addr 172.21.15.28:0/1954799314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa594ad4700 1 -- 172.21.15.28:0/1954799314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59019b9b0 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa594ad4700 1 --2- 172.21.15.28:0/1954799314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa59019b470 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa584000c00 tx=0x7fa584004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa58dffb700 1 -- 172.21.15.28:0/1954799314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa584003db0 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa58dffb700 1 -- 172.21.15.28:0/1954799314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa584011630 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.924+0000 7fa596d38700 1 -- 172.21.15.28:0/1954799314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa59019bc40 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.925+0000 7fa58dffb700 1 -- 172.21.15.28:0/1954799314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa584011950 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.925+0000 7fa596d38700 1 -- 172.21.15.28:0/1954799314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa59019e930 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.925+0000 7fa58dffb700 1 -- 172.21.15.28:0/1954799314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa584011af0 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.925+0000 7fa58dffb700 1 -- 172.21.15.28:0/1954799314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa58401c450 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.926+0000 7fa596d38700 1 -- 172.21.15.28:0/1954799314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa590063980 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.928+0000 7fa58dffb700 1 -- 172.21.15.28:0/1954799314 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa584032020 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:56.999+0000 7fa596d38700 1 -- 172.21.15.28:0/1954799314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fa59002ddd0 con 0x7fa590106ce0 2024-05-09T08:15:57.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:57.002+0000 7fa58dffb700 1 -- 172.21.15.28:0/1954799314 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7fa584011de0 con 0x7fa590106ce0 2024-05-09T08:15:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:57.003+0000 7fa58dffb700 1 -- 172.21.15.28:0/1954799314 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa584004260 con 0x7fa590106ce0 2024-05-09T08:15:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:57.003+0000 7fa596d38700 1 -- 172.21.15.28:0/1954799314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 msgr2=0x7fa59019b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:15:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:57.003+0000 7fa596d38700 1 --2- 172.21.15.28:0/1954799314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa59019b470 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa584000c00 tx=0x7fa584004750 comp rx=0 tx=0).stop 2024-05-09T08:15:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:57.003+0000 7fa596d38700 1 -- 172.21.15.28:0/1954799314 shutdown_connections 2024-05-09T08:15:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:57.003+0000 7fa596d38700 1 --2- 172.21.15.28:0/1954799314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa590106ce0 0x7fa59019b470 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:15:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:57.003+0000 7fa596d38700 1 -- 172.21.15.28:0/1954799314 >> 172.21.15.28:0/1954799314 conn(0x7fa590102260 msgr2=0x7fa590192100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:15:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:57.003+0000 7fa596d38700 1 -- 172.21.15.28:0/1954799314 shutdown_connections 2024-05-09T08:15:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:57.003+0000 7fa596d38700 1 -- 172.21.15.28:0/1954799314 wait complete. 2024-05-09T08:15:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-05-09T08:15:58.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:58 smithi028 ceph-mon[29738]: from='client.? 172.21.15.28:0/3059242441' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:15:58.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.027+0000 7f3541ec4700 1 Processor -- start 2024-05-09T08:15:58.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.027+0000 7f3541ec4700 1 -- start start 2024-05-09T08:15:58.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.027+0000 7f3541ec4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c07c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:15:58.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.027+0000 7f3541ec4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f353c07c930 con 0x7f353c07deb0 2024-05-09T08:15:58.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.028+0000 7f353b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c07c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.028+0000 7f353b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c07c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57406/0 (socket says 172.21.15.28:57406) 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.028+0000 7f353b7fe700 1 -- 172.21.15.28:0/1734596227 learned_addr learned my addr 172.21.15.28:0/1734596227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.028+0000 7f353b7fe700 1 -- 172.21.15.28:0/1734596227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353c07cab0 con 0x7f353c07deb0 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.028+0000 7f353b7fe700 1 --2- 172.21.15.28:0/1734596227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c07c360 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f352c0098a0 tx=0x7f352c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19092ca56e7ff3ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.028+0000 7f353a7fc700 1 -- 172.21.15.28:0/1734596227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f352c004160 con 0x7f353c07deb0 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.028+0000 7f353a7fc700 1 -- 172.21.15.28:0/1734596227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f352c004300 con 0x7f353c07deb0 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.028+0000 7f353a7fc700 1 -- 172.21.15.28:0/1734596227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f352c011b50 con 0x7f353c07deb0 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.029+0000 7f3541ec4700 1 -- 172.21.15.28:0/1734596227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 msgr2=0x7f353c07c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.029+0000 7f3541ec4700 1 --2- 172.21.15.28:0/1734596227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c07c360 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f352c0098a0 tx=0x7f352c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.029+0000 7f3541ec4700 1 -- 172.21.15.28:0/1734596227 shutdown_connections 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.029+0000 7f3541ec4700 1 --2- 172.21.15.28:0/1734596227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c07c360 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.029+0000 7f3541ec4700 1 -- 172.21.15.28:0/1734596227 >> 172.21.15.28:0/1734596227 conn(0x7f353c103400 msgr2=0x7f353c105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.029+0000 7f3541ec4700 1 -- 172.21.15.28:0/1734596227 shutdown_connections 2024-05-09T08:15:58.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.029+0000 7f3541ec4700 1 -- 172.21.15.28:0/1734596227 wait complete. 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.029+0000 7f3541ec4700 1 Processor -- start 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.030+0000 7f3541ec4700 1 -- start start 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.030+0000 7f3541ec4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.030+0000 7f3541ec4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f353c10e3f0 con 0x7f353c07deb0 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.030+0000 7f353b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.030+0000 7f353b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c19dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57414/0 (socket says 172.21.15.28:57414) 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.030+0000 7f353b7fe700 1 -- 172.21.15.28:0/3059242441 learned_addr learned my addr 172.21.15.28:0/3059242441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.030+0000 7f353b7fe700 1 -- 172.21.15.28:0/3059242441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353c19e420 con 0x7f353c07deb0 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.030+0000 7f353b7fe700 1 --2- 172.21.15.28:0/3059242441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c19dee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f352c011b60 tx=0x7f352c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.031+0000 7f3538ff9700 1 -- 172.21.15.28:0/3059242441 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f352c003970 con 0x7f353c07deb0 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.031+0000 7f3538ff9700 1 -- 172.21.15.28:0/3059242441 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f352c003b10 con 0x7f353c07deb0 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.031+0000 7f3541ec4700 1 -- 172.21.15.28:0/3059242441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353c19e6b0 con 0x7f353c07deb0 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.031+0000 7f3538ff9700 1 -- 172.21.15.28:0/3059242441 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f352c024610 con 0x7f353c07deb0 2024-05-09T08:15:58.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.031+0000 7f3541ec4700 1 -- 172.21.15.28:0/3059242441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f353c1a13a0 con 0x7f353c07deb0 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.031+0000 7f3538ff9700 1 -- 172.21.15.28:0/3059242441 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f352c024c50 con 0x7f353c07deb0 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.031+0000 7f3538ff9700 1 -- 172.21.15.28:0/3059242441 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f352c011e00 con 0x7f353c07deb0 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.032+0000 7f3541ec4700 1 -- 172.21.15.28:0/3059242441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f353c063980 con 0x7f353c07deb0 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.034+0000 7f3538ff9700 1 -- 172.21.15.28:0/3059242441 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f352c01a9f0 con 0x7f353c07deb0 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.104+0000 7f3541ec4700 1 -- 172.21.15.28:0/3059242441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f353c1a1570 con 0x7f353c07deb0 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.105+0000 7f3538ff9700 1 -- 172.21.15.28:0/3059242441 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f352c01a430 con 0x7f353c07deb0 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.105+0000 7f3541ec4700 1 -- 172.21.15.28:0/3059242441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 msgr2=0x7f353c19dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.105+0000 7f3541ec4700 1 --2- 172.21.15.28:0/3059242441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c19dee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f352c011b60 tx=0x7f352c011c40 comp rx=0 tx=0).stop 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.106+0000 7f3541ec4700 1 -- 172.21.15.28:0/3059242441 shutdown_connections 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.106+0000 7f3541ec4700 1 --2- 172.21.15.28:0/3059242441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c07deb0 0x7f353c19dee0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.106+0000 7f3541ec4700 1 -- 172.21.15.28:0/3059242441 >> 172.21.15.28:0/3059242441 conn(0x7f353c103400 msgr2=0x7f353c103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.106+0000 7f3541ec4700 1 -- 172.21.15.28:0/3059242441 shutdown_connections 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:15:58.106+0000 7f3541ec4700 1 -- 172.21.15.28:0/3059242441 wait complete. 2024-05-09T08:15:58.682 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-05-09T08:15:58.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:58 smithi028 systemd[1]: Stopping Ceph mon.a for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:15:59.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:58 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[29734]: 2024-05-09T08:15:58.931+0000 7f40d9097700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:15:59.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:58 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[29734]: 2024-05-09T08:15:58.931+0000 7f40d9097700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-09T08:15:59.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:59 smithi028 podman[29983]: 2024-05-09 08:15:59.074359853 +0000 UTC m=+0.227884564 container died 15e512c044a53c703cd99005a844bcaf46c59cfefe4407b248440d1bd55368ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-09T08:15:59.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:59 smithi028 podman[29983]: 2024-05-09 08:15:59.479998679 +0000 UTC m=+0.633523398 container cleanup 15e512c044a53c703cd99005a844bcaf46c59cfefe4407b248440d1bd55368ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:15:59.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:59 smithi028 bash[29983]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a 2024-05-09T08:16:00.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:59 smithi028 podman[29997]: 2024-05-09 08:15:59.68918107 +0000 UTC m=+0.615247939 container remove 15e512c044a53c703cd99005a844bcaf46c59cfefe4407b248440d1bd55368ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-96805dc) 2024-05-09T08:16:00.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:59 smithi028 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-09T08:16:00.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:59 smithi028 systemd[1]: Stopped Ceph mon.a for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:16:00.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:15:59 smithi028 systemd[1]: Starting Ceph mon.a for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:16:00.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 podman[30100]: 2024-05-09 08:16:00.048335815 +0000 UTC m=+0.114097769 container create ba577546d1881252c883862af96b2db20369410d2ca7bc7bf4a21d62ca6b236f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-96805dc, ceph=True) 2024-05-09T08:16:00.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 podman[30100]: 2024-05-09 08:15:59.959238741 +0000 UTC m=+0.025000694 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:16:00.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 podman[30100]: 2024-05-09 08:16:00.233923275 +0000 UTC m=+0.299685229 container init ba577546d1881252c883862af96b2db20369410d2ca7bc7bf4a21d62ca6b236f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:16:00.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 podman[30100]: 2024-05-09 08:16:00.237359422 +0000 UTC m=+0.303121368 container start ba577546d1881252c883862af96b2db20369410d2ca7bc7bf4a21d62ca6b236f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-09T08:16:00.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: set uid:gid to 167:167 (ceph:ceph) 2024-05-09T08:16:00.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev), process ceph-mon, pid 2 2024-05-09T08:16:00.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: pidfile_write: ignore empty --pid-file 2024-05-09T08:16:00.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: load: jerasure load: lrc 2024-05-09T08:16:00.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: RocksDB version: 7.9.2 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Git sha 0 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Compile date 2024-05-07 19:19:25 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: DB SUMMARY 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: DB Session ID: NNLVX11EKH513ASYFLLS 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: CURRENT file: CURRENT 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: IDENTITY file: IDENTITY 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105581 ; 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.error_if_exists: 0 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.create_if_missing: 0 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.paranoid_checks: 1 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-09T08:16:00.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-09T08:16:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-09T08:16:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.env: 0x562aef2fac60 2024-05-09T08:16:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.fs: PosixFileSystem 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.info_log: 0x562aefe59ac0 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_file_opening_threads: 16 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.statistics: (nil) 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.use_fsync: 0 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_log_file_size: 0 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.keep_log_file_num: 1000 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.recycle_log_file_num: 0 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.allow_fallocate: 1 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.allow_mmap_reads: 0 2024-05-09T08:16:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.allow_mmap_writes: 0 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.use_direct_reads: 0 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.create_missing_column_families: 0 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.db_log_dir: 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.wal_dir: 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.advise_random_on_open: 1 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.db_write_buffer_size: 0 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.write_buffer_manager: 0x562aefe2c1e0 2024-05-09T08:16:00.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.rate_limiter: (nil) 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.wal_recovery_mode: 2 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.enable_thread_tracking: 0 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.enable_pipelined_write: 0 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.unordered_write: 0 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.row_cache: None 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.wal_filter: None 2024-05-09T08:16:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.allow_ingest_behind: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.two_write_queues: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.manual_wal_flush: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.wal_compression: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.atomic_flush: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.log_readahead_size: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.best_efforts_recovery: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.allow_data_in_errors: 0 2024-05-09T08:16:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.db_host_id: __hostname__ 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_background_jobs: 2 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_background_compactions: -1 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_subcompactions: 1 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_total_wal_size: 0 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_open_files: -1 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bytes_per_sync: 0 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-09T08:16:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_readahead_size: 0 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_background_flushes: -1 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Compression algorithms supported: 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: kZSTD supported: 0 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: kXpressCompression supported: 0 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: kLZ4HCCompression supported: 1 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: kZlibCompression supported: 1 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: kSnappyCompression supported: 1 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: kLZ4Compression supported: 1 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: kBZip2Compression supported: 0 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-09T08:16:00.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.merge_operator: 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_filter: None 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_filter_factory: None 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.sst_partitioner_factory: None 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562aefe59bc0) 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-05-09T08:16:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x562aefe5d090 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-05-09T08:16:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-05-09T08:16:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.write_buffer_size: 33554432 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_write_buffer_number: 2 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression: NoCompression 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression: Disabled 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.prefix_extractor: nullptr 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.num_levels: 7 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-09T08:16:00.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression_opts.level: 32767 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression_opts.strategy: 0 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression_opts.enabled: false 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:16:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.target_file_size_base: 67108864 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-09T08:16:00.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.arena_block_size: 1048576 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.disable_auto_compactions: 0 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-09T08:16:00.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.table_properties_collectors: 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.inplace_update_support: 0 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.bloom_locality: 0 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.max_successive_merges: 0 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.paranoid_file_checks: 0 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.force_consistency_checks: 1 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.report_bg_io_stats: 0 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.ttl: 2592000 2024-05-09T08:16:00.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-09T08:16:00.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-09T08:16:00.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-09T08:16:00.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.enable_blob_files: false 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.min_blob_size: 0 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.blob_file_size: 268435456 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.blob_file_starting_level: 0 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3baa1423-2f75-4a2d-a3c4-44405cd6a8a1 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715242560268342, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-09T08:16:00.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715242560269440, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100462, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 334, "table_properties": {"data_size": 98382, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14328, "raw_average_key_size": 49, "raw_value_size": 90624, "raw_average_value_size": 313, "num_data_blocks": 12, "num_entries": 289, "num_filter_entries": 289, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715242560, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3baa1423-2f75-4a2d-a3c4-44405cd6a8a1", "db_session_id": "NNLVX11EKH513ASYFLLS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715242560269525, "job": 1, "event": "recovery_finished"} 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562aeff4e000 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: DB pointer 0x562aeff38000 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:16:00.331 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 99.94 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 99.94 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-09T08:16:00.332 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 139.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 30.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:16:00.333 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 30.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x562aefe5d090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mon.a@-1(???) e1 preinit fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mon.a@-1(???).mds e1 new map 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mon.a@-1(???).mds e1 print_map 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T08:16:00.334 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 bash[30100]: ba577546d1881252c883862af96b2db20369410d2ca7bc7bf4a21d62ca6b236f 2024-05-09T08:16:00.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 systemd[1]: Started Ceph mon.a for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:16:00.335 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-09T08:16:00.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T08:16:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: monmap epoch 1 2024-05-09T08:16:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:16:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: last_changed 2024-05-09T08:15:41.070448+0000 2024-05-09T08:16:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: created 2024-05-09T08:15:41.070448+0000 2024-05-09T08:16:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: min_mon_release 19 (squid) 2024-05-09T08:16:00.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: election_strategy: 1 2024-05-09T08:16:00.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-09T08:16:00.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: fsmap 2024-05-09T08:16:00.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: osdmap e1: 0 total, 0 up, 0 in 2024-05-09T08:16:00.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:00 smithi028 ceph-mon[30119]: mgrmap e1: no daemons active 2024-05-09T08:16:01.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.736+0000 7fed449df700 1 Processor -- start 2024-05-09T08:16:01.425 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.737+0000 7fed449df700 1 -- start start 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.737+0000 7fed449df700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4007c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.737+0000 7fed449df700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed4007c840 con 0x7fed4007de50 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.737+0000 7fed3e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4007c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.737+0000 7fed3e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4007c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57428/0 (socket says 172.21.15.28:57428) 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.737+0000 7fed3e59c700 1 -- 172.21.15.28:0/565984428 learned_addr learned my addr 172.21.15.28:0/565984428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.738+0000 7fed3e59c700 1 -- 172.21.15.28:0/565984428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4007c9c0 con 0x7fed4007de50 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.738+0000 7fed3e59c700 1 --2- 172.21.15.28:0/565984428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4007c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed300098a0 tx=0x7fed30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6800d254fbdffe7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.738+0000 7fed3d59a700 1 -- 172.21.15.28:0/565984428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed30004160 con 0x7fed4007de50 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.738+0000 7fed3d59a700 1 -- 172.21.15.28:0/565984428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fed30004300 con 0x7fed4007de50 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.738+0000 7fed3d59a700 1 -- 172.21.15.28:0/565984428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed30011b50 con 0x7fed4007de50 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.739+0000 7fed449df700 1 -- 172.21.15.28:0/565984428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 msgr2=0x7fed4007c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.739+0000 7fed449df700 1 --2- 172.21.15.28:0/565984428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4007c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed300098a0 tx=0x7fed30009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:01.426 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.739+0000 7fed449df700 1 -- 172.21.15.28:0/565984428 shutdown_connections 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.739+0000 7fed449df700 1 --2- 172.21.15.28:0/565984428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4007c300 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.739+0000 7fed449df700 1 -- 172.21.15.28:0/565984428 >> 172.21.15.28:0/565984428 conn(0x7fed40103400 msgr2=0x7fed40105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.739+0000 7fed449df700 1 -- 172.21.15.28:0/565984428 shutdown_connections 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.739+0000 7fed449df700 1 -- 172.21.15.28:0/565984428 wait complete. 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed449df700 1 Processor -- start 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed449df700 1 -- start start 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed449df700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed449df700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed4010e3f0 con 0x7fed4007de50 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed3e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed3e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4019dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57438/0 (socket says 172.21.15.28:57438) 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed3e59c700 1 -- 172.21.15.28:0/1117246343 learned_addr learned my addr 172.21.15.28:0/1117246343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed3e59c700 1 -- 172.21.15.28:0/1117246343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4019e420 con 0x7fed4007de50 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed3e59c700 1 --2- 172.21.15.28:0/1117246343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4019dee0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed30011b60 tx=0x7fed30011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.740+0000 7fed377fe700 1 -- 172.21.15.28:0/1117246343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed30003970 con 0x7fed4007de50 2024-05-09T08:16:01.427 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.741+0000 7fed377fe700 1 -- 172.21.15.28:0/1117246343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fed30003b10 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.741+0000 7fed449df700 1 -- 172.21.15.28:0/1117246343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4019e6b0 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.741+0000 7fed377fe700 1 -- 172.21.15.28:0/1117246343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed30024610 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.741+0000 7fed449df700 1 -- 172.21.15.28:0/1117246343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed401a13a0 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.741+0000 7fed377fe700 1 -- 172.21.15.28:0/1117246343 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fed30024c50 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.741+0000 7fed377fe700 1 -- 172.21.15.28:0/1117246343 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed30011e00 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.742+0000 7fed449df700 1 -- 172.21.15.28:0/1117246343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed40063980 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.744+0000 7fed377fe700 1 -- 172.21.15.28:0/1117246343 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fed3001a9f0 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.813+0000 7fed449df700 1 -- 172.21.15.28:0/1117246343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fed401a1570 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.815+0000 7fed377fe700 1 -- 172.21.15.28:0/1117246343 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fed3001a430 con 0x7fed4007de50 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.816+0000 7fed449df700 1 -- 172.21.15.28:0/1117246343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 msgr2=0x7fed4019dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.816+0000 7fed449df700 1 --2- 172.21.15.28:0/1117246343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4019dee0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed30011b60 tx=0x7fed30011c40 comp rx=0 tx=0).stop 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.816+0000 7fed449df700 1 -- 172.21.15.28:0/1117246343 shutdown_connections 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.816+0000 7fed449df700 1 --2- 172.21.15.28:0/1117246343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed4007de50 0x7fed4019dee0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.816+0000 7fed449df700 1 -- 172.21.15.28:0/1117246343 >> 172.21.15.28:0/1117246343 conn(0x7fed40103400 msgr2=0x7fed40103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.816+0000 7fed449df700 1 -- 172.21.15.28:0/1117246343 shutdown_connections 2024-05-09T08:16:01.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:00.816+0000 7fed449df700 1 -- 172.21.15.28:0/1117246343 wait complete. 2024-05-09T08:16:01.429 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-09T08:16:01.450 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:16:01.450 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-05-09T08:16:01.451 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-09T08:16:01.451 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-09T08:16:01.715 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.x 2024-05-09T08:16:01.715 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.x.service: Unit ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.x.service not loaded. 2024-05-09T08:16:01.943 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3891b212-0ddc-11ef-bc98-c7b262605968.target.wants/ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-3891b212-0ddc-11ef-bc98-c7b262605968@.service. 2024-05-09T08:16:02.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:01 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1117246343' entity='client.admin' 2024-05-09T08:16:02.183 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:01 smithi028 systemd[1]: Starting Ceph mgr.x for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:16:02.436 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:02 smithi028 podman[30334]: 2024-05-09 08:16:02.083340157 +0000 UTC m=+0.018632828 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:16:03.295 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:03 smithi028 podman[30334]: 2024-05-09 08:16:03.015367418 +0000 UTC m=+0.950660101 container create d33853de27af83dfa677d3a8b81389f9a0167c9512d9fea242855c47ab5d6ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:16:03.295 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:03 smithi028 podman[30334]: 2024-05-09 08:16:03.208992837 +0000 UTC m=+1.144285482 container init d33853de27af83dfa677d3a8b81389f9a0167c9512d9fea242855c47ab5d6ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-09T08:16:03.295 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:03 smithi028 podman[30334]: 2024-05-09 08:16:03.212051604 +0000 UTC m=+1.147344253 container start d33853de27af83dfa677d3a8b81389f9a0167c9512d9fea242855c47ab5d6ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:16:03.295 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:03 smithi028 bash[30334]: d33853de27af83dfa677d3a8b81389f9a0167c9512d9fea242855c47ab5d6ce0 2024-05-09T08:16:03.312 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:16:03.312 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-05-09T08:16:03.322 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:16:03.322 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-09T08:16:03.322 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-05-09T08:16:03.322 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-05-09T08:16:03.333 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:16:03.333 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-05-09T08:16:03.342 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:16:03.342 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-09T08:16:03.342 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-05-09T08:16:03.342 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-09T08:16:03.342 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-05-09T08:16:03.342 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-05-09T08:16:03.580 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:03 smithi028 systemd[1]: Started Ceph mgr.x for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:16:03.581 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:03 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:03.383+0000 7fea7a27c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:16:03.581 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:03 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:03.577+0000 7fea7a27c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:16:03.915 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:03 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:03.685+0000 7fea7a27c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:16:04.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:03 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2572640209' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:16:04.176 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:04 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:04.055+0000 7fea7a27c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:16:04.176 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:04 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:04.145+0000 7fea7a27c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:16:04.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:16:04.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:16:04.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:16:04.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:16:04.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:16:04.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:15:46.667810+0000", 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:16:04.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.799+0000 7f66c8c51700 1 Processor -- start 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.799+0000 7f66c8c51700 1 -- start start 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.800+0000 7f66c8c51700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c4106ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.800+0000 7f66c8c51700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66c407be50 con 0x7f66c4106690 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.800+0000 7f66c259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c4106ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.800+0000 7f66c259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c4106ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57480/0 (socket says 172.21.15.28:57480) 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.800+0000 7f66c259c700 1 -- 172.21.15.28:0/3946414269 learned_addr learned my addr 172.21.15.28:0/3946414269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.800+0000 7f66c259c700 1 -- 172.21.15.28:0/3946414269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c4106ff0 con 0x7f66c4106690 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.800+0000 7f66c259c700 1 --2- 172.21.15.28:0/3946414269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c4106ab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f66b40098a0 tx=0x7f66b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df675b4877b2d20 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.801+0000 7f66c159a700 1 -- 172.21.15.28:0/3946414269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66b4004160 con 0x7f66c4106690 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.801+0000 7f66c159a700 1 -- 172.21.15.28:0/3946414269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f66b4004300 con 0x7f66c4106690 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.801+0000 7f66c159a700 1 -- 172.21.15.28:0/3946414269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66b4011b50 con 0x7f66c4106690 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.801+0000 7f66c8c51700 1 -- 172.21.15.28:0/3946414269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 msgr2=0x7f66c4106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:04.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.802+0000 7f66c8c51700 1 --2- 172.21.15.28:0/3946414269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c4106ab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f66b40098a0 tx=0x7f66b4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.802+0000 7f66c8c51700 1 -- 172.21.15.28:0/3946414269 shutdown_connections 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.802+0000 7f66c8c51700 1 --2- 172.21.15.28:0/3946414269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c4106ab0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.802+0000 7f66c8c51700 1 -- 172.21.15.28:0/3946414269 >> 172.21.15.28:0/3946414269 conn(0x7f66c4102210 msgr2=0x7f66c4104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.802+0000 7f66c8c51700 1 -- 172.21.15.28:0/3946414269 shutdown_connections 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.802+0000 7f66c8c51700 1 -- 172.21.15.28:0/3946414269 wait complete. 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.803+0000 7f66c8c51700 1 Processor -- start 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.803+0000 7f66c8c51700 1 -- start start 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.803+0000 7f66c8c51700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.803+0000 7f66c8c51700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66c4108080 con 0x7f66c4106690 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.803+0000 7f66c259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:04.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.803+0000 7f66c259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c419df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57486/0 (socket says 172.21.15.28:57486) 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.803+0000 7f66c259c700 1 -- 172.21.15.28:0/2572640209 learned_addr learned my addr 172.21.15.28:0/2572640209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.803+0000 7f66c259c700 1 -- 172.21.15.28:0/2572640209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c419e480 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.804+0000 7f66c259c700 1 --2- 172.21.15.28:0/2572640209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c419df40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f66b4011b60 tx=0x7f66b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.804+0000 7f66b37fe700 1 -- 172.21.15.28:0/2572640209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66b4003b20 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.804+0000 7f66b37fe700 1 -- 172.21.15.28:0/2572640209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f66b4003cc0 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.804+0000 7f66c8c51700 1 -- 172.21.15.28:0/2572640209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c419e710 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.804+0000 7f66b37fe700 1 -- 172.21.15.28:0/2572640209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66b4024840 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.804+0000 7f66c8c51700 1 -- 172.21.15.28:0/2572640209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c41a1400 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.804+0000 7f66b37fe700 1 -- 172.21.15.28:0/2572640209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f66b4022020 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.804+0000 7f66b37fe700 1 -- 172.21.15.28:0/2572640209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f66b4011e00 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.805+0000 7f66c8c51700 1 -- 172.21.15.28:0/2572640209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c4063980 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.807+0000 7f66b37fe700 1 -- 172.21.15.28:0/2572640209 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f66b4043dd0 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.911+0000 7f66c8c51700 1 -- 172.21.15.28:0/2572640209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f66c40008d0 con 0x7f66c4106690 2024-05-09T08:16:04.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.912+0000 7f66b37fe700 1 -- 172.21.15.28:0/2572640209 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f66b4032020 con 0x7f66c4106690 2024-05-09T08:16:04.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.913+0000 7f66c8c51700 1 -- 172.21.15.28:0/2572640209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 msgr2=0x7f66c419df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:04.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.913+0000 7f66c8c51700 1 --2- 172.21.15.28:0/2572640209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c419df40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f66b4011b60 tx=0x7f66b4011c40 comp rx=0 tx=0).stop 2024-05-09T08:16:04.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.913+0000 7f66c8c51700 1 -- 172.21.15.28:0/2572640209 shutdown_connections 2024-05-09T08:16:04.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.913+0000 7f66c8c51700 1 --2- 172.21.15.28:0/2572640209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c4106690 0x7f66c419df40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:04.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.913+0000 7f66c8c51700 1 -- 172.21.15.28:0/2572640209 >> 172.21.15.28:0/2572640209 conn(0x7f66c4102210 msgr2=0x7f66c4194ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:04.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.913+0000 7f66c8c51700 1 -- 172.21.15.28:0/2572640209 shutdown_connections 2024-05-09T08:16:04.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:03.913+0000 7f66c8c51700 1 -- 172.21.15.28:0/2572640209 wait complete. 2024-05-09T08:16:04.405 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-05-09T08:16:05.686 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:05 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:05.408+0000 7fea7a27c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:16:05.686 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:05 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:05.584+0000 7fea7a27c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:16:06.065 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:05 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:05.762+0000 7fea7a27c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:16:06.065 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:06 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:06.061+0000 7fea7a27c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:16:06.371 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:06 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:06.156+0000 7fea7a27c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:16:06.371 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:06 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:06.367+0000 7fea7a27c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:16:06.678 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:06 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:06.574+0000 7fea7a27c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:16:06.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2868768804' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:16:07.507 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:07 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:07.228+0000 7fea7a27c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:16:07.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:16:07.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:16:07.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:16:07.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:16:07.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:16:07.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:16:07.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:16:07.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:07.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:16:07.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:07.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:16:07.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:16:07.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:16:07.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:16:07.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:16:07.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:07.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:15:46.667810+0000", 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.777+0000 7f706baba700 1 Processor -- start 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.777+0000 7f706baba700 1 -- start start 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.777+0000 7f706baba700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f7064108200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.777+0000 7f706baba700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70641087d0 con 0x7f7064107de0 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.778+0000 7f7069856700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f7064108200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.778+0000 7f7069856700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f7064108200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42132/0 (socket says 172.21.15.28:42132) 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.778+0000 7f7069856700 1 -- 172.21.15.28:0/425846683 learned_addr learned my addr 172.21.15.28:0/425846683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.778+0000 7f7069856700 1 -- 172.21.15.28:0/425846683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7064108f90 con 0x7f7064107de0 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.778+0000 7f7069856700 1 --2- 172.21.15.28:0/425846683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f7064108200 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70600098a0 tx=0x7f7060009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5466908ee1a05382 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.778+0000 7f7068854700 1 -- 172.21.15.28:0/425846683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7060004160 con 0x7f7064107de0 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.778+0000 7f7068854700 1 -- 172.21.15.28:0/425846683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7060004300 con 0x7f7064107de0 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.778+0000 7f7068854700 1 -- 172.21.15.28:0/425846683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7060011b50 con 0x7f7064107de0 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.779+0000 7f706baba700 1 -- 172.21.15.28:0/425846683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 msgr2=0x7f7064108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.779+0000 7f706baba700 1 --2- 172.21.15.28:0/425846683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f7064108200 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70600098a0 tx=0x7f7060009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.779+0000 7f706baba700 1 -- 172.21.15.28:0/425846683 shutdown_connections 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.779+0000 7f706baba700 1 --2- 172.21.15.28:0/425846683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f7064108200 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:07.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.779+0000 7f706baba700 1 -- 172.21.15.28:0/425846683 >> 172.21.15.28:0/425846683 conn(0x7f70641033a0 msgr2=0x7f70641057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.779+0000 7f706baba700 1 -- 172.21.15.28:0/425846683 shutdown_connections 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.779+0000 7f706baba700 1 -- 172.21.15.28:0/425846683 wait complete. 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.780+0000 7f706baba700 1 Processor -- start 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.780+0000 7f706baba700 1 -- start start 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.780+0000 7f706baba700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f706419bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.780+0000 7f706baba700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f706410c340 con 0x7f7064107de0 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.780+0000 7f7069856700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f706419bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.780+0000 7f7069856700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f706419bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42142/0 (socket says 172.21.15.28:42142) 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.780+0000 7f7069856700 1 -- 172.21.15.28:0/2868768804 learned_addr learned my addr 172.21.15.28:0/2868768804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.781+0000 7f7069856700 1 -- 172.21.15.28:0/2868768804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706419c200 con 0x7f7064107de0 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.781+0000 7f7069856700 1 --2- 172.21.15.28:0/2868768804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f706419bcc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7060011670 tx=0x7f7060011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.781+0000 7f705affd700 1 -- 172.21.15.28:0/2868768804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70600039c0 con 0x7f7064107de0 2024-05-09T08:16:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.781+0000 7f705affd700 1 -- 172.21.15.28:0/2868768804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7060011e50 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.781+0000 7f705affd700 1 -- 172.21.15.28:0/2868768804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f706001a5c0 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.781+0000 7f706baba700 1 -- 172.21.15.28:0/2868768804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706419c490 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.781+0000 7f706baba700 1 -- 172.21.15.28:0/2868768804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706419f180 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.781+0000 7f705affd700 1 -- 172.21.15.28:0/2868768804 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f706001ac50 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.781+0000 7f705affd700 1 -- 172.21.15.28:0/2868768804 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7060019700 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.782+0000 7f706baba700 1 -- 172.21.15.28:0/2868768804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7064063980 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.784+0000 7f705affd700 1 -- 172.21.15.28:0/2868768804 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7060003b60 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.858+0000 7f706baba700 1 -- 172.21.15.28:0/2868768804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f706419f350 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.858+0000 7f705affd700 1 -- 172.21.15.28:0/2868768804 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f7060019950 con 0x7f7064107de0 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.859+0000 7f706baba700 1 -- 172.21.15.28:0/2868768804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 msgr2=0x7f706419bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.859+0000 7f706baba700 1 --2- 172.21.15.28:0/2868768804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f706419bcc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7060011670 tx=0x7f7060011b60 comp rx=0 tx=0).stop 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.859+0000 7f706baba700 1 -- 172.21.15.28:0/2868768804 shutdown_connections 2024-05-09T08:16:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.859+0000 7f706baba700 1 --2- 172.21.15.28:0/2868768804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7064107de0 0x7f706419bcc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.859+0000 7f706baba700 1 -- 172.21.15.28:0/2868768804 >> 172.21.15.28:0/2868768804 conn(0x7f70641033a0 msgr2=0x7f7064103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.859+0000 7f706baba700 1 -- 172.21.15.28:0/2868768804 shutdown_connections 2024-05-09T08:16:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:06.859+0000 7f706baba700 1 -- 172.21.15.28:0/2868768804 wait complete. 2024-05-09T08:16:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-05-09T08:16:07.769 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:07 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:07.665+0000 7fea7a27c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:16:07.770 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:07 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:07.765+0000 7fea7a27c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:16:08.186 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:07 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:07.972+0000 7fea7a27c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:16:08.885 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:08 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:08.533+0000 7fea7a27c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:16:08.885 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:08 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:08.628+0000 7fea7a27c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:16:09.186 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:08 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:08.881+0000 7fea7a27c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:16:09.186 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:09.019+0000 7fea7a27c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:16:09.186 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:09.117+0000 7fea7a27c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:16:09.555 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:09.236+0000 7fea7a27c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:16:09.555 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:09.551+0000 7fea7a27c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:16:10.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:10 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3657371406' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:16:10.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:10.210+0000 7fea7a27c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:16:10.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:10.438+0000 7fea7a27c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:16:10.576 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:16:10.576 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:16:10.576 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:16:10.576 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:16:10.576 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:16:10.576 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:16:10.577 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:16:10.578 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:16:10.579 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:15:46.667810+0000", 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:16:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.918+0000 7eff6dbf8700 1 Processor -- start 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.918+0000 7eff6dbf8700 1 -- start start 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.919+0000 7eff6dbf8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff681086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.919+0000 7eff6dbf8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff68108c70 con 0x7eff68108280 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.919+0000 7eff677fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff681086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.919+0000 7eff677fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff681086a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42146/0 (socket says 172.21.15.28:42146) 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.919+0000 7eff677fe700 1 -- 172.21.15.28:0/2983018674 learned_addr learned my addr 172.21.15.28:0/2983018674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.919+0000 7eff677fe700 1 -- 172.21.15.28:0/2983018674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff68106ee0 con 0x7eff68108280 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.919+0000 7eff677fe700 1 --2- 172.21.15.28:0/2983018674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff681086a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff580098a0 tx=0x7eff58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8419adb1d3505cda server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff66ffd700 1 -- 172.21.15.28:0/2983018674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff58004160 con 0x7eff68108280 2024-05-09T08:16:10.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff66ffd700 1 -- 172.21.15.28:0/2983018674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7eff58004300 con 0x7eff68108280 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff66ffd700 1 -- 172.21.15.28:0/2983018674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff58011b50 con 0x7eff68108280 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff6dbf8700 1 -- 172.21.15.28:0/2983018674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 msgr2=0x7eff681086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff6dbf8700 1 --2- 172.21.15.28:0/2983018674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff681086a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff580098a0 tx=0x7eff58009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff6dbf8700 1 -- 172.21.15.28:0/2983018674 shutdown_connections 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff6dbf8700 1 --2- 172.21.15.28:0/2983018674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff681086a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff6dbf8700 1 -- 172.21.15.28:0/2983018674 >> 172.21.15.28:0/2983018674 conn(0x7eff681019f0 msgr2=0x7eff68103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff6dbf8700 1 -- 172.21.15.28:0/2983018674 shutdown_connections 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.920+0000 7eff6dbf8700 1 -- 172.21.15.28:0/2983018674 wait complete. 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.921+0000 7eff6dbf8700 1 Processor -- start 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.921+0000 7eff6dbf8700 1 -- start start 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.921+0000 7eff6dbf8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff6819bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.921+0000 7eff6dbf8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff6810b2a0 con 0x7eff68108280 2024-05-09T08:16:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff677fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff6819bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff677fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff6819bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42148/0 (socket says 172.21.15.28:42148) 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff677fe700 1 -- 172.21.15.28:0/3657371406 learned_addr learned my addr 172.21.15.28:0/3657371406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff677fe700 1 -- 172.21.15.28:0/3657371406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6819c220 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff677fe700 1 --2- 172.21.15.28:0/3657371406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff6819bce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7eff58000c00 tx=0x7eff58011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff657fa700 1 -- 172.21.15.28:0/3657371406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff58003b20 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff657fa700 1 -- 172.21.15.28:0/3657371406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7eff58003cc0 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff6dbf8700 1 -- 172.21.15.28:0/3657371406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6819c4b0 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff657fa700 1 -- 172.21.15.28:0/3657371406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff58024910 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.922+0000 7eff6dbf8700 1 -- 172.21.15.28:0/3657371406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6819f190 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.923+0000 7eff657fa700 1 -- 172.21.15.28:0/3657371406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7eff58022020 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.923+0000 7eff657fa700 1 -- 172.21.15.28:0/3657371406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7eff58011e00 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.923+0000 7eff6dbf8700 1 -- 172.21.15.28:0/3657371406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff68063980 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:09.927+0000 7eff657fa700 1 -- 172.21.15.28:0/3657371406 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7eff58042dd0 con 0x7eff68108280 2024-05-09T08:16:10.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:10.001+0000 7eff6dbf8700 1 -- 172.21.15.28:0/3657371406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7eff6802ddd0 con 0x7eff68108280 2024-05-09T08:16:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:10.002+0000 7eff657fa700 1 -- 172.21.15.28:0/3657371406 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7eff58031020 con 0x7eff68108280 2024-05-09T08:16:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:10.003+0000 7eff6dbf8700 1 -- 172.21.15.28:0/3657371406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 msgr2=0x7eff6819bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:10.003+0000 7eff6dbf8700 1 --2- 172.21.15.28:0/3657371406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff6819bce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7eff58000c00 tx=0x7eff58011c10 comp rx=0 tx=0).stop 2024-05-09T08:16:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:10.003+0000 7eff6dbf8700 1 -- 172.21.15.28:0/3657371406 shutdown_connections 2024-05-09T08:16:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:10.003+0000 7eff6dbf8700 1 --2- 172.21.15.28:0/3657371406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68108280 0x7eff6819bce0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:10.003+0000 7eff6dbf8700 1 -- 172.21.15.28:0/3657371406 >> 172.21.15.28:0/3657371406 conn(0x7eff681019f0 msgr2=0x7eff681027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:10.003+0000 7eff6dbf8700 1 -- 172.21.15.28:0/3657371406 shutdown_connections 2024-05-09T08:16:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:10.003+0000 7eff6dbf8700 1 -- 172.21.15.28:0/3657371406 wait complete. 2024-05-09T08:16:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (3/15)... 2024-05-09T08:16:11.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: Activating manager daemon x 2024-05-09T08:16:11.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: mgrmap e2: x(active, starting, since 0.00321626s) 2024-05-09T08:16:11.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:16:11.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:16:11.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:16:11.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:16:11.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:16:11.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: Manager daemon x is now available 2024-05-09T08:16:11.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:16:11.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' 2024-05-09T08:16:11.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:16:11.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' 2024-05-09T08:16:11.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:11 smithi028 ceph-mon[30119]: from='mgr.14100 172.21.15.28:0/483974024' entity='mgr.x' 2024-05-09T08:16:12.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:12 smithi028 ceph-mon[30119]: mgrmap e3: x(active, since 1.00749s) 2024-05-09T08:16:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:13 smithi028 ceph-mon[30119]: mgrmap e4: x(active, since 2s) 2024-05-09T08:16:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:13 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/428133118' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:16:13.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:16:13.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:16:13.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:16:13.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:16:13.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:16:13.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:16:13.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:13.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:16:13.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:13.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:16:13.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:16:13.861 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:16:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:15:46.667810+0000", 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.005+0000 7f3a5891d700 1 Processor -- start 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.005+0000 7f3a5891d700 1 -- start start 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.005+0000 7f3a5891d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5407c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.005+0000 7f3a5891d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5407c840 con 0x7f3a5407de50 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.005+0000 7f3a5259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5407c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.005+0000 7f3a5259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5407c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42220/0 (socket says 172.21.15.28:42220) 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.005+0000 7f3a5259c700 1 -- 172.21.15.28:0/1704005840 learned_addr learned my addr 172.21.15.28:0/1704005840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.006+0000 7f3a5259c700 1 -- 172.21.15.28:0/1704005840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5407c9c0 con 0x7f3a5407de50 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.006+0000 7f3a5259c700 1 --2- 172.21.15.28:0/1704005840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5407c300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3a440098a0 tx=0x7f3a44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab982eb1f9e552a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.006+0000 7f3a5159a700 1 -- 172.21.15.28:0/1704005840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a44004160 con 0x7f3a5407de50 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.006+0000 7f3a5159a700 1 -- 172.21.15.28:0/1704005840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3a44004300 con 0x7f3a5407de50 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.006+0000 7f3a5159a700 1 -- 172.21.15.28:0/1704005840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a44011b50 con 0x7f3a5407de50 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.007+0000 7f3a5891d700 1 -- 172.21.15.28:0/1704005840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 msgr2=0x7f3a5407c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.007+0000 7f3a5891d700 1 --2- 172.21.15.28:0/1704005840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5407c300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3a440098a0 tx=0x7f3a44009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.007+0000 7f3a5891d700 1 -- 172.21.15.28:0/1704005840 shutdown_connections 2024-05-09T08:16:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.007+0000 7f3a5891d700 1 --2- 172.21.15.28:0/1704005840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5407c300 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.007+0000 7f3a5891d700 1 -- 172.21.15.28:0/1704005840 >> 172.21.15.28:0/1704005840 conn(0x7f3a54103400 msgr2=0x7f3a54105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.007+0000 7f3a5891d700 1 -- 172.21.15.28:0/1704005840 shutdown_connections 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.007+0000 7f3a5891d700 1 -- 172.21.15.28:0/1704005840 wait complete. 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.008+0000 7f3a5891d700 1 Processor -- start 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.008+0000 7f3a5891d700 1 -- start start 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.008+0000 7f3a5891d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.008+0000 7f3a5891d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5410e3f0 con 0x7f3a5407de50 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.008+0000 7f3a5259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.008+0000 7f3a5259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5419dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42232/0 (socket says 172.21.15.28:42232) 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.008+0000 7f3a5259c700 1 -- 172.21.15.28:0/428133118 learned_addr learned my addr 172.21.15.28:0/428133118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.008+0000 7f3a5259c700 1 -- 172.21.15.28:0/428133118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5419e420 con 0x7f3a5407de50 2024-05-09T08:16:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.008+0000 7f3a5259c700 1 --2- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5419dee0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009fb0 tx=0x7f3a44011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.009+0000 7f3a437fe700 1 -- 172.21.15.28:0/428133118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a440038b0 con 0x7f3a5407de50 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.009+0000 7f3a437fe700 1 -- 172.21.15.28:0/428133118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3a44011e50 con 0x7f3a5407de50 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.009+0000 7f3a437fe700 1 -- 172.21.15.28:0/428133118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a44019420 con 0x7f3a5407de50 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.009+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5419e6b0 con 0x7f3a5407de50 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.009+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a541a13a0 con 0x7f3a5407de50 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.009+0000 7f3a437fe700 1 -- 172.21.15.28:0/428133118 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f3a44019720 con 0x7f3a5407de50 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.010+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a54063980 con 0x7f3a5407de50 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.010+0000 7f3a437fe700 1 --2- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7f3a3c039070 0x7f3a3c03b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.010+0000 7f3a437fe700 1 -- 172.21.15.28:0/428133118 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3a4404d1d0 con 0x7f3a5407de50 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.010+0000 7f3a51d9b700 1 --2- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7f3a3c039070 0x7f3a3c03b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.010+0000 7f3a51d9b700 1 --2- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7f3a3c039070 0x7f3a3c03b530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3a48006fd0 tx=0x7f3a48006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.015+0000 7f3a437fe700 1 -- 172.21.15.28:0/428133118 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3a440172b0 con 0x7f3a5407de50 2024-05-09T08:16:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.304+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f3a541a1570 con 0x7f3a5407de50 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.305+0000 7f3a437fe700 1 -- 172.21.15.28:0/428133118 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f3a541a1570 con 0x7f3a5407de50 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7f3a3c039070 msgr2=0x7f3a3c03b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 --2- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7f3a3c039070 0x7f3a3c03b530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3a48006fd0 tx=0x7f3a48006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 msgr2=0x7f3a5419dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 --2- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5419dee0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009fb0 tx=0x7f3a44011bf0 comp rx=0 tx=0).stop 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 shutdown_connections 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 --2- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7f3a3c039070 0x7f3a3c03b530 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 --2- 172.21.15.28:0/428133118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a5407de50 0x7f3a5419dee0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 >> 172.21.15.28:0/428133118 conn(0x7f3a54103400 msgr2=0x7f3a54103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 shutdown_connections 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:13.307+0000 7f3a5891d700 1 -- 172.21.15.28:0/428133118 wait complete. 2024-05-09T08:16:13.867 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-05-09T08:16:14.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:14 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1741730475' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-09T08:16:15.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:16:15.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-05-09T08:16:15.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-09T08:16:15.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.267+0000 7fbbae4bb700 1 Processor -- start 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.267+0000 7fbbae4bb700 1 -- start start 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.267+0000 7fbbae4bb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba810ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.267+0000 7fbbae4bb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbba807be90 con 0x7fbba8108880 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.267+0000 7fbba7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba810ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.267+0000 7fbba7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba810ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42248/0 (socket says 172.21.15.28:42248) 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.267+0000 7fbba7fff700 1 -- 172.21.15.28:0/1541129452 learned_addr learned my addr 172.21.15.28:0/1541129452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.267+0000 7fbba7fff700 1 -- 172.21.15.28:0/1541129452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba810b1b0 con 0x7fbba8108880 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbba7fff700 1 --2- 172.21.15.28:0/1541129452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba810ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbb980098a0 tx=0x7fbb98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9417edeb8dacac06 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbba6ffd700 1 -- 172.21.15.28:0/1541129452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98004160 con 0x7fbba8108880 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbba6ffd700 1 -- 172.21.15.28:0/1541129452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb98004300 con 0x7fbba8108880 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbba6ffd700 1 -- 172.21.15.28:0/1541129452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98011b50 con 0x7fbba8108880 2024-05-09T08:16:15.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1541129452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 msgr2=0x7fbba810ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbbae4bb700 1 --2- 172.21.15.28:0/1541129452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba810ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbb980098a0 tx=0x7fbb98009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1541129452 shutdown_connections 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbbae4bb700 1 --2- 172.21.15.28:0/1541129452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba810ac70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1541129452 >> 172.21.15.28:0/1541129452 conn(0x7fbba8102230 msgr2=0x7fbba81046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1541129452 shutdown_connections 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.268+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1541129452 wait complete. 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.269+0000 7fbbae4bb700 1 Processor -- start 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.269+0000 7fbbae4bb700 1 -- start start 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.269+0000 7fbbae4bb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba819bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.269+0000 7fbbae4bb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbba810c1d0 con 0x7fbba8108880 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbba7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba819bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbba7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba819bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42256/0 (socket says 172.21.15.28:42256) 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbba7fff700 1 -- 172.21.15.28:0/1741730475 learned_addr learned my addr 172.21.15.28:0/1741730475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbba7fff700 1 -- 172.21.15.28:0/1741730475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba819c1c0 con 0x7fbba8108880 2024-05-09T08:16:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbba7fff700 1 --2- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba819bc80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbb98011b60 tx=0x7fbb98011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbba57fa700 1 -- 172.21.15.28:0/1741730475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98003b20 con 0x7fbba8108880 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba819c450 con 0x7fbba8108880 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba819f140 con 0x7fbba8108880 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbba57fa700 1 -- 172.21.15.28:0/1741730475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb98003cc0 con 0x7fbba8108880 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.270+0000 7fbba57fa700 1 -- 172.21.15.28:0/1741730475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98024840 con 0x7fbba8108880 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.271+0000 7fbba57fa700 1 -- 172.21.15.28:0/1741730475 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fbb98022020 con 0x7fbba8108880 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.272+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb94005360 con 0x7fbba8108880 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.272+0000 7fbba57fa700 1 --2- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fbb900390c0 0x7fbb9003b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.272+0000 7fbba57fa700 1 -- 172.21.15.28:0/1741730475 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbb98011e00 con 0x7fbba8108880 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.272+0000 7fbba77fe700 1 --2- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fbb900390c0 0x7fbb9003b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.272+0000 7fbba77fe700 1 --2- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fbb900390c0 0x7fbb9003b580 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c006fd0 tx=0x7fbb9c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.277+0000 7fbba57fa700 1 -- 172.21.15.28:0/1741730475 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbb9804cbd0 con 0x7fbba8108880 2024-05-09T08:16:15.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.480+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fbb94002430 con 0x7fbba8108880 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.481+0000 7fbba57fa700 1 -- 172.21.15.28:0/1741730475 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fbb9802d3f0 con 0x7fbba8108880 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fbb900390c0 msgr2=0x7fbb9003b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 --2- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fbb900390c0 0x7fbb9003b580 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c006fd0 tx=0x7fbb9c006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 msgr2=0x7fbba819bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 --2- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba819bc80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbb98011b60 tx=0x7fbb98011c40 comp rx=0 tx=0).stop 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 shutdown_connections 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 --2- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fbb900390c0 0x7fbb9003b580 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 --2- 172.21.15.28:0/1741730475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbba8108880 0x7fbba819bc80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 >> 172.21.15.28:0/1741730475 conn(0x7fbba8102230 msgr2=0x7fbba8102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 shutdown_connections 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:14.483+0000 7fbbae4bb700 1 -- 172.21.15.28:0/1741730475 wait complete. 2024-05-09T08:16:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-05-09T08:16:16.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:15 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3783680974' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-09T08:16:16.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: ignoring --setuser ceph since I am not root 2024-05-09T08:16:16.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: ignoring --setgroup ceph since I am not root 2024-05-09T08:16:16.048 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:15.907+0000 7f235ca99200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:16:16.314 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:16.100+0000 7f235ca99200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:16:16.315 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:16.208+0000 7f235ca99200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:16:16.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.409+0000 7ff6e6caa700 1 Processor -- start 2024-05-09T08:16:16.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.409+0000 7ff6e6caa700 1 -- start start 2024-05-09T08:16:16.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.409+0000 7ff6e6caa700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e01070f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:16.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.409+0000 7ff6e6caa700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e01076c0 con 0x7ff6e0106cd0 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.409+0000 7ff6e4a46700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e01070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.409+0000 7ff6e4a46700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e01070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59850/0 (socket says 172.21.15.28:59850) 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.409+0000 7ff6e4a46700 1 -- 172.21.15.28:0/207907816 learned_addr learned my addr 172.21.15.28:0/207907816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.410+0000 7ff6e4a46700 1 -- 172.21.15.28:0/207907816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e0107ed0 con 0x7ff6e0106cd0 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.410+0000 7ff6e4a46700 1 --2- 172.21.15.28:0/207907816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e01070f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40098a0 tx=0x7ff6d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e17a7478d70f5ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.410+0000 7ff6df7fe700 1 -- 172.21.15.28:0/207907816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6d4004160 con 0x7ff6e0106cd0 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.410+0000 7ff6df7fe700 1 -- 172.21.15.28:0/207907816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff6d4004300 con 0x7ff6e0106cd0 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.410+0000 7ff6df7fe700 1 -- 172.21.15.28:0/207907816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6d4011b50 con 0x7ff6e0106cd0 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.411+0000 7ff6e6caa700 1 -- 172.21.15.28:0/207907816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 msgr2=0x7ff6e01070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.411+0000 7ff6e6caa700 1 --2- 172.21.15.28:0/207907816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e01070f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40098a0 tx=0x7ff6d4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.411+0000 7ff6e6caa700 1 -- 172.21.15.28:0/207907816 shutdown_connections 2024-05-09T08:16:16.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.411+0000 7ff6e6caa700 1 --2- 172.21.15.28:0/207907816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e01070f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:16.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.411+0000 7ff6e6caa700 1 -- 172.21.15.28:0/207907816 >> 172.21.15.28:0/207907816 conn(0x7ff6e0102230 msgr2=0x7ff6e01046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:16.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.411+0000 7ff6e6caa700 1 -- 172.21.15.28:0/207907816 shutdown_connections 2024-05-09T08:16:16.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.411+0000 7ff6e6caa700 1 -- 172.21.15.28:0/207907816 wait complete. 2024-05-09T08:16:16.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.411+0000 7ff6e6caa700 1 Processor -- start 2024-05-09T08:16:16.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6e6caa700 1 -- start start 2024-05-09T08:16:16.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6e6caa700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:16.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6e6caa700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e010b280 con 0x7ff6e0106cd0 2024-05-09T08:16:16.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6e4a46700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:16.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6e4a46700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e019df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59866/0 (socket says 172.21.15.28:59866) 2024-05-09T08:16:16.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6e4a46700 1 -- 172.21.15.28:0/3783680974 learned_addr learned my addr 172.21.15.28:0/3783680974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:16.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6e4a46700 1 -- 172.21.15.28:0/3783680974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e019e4a0 con 0x7ff6e0106cd0 2024-05-09T08:16:16.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6e4a46700 1 --2- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e019df60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4011850 tx=0x7ff6d4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:16.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6ddffb700 1 -- 172.21.15.28:0/3783680974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6d4003af0 con 0x7ff6e0106cd0 2024-05-09T08:16:16.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6ddffb700 1 -- 172.21.15.28:0/3783680974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff6d4020070 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6ddffb700 1 -- 172.21.15.28:0/3783680974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6d40195a0 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.412+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e019e730 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.413+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e01a1420 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.414+0000 7ff6ddffb700 1 -- 172.21.15.28:0/3783680974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7ff6d4019850 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.414+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e0063980 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.414+0000 7ff6ddffb700 1 --2- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7ff6d0039110 0x7ff6d003b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.414+0000 7ff6ddffb700 1 -- 172.21.15.28:0/3783680974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff6d4011e00 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.414+0000 7ff6dffff700 1 --2- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7ff6d0039110 0x7ff6d003b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.414+0000 7ff6dffff700 1 --2- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7ff6d0039110 0x7ff6d003b5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc006fd0 tx=0x7ff6cc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.419+0000 7ff6ddffb700 1 -- 172.21.15.28:0/3783680974 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff6d400e260 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.683+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7ff6e00008d0 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.734+0000 7ff6ddffb700 1 -- 172.21.15.28:0/3783680974 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7ff6d4003c90 con 0x7ff6e0106cd0 2024-05-09T08:16:16.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.736+0000 7ff6ddffb700 1 -- 172.21.15.28:0/3783680974 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7ff6d401a430 con 0x7ff6e0106cd0 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7ff6d0039110 msgr2=0x7ff6d003b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 --2- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7ff6d0039110 0x7ff6d003b5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc006fd0 tx=0x7ff6cc006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 msgr2=0x7ff6e019df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 --2- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e019df60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4011850 tx=0x7ff6d4011c10 comp rx=0 tx=0).stop 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 shutdown_connections 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 --2- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7ff6d0039110 0x7ff6d003b5d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 --2- 172.21.15.28:0/3783680974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6e0106cd0 0x7ff6e019df60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 >> 172.21.15.28:0/3783680974 conn(0x7ff6e0102230 msgr2=0x7ff6e0103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 shutdown_connections 2024-05-09T08:16:16.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:15.737+0000 7ff6e6caa700 1 -- 172.21.15.28:0/3783680974 wait complete. 2024-05-09T08:16:16.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:16.533+0000 7f235ca99200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:16:16.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:16 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3783680974' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-09T08:16:16.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:16 smithi028 ceph-mon[30119]: mgrmap e5: x(active, since 5s) 2024-05-09T08:16:16.936 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:16.624+0000 7f235ca99200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:16:17.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:16:17.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-09T08:16:17.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T08:16:17.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-09T08:16:17.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-09T08:16:17.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.758+0000 7faedccb4700 1 Processor -- start 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.758+0000 7faedccb4700 1 -- start start 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.759+0000 7faedccb4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed810ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.759+0000 7faedccb4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faed807bed0 con 0x7faed8108820 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.759+0000 7faed659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed810ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.759+0000 7faed659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed810ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59898/0 (socket says 172.21.15.28:59898) 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.759+0000 7faed659c700 1 -- 172.21.15.28:0/2296526277 learned_addr learned my addr 172.21.15.28:0/2296526277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.759+0000 7faed659c700 1 -- 172.21.15.28:0/2296526277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed810b150 con 0x7faed8108820 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.760+0000 7faed659c700 1 --2- 172.21.15.28:0/2296526277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed810ac10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faec80098a0 tx=0x7faec8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f4ab392e5b125a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.760+0000 7faed559a700 1 -- 172.21.15.28:0/2296526277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faec8004160 con 0x7faed8108820 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.760+0000 7faed559a700 1 -- 172.21.15.28:0/2296526277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7faec8004300 con 0x7faed8108820 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.760+0000 7faed559a700 1 -- 172.21.15.28:0/2296526277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faec8011b50 con 0x7faed8108820 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.760+0000 7faedccb4700 1 -- 172.21.15.28:0/2296526277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 msgr2=0x7faed810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:17.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.760+0000 7faedccb4700 1 --2- 172.21.15.28:0/2296526277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed810ac10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faec80098a0 tx=0x7faec8009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.760+0000 7faedccb4700 1 -- 172.21.15.28:0/2296526277 shutdown_connections 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.760+0000 7faedccb4700 1 --2- 172.21.15.28:0/2296526277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed810ac10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.760+0000 7faedccb4700 1 -- 172.21.15.28:0/2296526277 >> 172.21.15.28:0/2296526277 conn(0x7faed8102210 msgr2=0x7faed8104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.761+0000 7faedccb4700 1 -- 172.21.15.28:0/2296526277 shutdown_connections 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.761+0000 7faedccb4700 1 -- 172.21.15.28:0/2296526277 wait complete. 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.761+0000 7faedccb4700 1 Processor -- start 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.761+0000 7faedccb4700 1 -- start start 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.761+0000 7faedccb4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faedccb4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faed810c180 con 0x7faed8108820 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faed659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faed659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed819bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59908/0 (socket says 172.21.15.28:59908) 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faed659c700 1 -- 172.21.15.28:0/2995055207 learned_addr learned my addr 172.21.15.28:0/2995055207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:17.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faed659c700 1 -- 172.21.15.28:0/2995055207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed819c2e0 con 0x7faed8108820 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faed659c700 1 --2- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed819bda0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faec8011670 tx=0x7faec8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faec77fe700 1 -- 172.21.15.28:0/2995055207 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faec80039c0 con 0x7faed8108820 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faec77fe700 1 -- 172.21.15.28:0/2995055207 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7faec8011e50 con 0x7faed8108820 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faec77fe700 1 -- 172.21.15.28:0/2995055207 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faec801a5c0 con 0x7faed8108820 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed819c570 con 0x7faed8108820 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.762+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed819f220 con 0x7faed8108820 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.763+0000 7faec77fe700 1 -- 172.21.15.28:0/2995055207 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7faec801a8c0 con 0x7faed8108820 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.764+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed8063980 con 0x7faed8108820 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.764+0000 7faec77fe700 1 --2- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7faec0039000 0x7faec003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.764+0000 7faec77fe700 1 -- 172.21.15.28:0/2995055207 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7faec804d440 con 0x7faed8108820 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.764+0000 7faed5d9b700 1 -- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7faec0039000 msgr2=0x7faec003b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1179097296 2024-05-09T08:16:17.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.764+0000 7faed5d9b700 1 --2- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7faec0039000 0x7faec003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.769+0000 7faec77fe700 1 -- 172.21.15.28:0/2995055207 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faec8019c70 con 0x7faed8108820 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.965+0000 7faed5d9b700 1 -- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7faec0039000 msgr2=0x7faec003b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1179097296 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:16.965+0000 7faed5d9b700 1 --2- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7faec0039000 0x7faec003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.063+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7faed819f440 con 0x7faed8108820 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.065+0000 7faec77fe700 1 -- 172.21.15.28:0/2995055207 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7faed819f440 con 0x7faed8108820 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.069+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7faec0039000 msgr2=0x7faec003b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.069+0000 7faedccb4700 1 --2- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7faec0039000 0x7faec003b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.069+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 msgr2=0x7faed819bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.069+0000 7faedccb4700 1 --2- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed819bda0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faec8011670 tx=0x7faec8011b60 comp rx=0 tx=0).stop 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.070+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 shutdown_connections 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.070+0000 7faedccb4700 1 --2- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7faec0039000 0x7faec003b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:17.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.070+0000 7faedccb4700 1 --2- 172.21.15.28:0/2995055207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8108820 0x7faed819bda0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:17.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.070+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 >> 172.21.15.28:0/2995055207 conn(0x7faed8102210 msgr2=0x7faed8102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:17.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.071+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 shutdown_connections 2024-05-09T08:16:17.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:17.072+0000 7faedccb4700 1 -- 172.21.15.28:0/2995055207 wait complete. 2024-05-09T08:16:17.644 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-05-09T08:16:17.644 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-05-09T08:16:17.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2995055207' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-09T08:16:18.186 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:17.890+0000 7f235ca99200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:16:18.186 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:18.070+0000 7f235ca99200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:16:18.554 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:18.248+0000 7f235ca99200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:16:18.554 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:18.550+0000 7f235ca99200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:16:18.862 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:18.646+0000 7f235ca99200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:16:18.862 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:18.857+0000 7f235ca99200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:16:19.186 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:19 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:19.060+0000 7f235ca99200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:16:19.935 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:19 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:19.690+0000 7f235ca99200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:16:20.436 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:20.129+0000 7f235ca99200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:16:20.436 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:20.230+0000 7f235ca99200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:16:20.935 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:20.439+0000 7f235ca99200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:16:21.338 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:20.984+0000 7f235ca99200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:16:21.338 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:21.080+0000 7f235ca99200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:16:21.685 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:21.334+0000 7f235ca99200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:16:21.686 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:21.473+0000 7f235ca99200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:16:21.686 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:21.572+0000 7f235ca99200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:16:22.008 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:21.693+0000 7f235ca99200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:16:22.008 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:22.004+0000 7f235ca99200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:16:22.913 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:22.633+0000 7f235ca99200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:16:22.913 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:22.857+0000 7f235ca99200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:16:23.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: Active manager daemon x restarted 2024-05-09T08:16:23.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: Activating manager daemon x 2024-05-09T08:16:23.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: osdmap e2: 0 total, 0 up, 0 in 2024-05-09T08:16:23.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: mgrmap e6: x(active, starting, since 0.00326935s) 2024-05-09T08:16:23.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:16:23.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:16:23.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:16:23.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:16:23.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:16:23.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: Manager daemon x is now available 2024-05-09T08:16:23.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:23.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:23.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:16:23.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:22 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:16:24.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:23 smithi028 ceph-mon[30119]: Found migration_current of "None". Setting to last migration. 2024-05-09T08:16:24.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:23 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:16:24.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:23 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:16:24.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:23 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:24.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:23 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:24.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:23 smithi028 ceph-mon[30119]: mgrmap e7: x(active, since 1.00561s) 2024-05-09T08:16:24.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:16:24.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.092+0000 7fcf2ede5700 1 Processor -- start 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.092+0000 7fcf2ede5700 1 -- start start 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.092+0000 7fcf2ede5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2810ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.092+0000 7fcf2ede5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2807bed0 con 0x7fcf28108830 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.092+0000 7fcf2cb81700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2810ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.092+0000 7fcf2cb81700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2810ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59922/0 (socket says 172.21.15.28:59922) 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.092+0000 7fcf2cb81700 1 -- 172.21.15.28:0/3432204151 learned_addr learned my addr 172.21.15.28:0/3432204151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.093+0000 7fcf2cb81700 1 -- 172.21.15.28:0/3432204151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf2810b160 con 0x7fcf28108830 2024-05-09T08:16:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.093+0000 7fcf2cb81700 1 --2- 172.21.15.28:0/3432204151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2810ac20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0098a0 tx=0x7fcf1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ce70647b32b341 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.093+0000 7fcf277fe700 1 -- 172.21.15.28:0/3432204151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf1c004160 con 0x7fcf28108830 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.093+0000 7fcf277fe700 1 -- 172.21.15.28:0/3432204151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcf1c004300 con 0x7fcf28108830 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.093+0000 7fcf277fe700 1 -- 172.21.15.28:0/3432204151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf1c011b50 con 0x7fcf28108830 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.093+0000 7fcf2ede5700 1 -- 172.21.15.28:0/3432204151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 msgr2=0x7fcf2810ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.093+0000 7fcf2ede5700 1 --2- 172.21.15.28:0/3432204151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2810ac20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0098a0 tx=0x7fcf1c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.094+0000 7fcf2ede5700 1 -- 172.21.15.28:0/3432204151 shutdown_connections 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.094+0000 7fcf2ede5700 1 --2- 172.21.15.28:0/3432204151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2810ac20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.094+0000 7fcf2ede5700 1 -- 172.21.15.28:0/3432204151 >> 172.21.15.28:0/3432204151 conn(0x7fcf28102240 msgr2=0x7fcf281046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.094+0000 7fcf2ede5700 1 -- 172.21.15.28:0/3432204151 shutdown_connections 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.094+0000 7fcf2ede5700 1 -- 172.21.15.28:0/3432204151 wait complete. 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.094+0000 7fcf2ede5700 1 Processor -- start 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.094+0000 7fcf2ede5700 1 -- start start 2024-05-09T08:16:24.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf2ede5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf2ede5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2810c190 con 0x7fcf28108830 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf2cb81700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf2cb81700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2819bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59936/0 (socket says 172.21.15.28:59936) 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf2cb81700 1 -- 172.21.15.28:0/2289750619 learned_addr learned my addr 172.21.15.28:0/2289750619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf2cb81700 1 -- 172.21.15.28:0/2289750619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf2819c270 con 0x7fcf28108830 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf2cb81700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2819bd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c011b60 tx=0x7fcf1c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf1c003b20 con 0x7fcf28108830 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf2ede5700 1 -- 172.21.15.28:0/2289750619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf2819c500 con 0x7fcf28108830 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcf1c003cc0 con 0x7fcf28108830 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.095+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf1c024840 con 0x7fcf28108830 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.096+0000 7fcf2ede5700 1 -- 172.21.15.28:0/2289750619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf2819f1f0 con 0x7fcf28108830 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.096+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fcf1c022020 con 0x7fcf28108830 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.097+0000 7fcf25ffb700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 0x7fcf1803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:24.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.097+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 --> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcf1803bc20 con 0x7fcf18039050 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.097+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcf1c011e00 con 0x7fcf28108830 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.097+0000 7fcf27fff700 1 -- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 msgr2=0x7fcf1803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1179097296 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.097+0000 7fcf27fff700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 0x7fcf1803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.298+0000 7fcf27fff700 1 -- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 msgr2=0x7fcf1803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1179097296 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.298+0000 7fcf27fff700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 0x7fcf1803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.699+0000 7fcf27fff700 1 -- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 msgr2=0x7fcf1803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1179097296 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:18.699+0000 7fcf27fff700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 0x7fcf1803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:19.500+0000 7fcf27fff700 1 -- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 msgr2=0x7fcf1803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1179097296 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:19.500+0000 7fcf27fff700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 0x7fcf1803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:21.101+0000 7fcf27fff700 1 -- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 msgr2=0x7fcf1803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1179097296 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:21.101+0000 7fcf27fff700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 0x7fcf1803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:22.860+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7fcf1c02d910 con 0x7fcf28108830 2024-05-09T08:16:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:22.860+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 msgr2=0x7fcf1803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:22.860+0000 7fcf25ffb700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 0x7fcf1803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.861+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fcf1c04eca0 con 0x7fcf28108830 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.861+0000 7fcf25ffb700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcf1803c980 0x7fcf1803ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.861+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcf1c012ea0 con 0x7fcf1803c980 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.862+0000 7fcf27fff700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcf1803c980 0x7fcf1803ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.863+0000 7fcf27fff700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcf1803c980 0x7fcf1803ed70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcf14003a10 tx=0x7fcf14009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.864+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 <== mgr.14120 v2:172.21.15.28:6800/2931568341 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fcf1c012ea0 con 0x7fcf1803c980 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.870+0000 7fcf2ede5700 1 -- 172.21.15.28:0/2289750619 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fcf28069450 con 0x7fcf1803c980 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.870+0000 7fcf25ffb700 1 -- 172.21.15.28:0/2289750619 <== mgr.14120 v2:172.21.15.28:6800/2931568341 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fcf28069450 con 0x7fcf1803c980 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 -- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcf1803c980 msgr2=0x7fcf1803ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcf1803c980 0x7fcf1803ed70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcf14003a10 tx=0x7fcf14009290 comp rx=0 tx=0).stop 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 -- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 msgr2=0x7fcf2819bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2819bd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c011b60 tx=0x7fcf1c011c40 comp rx=0 tx=0).stop 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 -- 172.21.15.28:0/2289750619 shutdown_connections 2024-05-09T08:16:24.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcf1803c980 0x7fcf1803ed70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf28108830 0x7fcf2819bd30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 --2- 172.21.15.28:0/2289750619 >> [v2:172.21.15.28:6800/1179097296,v1:172.21.15.28:6801/1179097296] conn(0x7fcf18039050 0x7fcf1803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 -- 172.21.15.28:0/2289750619 >> 172.21.15.28:0/2289750619 conn(0x7fcf28102240 msgr2=0x7fcf28102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 -- 172.21.15.28:0/2289750619 shutdown_connections 2024-05-09T08:16:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:23.871+0000 7fcf2ede5700 1 -- 172.21.15.28:0/2289750619 wait complete. 2024-05-09T08:16:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-05-09T08:16:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-05-09T08:16:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:24 smithi028 ceph-mon[30119]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-09T08:16:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:24 smithi028 ceph-mon[30119]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-09T08:16:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:24 smithi028 ceph-mon[30119]: [09/May/2024:08:16:24] ENGINE Bus STARTING 2024-05-09T08:16:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:24 smithi028 ceph-mon[30119]: [09/May/2024:08:16:24] ENGINE Serving on https://172.21.15.28:7150 2024-05-09T08:16:25.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:24 smithi028 ceph-mon[30119]: [09/May/2024:08:16:24] ENGINE Serving on http://172.21.15.28:8765 2024-05-09T08:16:25.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:24 smithi028 ceph-mon[30119]: [09/May/2024:08:16:24] ENGINE Bus STARTED 2024-05-09T08:16:25.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:24 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:16:25.841 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.017+0000 7fcdd02c5700 1 Processor -- start 2024-05-09T08:16:25.841 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.017+0000 7fcdd02c5700 1 -- start start 2024-05-09T08:16:25.841 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.018+0000 7fcdd02c5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc810ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:25.841 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.018+0000 7fcdd02c5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc807bed0 con 0x7fcdc8108820 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.018+0000 7fcdce061700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc810ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.018+0000 7fcdce061700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc810ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48298/0 (socket says 172.21.15.28:48298) 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.018+0000 7fcdce061700 1 -- 172.21.15.28:0/3812046831 learned_addr learned my addr 172.21.15.28:0/3812046831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.018+0000 7fcdce061700 1 -- 172.21.15.28:0/3812046831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc810b150 con 0x7fcdc8108820 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.018+0000 7fcdce061700 1 --2- 172.21.15.28:0/3812046831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc810ac10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcdc40098a0 tx=0x7fcdc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f163e8bf66ff0f99 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.018+0000 7fcdcd05f700 1 -- 172.21.15.28:0/3812046831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc4004160 con 0x7fcdc8108820 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.019+0000 7fcdcd05f700 1 -- 172.21.15.28:0/3812046831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcdc4004300 con 0x7fcdc8108820 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.019+0000 7fcdcd05f700 1 -- 172.21.15.28:0/3812046831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc4011b50 con 0x7fcdc8108820 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.019+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3812046831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 msgr2=0x7fcdc810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.019+0000 7fcdd02c5700 1 --2- 172.21.15.28:0/3812046831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc810ac10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcdc40098a0 tx=0x7fcdc4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.019+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3812046831 shutdown_connections 2024-05-09T08:16:25.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.019+0000 7fcdd02c5700 1 --2- 172.21.15.28:0/3812046831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc810ac10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.019+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3812046831 >> 172.21.15.28:0/3812046831 conn(0x7fcdc8102210 msgr2=0x7fcdc8104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.019+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3812046831 shutdown_connections 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.019+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3812046831 wait complete. 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.020+0000 7fcdd02c5700 1 Processor -- start 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.020+0000 7fcdd02c5700 1 -- start start 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.020+0000 7fcdd02c5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc819bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.020+0000 7fcdd02c5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc810c180 con 0x7fcdc8108820 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.020+0000 7fcdce061700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc819bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.020+0000 7fcdce061700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc819bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48302/0 (socket says 172.21.15.28:48302) 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.020+0000 7fcdce061700 1 -- 172.21.15.28:0/3492303509 learned_addr learned my addr 172.21.15.28:0/3492303509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.021+0000 7fcdce061700 1 -- 172.21.15.28:0/3492303509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc819c2a0 con 0x7fcdc8108820 2024-05-09T08:16:25.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.021+0000 7fcdce061700 1 --2- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc819bd60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4011b60 tx=0x7fcdc4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.021+0000 7fcdbf7fe700 1 -- 172.21.15.28:0/3492303509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc4003b20 con 0x7fcdc8108820 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.021+0000 7fcdbf7fe700 1 -- 172.21.15.28:0/3492303509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcdc4003cc0 con 0x7fcdc8108820 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.021+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc819c530 con 0x7fcdc8108820 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.021+0000 7fcdbf7fe700 1 -- 172.21.15.28:0/3492303509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc4024840 con 0x7fcdc8108820 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.021+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc819f220 con 0x7fcdc8108820 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.022+0000 7fcdbf7fe700 1 -- 172.21.15.28:0/3492303509 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fcdc4022020 con 0x7fcdc8108820 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.022+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc8063980 con 0x7fcdc8108820 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.022+0000 7fcdbf7fe700 1 --2- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcdb40390a0 0x7fcdb403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.022+0000 7fcdbf7fe700 1 -- 172.21.15.28:0/3492303509 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcdc4011e00 con 0x7fcdc8108820 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.022+0000 7fcdcd860700 1 --2- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcdb40390a0 0x7fcdb403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.023+0000 7fcdcd860700 1 --2- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcdb40390a0 0x7fcdb403b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8006fd0 tx=0x7fcdb8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.027+0000 7fcdbf7fe700 1 -- 172.21.15.28:0/3492303509 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcdc400b3e0 con 0x7fcdc8108820 2024-05-09T08:16:25.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.249+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fcdc8103510 con 0x7fcdb40390a0 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.254+0000 7fcdbf7fe700 1 -- 172.21.15.28:0/3492303509 <== mgr.14120 v2:172.21.15.28:6800/2931568341 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcdc8103510 con 0x7fcdb40390a0 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcdb40390a0 msgr2=0x7fcdb403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 --2- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcdb40390a0 0x7fcdb403b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8006fd0 tx=0x7fcdb8006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 msgr2=0x7fcdc819bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 --2- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc819bd60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4011b60 tx=0x7fcdc4011c40 comp rx=0 tx=0).stop 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 shutdown_connections 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 --2- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fcdb40390a0 0x7fcdb403b560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 --2- 172.21.15.28:0/3492303509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdc8108820 0x7fcdc819bd60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 >> 172.21.15.28:0/3492303509 conn(0x7fcdc8102210 msgr2=0x7fcdc8102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 shutdown_connections 2024-05-09T08:16:25.845 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:25.257+0000 7fcdd02c5700 1 -- 172.21.15.28:0/3492303509 wait complete. 2024-05-09T08:16:26.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:25 smithi028 ceph-mon[30119]: mgrmap e8: x(active, since 2s) 2024-05-09T08:16:26.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:25 smithi028 ceph-mon[30119]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:16:26.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:25 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:26.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:25 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:16:27.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:26 smithi028 ceph-mon[30119]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:16:27.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-09T08:16:27.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.221+0000 7f247f80f700 1 Processor -- start 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.222+0000 7f247f80f700 1 -- start start 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.222+0000 7f247f80f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f2478108260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.222+0000 7f247f80f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2478108830 con 0x7f2478107e40 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.222+0000 7f247d5ab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f2478108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.222+0000 7f247d5ab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f2478108260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48314/0 (socket says 172.21.15.28:48314) 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.222+0000 7f247d5ab700 1 -- 172.21.15.28:0/2246284593 learned_addr learned my addr 172.21.15.28:0/2246284593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.222+0000 7f247d5ab700 1 -- 172.21.15.28:0/2246284593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2478108ff0 con 0x7f2478107e40 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.222+0000 7f247d5ab700 1 --2- 172.21.15.28:0/2246284593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f2478108260 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f24680098a0 tx=0x7f2468009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce2073d030cbb19 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.222+0000 7f2477fff700 1 -- 172.21.15.28:0/2246284593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2468004160 con 0x7f2478107e40 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.223+0000 7f2477fff700 1 -- 172.21.15.28:0/2246284593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2468004300 con 0x7f2478107e40 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.223+0000 7f2477fff700 1 -- 172.21.15.28:0/2246284593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2468011b50 con 0x7f2478107e40 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.223+0000 7f247f80f700 1 -- 172.21.15.28:0/2246284593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 msgr2=0x7f2478108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:27.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.223+0000 7f247f80f700 1 --2- 172.21.15.28:0/2246284593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f2478108260 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f24680098a0 tx=0x7f2468009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.223+0000 7f247f80f700 1 -- 172.21.15.28:0/2246284593 shutdown_connections 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.223+0000 7f247f80f700 1 --2- 172.21.15.28:0/2246284593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f2478108260 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.223+0000 7f247f80f700 1 -- 172.21.15.28:0/2246284593 >> 172.21.15.28:0/2246284593 conn(0x7f2478103400 msgr2=0x7f2478105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.223+0000 7f247f80f700 1 -- 172.21.15.28:0/2246284593 shutdown_connections 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.223+0000 7f247f80f700 1 -- 172.21.15.28:0/2246284593 wait complete. 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.224+0000 7f247f80f700 1 Processor -- start 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.224+0000 7f247f80f700 1 -- start start 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.224+0000 7f247f80f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f247807db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.224+0000 7f247f80f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f247810c3a0 con 0x7f2478107e40 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.224+0000 7f247d5ab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f247807db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.224+0000 7f247d5ab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f247807db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48324/0 (socket says 172.21.15.28:48324) 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.224+0000 7f247d5ab700 1 -- 172.21.15.28:0/3082237701 learned_addr learned my addr 172.21.15.28:0/3082237701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.224+0000 7f247d5ab700 1 -- 172.21.15.28:0/3082237701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247807e0d0 con 0x7f2478107e40 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.225+0000 7f247d5ab700 1 --2- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f247807db90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2468011850 tx=0x7f2468011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:27.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.225+0000 7f24767fc700 1 -- 172.21.15.28:0/3082237701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2468003af0 con 0x7f2478107e40 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.225+0000 7f24767fc700 1 -- 172.21.15.28:0/3082237701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2468020070 con 0x7f2478107e40 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.225+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247807c2c0 con 0x7f2478107e40 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.225+0000 7f24767fc700 1 -- 172.21.15.28:0/3082237701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24680195a0 con 0x7f2478107e40 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.225+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247807c800 con 0x7f2478107e40 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.226+0000 7f24767fc700 1 -- 172.21.15.28:0/3082237701 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f246801a430 con 0x7f2478107e40 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.226+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2478069450 con 0x7f2478107e40 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.226+0000 7f24767fc700 1 --2- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f2460039160 0x7f246003b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.226+0000 7f24767fc700 1 -- 172.21.15.28:0/3082237701 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2468011e00 con 0x7f2478107e40 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.226+0000 7f247cdaa700 1 --2- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f2460039160 0x7f246003b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.226+0000 7f247cdaa700 1 --2- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f2460039160 0x7f246003b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f246c006fd0 tx=0x7f246c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.231+0000 7f24767fc700 1 -- 172.21.15.28:0/3082237701 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f246800e260 con 0x7f2478107e40 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.440+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f2478104600 con 0x7f2460039160 2024-05-09T08:16:27.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.441+0000 7f24767fc700 1 -- 172.21.15.28:0/3082237701 <== mgr.14120 v2:172.21.15.28:6800/2931568341 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f2478104600 con 0x7f2460039160 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f2460039160 msgr2=0x7f246003b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 --2- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f2460039160 0x7f246003b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f246c006fd0 tx=0x7f246c006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 msgr2=0x7f247807db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 --2- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f247807db90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2468011850 tx=0x7f2468011c10 comp rx=0 tx=0).stop 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 shutdown_connections 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 --2- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f2460039160 0x7f246003b620 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 --2- 172.21.15.28:0/3082237701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2478107e40 0x7f247807db90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 >> 172.21.15.28:0/3082237701 conn(0x7f2478103400 msgr2=0x7f2478103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 shutdown_connections 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:26.443+0000 7f247f80f700 1 -- 172.21.15.28:0/3082237701 wait complete. 2024-05-09T08:16:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-05-09T08:16:27.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: Generating public/private rsa key pair. 2024-05-09T08:16:27.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: Your identification has been saved in /tmp/tmplgq2h51w/key. 2024-05-09T08:16:27.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: Your public key has been saved in /tmp/tmplgq2h51w/key.pub. 2024-05-09T08:16:27.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: The key fingerprint is: 2024-05-09T08:16:27.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: SHA256:QBPKSBP00+Wcvcdzbq/UQroO1VjhI0d4n24pEGXwIxg ceph-3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:16:27.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: The key's randomart image is: 2024-05-09T08:16:27.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: +---[RSA 3072]----+ 2024-05-09T08:16:27.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: | .=. +..E .o+o | 2024-05-09T08:16:27.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: | . = + = oo.+o.. | 2024-05-09T08:16:27.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: | . = o +...o+=..| 2024-05-09T08:16:27.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: | . . +.*.o.| 2024-05-09T08:16:27.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: | S . B = .| 2024-05-09T08:16:27.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: | o B = | 2024-05-09T08:16:27.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: | . . B .| 2024-05-09T08:16:27.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: | . + o | 2024-05-09T08:16:27.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: | .o ...| 2024-05-09T08:16:27.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: +----[SHA256]-----+ 2024-05-09T08:16:28.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:27 smithi028 ceph-mon[30119]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:16:28.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:27 smithi028 ceph-mon[30119]: Generating ssh key... 2024-05-09T08:16:28.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:27 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:28.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:27 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:28.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.436+0000 7f84b8b3a700 1 Processor -- start 2024-05-09T08:16:28.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.436+0000 7f84b8b3a700 1 -- start start 2024-05-09T08:16:28.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b8b3a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b410ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:28.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b8b3a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84b407be90 con 0x7f84b4108880 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b410ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b410ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48330/0 (socket says 172.21.15.28:48330) 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b259c700 1 -- 172.21.15.28:0/2322431373 learned_addr learned my addr 172.21.15.28:0/2322431373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b259c700 1 -- 172.21.15.28:0/2322431373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b410b1b0 con 0x7f84b4108880 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b259c700 1 --2- 172.21.15.28:0/2322431373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b410ac70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84a40098a0 tx=0x7f84a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02d527c9e5355fc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b159a700 1 -- 172.21.15.28:0/2322431373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84a4004160 con 0x7f84b4108880 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b159a700 1 -- 172.21.15.28:0/2322431373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f84a4004300 con 0x7f84b4108880 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.437+0000 7f84b159a700 1 -- 172.21.15.28:0/2322431373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84a4011b50 con 0x7f84b4108880 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.438+0000 7f84b8b3a700 1 -- 172.21.15.28:0/2322431373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 msgr2=0x7f84b410ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.438+0000 7f84b8b3a700 1 --2- 172.21.15.28:0/2322431373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b410ac70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84a40098a0 tx=0x7f84a4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.438+0000 7f84b8b3a700 1 -- 172.21.15.28:0/2322431373 shutdown_connections 2024-05-09T08:16:28.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.438+0000 7f84b8b3a700 1 --2- 172.21.15.28:0/2322431373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b410ac70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.438+0000 7f84b8b3a700 1 -- 172.21.15.28:0/2322431373 >> 172.21.15.28:0/2322431373 conn(0x7f84b4102230 msgr2=0x7f84b41046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.438+0000 7f84b8b3a700 1 -- 172.21.15.28:0/2322431373 shutdown_connections 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.438+0000 7f84b8b3a700 1 -- 172.21.15.28:0/2322431373 wait complete. 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.439+0000 7f84b8b3a700 1 Processor -- start 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.439+0000 7f84b8b3a700 1 -- start start 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.439+0000 7f84b8b3a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b419bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.439+0000 7f84b8b3a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84b410c1d0 con 0x7f84b4108880 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.439+0000 7f84b259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b419bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.439+0000 7f84b259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b419bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48336/0 (socket says 172.21.15.28:48336) 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.439+0000 7f84b259c700 1 -- 172.21.15.28:0/3826457744 learned_addr learned my addr 172.21.15.28:0/3826457744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.440+0000 7f84b259c700 1 -- 172.21.15.28:0/3826457744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b419c1c0 con 0x7f84b4108880 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.440+0000 7f84b259c700 1 --2- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b419bc80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f84a4011670 tx=0x7f84a4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.440+0000 7f84ab7fe700 1 -- 172.21.15.28:0/3826457744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84a40039c0 con 0x7f84b4108880 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.440+0000 7f84ab7fe700 1 -- 172.21.15.28:0/3826457744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f84a4011e50 con 0x7f84b4108880 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.440+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84b419c450 con 0x7f84b4108880 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.440+0000 7f84ab7fe700 1 -- 172.21.15.28:0/3826457744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84a401a5c0 con 0x7f84b4108880 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.440+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84b419f140 con 0x7f84b4108880 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.441+0000 7f84ab7fe700 1 -- 172.21.15.28:0/3826457744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f84a401a8c0 con 0x7f84b4108880 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.441+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84b4063980 con 0x7f84b4108880 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.441+0000 7f84ab7fe700 1 --2- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f84a00390c0 0x7f84a003b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.441+0000 7f84ab7fe700 1 -- 172.21.15.28:0/3826457744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f84a404d440 con 0x7f84b4108880 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.441+0000 7f84b1d9b700 1 --2- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f84a00390c0 0x7f84a003b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.442+0000 7f84b1d9b700 1 --2- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f84a00390c0 0x7f84a003b580 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f849c006fd0 tx=0x7f849c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.446+0000 7f84ab7fe700 1 -- 172.21.15.28:0/3826457744 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84a4019c70 con 0x7f84b4108880 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.654+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f84b41033b0 con 0x7f84a00390c0 2024-05-09T08:16:28.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.885+0000 7f84ab7fe700 1 -- 172.21.15.28:0/3826457744 <== mgr.14120 v2:172.21.15.28:6800/2931568341 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f84b41033b0 con 0x7f84a00390c0 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f84a00390c0 msgr2=0x7f84a003b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 --2- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f84a00390c0 0x7f84a003b580 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f849c006fd0 tx=0x7f849c006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 msgr2=0x7f84b419bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 --2- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b419bc80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f84a4011670 tx=0x7f84a4011b60 comp rx=0 tx=0).stop 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 shutdown_connections 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 --2- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f84a00390c0 0x7f84a003b580 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 --2- 172.21.15.28:0/3826457744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84b4108880 0x7f84b419bc80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 >> 172.21.15.28:0/3826457744 conn(0x7f84b4102230 msgr2=0x7f84b4102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 shutdown_connections 2024-05-09T08:16:28.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:27.887+0000 7f84b8b3a700 1 -- 172.21.15.28:0/3826457744 wait complete. 2024-05-09T08:16:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHPY9HBR6azUX3GTzSOsY5TfO2p12KnSrrhErh5jrZWMyTJCGy8XoQ9DhSdUM+cv9nyStF/0NPhPpwp3zXVx+XmhOmS/p56y/D0Liz1gLcoC3iW78OuolUXMObP9fz8IGEyqCVl1QBDLemGFf6B74GBB/c9uCSkYO7afQEc4tmX/d13YcdScguSWRv1eBIoLu5z9lb8fl/0ygGvGP0nZXqjj7ssQHaJxCpqAXk4TPC2so1FRiEV5C0mCazBxbPqDWGrioKnIGzwDi12k1sGv8EVB2vnaiBKUFS7srHdA8Qz73PJpl1x40q8vt8H6zYq1GmuHinPxyKCGsQFgAjIWmEz5foJPDljt08khW8cdTQ/08btK8FQYE0+uOoQhpZbYuwUkIaHZZoFblBD8L1aifLW5iasBmWpdtOFTFE54ZPSauodIcVRGtml8IlI5aHgaJrjFs8LVnfDPjUG4E3y0zqM5MFjjh0cFmLhQJU86vEQem18HXZO1d69aO5NdsZaZs= ceph-3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:16:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.780+0000 7fe323dca700 1 Processor -- start 2024-05-09T08:16:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.780+0000 7fe323dca700 1 -- start start 2024-05-09T08:16:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.780+0000 7fe323dca700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.780+0000 7fe323dca700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c07be50 con 0x7fe31c1066a0 2024-05-09T08:16:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.780+0000 7fe321b66700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.780+0000 7fe321b66700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48344/0 (socket says 172.21.15.28:48344) 2024-05-09T08:16:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.780+0000 7fe321b66700 1 -- 172.21.15.28:0/2939360025 learned_addr learned my addr 172.21.15.28:0/2939360025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.781+0000 7fe321b66700 1 -- 172.21.15.28:0/2939360025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c107000 con 0x7fe31c1066a0 2024-05-09T08:16:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.781+0000 7fe321b66700 1 --2- 172.21.15.28:0/2939360025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c106ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3100098a0 tx=0x7fe310009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6670b8954b42f12 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.781+0000 7fe320b64700 1 -- 172.21.15.28:0/2939360025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe310004160 con 0x7fe31c1066a0 2024-05-09T08:16:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.781+0000 7fe320b64700 1 -- 172.21.15.28:0/2939360025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe310004300 con 0x7fe31c1066a0 2024-05-09T08:16:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.781+0000 7fe320b64700 1 -- 172.21.15.28:0/2939360025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe310011b50 con 0x7fe31c1066a0 2024-05-09T08:16:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.782+0000 7fe323dca700 1 -- 172.21.15.28:0/2939360025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 msgr2=0x7fe31c106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.782+0000 7fe323dca700 1 --2- 172.21.15.28:0/2939360025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c106ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3100098a0 tx=0x7fe310009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.782+0000 7fe323dca700 1 -- 172.21.15.28:0/2939360025 shutdown_connections 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.782+0000 7fe323dca700 1 --2- 172.21.15.28:0/2939360025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c106ac0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.782+0000 7fe323dca700 1 -- 172.21.15.28:0/2939360025 >> 172.21.15.28:0/2939360025 conn(0x7fe31c102260 msgr2=0x7fe31c1046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.782+0000 7fe323dca700 1 -- 172.21.15.28:0/2939360025 shutdown_connections 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.782+0000 7fe323dca700 1 -- 172.21.15.28:0/2939360025 wait complete. 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.782+0000 7fe323dca700 1 Processor -- start 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.782+0000 7fe323dca700 1 -- start start 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe323dca700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c19dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe323dca700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c108090 con 0x7fe31c1066a0 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe321b66700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c19dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe321b66700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c19dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48350/0 (socket says 172.21.15.28:48350) 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe321b66700 1 -- 172.21.15.28:0/437429511 learned_addr learned my addr 172.21.15.28:0/437429511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe321b66700 1 -- 172.21.15.28:0/437429511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c19e3e0 con 0x7fe31c1066a0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe321b66700 1 --2- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c19dea0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe310011670 tx=0x7fe310011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe30effd700 1 -- 172.21.15.28:0/437429511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3100039c0 con 0x7fe31c1066a0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe30effd700 1 -- 172.21.15.28:0/437429511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe310011e50 con 0x7fe31c1066a0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31c19e670 con 0x7fe31c1066a0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe30effd700 1 -- 172.21.15.28:0/437429511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe31001a5c0 con 0x7fe31c1066a0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.783+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31c1a1360 con 0x7fe31c1066a0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.784+0000 7fe30effd700 1 -- 172.21.15.28:0/437429511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe31001a870 con 0x7fe31c1066a0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.784+0000 7fe30effd700 1 --2- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe308039110 0x7fe30803b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.784+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe31c063980 con 0x7fe31c1066a0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.784+0000 7fe30effd700 1 -- 172.21.15.28:0/437429511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe31004d3a0 con 0x7fe31c1066a0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.785+0000 7fe321365700 1 --2- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe308039110 0x7fe30803b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.785+0000 7fe321365700 1 --2- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe308039110 0x7fe30803b5d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe318006fd0 tx=0x7fe318006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.790+0000 7fe30effd700 1 -- 172.21.15.28:0/437429511 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe310019c70 con 0x7fe31c1066a0 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:28.999+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fe31c02def0 con 0x7fe308039110 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.000+0000 7fe30effd700 1 -- 172.21.15.28:0/437429511 <== mgr.14120 v2:172.21.15.28:6800/2931568341 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fe31c02def0 con 0x7fe308039110 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.002+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe308039110 msgr2=0x7fe30803b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.002+0000 7fe323dca700 1 --2- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe308039110 0x7fe30803b5d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe318006fd0 tx=0x7fe318006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.002+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 msgr2=0x7fe31c19dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.002+0000 7fe323dca700 1 --2- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c19dea0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe310011670 tx=0x7fe310011b60 comp rx=0 tx=0).stop 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.002+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 shutdown_connections 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.002+0000 7fe323dca700 1 --2- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe308039110 0x7fe30803b5d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.002+0000 7fe323dca700 1 --2- 172.21.15.28:0/437429511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c1066a0 0x7fe31c19dea0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.002+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 >> 172.21.15.28:0/437429511 conn(0x7fe31c102260 msgr2=0x7fe31c194b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:29.589 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.003+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 shutdown_connections 2024-05-09T08:16:29.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.003+0000 7fe323dca700 1 -- 172.21.15.28:0/437429511 wait complete. 2024-05-09T08:16:29.590 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-09T08:16:29.590 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-05-09T08:16:29.590 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-05-09T08:16:30.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:29 smithi028 ceph-mon[30119]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:16:31.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:30 smithi028 ceph-mon[30119]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:16:31.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:30 smithi028 ceph-mon[30119]: Deploying cephadm binary to smithi028 2024-05-09T08:16:34.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-05-09T08:16:34.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.991+0000 7f2805a97700 1 Processor -- start 2024-05-09T08:16:34.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.991+0000 7f2805a97700 1 -- start start 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f2805a97700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f28001070f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f2805a97700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28001076c0 con 0x7f2800106cd0 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f27ff7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f28001070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f27ff7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f28001070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48352/0 (socket says 172.21.15.28:48352) 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f27ff7fe700 1 -- 172.21.15.28:0/1502509840 learned_addr learned my addr 172.21.15.28:0/1502509840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f27ff7fe700 1 -- 172.21.15.28:0/1502509840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2800107ed0 con 0x7f2800106cd0 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f27ff7fe700 1 --2- 172.21.15.28:0/1502509840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f28001070f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f27f0009e80 tx=0x7f27f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e163601b80eb93c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f27feffd700 1 -- 172.21.15.28:0/1502509840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27f0017070 con 0x7f2800106cd0 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f27feffd700 1 -- 172.21.15.28:0/1502509840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f27f000b8c0 con 0x7f2800106cd0 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.992+0000 7f27feffd700 1 -- 172.21.15.28:0/1502509840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27f00045c0 con 0x7f2800106cd0 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.993+0000 7f2805a97700 1 -- 172.21.15.28:0/1502509840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 msgr2=0x7f28001070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.993+0000 7f2805a97700 1 --2- 172.21.15.28:0/1502509840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f28001070f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f27f0009e80 tx=0x7f27f000b330 comp rx=0 tx=0).stop 2024-05-09T08:16:35.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.993+0000 7f2805a97700 1 -- 172.21.15.28:0/1502509840 shutdown_connections 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.993+0000 7f2805a97700 1 --2- 172.21.15.28:0/1502509840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f28001070f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.993+0000 7f2805a97700 1 -- 172.21.15.28:0/1502509840 >> 172.21.15.28:0/1502509840 conn(0x7f2800102270 msgr2=0x7f28001046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.993+0000 7f2805a97700 1 -- 172.21.15.28:0/1502509840 shutdown_connections 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.993+0000 7f2805a97700 1 -- 172.21.15.28:0/1502509840 wait complete. 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.994+0000 7f2805a97700 1 Processor -- start 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.994+0000 7f2805a97700 1 -- start start 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.994+0000 7f2805a97700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f280019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.994+0000 7f2805a97700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f280007aff0 con 0x7f2800106cd0 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.994+0000 7f27ff7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f280019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.994+0000 7f27ff7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f280019dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48360/0 (socket says 172.21.15.28:48360) 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.994+0000 7f27ff7fe700 1 -- 172.21.15.28:0/2525297568 learned_addr learned my addr 172.21.15.28:0/2525297568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.995+0000 7f27ff7fe700 1 -- 172.21.15.28:0/2525297568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280019e420 con 0x7f2800106cd0 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.995+0000 7f27ff7fe700 1 --2- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f280019dee0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f27f0009530 tx=0x7f27f000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:35.001 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.995+0000 7f27fd7fa700 1 -- 172.21.15.28:0/2525297568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27f0020020 con 0x7f2800106cd0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.995+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280019e6b0 con 0x7f2800106cd0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.995+0000 7f27fd7fa700 1 -- 172.21.15.28:0/2525297568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f27f0003b50 con 0x7f2800106cd0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.995+0000 7f27fd7fa700 1 -- 172.21.15.28:0/2525297568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27f0017040 con 0x7f2800106cd0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.995+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28001a1350 con 0x7f2800106cd0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.996+0000 7f27fd7fa700 1 -- 172.21.15.28:0/2525297568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f27f0023020 con 0x7f2800106cd0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.996+0000 7f27fd7fa700 1 --2- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f27ec039110 0x7f27ec03b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.996+0000 7f27fd7fa700 1 -- 172.21.15.28:0/2525297568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f27f000f050 con 0x7f2800106cd0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.996+0000 7f27f6ffd700 1 --2- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f27ec039110 0x7f27ec03b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.996+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2800063980 con 0x7f2800106cd0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:29.997+0000 7f27f6ffd700 1 --2- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f27ec039110 0x7f27ec03b5d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27e8006fd0 tx=0x7f27e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:30.001+0000 7f27fd7fa700 1 -- 172.21.15.28:0/2525297568 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f27f002a020 con 0x7f2800106cd0 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:30.211+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7f2800191890 con 0x7f27ec039110 2024-05-09T08:16:35.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.423+0000 7f27fd7fa700 1 -- 172.21.15.28:0/2525297568 <== mgr.14120 v2:172.21.15.28:6800/2931568341 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f2800191890 con 0x7f27ec039110 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f27ec039110 msgr2=0x7f27ec03b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 --2- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f27ec039110 0x7f27ec03b5d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27e8006fd0 tx=0x7f27e8006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 msgr2=0x7f280019dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 --2- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f280019dee0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f27f0009530 tx=0x7f27f000bd50 comp rx=0 tx=0).stop 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 shutdown_connections 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 --2- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f27ec039110 0x7f27ec03b5d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 --2- 172.21.15.28:0/2525297568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106cd0 0x7f280019dee0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 >> 172.21.15.28:0/2525297568 conn(0x7f2800102270 msgr2=0x7f28001900f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 shutdown_connections 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:34.425+0000 7f2805a97700 1 -- 172.21.15.28:0/2525297568 wait complete. 2024-05-09T08:16:35.003 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-05-09T08:16:35.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:35 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:35.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:35 smithi028 ceph-mon[30119]: Added host smithi028 2024-05-09T08:16:35.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:35 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:16:36.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:36 smithi028 ceph-mon[30119]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:16:36.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:36 smithi028 ceph-mon[30119]: Saving service mon spec with placement count:5 2024-05-09T08:16:36.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:36 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:37.222 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-09T08:16:37.222 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.415+0000 7f8633297700 1 Processor -- start 2024-05-09T08:16:37.222 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.416+0000 7f8633297700 1 -- start start 2024-05-09T08:16:37.222 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.416+0000 7f8633297700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c108240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.416+0000 7f8633297700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f862c108810 con 0x7f862c107e20 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.416+0000 7f8631033700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.416+0000 7f8631033700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41400/0 (socket says 172.21.15.28:41400) 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.416+0000 7f8631033700 1 -- 172.21.15.28:0/226764919 learned_addr learned my addr 172.21.15.28:0/226764919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.416+0000 7f8631033700 1 -- 172.21.15.28:0/226764919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862c108fd0 con 0x7f862c107e20 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8631033700 1 --2- 172.21.15.28:0/226764919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c108240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f86280098a0 tx=0x7f8628009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab37f0b41e5a6cd9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8623fff700 1 -- 172.21.15.28:0/226764919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8628004160 con 0x7f862c107e20 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8623fff700 1 -- 172.21.15.28:0/226764919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8628004300 con 0x7f862c107e20 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8623fff700 1 -- 172.21.15.28:0/226764919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8628011b50 con 0x7f862c107e20 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8633297700 1 -- 172.21.15.28:0/226764919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 msgr2=0x7f862c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8633297700 1 --2- 172.21.15.28:0/226764919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c108240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f86280098a0 tx=0x7f8628009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8633297700 1 -- 172.21.15.28:0/226764919 shutdown_connections 2024-05-09T08:16:37.223 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8633297700 1 --2- 172.21.15.28:0/226764919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c108240 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8633297700 1 -- 172.21.15.28:0/226764919 >> 172.21.15.28:0/226764919 conn(0x7f862c103400 msgr2=0x7f862c105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8633297700 1 -- 172.21.15.28:0/226764919 shutdown_connections 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.417+0000 7f8633297700 1 -- 172.21.15.28:0/226764919 wait complete. 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.418+0000 7f8633297700 1 Processor -- start 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.418+0000 7f8633297700 1 -- start start 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.418+0000 7f8633297700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c07db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.418+0000 7f8633297700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f862c10c380 con 0x7f862c107e20 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.418+0000 7f8631033700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c07db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.418+0000 7f8631033700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c07db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41414/0 (socket says 172.21.15.28:41414) 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.418+0000 7f8631033700 1 -- 172.21.15.28:0/4140177310 learned_addr learned my addr 172.21.15.28:0/4140177310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.419+0000 7f8631033700 1 -- 172.21.15.28:0/4140177310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862c07e0d0 con 0x7f862c107e20 2024-05-09T08:16:37.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.419+0000 7f8631033700 1 --2- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c07db90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8628011850 tx=0x7f8628011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.419+0000 7f86227fc700 1 -- 172.21.15.28:0/4140177310 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8628003af0 con 0x7f862c107e20 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.419+0000 7f86227fc700 1 -- 172.21.15.28:0/4140177310 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8628020070 con 0x7f862c107e20 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.419+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f862c07c2c0 con 0x7f862c107e20 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.419+0000 7f86227fc700 1 -- 172.21.15.28:0/4140177310 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86280195a0 con 0x7f862c107e20 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.419+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f862c07c800 con 0x7f862c107e20 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.420+0000 7f86227fc700 1 -- 172.21.15.28:0/4140177310 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f8628019850 con 0x7f862c107e20 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.420+0000 7f86227fc700 1 --2- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f8618039160 0x7f861803b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.420+0000 7f86227fc700 1 -- 172.21.15.28:0/4140177310 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8628011e00 con 0x7f862c107e20 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.420+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f862c063980 con 0x7f862c107e20 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.420+0000 7f8630832700 1 --2- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f8618039160 0x7f861803b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:37.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.421+0000 7f8630832700 1 --2- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f8618039160 0x7f861803b620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f861c006fd0 tx=0x7f861c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.425+0000 7f86227fc700 1 -- 172.21.15.28:0/4140177310 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8628003c90 con 0x7f862c107e20 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.649+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f862c07d080 con 0x7f8618039160 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.653+0000 7f86227fc700 1 -- 172.21.15.28:0/4140177310 <== mgr.14120 v2:172.21.15.28:6800/2931568341 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f862c07d080 con 0x7f8618039160 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.655+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f8618039160 msgr2=0x7f861803b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.655+0000 7f8633297700 1 --2- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f8618039160 0x7f861803b620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f861c006fd0 tx=0x7f861c006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.655+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 msgr2=0x7f862c07db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.655+0000 7f8633297700 1 --2- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c07db90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8628011850 tx=0x7f8628011c10 comp rx=0 tx=0).stop 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.656+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 shutdown_connections 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.656+0000 7f8633297700 1 --2- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f8618039160 0x7f861803b620 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.656+0000 7f8633297700 1 --2- 172.21.15.28:0/4140177310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c107e20 0x7f862c07db90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.656+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 >> 172.21.15.28:0/4140177310 conn(0x7f862c103400 msgr2=0x7f862c103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.656+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 shutdown_connections 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:35.656+0000 7f8633297700 1 -- 172.21.15.28:0/4140177310 wait complete. 2024-05-09T08:16:37.226 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-05-09T08:16:38.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:37 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:38.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:37 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:38.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-09T08:16:38.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.645+0000 7f50c12d7700 1 Processor -- start 2024-05-09T08:16:38.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.645+0000 7f50c12d7700 1 -- start start 2024-05-09T08:16:38.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.645+0000 7f50c12d7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc106ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:38.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.645+0000 7f50c12d7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc07be50 con 0x7f50bc106690 2024-05-09T08:16:38.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.645+0000 7f50baffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc106ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:38.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.646+0000 7f50baffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc106ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41430/0 (socket says 172.21.15.28:41430) 2024-05-09T08:16:38.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.646+0000 7f50baffd700 1 -- 172.21.15.28:0/582158804 learned_addr learned my addr 172.21.15.28:0/582158804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:38.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.646+0000 7f50baffd700 1 -- 172.21.15.28:0/582158804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50bc106ff0 con 0x7f50bc106690 2024-05-09T08:16:38.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.646+0000 7f50baffd700 1 --2- 172.21.15.28:0/582158804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc106ab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0098a0 tx=0x7f50ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ad94356d07b305 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:38.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.646+0000 7f50b9ffb700 1 -- 172.21.15.28:0/582158804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50ac004160 con 0x7f50bc106690 2024-05-09T08:16:38.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.646+0000 7f50b9ffb700 1 -- 172.21.15.28:0/582158804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ac004300 con 0x7f50bc106690 2024-05-09T08:16:38.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.646+0000 7f50b9ffb700 1 -- 172.21.15.28:0/582158804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50ac011b20 con 0x7f50bc106690 2024-05-09T08:16:38.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.647+0000 7f50c12d7700 1 -- 172.21.15.28:0/582158804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 msgr2=0x7f50bc106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:38.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.647+0000 7f50c12d7700 1 --2- 172.21.15.28:0/582158804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc106ab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0098a0 tx=0x7f50ac009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:38.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.647+0000 7f50c12d7700 1 -- 172.21.15.28:0/582158804 shutdown_connections 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.647+0000 7f50c12d7700 1 --2- 172.21.15.28:0/582158804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc106ab0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.647+0000 7f50c12d7700 1 -- 172.21.15.28:0/582158804 >> 172.21.15.28:0/582158804 conn(0x7f50bc102210 msgr2=0x7f50bc104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.647+0000 7f50c12d7700 1 -- 172.21.15.28:0/582158804 shutdown_connections 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.647+0000 7f50c12d7700 1 -- 172.21.15.28:0/582158804 wait complete. 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.648+0000 7f50c12d7700 1 Processor -- start 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.648+0000 7f50c12d7700 1 -- start start 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.648+0000 7f50c12d7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.648+0000 7f50c12d7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc1080c0 con 0x7f50bc106690 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.648+0000 7f50baffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.648+0000 7f50baffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc19df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41436/0 (socket says 172.21.15.28:41436) 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.648+0000 7f50baffd700 1 -- 172.21.15.28:0/3956150234 learned_addr learned my addr 172.21.15.28:0/3956150234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.649+0000 7f50baffd700 1 -- 172.21.15.28:0/3956150234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50bc19e460 con 0x7f50bc106690 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.649+0000 7f50baffd700 1 --2- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc19df20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0117e0 tx=0x7f50ac011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.649+0000 7f50b3fff700 1 -- 172.21.15.28:0/3956150234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50ac003bf0 con 0x7f50bc106690 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.649+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50bc19e6f0 con 0x7f50bc106690 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.649+0000 7f50b3fff700 1 -- 172.21.15.28:0/3956150234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ac01a910 con 0x7f50bc106690 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.649+0000 7f50b3fff700 1 -- 172.21.15.28:0/3956150234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50ac024970 con 0x7f50bc106690 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.649+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50bc19eb50 con 0x7f50bc106690 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.650+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50bc063980 con 0x7f50bc106690 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.650+0000 7f50b3fff700 1 -- 172.21.15.28:0/3956150234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f50ac022020 con 0x7f50bc106690 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.651+0000 7f50b3fff700 1 --2- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f50a8039000 0x7f50a803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.651+0000 7f50b3fff700 1 -- 172.21.15.28:0/3956150234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f50ac04d3b0 con 0x7f50bc106690 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.651+0000 7f50ba7fc700 1 --2- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f50a8039000 0x7f50a803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.651+0000 7f50ba7fc700 1 --2- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f50a8039000 0x7f50a803b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f50a4006fd0 tx=0x7f50a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.656+0000 7f50b3fff700 1 -- 172.21.15.28:0/3956150234 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f50ac011e20 con 0x7f50bc106690 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.878+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f50bc062760 con 0x7f50a8039000 2024-05-09T08:16:38.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.883+0000 7f50b3fff700 1 -- 172.21.15.28:0/3956150234 <== mgr.14120 v2:172.21.15.28:6800/2931568341 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f50bc062760 con 0x7f50a8039000 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.885+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f50a8039000 msgr2=0x7f50a803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.885+0000 7f50c12d7700 1 --2- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f50a8039000 0x7f50a803b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f50a4006fd0 tx=0x7f50a4006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.885+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 msgr2=0x7f50bc19df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.885+0000 7f50c12d7700 1 --2- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc19df20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0117e0 tx=0x7f50ac011c10 comp rx=0 tx=0).stop 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.885+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 shutdown_connections 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.885+0000 7f50c12d7700 1 --2- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f50a8039000 0x7f50a803b4c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.885+0000 7f50c12d7700 1 --2- 172.21.15.28:0/3956150234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc106690 0x7f50bc19df20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.885+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 >> 172.21.15.28:0/3956150234 conn(0x7f50bc102210 msgr2=0x7f50bc194b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.886+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 shutdown_connections 2024-05-09T08:16:38.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:37.886+0000 7f50c12d7700 1 -- 172.21.15.28:0/3956150234 wait complete. 2024-05-09T08:16:39.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:38 smithi028 ceph-mon[30119]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:16:39.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:38 smithi028 ceph-mon[30119]: Saving service mgr spec with placement count:2 2024-05-09T08:16:39.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:38 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:39.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.837+0000 7ff62be4f700 1 Processor -- start 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.837+0000 7ff62be4f700 1 -- start start 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff62be4f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff6241070d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff62be4f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6241076a0 con 0x7ff624106cb0 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff629beb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff6241070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff629beb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff6241070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41438/0 (socket says 172.21.15.28:41438) 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff629beb700 1 -- 172.21.15.28:0/750206162 learned_addr learned my addr 172.21.15.28:0/750206162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff629beb700 1 -- 172.21.15.28:0/750206162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff624107eb0 con 0x7ff624106cb0 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff629beb700 1 --2- 172.21.15.28:0/750206162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff6241070d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff62000d180 tx=0x7ff62000d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3985c6361916d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff628be9700 1 -- 172.21.15.28:0/750206162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff620019070 con 0x7ff624106cb0 2024-05-09T08:16:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff628be9700 1 -- 172.21.15.28:0/750206162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff62000de20 con 0x7ff624106cb0 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.838+0000 7ff628be9700 1 -- 172.21.15.28:0/750206162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff620009af0 con 0x7ff624106cb0 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.839+0000 7ff62be4f700 1 -- 172.21.15.28:0/750206162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 msgr2=0x7ff6241070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.839+0000 7ff62be4f700 1 --2- 172.21.15.28:0/750206162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff6241070d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff62000d180 tx=0x7ff62000d490 comp rx=0 tx=0).stop 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.839+0000 7ff62be4f700 1 -- 172.21.15.28:0/750206162 shutdown_connections 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.839+0000 7ff62be4f700 1 --2- 172.21.15.28:0/750206162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff6241070d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.839+0000 7ff62be4f700 1 -- 172.21.15.28:0/750206162 >> 172.21.15.28:0/750206162 conn(0x7ff624102210 msgr2=0x7ff624104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.839+0000 7ff62be4f700 1 -- 172.21.15.28:0/750206162 shutdown_connections 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.839+0000 7ff62be4f700 1 -- 172.21.15.28:0/750206162 wait complete. 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.840+0000 7ff62be4f700 1 Processor -- start 2024-05-09T08:16:39.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.840+0000 7ff62be4f700 1 -- start start 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.840+0000 7ff62be4f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff624199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.840+0000 7ff62be4f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff62407aff0 con 0x7ff624106cb0 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.840+0000 7ff629beb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff624199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.840+0000 7ff629beb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff624199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41440/0 (socket says 172.21.15.28:41440) 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.840+0000 7ff629beb700 1 -- 172.21.15.28:0/4140005102 learned_addr learned my addr 172.21.15.28:0/4140005102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.840+0000 7ff629beb700 1 -- 172.21.15.28:0/4140005102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62419a0b0 con 0x7ff624106cb0 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.841+0000 7ff629beb700 1 --2- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff624199b70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff62000b740 tx=0x7ff620003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.841+0000 7ff61affd700 1 -- 172.21.15.28:0/4140005102 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff620023020 con 0x7ff624106cb0 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.841+0000 7ff61affd700 1 -- 172.21.15.28:0/4140005102 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff62000b200 con 0x7ff624106cb0 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.841+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62419a340 con 0x7ff624106cb0 2024-05-09T08:16:39.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.841+0000 7ff61affd700 1 -- 172.21.15.28:0/4140005102 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff620019040 con 0x7ff624106cb0 2024-05-09T08:16:39.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.841+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62419d030 con 0x7ff624106cb0 2024-05-09T08:16:39.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.842+0000 7ff61affd700 1 -- 172.21.15.28:0/4140005102 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7ff620026020 con 0x7ff624106cb0 2024-05-09T08:16:39.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.842+0000 7ff61affd700 1 --2- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7ff610039050 0x7ff61003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:39.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.842+0000 7ff61affd700 1 -- 172.21.15.28:0/4140005102 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff620011050 con 0x7ff624106cb0 2024-05-09T08:16:39.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.842+0000 7ff6293ea700 1 --2- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7ff610039050 0x7ff61003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:39.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.842+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff624063980 con 0x7ff624106cb0 2024-05-09T08:16:39.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.843+0000 7ff6293ea700 1 --2- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7ff610039050 0x7ff61003b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff614006fd0 tx=0x7ff614006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:39.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:38.848+0000 7ff61affd700 1 -- 172.21.15.28:0/4140005102 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff62002d020 con 0x7ff624106cb0 2024-05-09T08:16:39.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.055+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7ff6241930d0 con 0x7ff624106cb0 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.058+0000 7ff61affd700 1 -- 172.21.15.28:0/4140005102 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7ff6200273b0 con 0x7ff624106cb0 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7ff610039050 msgr2=0x7ff61003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 --2- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7ff610039050 0x7ff61003b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff614006fd0 tx=0x7ff614006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 msgr2=0x7ff624199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 --2- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff624199b70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff62000b740 tx=0x7ff620003d00 comp rx=0 tx=0).stop 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 shutdown_connections 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 --2- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7ff610039050 0x7ff61003b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 --2- 172.21.15.28:0/4140005102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff624106cb0 0x7ff624199b70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 >> 172.21.15.28:0/4140005102 conn(0x7ff624102210 msgr2=0x7ff624190190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 shutdown_connections 2024-05-09T08:16:39.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:39.060+0000 7ff62be4f700 1 -- 172.21.15.28:0/4140005102 wait complete. 2024-05-09T08:16:40.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:40 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4140005102' entity='client.admin' 2024-05-09T08:16:41.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.116+0000 7fc39b9e5700 1 Processor -- start 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.117+0000 7fc39b9e5700 1 -- start start 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.117+0000 7fc39b9e5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc3941086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.117+0000 7fc39b9e5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc394108c70 con 0x7fc394108280 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.117+0000 7fc399781700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc3941086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.117+0000 7fc399781700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc3941086a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41452/0 (socket says 172.21.15.28:41452) 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.117+0000 7fc399781700 1 -- 172.21.15.28:0/1617661879 learned_addr learned my addr 172.21.15.28:0/1617661879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.117+0000 7fc399781700 1 -- 172.21.15.28:0/1617661879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc394106ee0 con 0x7fc394108280 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.117+0000 7fc399781700 1 --2- 172.21.15.28:0/1617661879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc3941086a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc39000bb40 tx=0x7fc39000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5121f7b8d596a04c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc38bfff700 1 -- 172.21.15.28:0/1617661879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc39000dbe0 con 0x7fc394108280 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc38bfff700 1 -- 172.21.15.28:0/1617661879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc390004510 con 0x7fc394108280 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc38bfff700 1 -- 172.21.15.28:0/1617661879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc390010470 con 0x7fc394108280 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc39b9e5700 1 -- 172.21.15.28:0/1617661879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 msgr2=0x7fc3941086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:41.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc39b9e5700 1 --2- 172.21.15.28:0/1617661879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc3941086a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc39000bb40 tx=0x7fc39000be50 comp rx=0 tx=0).stop 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc39b9e5700 1 -- 172.21.15.28:0/1617661879 shutdown_connections 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc39b9e5700 1 --2- 172.21.15.28:0/1617661879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc3941086a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc39b9e5700 1 -- 172.21.15.28:0/1617661879 >> 172.21.15.28:0/1617661879 conn(0x7fc3941019f0 msgr2=0x7fc394103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc39b9e5700 1 -- 172.21.15.28:0/1617661879 shutdown_connections 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.118+0000 7fc39b9e5700 1 -- 172.21.15.28:0/1617661879 wait complete. 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.119+0000 7fc39b9e5700 1 Processor -- start 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.119+0000 7fc39b9e5700 1 -- start start 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.119+0000 7fc39b9e5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc39419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.119+0000 7fc39b9e5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc394105dd0 con 0x7fc394108280 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.119+0000 7fc399781700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc39419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.119+0000 7fc399781700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc39419df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41456/0 (socket says 172.21.15.28:41456) 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.119+0000 7fc399781700 1 -- 172.21.15.28:0/2040835374 learned_addr learned my addr 172.21.15.28:0/2040835374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.119+0000 7fc399781700 1 -- 172.21.15.28:0/2040835374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39419e440 con 0x7fc394108280 2024-05-09T08:16:41.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.120+0000 7fc399781700 1 --2- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc39419df00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3900046e0 tx=0x7fc3900040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.120+0000 7fc38a7fc700 1 -- 172.21.15.28:0/2040835374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc390010720 con 0x7fc394108280 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.120+0000 7fc38a7fc700 1 -- 172.21.15.28:0/2040835374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc39001d070 con 0x7fc394108280 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.120+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39419e6d0 con 0x7fc394108280 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.120+0000 7fc38a7fc700 1 -- 172.21.15.28:0/2040835374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc390021b10 con 0x7fc394108280 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.120+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3941a13c0 con 0x7fc394108280 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.121+0000 7fc38a7fc700 1 -- 172.21.15.28:0/2040835374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc39001f030 con 0x7fc394108280 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.121+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc394063980 con 0x7fc394108280 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.121+0000 7fc38a7fc700 1 --2- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fc3800391b0 0x7fc38003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.121+0000 7fc38a7fc700 1 -- 172.21.15.28:0/2040835374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc39004d4b0 con 0x7fc394108280 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.121+0000 7fc398f80700 1 --2- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fc3800391b0 0x7fc38003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.122+0000 7fc398f80700 1 --2- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fc3800391b0 0x7fc38003b670 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc384006fd0 tx=0x7fc384006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.126+0000 7fc38a7fc700 1 -- 172.21.15.28:0/2040835374 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc390015070 con 0x7fc394108280 2024-05-09T08:16:41.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.333+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fc394193050 con 0x7fc394108280 2024-05-09T08:16:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.336+0000 7fc38a7fc700 1 -- 172.21.15.28:0/2040835374 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fc394193050 con 0x7fc394108280 2024-05-09T08:16:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fc3800391b0 msgr2=0x7fc38003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 --2- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fc3800391b0 0x7fc38003b670 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc384006fd0 tx=0x7fc384006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 msgr2=0x7fc39419df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 --2- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc39419df00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3900046e0 tx=0x7fc3900040b0 comp rx=0 tx=0).stop 2024-05-09T08:16:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 shutdown_connections 2024-05-09T08:16:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 --2- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fc3800391b0 0x7fc38003b670 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 --2- 172.21.15.28:0/2040835374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc394108280 0x7fc39419df00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 >> 172.21.15.28:0/2040835374 conn(0x7fc3941019f0 msgr2=0x7fc394190110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:41.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 shutdown_connections 2024-05-09T08:16:41.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:40.338+0000 7fc39b9e5700 1 -- 172.21.15.28:0/2040835374 wait complete. 2024-05-09T08:16:41.048 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-05-09T08:16:41.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:41 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2040835374' entity='client.admin' 2024-05-09T08:16:41.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:41 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:42.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:42 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3035830044' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-09T08:16:42.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:42 smithi028 ceph-mon[30119]: from='mgr.14120 172.21.15.28:0/2928660876' entity='mgr.x' 2024-05-09T08:16:42.650 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: ignoring --setuser ceph since I am not root 2024-05-09T08:16:42.651 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: ignoring --setgroup ceph since I am not root 2024-05-09T08:16:42.651 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:42.522+0000 7fa66b895200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:16:42.935 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:42.721+0000 7fa66b895200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:16:42.936 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:42.828+0000 7fa66b895200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:16:42.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.499+0000 7f1824b58700 1 Processor -- start 2024-05-09T08:16:42.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.499+0000 7f1824b58700 1 -- start start 2024-05-09T08:16:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.499+0000 7f1824b58700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f18201064c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.499+0000 7f1824b58700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1820106a90 con 0x7f18201060a0 2024-05-09T08:16:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.499+0000 7f181e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f18201064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.499+0000 7f181e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f18201064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41470/0 (socket says 172.21.15.28:41470) 2024-05-09T08:16:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.499+0000 7f181e59c700 1 -- 172.21.15.28:0/887361341 learned_addr learned my addr 172.21.15.28:0/887361341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.500+0000 7f181e59c700 1 -- 172.21.15.28:0/887361341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1820109030 con 0x7f18201060a0 2024-05-09T08:16:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.500+0000 7f181e59c700 1 --2- 172.21.15.28:0/887361341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f18201064c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f18100098a0 tx=0x7f1810009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa72c9a6029b6fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.500+0000 7f181d59a700 1 -- 172.21.15.28:0/887361341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1810004160 con 0x7f18201060a0 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.500+0000 7f181d59a700 1 -- 172.21.15.28:0/887361341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1810004300 con 0x7f18201060a0 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.500+0000 7f181d59a700 1 -- 172.21.15.28:0/887361341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1810011b20 con 0x7f18201060a0 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.501+0000 7f1824b58700 1 -- 172.21.15.28:0/887361341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 msgr2=0x7f18201064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.501+0000 7f1824b58700 1 --2- 172.21.15.28:0/887361341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f18201064c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f18100098a0 tx=0x7f1810009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.501+0000 7f1824b58700 1 -- 172.21.15.28:0/887361341 shutdown_connections 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.501+0000 7f1824b58700 1 --2- 172.21.15.28:0/887361341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f18201064c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.501+0000 7f1824b58700 1 -- 172.21.15.28:0/887361341 >> 172.21.15.28:0/887361341 conn(0x7f18201019f0 msgr2=0x7f1820103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.501+0000 7f1824b58700 1 -- 172.21.15.28:0/887361341 shutdown_connections 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.501+0000 7f1824b58700 1 -- 172.21.15.28:0/887361341 wait complete. 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.501+0000 7f1824b58700 1 Processor -- start 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.502+0000 7f1824b58700 1 -- start start 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.502+0000 7f1824b58700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f182019bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.502+0000 7f1824b58700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f182010c3e0 con 0x7f18201060a0 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.502+0000 7f181e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f182019bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.502+0000 7f181e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f182019bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41480/0 (socket says 172.21.15.28:41480) 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.502+0000 7f181e59c700 1 -- 172.21.15.28:0/3035830044 learned_addr learned my addr 172.21.15.28:0/3035830044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.502+0000 7f181e59c700 1 -- 172.21.15.28:0/3035830044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182019c1b0 con 0x7f18201060a0 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.502+0000 7f181e59c700 1 --2- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f182019bc70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f18100117e0 tx=0x7f1810011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.503+0000 7f180f7fe700 1 -- 172.21.15.28:0/3035830044 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1810003bf0 con 0x7f18201060a0 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.503+0000 7f180f7fe700 1 -- 172.21.15.28:0/3035830044 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f181001a910 con 0x7f18201060a0 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.503+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f182019c440 con 0x7f18201060a0 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.503+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f182019c8a0 con 0x7f18201060a0 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.504+0000 7f180f7fe700 1 -- 172.21.15.28:0/3035830044 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1810024970 con 0x7f18201060a0 2024-05-09T08:16:42.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.504+0000 7f180f7fe700 1 -- 172.21.15.28:0/3035830044 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1810022020 con 0x7f18201060a0 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.504+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1820063980 con 0x7f18201060a0 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.504+0000 7f180f7fe700 1 --2- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f1808039110 0x7f180803b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.504+0000 7f180f7fe700 1 -- 172.21.15.28:0/3035830044 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1810052580 con 0x7f18201060a0 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.504+0000 7f181dd9b700 1 --2- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f1808039110 0x7f180803b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.505+0000 7f181dd9b700 1 --2- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f1808039110 0x7f180803b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1814006fd0 tx=0x7f1814006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.509+0000 7f180f7fe700 1 -- 172.21.15.28:0/3035830044 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f181001fe40 con 0x7f18201060a0 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:41.784+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f182019f310 con 0x7f18201060a0 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.338+0000 7f180f7fe700 1 -- 172.21.15.28:0/3035830044 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f1810018d60 con 0x7f18201060a0 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.340+0000 7f180f7fe700 1 -- 172.21.15.28:0/3035830044 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f1810019c60 con 0x7f18201060a0 2024-05-09T08:16:42.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f1808039110 msgr2=0x7f180803b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:42.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 --2- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f1808039110 0x7f180803b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1814006fd0 tx=0x7f1814006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:42.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 msgr2=0x7f182019bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:42.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 --2- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f182019bc70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f18100117e0 tx=0x7f1810011ba0 comp rx=0 tx=0).stop 2024-05-09T08:16:42.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 shutdown_connections 2024-05-09T08:16:42.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 --2- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7f1808039110 0x7f180803b5d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:42.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 --2- 172.21.15.28:0/3035830044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18201060a0 0x7f182019bc70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:42.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 >> 172.21.15.28:0/3035830044 conn(0x7f18201019f0 msgr2=0x7f18201023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:42.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 shutdown_connections 2024-05-09T08:16:42.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:42.342+0000 7f1824b58700 1 -- 172.21.15.28:0/3035830044 wait complete. 2024-05-09T08:16:43.253 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:43 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:43.158+0000 7fa66b895200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:16:43.254 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:43 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:43.250+0000 7fa66b895200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:16:43.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:43 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3035830044' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-09T08:16:43.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:43 smithi028 ceph-mon[30119]: mgrmap e9: x(active, since 19s) 2024-05-09T08:16:44.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:44 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/305749011' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-09T08:16:44.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.458+0000 7fe6e5ed2700 1 Processor -- start 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.458+0000 7fe6e5ed2700 1 -- start start 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.458+0000 7fe6e5ed2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e0106a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.458+0000 7fe6e5ed2700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e007bed0 con 0x7fe6e0106630 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.459+0000 7fe6df7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e0106a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.459+0000 7fe6df7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e0106a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41504/0 (socket says 172.21.15.28:41504) 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.459+0000 7fe6df7fe700 1 -- 172.21.15.28:0/4267839096 learned_addr learned my addr 172.21.15.28:0/4267839096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.459+0000 7fe6df7fe700 1 -- 172.21.15.28:0/4267839096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e0106f90 con 0x7fe6e0106630 2024-05-09T08:16:44.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.459+0000 7fe6df7fe700 1 --2- 172.21.15.28:0/4267839096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e0106a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00098a0 tx=0x7fe6d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ba00824c085b24 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.459+0000 7fe6deffd700 1 -- 172.21.15.28:0/4267839096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0004160 con 0x7fe6e0106630 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.459+0000 7fe6deffd700 1 -- 172.21.15.28:0/4267839096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6d0004300 con 0x7fe6e0106630 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.459+0000 7fe6deffd700 1 -- 172.21.15.28:0/4267839096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0011b20 con 0x7fe6e0106630 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.460+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/4267839096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 msgr2=0x7fe6e0106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.460+0000 7fe6e5ed2700 1 --2- 172.21.15.28:0/4267839096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e0106a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00098a0 tx=0x7fe6d0009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.460+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/4267839096 shutdown_connections 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.460+0000 7fe6e5ed2700 1 --2- 172.21.15.28:0/4267839096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e0106a50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.460+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/4267839096 >> 172.21.15.28:0/4267839096 conn(0x7fe6e0102270 msgr2=0x7fe6e01046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.461+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/4267839096 shutdown_connections 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.461+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/4267839096 wait complete. 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.461+0000 7fe6e5ed2700 1 Processor -- start 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.461+0000 7fe6e5ed2700 1 -- start start 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6e5ed2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:44.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6e5ed2700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e0108060 con 0x7fe6e0106630 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6df7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6df7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41518/0 (socket says 172.21.15.28:41518) 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6df7fe700 1 -- 172.21.15.28:0/305749011 learned_addr learned my addr 172.21.15.28:0/305749011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6df7fe700 1 -- 172.21.15.28:0/305749011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e019e360 con 0x7fe6e0106630 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6df7fe700 1 --2- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e019de20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00117e0 tx=0x7fe6d0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6dd7fa700 1 -- 172.21.15.28:0/305749011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0003bf0 con 0x7fe6e0106630 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6dd7fa700 1 -- 172.21.15.28:0/305749011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6d001a910 con 0x7fe6e0106630 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.462+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6e019e5f0 con 0x7fe6e0106630 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.463+0000 7fe6dd7fa700 1 -- 172.21.15.28:0/305749011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0024970 con 0x7fe6e0106630 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.463+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6e019ea50 con 0x7fe6e0106630 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.463+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6e0063980 con 0x7fe6e0106630 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.463+0000 7fe6dd7fa700 1 -- 172.21.15.28:0/305749011 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fe6d0022020 con 0x7fe6e0106630 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.464+0000 7fe6dd7fa700 1 --2- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe6cc039170 0x7fe6cc03b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:44.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.464+0000 7fe6dd7fa700 1 -- 172.21.15.28:0/305749011 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe6d004d3b0 con 0x7fe6e0106630 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.464+0000 7fe6d7fff700 1 -- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe6cc039170 msgr2=0x7fe6cc03b630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2931568341 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.464+0000 7fe6d7fff700 1 --2- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe6cc039170 0x7fe6cc03b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.468+0000 7fe6dd7fa700 1 -- 172.21.15.28:0/305749011 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe6d0011e20 con 0x7fe6e0106630 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.665+0000 7fe6d7fff700 1 -- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe6cc039170 msgr2=0x7fe6cc03b630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2931568341 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.665+0000 7fe6d7fff700 1 --2- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe6cc039170 0x7fe6cc03b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.779+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fe6e0062f30 con 0x7fe6e0106630 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.779+0000 7fe6dd7fa700 1 -- 172.21.15.28:0/305749011 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fe6d000b3e0 con 0x7fe6e0106630 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.781+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe6cc039170 msgr2=0x7fe6cc03b630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.781+0000 7fe6e5ed2700 1 --2- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe6cc039170 0x7fe6cc03b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.781+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 msgr2=0x7fe6e019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.781+0000 7fe6e5ed2700 1 --2- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e019de20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00117e0 tx=0x7fe6d0011c10 comp rx=0 tx=0).stop 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.781+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 shutdown_connections 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.781+0000 7fe6e5ed2700 1 --2- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe6cc039170 0x7fe6cc03b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:44.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.781+0000 7fe6e5ed2700 1 --2- 172.21.15.28:0/305749011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6e0106630 0x7fe6e019de20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:44.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.781+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 >> 172.21.15.28:0/305749011 conn(0x7fe6e0102270 msgr2=0x7fe6e0194a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:44.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.781+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 shutdown_connections 2024-05-09T08:16:44.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:43.782+0000 7fe6e5ed2700 1 -- 172.21.15.28:0/305749011 wait complete. 2024-05-09T08:16:44.404 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-05-09T08:16:44.404 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-05-09T08:16:44.685 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:44.527+0000 7fa66b895200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:16:45.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:44.709+0000 7fa66b895200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:16:45.033 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:44.891+0000 7fa66b895200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:16:45.298 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:45.196+0000 7fa66b895200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:16:45.685 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:45.294+0000 7fa66b895200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:16:45.686 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:45.506+0000 7fa66b895200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:16:46.185 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:45.709+0000 7fa66b895200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:16:46.685 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:46 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:46.342+0000 7fa66b895200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:16:47.102 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:46 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:46.785+0000 7fa66b895200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:16:47.102 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:46 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:46.886+0000 7fa66b895200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:16:47.103 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:47 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:47.098+0000 7fa66b895200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:16:47.935 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:47 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:47.655+0000 7fa66b895200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:16:47.936 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:47 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:47.752+0000 7fa66b895200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:16:48.378 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:48.011+0000 7fa66b895200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:16:48.379 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:48.152+0000 7fa66b895200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:16:48.379 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:48.253+0000 7fa66b895200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:16:48.379 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:48.375+0000 7fa66b895200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:16:49.185 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:48.692+0000 7fa66b895200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:16:49.606 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:49 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:49.324+0000 7fa66b895200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:16:49.606 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:16:49 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:16:49.550+0000 7fa66b895200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:16:49.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: Active manager daemon x restarted 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: Activating manager daemon x 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: osdmap e3: 0 total, 0 up, 0 in 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: mgrmap e10: x(active, starting, since 0.00333276s) 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: Manager daemon x is now available 2024-05-09T08:16:49.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:49 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:16:50.773 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:16:50.773 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:16:50.773 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:50 smithi028 ceph-mon[30119]: mgrmap e11: x(active, since 1.00639s) 2024-05-09T08:16:51.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:16:51.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.823+0000 7fe8a51c8700 1 Processor -- start 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.823+0000 7fe8a51c8700 1 -- start start 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.823+0000 7fe8a51c8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a010acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.823+0000 7fe8a51c8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a007be70 con 0x7fe8a01088d0 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.824+0000 7fe89ed9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a010acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.824+0000 7fe89ed9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a010acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48636/0 (socket says 172.21.15.28:48636) 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.824+0000 7fe89ed9d700 1 -- 172.21.15.28:0/3115495096 learned_addr learned my addr 172.21.15.28:0/3115495096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.824+0000 7fe89ed9d700 1 -- 172.21.15.28:0/3115495096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a010b200 con 0x7fe8a01088d0 2024-05-09T08:16:51.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.824+0000 7fe89ed9d700 1 --2- 172.21.15.28:0/3115495096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a010acc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe8900098a0 tx=0x7fe890009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51011da6fcbb5c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.824+0000 7fe89dd9b700 1 -- 172.21.15.28:0/3115495096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe890004160 con 0x7fe8a01088d0 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.824+0000 7fe89dd9b700 1 -- 172.21.15.28:0/3115495096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe890004300 con 0x7fe8a01088d0 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.824+0000 7fe89dd9b700 1 -- 172.21.15.28:0/3115495096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe890011b20 con 0x7fe8a01088d0 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.825+0000 7fe8a51c8700 1 -- 172.21.15.28:0/3115495096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 msgr2=0x7fe8a010acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.825+0000 7fe8a51c8700 1 --2- 172.21.15.28:0/3115495096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a010acc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe8900098a0 tx=0x7fe890009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.825+0000 7fe8a51c8700 1 -- 172.21.15.28:0/3115495096 shutdown_connections 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.825+0000 7fe8a51c8700 1 --2- 172.21.15.28:0/3115495096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a010acc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.825+0000 7fe8a51c8700 1 -- 172.21.15.28:0/3115495096 >> 172.21.15.28:0/3115495096 conn(0x7fe8a01022a0 msgr2=0x7fe8a01046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.825+0000 7fe8a51c8700 1 -- 172.21.15.28:0/3115495096 shutdown_connections 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.825+0000 7fe8a51c8700 1 -- 172.21.15.28:0/3115495096 wait complete. 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.826+0000 7fe8a51c8700 1 Processor -- start 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.826+0000 7fe8a51c8700 1 -- start start 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.826+0000 7fe8a51c8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.826+0000 7fe8a51c8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a010c210 con 0x7fe8a01088d0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.826+0000 7fe89ed9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.826+0000 7fe89ed9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48640/0 (socket says 172.21.15.28:48640) 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.826+0000 7fe89ed9d700 1 -- 172.21.15.28:0/641915797 learned_addr learned my addr 172.21.15.28:0/641915797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.827+0000 7fe89ed9d700 1 -- 172.21.15.28:0/641915797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a019e3e0 con 0x7fe8a01088d0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.827+0000 7fe89ed9d700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a019dea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8900116a0 tx=0x7fe890011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.827+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe890003a40 con 0x7fe8a01088d0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.827+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe890003be0 con 0x7fe8a01088d0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.827+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe89001a750 con 0x7fe8a01088d0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.827+0000 7fe8a51c8700 1 -- 172.21.15.28:0/641915797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a019e670 con 0x7fe8a01088d0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.827+0000 7fe8a51c8700 1 -- 172.21.15.28:0/641915797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a019ead0 con 0x7fe8a01088d0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.828+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fe89001aa00 con 0x7fe8a01088d0 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.828+0000 7fe887fff700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 0x7fe88803b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.828+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 --> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe88803bd40 con 0x7fe888039170 2024-05-09T08:16:51.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.828+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe89004d620 con 0x7fe8a01088d0 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.828+0000 7fe89e59c700 1 -- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 msgr2=0x7fe88803b630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2931568341 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:44.828+0000 7fe89e59c700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 0x7fe88803b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:45.029+0000 7fe89e59c700 1 -- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 msgr2=0x7fe88803b630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2931568341 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:45.029+0000 7fe89e59c700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 0x7fe88803b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:45.430+0000 7fe89e59c700 1 -- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 msgr2=0x7fe88803b630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2931568341 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:45.430+0000 7fe89e59c700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 0x7fe88803b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:46.231+0000 7fe89e59c700 1 -- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 msgr2=0x7fe88803b630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2931568341 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:46.231+0000 7fe89e59c700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 0x7fe88803b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:47.832+0000 7fe89e59c700 1 -- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 msgr2=0x7fe88803b630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2931568341 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:47.832+0000 7fe89e59c700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 0x7fe88803b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:49.553+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7fe89002d910 con 0x7fe8a01088d0 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:49.553+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 msgr2=0x7fe88803b630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:16:51.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:49.553+0000 7fe887fff700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 0x7fe88803b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.555+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7fe89004f750 con 0x7fe8a01088d0 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.555+0000 7fe887fff700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe88803cc30 0x7fe88803f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.556+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe890001d30 con 0x7fe88803cc30 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.556+0000 7fe89e59c700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe88803cc30 0x7fe88803f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.556+0000 7fe89e59c700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe88803cc30 0x7fe88803f020 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe894003a10 tx=0x7fe894009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.557+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fe890001d30 con 0x7fe88803cc30 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.564+0000 7fe8a51c8700 1 -- 172.21.15.28:0/641915797 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe8a0069450 con 0x7fe88803cc30 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.564+0000 7fe887fff700 1 -- 172.21.15.28:0/641915797 <== mgr.14152 v2:172.21.15.28:6800/953899001 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fe8a0069450 con 0x7fe88803cc30 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 -- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe88803cc30 msgr2=0x7fe88803f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe88803cc30 0x7fe88803f020 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe894003a10 tx=0x7fe894009270 comp rx=0 tx=0).stop 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 -- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 msgr2=0x7fe8a019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a019dea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8900116a0 tx=0x7fe890011ba0 comp rx=0 tx=0).stop 2024-05-09T08:16:51.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 -- 172.21.15.28:0/641915797 shutdown_connections 2024-05-09T08:16:51.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe88803cc30 0x7fe88803f020 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:51.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a01088d0 0x7fe8a019dea0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:51.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 --2- 172.21.15.28:0/641915797 >> [v2:172.21.15.28:6800/2931568341,v1:172.21.15.28:6801/2931568341] conn(0x7fe888039170 0x7fe88803b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:51.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 -- 172.21.15.28:0/641915797 >> 172.21.15.28:0/641915797 conn(0x7fe8a01022a0 msgr2=0x7fe8a01066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:51.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 -- 172.21.15.28:0/641915797 shutdown_connections 2024-05-09T08:16:51.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:50.565+0000 7fe8a51c8700 1 -- 172.21.15.28:0/641915797 wait complete. 2024-05-09T08:16:51.511 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-05-09T08:16:51.511 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-05-09T08:16:52.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:52 smithi028 ceph-mon[30119]: [09/May/2024:08:16:50] ENGINE Bus STARTING 2024-05-09T08:16:52.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:52 smithi028 ceph-mon[30119]: [09/May/2024:08:16:50] ENGINE Serving on http://172.21.15.28:8765 2024-05-09T08:16:52.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:52 smithi028 ceph-mon[30119]: [09/May/2024:08:16:50] ENGINE Serving on https://172.21.15.28:7150 2024-05-09T08:16:52.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:52 smithi028 ceph-mon[30119]: [09/May/2024:08:16:50] ENGINE Bus STARTED 2024-05-09T08:16:52.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:52 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:16:52.807 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-09T08:16:52.807 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.980+0000 7fc08e302700 1 Processor -- start 2024-05-09T08:16:52.807 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.980+0000 7fc08e302700 1 -- start start 2024-05-09T08:16:52.807 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.981+0000 7fc08e302700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc0881070d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:52.807 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.981+0000 7fc08e302700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0881076a0 con 0x7fc088106cb0 2024-05-09T08:16:52.807 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.981+0000 7fc087fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc0881070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.981+0000 7fc087fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc0881070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48710/0 (socket says 172.21.15.28:48710) 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.981+0000 7fc087fff700 1 -- 172.21.15.28:0/970277848 learned_addr learned my addr 172.21.15.28:0/970277848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.981+0000 7fc087fff700 1 -- 172.21.15.28:0/970277848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc088107eb0 con 0x7fc088106cb0 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.981+0000 7fc087fff700 1 --2- 172.21.15.28:0/970277848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc0881070d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0780098a0 tx=0x7fc078009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362ed2ef9f7a429b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.981+0000 7fc086ffd700 1 -- 172.21.15.28:0/970277848 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc078004160 con 0x7fc088106cb0 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.982+0000 7fc086ffd700 1 -- 172.21.15.28:0/970277848 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc078004300 con 0x7fc088106cb0 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.982+0000 7fc086ffd700 1 -- 172.21.15.28:0/970277848 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc078011b20 con 0x7fc088106cb0 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.982+0000 7fc08e302700 1 -- 172.21.15.28:0/970277848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 msgr2=0x7fc0881070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.982+0000 7fc08e302700 1 --2- 172.21.15.28:0/970277848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc0881070d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0780098a0 tx=0x7fc078009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.982+0000 7fc08e302700 1 -- 172.21.15.28:0/970277848 shutdown_connections 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.982+0000 7fc08e302700 1 --2- 172.21.15.28:0/970277848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc0881070d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:52.808 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.982+0000 7fc08e302700 1 -- 172.21.15.28:0/970277848 >> 172.21.15.28:0/970277848 conn(0x7fc088102210 msgr2=0x7fc088104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.982+0000 7fc08e302700 1 -- 172.21.15.28:0/970277848 shutdown_connections 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.982+0000 7fc08e302700 1 -- 172.21.15.28:0/970277848 wait complete. 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.983+0000 7fc08e302700 1 Processor -- start 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.983+0000 7fc08e302700 1 -- start start 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.983+0000 7fc08e302700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc08819bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.983+0000 7fc08e302700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc08810b260 con 0x7fc088106cb0 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.983+0000 7fc087fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc08819bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.984+0000 7fc087fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc08819bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48722/0 (socket says 172.21.15.28:48722) 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.984+0000 7fc087fff700 1 -- 172.21.15.28:0/2579424557 learned_addr learned my addr 172.21.15.28:0/2579424557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:52.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.984+0000 7fc087fff700 1 -- 172.21.15.28:0/2579424557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08819c320 con 0x7fc088106cb0 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.984+0000 7fc087fff700 1 --2- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc08819bde0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0780117e0 tx=0x7fc078011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.984+0000 7fc0857fa700 1 -- 172.21.15.28:0/2579424557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc078003bf0 con 0x7fc088106cb0 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.984+0000 7fc0857fa700 1 -- 172.21.15.28:0/2579424557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc07801a910 con 0x7fc088106cb0 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.984+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc08819c5b0 con 0x7fc088106cb0 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.984+0000 7fc0857fa700 1 -- 172.21.15.28:0/2579424557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0780249a0 con 0x7fc088106cb0 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.984+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc08819f210 con 0x7fc088106cb0 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.985+0000 7fc0857fa700 1 -- 172.21.15.28:0/2579424557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7fc078022020 con 0x7fc088106cb0 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.985+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc088063980 con 0x7fc088106cb0 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.985+0000 7fc0857fa700 1 --2- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc070038f90 0x7fc07003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:52.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.985+0000 7fc0857fa700 1 -- 172.21.15.28:0/2579424557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc07804d2e0 con 0x7fc088106cb0 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.986+0000 7fc0877fe700 1 --2- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc070038f90 0x7fc07003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.986+0000 7fc0877fe700 1 --2- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc070038f90 0x7fc07003b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc07c006fd0 tx=0x7fc07c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:51.991+0000 7fc0857fa700 1 -- 172.21.15.28:0/2579424557 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc078011e20 con 0x7fc088106cb0 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.165+0000 7fc0857fa700 1 -- 172.21.15.28:0/2579424557 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fc07802d910 con 0x7fc088106cb0 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.208+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fc0881034c0 con 0x7fc070038f90 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.322+0000 7fc0857fa700 1 -- 172.21.15.28:0/2579424557 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fc0881034c0 con 0x7fc070038f90 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc070038f90 msgr2=0x7fc07003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 --2- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc070038f90 0x7fc07003b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc07c006fd0 tx=0x7fc07c006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 msgr2=0x7fc08819bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 --2- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc08819bde0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0780117e0 tx=0x7fc078011c10 comp rx=0 tx=0).stop 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 shutdown_connections 2024-05-09T08:16:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 --2- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc070038f90 0x7fc07003b450 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:52.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 --2- 172.21.15.28:0/2579424557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088106cb0 0x7fc08819bde0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:52.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 >> 172.21.15.28:0/2579424557 conn(0x7fc088102210 msgr2=0x7fc088102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:52.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 shutdown_connections 2024-05-09T08:16:52.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:52.324+0000 7fc08e302700 1 -- 172.21.15.28:0/2579424557 wait complete. 2024-05-09T08:16:52.812 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-05-09T08:16:53.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:53 smithi028 ceph-mon[30119]: mgrmap e12: x(active, since 2s) 2024-05-09T08:16:53.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:53 smithi028 ceph-mon[30119]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:16:53.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:16:53.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:16:53.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:16:54.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:54 smithi028 ceph-mon[30119]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:16:54.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:54 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:16:55.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$xQquw/vumaFqpDDYCmBnDeyL596gmnx/SMEFZshpYmEFKAqyjPZUC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715242613, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-09T08:16:55.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.249+0000 7f4305638700 1 Processor -- start 2024-05-09T08:16:55.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.249+0000 7f4305638700 1 -- start start 2024-05-09T08:16:55.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.249+0000 7f4305638700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430010ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:55.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.249+0000 7f4305638700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f430007be70 con 0x7f4300108890 2024-05-09T08:16:55.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.249+0000 7f42feffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430010ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:55.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.250+0000 7f42feffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430010ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48726/0 (socket says 172.21.15.28:48726) 2024-05-09T08:16:55.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.250+0000 7f42feffd700 1 -- 172.21.15.28:0/2038719837 learned_addr learned my addr 172.21.15.28:0/2038719837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:55.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.250+0000 7f42feffd700 1 -- 172.21.15.28:0/2038719837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430010b1c0 con 0x7f4300108890 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.250+0000 7f42feffd700 1 --2- 172.21.15.28:0/2038719837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430010ac80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f42f00098a0 tx=0x7f42f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9630f2f2784955 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.250+0000 7f42fdffb700 1 -- 172.21.15.28:0/2038719837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f0004160 con 0x7f4300108890 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.250+0000 7f42fdffb700 1 -- 172.21.15.28:0/2038719837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42f0004300 con 0x7f4300108890 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.250+0000 7f42fdffb700 1 -- 172.21.15.28:0/2038719837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f0011b20 con 0x7f4300108890 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.251+0000 7f4305638700 1 -- 172.21.15.28:0/2038719837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 msgr2=0x7f430010ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.251+0000 7f4305638700 1 --2- 172.21.15.28:0/2038719837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430010ac80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f42f00098a0 tx=0x7f42f0009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.251+0000 7f4305638700 1 -- 172.21.15.28:0/2038719837 shutdown_connections 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.251+0000 7f4305638700 1 --2- 172.21.15.28:0/2038719837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430010ac80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.251+0000 7f4305638700 1 -- 172.21.15.28:0/2038719837 >> 172.21.15.28:0/2038719837 conn(0x7f4300102260 msgr2=0x7f43001046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.251+0000 7f4305638700 1 -- 172.21.15.28:0/2038719837 shutdown_connections 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.251+0000 7f4305638700 1 -- 172.21.15.28:0/2038719837 wait complete. 2024-05-09T08:16:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.252+0000 7f4305638700 1 Processor -- start 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.252+0000 7f4305638700 1 -- start start 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.252+0000 7f4305638700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430019bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.252+0000 7f4305638700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f430010c1d0 con 0x7f4300108890 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.252+0000 7f42feffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430019bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.252+0000 7f42feffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430019bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48742/0 (socket says 172.21.15.28:48742) 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.252+0000 7f42feffd700 1 -- 172.21.15.28:0/911326301 learned_addr learned my addr 172.21.15.28:0/911326301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.252+0000 7f42feffd700 1 -- 172.21.15.28:0/911326301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430019c170 con 0x7f4300108890 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.253+0000 7f42feffd700 1 --2- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430019bc30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f42f00117e0 tx=0x7f42f0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.253+0000 7f42e7fff700 1 -- 172.21.15.28:0/911326301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f0003bf0 con 0x7f4300108890 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.253+0000 7f42e7fff700 1 -- 172.21.15.28:0/911326301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42f001a910 con 0x7f4300108890 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.253+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f430019c400 con 0x7f4300108890 2024-05-09T08:16:55.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.253+0000 7f42e7fff700 1 -- 172.21.15.28:0/911326301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f0024970 con 0x7f4300108890 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.253+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f430019c860 con 0x7f4300108890 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.254+0000 7f42e7fff700 1 -- 172.21.15.28:0/911326301 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f42f0022020 con 0x7f4300108890 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.254+0000 7f42e7fff700 1 --2- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f42e8039110 0x7f42e803b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.254+0000 7f42e7fff700 1 -- 172.21.15.28:0/911326301 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f42f004e4d0 con 0x7f4300108890 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.254+0000 7f42fe7fc700 1 --2- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f42e8039110 0x7f42e803b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.254+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4300063980 con 0x7f4300108890 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.255+0000 7f42fe7fc700 1 --2- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f42e8039110 0x7f42e803b5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f42f4006fd0 tx=0x7f42f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.259+0000 7f42e7fff700 1 -- 172.21.15.28:0/911326301 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f42f0011e20 con 0x7f4300108890 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.496+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f4300103370 con 0x7f42e8039110 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.742+0000 7f42e7fff700 1 -- 172.21.15.28:0/911326301 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f4300103370 con 0x7f42e8039110 2024-05-09T08:16:55.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.745+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f42e8039110 msgr2=0x7f42e803b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.745+0000 7f4305638700 1 --2- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f42e8039110 0x7f42e803b5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f42f4006fd0 tx=0x7f42f4006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.745+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 msgr2=0x7f430019bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.745+0000 7f4305638700 1 --2- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430019bc30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f42f00117e0 tx=0x7f42f0011c10 comp rx=0 tx=0).stop 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.745+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 shutdown_connections 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.745+0000 7f4305638700 1 --2- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f42e8039110 0x7f42e803b5d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.745+0000 7f4305638700 1 --2- 172.21.15.28:0/911326301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4300108890 0x7f430019bc30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.745+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 >> 172.21.15.28:0/911326301 conn(0x7f4300102260 msgr2=0x7f4300102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.746+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 shutdown_connections 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:53.746+0000 7f4305638700 1 -- 172.21.15.28:0/911326301 wait complete. 2024-05-09T08:16:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-05-09T08:16:56.464 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-05-09T08:16:56.464 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.697+0000 7f46b2d7b700 1 Processor -- start 2024-05-09T08:16:56.464 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.698+0000 7f46b2d7b700 1 -- start start 2024-05-09T08:16:56.464 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.698+0000 7f46b2d7b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1064c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:56.464 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.698+0000 7f46b2d7b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac106a90 con 0x7f46ac1060a0 2024-05-09T08:16:56.464 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.698+0000 7f46b0b17700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:56.464 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.698+0000 7f46b0b17700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39758/0 (socket says 172.21.15.28:39758) 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.698+0000 7f46b0b17700 1 -- 172.21.15.28:0/3082057158 learned_addr learned my addr 172.21.15.28:0/3082057158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.698+0000 7f46b0b17700 1 -- 172.21.15.28:0/3082057158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ac109030 con 0x7f46ac1060a0 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.698+0000 7f46b0b17700 1 --2- 172.21.15.28:0/3082057158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1064c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009e80 tx=0x7f46a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e1d28c8e068139 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46a37fe700 1 -- 172.21.15.28:0/3082057158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46a8017070 con 0x7f46ac1060a0 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46a37fe700 1 -- 172.21.15.28:0/3082057158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46a800b8c0 con 0x7f46ac1060a0 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46a37fe700 1 -- 172.21.15.28:0/3082057158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46a80045c0 con 0x7f46ac1060a0 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3082057158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 msgr2=0x7f46ac1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46b2d7b700 1 --2- 172.21.15.28:0/3082057158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1064c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009e80 tx=0x7f46a800b330 comp rx=0 tx=0).stop 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3082057158 shutdown_connections 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46b2d7b700 1 --2- 172.21.15.28:0/3082057158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1064c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3082057158 >> 172.21.15.28:0/3082057158 conn(0x7f46ac1019f0 msgr2=0x7f46ac103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3082057158 shutdown_connections 2024-05-09T08:16:56.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.699+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3082057158 wait complete. 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.700+0000 7f46b2d7b700 1 Processor -- start 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.700+0000 7f46b2d7b700 1 -- start start 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.700+0000 7f46b2d7b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.700+0000 7f46b2d7b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac10c3e0 con 0x7f46ac1060a0 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.700+0000 7f46b0b17700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.700+0000 7f46b0b17700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39762/0 (socket says 172.21.15.28:39762) 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.700+0000 7f46b0b17700 1 -- 172.21.15.28:0/3350114430 learned_addr learned my addr 172.21.15.28:0/3350114430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.700+0000 7f46b0b17700 1 -- 172.21.15.28:0/3350114430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ac108df0 con 0x7f46ac1060a0 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.701+0000 7f46b0b17700 1 --2- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1088b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009fd0 tx=0x7f46a8003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.701+0000 7f46a1ffb700 1 -- 172.21.15.28:0/3350114430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46a8020020 con 0x7f46ac1060a0 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.701+0000 7f46a1ffb700 1 -- 172.21.15.28:0/3350114430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46a800bba0 con 0x7f46ac1060a0 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.701+0000 7f46a1ffb700 1 -- 172.21.15.28:0/3350114430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46a8017040 con 0x7f46ac1060a0 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.701+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ac106fe0 con 0x7f46ac1060a0 2024-05-09T08:16:56.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.701+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ac107520 con 0x7f46ac1060a0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.702+0000 7f46a1ffb700 1 -- 172.21.15.28:0/3350114430 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f46a8023020 con 0x7f46ac1060a0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.702+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ac063980 con 0x7f46ac1060a0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.702+0000 7f46a1ffb700 1 --2- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f46940391c0 0x7f469403b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.702+0000 7f46a1ffb700 1 -- 172.21.15.28:0/3350114430 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f46a800f050 con 0x7f46ac1060a0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.702+0000 7f46a3fff700 1 --2- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f46940391c0 0x7f469403b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.703+0000 7f46a3fff700 1 --2- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f46940391c0 0x7f469403b680 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f469c006fd0 tx=0x7f469c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.708+0000 7f46a1ffb700 1 -- 172.21.15.28:0/3350114430 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f46a802a020 con 0x7f46ac1060a0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.743+0000 7f46a1ffb700 1 -- 172.21.15.28:0/3350114430 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f46a8012950 con 0x7f46ac1060a0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.913+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f46ac107860 con 0x7f46ac1060a0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.914+0000 7f46a1ffb700 1 -- 172.21.15.28:0/3350114430 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f46ac107860 con 0x7f46ac1060a0 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f46940391c0 msgr2=0x7f469403b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 --2- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f46940391c0 0x7f469403b680 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f469c006fd0 tx=0x7f469c006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 msgr2=0x7f46ac1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:56.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 --2- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1088b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009fd0 tx=0x7f46a8003d00 comp rx=0 tx=0).stop 2024-05-09T08:16:56.468 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 shutdown_connections 2024-05-09T08:16:56.468 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 --2- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f46940391c0 0x7f469403b680 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:56.468 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 --2- 172.21.15.28:0/3350114430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ac1060a0 0x7f46ac1088b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:56.468 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 >> 172.21.15.28:0/3350114430 conn(0x7f46ac1019f0 msgr2=0x7f46ac1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:56.468 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 shutdown_connections 2024-05-09T08:16:56.468 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:55.917+0000 7f46b2d7b700 1 -- 172.21.15.28:0/3350114430 wait complete. 2024-05-09T08:16:56.469 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:16:56.469 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-05-09T08:16:56.473 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:16:56.473 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-09T08:16:56.473 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-05-09T08:16:56.473 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-09T08:16:56.475 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-05-09T08:16:56.475 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:56.475 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-05-09T08:16:56.475 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-05-09T08:16:56.475 INFO:teuthology.orchestra.run.smithi028.stdout: Password: 34wdcy5z2b 2024-05-09T08:16:56.475 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:56.476 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config directory 2024-05-09T08:16:56.476 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-05-09T08:16:56.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:56 smithi028 ceph-mon[30119]: mgrmap e13: x(active, since 6s) 2024-05-09T08:16:56.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:56 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3350114430' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-09T08:16:57.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.941+0000 7fafd8c6f700 1 Processor -- start 2024-05-09T08:16:57.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.941+0000 7fafd8c6f700 1 -- start start 2024-05-09T08:16:57.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.941+0000 7fafd8c6f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd4108240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:57.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.941+0000 7fafd8c6f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafd4108810 con 0x7fafd4107e20 2024-05-09T08:16:57.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.941+0000 7fafd259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd4108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:57.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.941+0000 7fafd259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd4108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39774/0 (socket says 172.21.15.28:39774) 2024-05-09T08:16:57.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.941+0000 7fafd259c700 1 -- 172.21.15.28:0/2038093732 learned_addr learned my addr 172.21.15.28:0/2038093732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.941+0000 7fafd259c700 1 -- 172.21.15.28:0/2038093732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd4108fd0 con 0x7fafd4107e20 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.942+0000 7fafd259c700 1 --2- 172.21.15.28:0/2038093732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd4108240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fafc40098a0 tx=0x7fafc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22253adc6be23dea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.942+0000 7fafd159a700 1 -- 172.21.15.28:0/2038093732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafc4004160 con 0x7fafd4107e20 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.942+0000 7fafd159a700 1 -- 172.21.15.28:0/2038093732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc4004300 con 0x7fafd4107e20 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.942+0000 7fafd159a700 1 -- 172.21.15.28:0/2038093732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafc4011b20 con 0x7fafd4107e20 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.942+0000 7fafd8c6f700 1 -- 172.21.15.28:0/2038093732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 msgr2=0x7fafd4108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.942+0000 7fafd8c6f700 1 --2- 172.21.15.28:0/2038093732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd4108240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fafc40098a0 tx=0x7fafc4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.942+0000 7fafd8c6f700 1 -- 172.21.15.28:0/2038093732 shutdown_connections 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.943+0000 7fafd8c6f700 1 --2- 172.21.15.28:0/2038093732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd4108240 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.943+0000 7fafd8c6f700 1 -- 172.21.15.28:0/2038093732 >> 172.21.15.28:0/2038093732 conn(0x7fafd4103400 msgr2=0x7fafd4105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.943+0000 7fafd8c6f700 1 -- 172.21.15.28:0/2038093732 shutdown_connections 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.943+0000 7fafd8c6f700 1 -- 172.21.15.28:0/2038093732 wait complete. 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.943+0000 7fafd8c6f700 1 Processor -- start 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.943+0000 7fafd8c6f700 1 -- start start 2024-05-09T08:16:57.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafd8c6f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd419bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafd8c6f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafd410c380 con 0x7fafd4107e20 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafd259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd419bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafd259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd419bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39780/0 (socket says 172.21.15.28:39780) 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafd259c700 1 -- 172.21.15.28:0/3693412376 learned_addr learned my addr 172.21.15.28:0/3693412376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafd259c700 1 -- 172.21.15.28:0/3693412376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd419c1c0 con 0x7fafd4107e20 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafd259c700 1 --2- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd419bc80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fafc4000c00 tx=0x7fafc4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafc37fe700 1 -- 172.21.15.28:0/3693412376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafc4003b60 con 0x7fafd4107e20 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd419c450 con 0x7fafd4107e20 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafc37fe700 1 -- 172.21.15.28:0/3693412376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc401a910 con 0x7fafd4107e20 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafc37fe700 1 -- 172.21.15.28:0/3693412376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafc4024930 con 0x7fafd4107e20 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.944+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd419c8b0 con 0x7fafd4107e20 2024-05-09T08:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.945+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd4063980 con 0x7fafd4107e20 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.946+0000 7fafc37fe700 1 -- 172.21.15.28:0/3693412376 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fafc4022020 con 0x7fafd4107e20 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.946+0000 7fafc37fe700 1 --2- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafbc039200 0x7fafbc03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.946+0000 7fafc37fe700 1 -- 172.21.15.28:0/3693412376 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fafc404d480 con 0x7fafd4107e20 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.946+0000 7fafd1d9b700 1 --2- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafbc039200 0x7fafbc03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.946+0000 7fafd1d9b700 1 --2- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafbc039200 0x7fafbc03b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fafc8006fd0 tx=0x7fafc8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:56.951+0000 7fafc37fe700 1 -- 172.21.15.28:0/3693412376 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fafc4010380 con 0x7fafd4107e20 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.155+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fafd419f3e0 con 0x7fafd4107e20 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.156+0000 7fafc37fe700 1 -- 172.21.15.28:0/3693412376 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fafc40212c0 con 0x7fafd4107e20 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafbc039200 msgr2=0x7fafbc03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 --2- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafbc039200 0x7fafbc03b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fafc8006fd0 tx=0x7fafc8006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 msgr2=0x7fafd419bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 --2- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd419bc80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fafc4000c00 tx=0x7fafc4011c10 comp rx=0 tx=0).stop 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 shutdown_connections 2024-05-09T08:16:57.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 --2- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafbc039200 0x7fafbc03b6c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:57.772 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 --2- 172.21.15.28:0/3693412376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafd4107e20 0x7fafd419bc80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:57.772 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 >> 172.21.15.28:0/3693412376 conn(0x7fafd4103400 msgr2=0x7fafd4103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:57.772 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 shutdown_connections 2024-05-09T08:16:57.772 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:57.158+0000 7fafd8c6f700 1 -- 172.21.15.28:0/3693412376 wait complete. 2024-05-09T08:16:59.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.288+0000 7f3e028ec700 1 Processor -- start 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.288+0000 7f3e028ec700 1 -- start start 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.288+0000 7f3e028ec700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc106ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.288+0000 7f3e028ec700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc07be30 con 0x7f3dfc1066b0 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.288+0000 7f3dfbfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc106ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.288+0000 7f3dfbfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc106ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39788/0 (socket says 172.21.15.28:39788) 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.288+0000 7f3dfbfff700 1 -- 172.21.15.28:0/507321911 learned_addr learned my addr 172.21.15.28:0/507321911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.289+0000 7f3dfbfff700 1 -- 172.21.15.28:0/507321911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc107010 con 0x7f3dfc1066b0 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.289+0000 7f3dfbfff700 1 --2- 172.21.15.28:0/507321911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc106ad0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0098a0 tx=0x7f3dec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fc8f96427d995d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.289+0000 7f3dfaffd700 1 -- 172.21.15.28:0/507321911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dec004160 con 0x7f3dfc1066b0 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.289+0000 7f3dfaffd700 1 -- 172.21.15.28:0/507321911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dec004300 con 0x7f3dfc1066b0 2024-05-09T08:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.289+0000 7f3dfaffd700 1 -- 172.21.15.28:0/507321911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dec011b20 con 0x7f3dfc1066b0 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.289+0000 7f3e028ec700 1 -- 172.21.15.28:0/507321911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 msgr2=0x7f3dfc106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.289+0000 7f3e028ec700 1 --2- 172.21.15.28:0/507321911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc106ad0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0098a0 tx=0x7f3dec009bb0 comp rx=0 tx=0).stop 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.290+0000 7f3e028ec700 1 -- 172.21.15.28:0/507321911 shutdown_connections 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.290+0000 7f3e028ec700 1 --2- 172.21.15.28:0/507321911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc106ad0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.290+0000 7f3e028ec700 1 -- 172.21.15.28:0/507321911 >> 172.21.15.28:0/507321911 conn(0x7f3dfc102210 msgr2=0x7f3dfc104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.290+0000 7f3e028ec700 1 -- 172.21.15.28:0/507321911 shutdown_connections 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.290+0000 7f3e028ec700 1 -- 172.21.15.28:0/507321911 wait complete. 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.290+0000 7f3e028ec700 1 Processor -- start 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.290+0000 7f3e028ec700 1 -- start start 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.291+0000 7f3e028ec700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.291+0000 7f3e028ec700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc1080e0 con 0x7f3dfc1066b0 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.291+0000 7f3dfbfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.291+0000 7f3dfbfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc19df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39798/0 (socket says 172.21.15.28:39798) 2024-05-09T08:16:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.291+0000 7f3dfbfff700 1 -- 172.21.15.28:0/3711963545 learned_addr learned my addr 172.21.15.28:0/3711963545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.291+0000 7f3dfbfff700 1 -- 172.21.15.28:0/3711963545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc19e450 con 0x7f3dfc1066b0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.291+0000 7f3dfbfff700 1 --2- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc19df10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0117e0 tx=0x7f3dec011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.291+0000 7f3df97fa700 1 -- 172.21.15.28:0/3711963545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dec003ba0 con 0x7f3dfc1066b0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.292+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dfc19e6e0 con 0x7f3dfc1066b0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.292+0000 7f3df97fa700 1 -- 172.21.15.28:0/3711963545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dec01a910 con 0x7f3dfc1066b0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.292+0000 7f3df97fa700 1 -- 172.21.15.28:0/3711963545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dec0248f0 con 0x7f3dfc1066b0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.292+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dfc19eb40 con 0x7f3dfc1066b0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.292+0000 7f3df97fa700 1 -- 172.21.15.28:0/3711963545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3dec022020 con 0x7f3dfc1066b0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.293+0000 7f3df97fa700 1 --2- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3de4039140 0x7f3de403b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.293+0000 7f3df97fa700 1 -- 172.21.15.28:0/3711963545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3dec04da90 con 0x7f3dfc1066b0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.293+0000 7f3dfb7fe700 1 --2- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3de4039140 0x7f3de403b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.293+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dfc063980 con 0x7f3dfc1066b0 2024-05-09T08:16:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.293+0000 7f3dfb7fe700 1 --2- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3de4039140 0x7f3de403b600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006fd0 tx=0x7f3df0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.298+0000 7f3df97fa700 1 -- 172.21.15.28:0/3711963545 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3dec020c10 con 0x7f3dfc1066b0 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.620+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f3dfc062f30 con 0x7f3dfc1066b0 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.622+0000 7f3df97fa700 1 -- 172.21.15.28:0/3711963545 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f3dec021d80 con 0x7f3dfc1066b0 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3de4039140 msgr2=0x7f3de403b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 --2- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3de4039140 0x7f3de403b600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006fd0 tx=0x7f3df0006e40 comp rx=0 tx=0).stop 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 msgr2=0x7f3dfc19df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 --2- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc19df10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0117e0 tx=0x7f3dec011ba0 comp rx=0 tx=0).stop 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 shutdown_connections 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 --2- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3de4039140 0x7f3de403b600 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 --2- 172.21.15.28:0/3711963545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dfc1066b0 0x7f3dfc19df10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 >> 172.21.15.28:0/3711963545 conn(0x7f3dfc102210 msgr2=0x7f3dfc194c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 shutdown_connections 2024-05-09T08:16:59.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-09T08:16:58.625+0000 7f3e028ec700 1 -- 172.21.15.28:0/3711963545 wait complete. 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-05-09T08:16:59.235 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:59.236 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-09T08:16:59.236 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:16:59.236 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-05-09T08:16:59.272 INFO:tasks.cephadm:Fetching config... 2024-05-09T08:16:59.273 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:16:59.273 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-09T08:16:59.290 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-09T08:16:59.290 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:16:59.290 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-09T08:16:59.347 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-09T08:16:59.347 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:16:59.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-09T08:16:59.414 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-09T08:16:59.414 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:16:59.414 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-09T08:16:59.471 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-09T08:16:59.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHPY9HBR6azUX3GTzSOsY5TfO2p12KnSrrhErh5jrZWMyTJCGy8XoQ9DhSdUM+cv9nyStF/0NPhPpwp3zXVx+XmhOmS/p56y/D0Liz1gLcoC3iW78OuolUXMObP9fz8IGEyqCVl1QBDLemGFf6B74GBB/c9uCSkYO7afQEc4tmX/d13YcdScguSWRv1eBIoLu5z9lb8fl/0ygGvGP0nZXqjj7ssQHaJxCpqAXk4TPC2so1FRiEV5C0mCazBxbPqDWGrioKnIGzwDi12k1sGv8EVB2vnaiBKUFS7srHdA8Qz73PJpl1x40q8vt8H6zYq1GmuHinPxyKCGsQFgAjIWmEz5foJPDljt08khW8cdTQ/08btK8FQYE0+uOoQhpZbYuwUkIaHZZoFblBD8L1aifLW5iasBmWpdtOFTFE54ZPSauodIcVRGtml8IlI5aHgaJrjFs8LVnfDPjUG4E3y0zqM5MFjjh0cFmLhQJU86vEQem18HXZO1d69aO5NdsZaZs= ceph-3891b212-0ddc-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T08:16:59.547 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHPY9HBR6azUX3GTzSOsY5TfO2p12KnSrrhErh5jrZWMyTJCGy8XoQ9DhSdUM+cv9nyStF/0NPhPpwp3zXVx+XmhOmS/p56y/D0Liz1gLcoC3iW78OuolUXMObP9fz8IGEyqCVl1QBDLemGFf6B74GBB/c9uCSkYO7afQEc4tmX/d13YcdScguSWRv1eBIoLu5z9lb8fl/0ygGvGP0nZXqjj7ssQHaJxCpqAXk4TPC2so1FRiEV5C0mCazBxbPqDWGrioKnIGzwDi12k1sGv8EVB2vnaiBKUFS7srHdA8Qz73PJpl1x40q8vt8H6zYq1GmuHinPxyKCGsQFgAjIWmEz5foJPDljt08khW8cdTQ/08btK8FQYE0+uOoQhpZbYuwUkIaHZZoFblBD8L1aifLW5iasBmWpdtOFTFE54ZPSauodIcVRGtml8IlI5aHgaJrjFs8LVnfDPjUG4E3y0zqM5MFjjh0cFmLhQJU86vEQem18HXZO1d69aO5NdsZaZs= ceph-3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:16:59.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHPY9HBR6azUX3GTzSOsY5TfO2p12KnSrrhErh5jrZWMyTJCGy8XoQ9DhSdUM+cv9nyStF/0NPhPpwp3zXVx+XmhOmS/p56y/D0Liz1gLcoC3iW78OuolUXMObP9fz8IGEyqCVl1QBDLemGFf6B74GBB/c9uCSkYO7afQEc4tmX/d13YcdScguSWRv1eBIoLu5z9lb8fl/0ygGvGP0nZXqjj7ssQHaJxCpqAXk4TPC2so1FRiEV5C0mCazBxbPqDWGrioKnIGzwDi12k1sGv8EVB2vnaiBKUFS7srHdA8Qz73PJpl1x40q8vt8H6zYq1GmuHinPxyKCGsQFgAjIWmEz5foJPDljt08khW8cdTQ/08btK8FQYE0+uOoQhpZbYuwUkIaHZZoFblBD8L1aifLW5iasBmWpdtOFTFE54ZPSauodIcVRGtml8IlI5aHgaJrjFs8LVnfDPjUG4E3y0zqM5MFjjh0cFmLhQJU86vEQem18HXZO1d69aO5NdsZaZs= ceph-3891b212-0ddc-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T08:16:59.599 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHPY9HBR6azUX3GTzSOsY5TfO2p12KnSrrhErh5jrZWMyTJCGy8XoQ9DhSdUM+cv9nyStF/0NPhPpwp3zXVx+XmhOmS/p56y/D0Liz1gLcoC3iW78OuolUXMObP9fz8IGEyqCVl1QBDLemGFf6B74GBB/c9uCSkYO7afQEc4tmX/d13YcdScguSWRv1eBIoLu5z9lb8fl/0ygGvGP0nZXqjj7ssQHaJxCpqAXk4TPC2so1FRiEV5C0mCazBxbPqDWGrioKnIGzwDi12k1sGv8EVB2vnaiBKUFS7srHdA8Qz73PJpl1x40q8vt8H6zYq1GmuHinPxyKCGsQFgAjIWmEz5foJPDljt08khW8cdTQ/08btK8FQYE0+uOoQhpZbYuwUkIaHZZoFblBD8L1aifLW5iasBmWpdtOFTFE54ZPSauodIcVRGtml8IlI5aHgaJrjFs8LVnfDPjUG4E3y0zqM5MFjjh0cFmLhQJU86vEQem18HXZO1d69aO5NdsZaZs= ceph-3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:16:59.611 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHPY9HBR6azUX3GTzSOsY5TfO2p12KnSrrhErh5jrZWMyTJCGy8XoQ9DhSdUM+cv9nyStF/0NPhPpwp3zXVx+XmhOmS/p56y/D0Liz1gLcoC3iW78OuolUXMObP9fz8IGEyqCVl1QBDLemGFf6B74GBB/c9uCSkYO7afQEc4tmX/d13YcdScguSWRv1eBIoLu5z9lb8fl/0ygGvGP0nZXqjj7ssQHaJxCpqAXk4TPC2so1FRiEV5C0mCazBxbPqDWGrioKnIGzwDi12k1sGv8EVB2vnaiBKUFS7srHdA8Qz73PJpl1x40q8vt8H6zYq1GmuHinPxyKCGsQFgAjIWmEz5foJPDljt08khW8cdTQ/08btK8FQYE0+uOoQhpZbYuwUkIaHZZoFblBD8L1aifLW5iasBmWpdtOFTFE54ZPSauodIcVRGtml8IlI5aHgaJrjFs8LVnfDPjUG4E3y0zqM5MFjjh0cFmLhQJU86vEQem18HXZO1d69aO5NdsZaZs= ceph-3891b212-0ddc-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T08:16:59.649 INFO:teuthology.orchestra.run.smithi184.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHPY9HBR6azUX3GTzSOsY5TfO2p12KnSrrhErh5jrZWMyTJCGy8XoQ9DhSdUM+cv9nyStF/0NPhPpwp3zXVx+XmhOmS/p56y/D0Liz1gLcoC3iW78OuolUXMObP9fz8IGEyqCVl1QBDLemGFf6B74GBB/c9uCSkYO7afQEc4tmX/d13YcdScguSWRv1eBIoLu5z9lb8fl/0ygGvGP0nZXqjj7ssQHaJxCpqAXk4TPC2so1FRiEV5C0mCazBxbPqDWGrioKnIGzwDi12k1sGv8EVB2vnaiBKUFS7srHdA8Qz73PJpl1x40q8vt8H6zYq1GmuHinPxyKCGsQFgAjIWmEz5foJPDljt08khW8cdTQ/08btK8FQYE0+uOoQhpZbYuwUkIaHZZoFblBD8L1aifLW5iasBmWpdtOFTFE54ZPSauodIcVRGtml8IlI5aHgaJrjFs8LVnfDPjUG4E3y0zqM5MFjjh0cFmLhQJU86vEQem18HXZO1d69aO5NdsZaZs= ceph-3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:16:59.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-09T08:16:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:16:59.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:16:59 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3711963545' entity='client.admin' 2024-05-09T08:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.011+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/495872748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 msgr2=0x7f2b581010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.011+0000 7f2b5c9b3700 1 --2- 172.21.15.28:0/495872748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 0x7f2b581010a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2b48009910 tx=0x7f2b48009c20 comp rx=0 tx=0).stop 2024-05-09T08:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.011+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/495872748 shutdown_connections 2024-05-09T08:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.011+0000 7f2b5c9b3700 1 --2- 172.21.15.28:0/495872748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 0x7f2b581010a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.011+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/495872748 >> 172.21.15.28:0/495872748 conn(0x7f2b580fc220 msgr2=0x7f2b580fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.011+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/495872748 shutdown_connections 2024-05-09T08:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.011+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/495872748 wait complete. 2024-05-09T08:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.012+0000 7f2b5c9b3700 1 Processor -- start 2024-05-09T08:17:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.012+0000 7f2b5c9b3700 1 -- start start 2024-05-09T08:17:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.012+0000 7f2b5c9b3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 0x7f2b580fe0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.012+0000 7f2b5c9b3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b58106d10 con 0x7f2b58100c80 2024-05-09T08:17:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.012+0000 7f2b5659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 0x7f2b580fe0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.012+0000 7f2b5659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 0x7f2b580fe0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39812/0 (socket says 172.21.15.28:39812) 2024-05-09T08:17:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.012+0000 7f2b5659c700 1 -- 172.21.15.28:0/544302391 learned_addr learned my addr 172.21.15.28:0/544302391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.012+0000 7f2b5659c700 1 -- 172.21.15.28:0/544302391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b580fe620 con 0x7f2b58100c80 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.012+0000 7f2b5659c700 1 --2- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 0x7f2b580fe0e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b48005850 tx=0x7f2b48005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.013+0000 7f2b4ffff700 1 -- 172.21.15.28:0/544302391 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b48026070 con 0x7f2b58100c80 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.013+0000 7f2b4ffff700 1 -- 172.21.15.28:0/544302391 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b480044a0 con 0x7f2b58100c80 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.013+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b580fc810 con 0x7f2b58100c80 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.013+0000 7f2b4ffff700 1 -- 172.21.15.28:0/544302391 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b48003ab0 con 0x7f2b58100c80 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.013+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b580fcd50 con 0x7f2b58100c80 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.014+0000 7f2b4ffff700 1 -- 172.21.15.28:0/544302391 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f2b48029070 con 0x7f2b58100c80 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.014+0000 7f2b4ffff700 1 --2- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2b440390f0 0x7f2b4403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.014+0000 7f2b4ffff700 1 -- 172.21.15.28:0/544302391 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2b4801e070 con 0x7f2b58100c80 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.014+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b58107490 con 0x7f2b58100c80 2024-05-09T08:17:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.014+0000 7f2b4dd9b700 1 --2- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2b440390f0 0x7f2b4403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:02.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.014+0000 7f2b4dd9b700 1 --2- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2b440390f0 0x7f2b4403b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.019+0000 7f2b4ffff700 1 -- 172.21.15.28:0/544302391 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b48030070 con 0x7f2b58100c80 2024-05-09T08:17:02.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.227+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f2b58190940 con 0x7f2b58100c80 2024-05-09T08:17:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.229+0000 7f2b4ffff700 1 -- 172.21.15.28:0/544302391 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f2b48013070 con 0x7f2b58100c80 2024-05-09T08:17:02.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2b440390f0 msgr2=0x7f2b4403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:02.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 --2- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2b440390f0 0x7f2b4403b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40006e40 comp rx=0 tx=0).stop 2024-05-09T08:17:02.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 msgr2=0x7f2b580fe0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:02.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 --2- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 0x7f2b580fe0e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b48005850 tx=0x7f2b48005030 comp rx=0 tx=0).stop 2024-05-09T08:17:02.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 shutdown_connections 2024-05-09T08:17:02.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 --2- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2b440390f0 0x7f2b4403b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:02.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 --2- 172.21.15.28:0/544302391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b58100c80 0x7f2b580fe0e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:02.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 >> 172.21.15.28:0/544302391 conn(0x7f2b580fc220 msgr2=0x7f2b58069b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:02.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 shutdown_connections 2024-05-09T08:17:02.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:02.232+0000 7f2b5c9b3700 1 -- 172.21.15.28:0/544302391 wait complete. 2024-05-09T08:17:02.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:02.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:02.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:17:02.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:02.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:02.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:02.829 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-09T08:17:02.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-09T08:17:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:17:03.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:03 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/544302391' entity='client.admin' 2024-05-09T08:17:03.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:03.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.593+0000 7fe969b28700 1 -- 172.21.15.28:0/977617497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 msgr2=0x7fe9641021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.593+0000 7fe969b28700 1 --2- 172.21.15.28:0/977617497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 0x7fe9641021e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe94c009910 tx=0x7fe94c009c20 comp rx=0 tx=0).stop 2024-05-09T08:17:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.593+0000 7fe969b28700 1 -- 172.21.15.28:0/977617497 shutdown_connections 2024-05-09T08:17:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.593+0000 7fe969b28700 1 --2- 172.21.15.28:0/977617497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 0x7fe9641021e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.593+0000 7fe969b28700 1 -- 172.21.15.28:0/977617497 >> 172.21.15.28:0/977617497 conn(0x7fe9640fd320 msgr2=0x7fe9640ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.593+0000 7fe969b28700 1 -- 172.21.15.28:0/977617497 shutdown_connections 2024-05-09T08:17:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.593+0000 7fe969b28700 1 -- 172.21.15.28:0/977617497 wait complete. 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.593+0000 7fe969b28700 1 Processor -- start 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.594+0000 7fe969b28700 1 -- start start 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.594+0000 7fe969b28700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 0x7fe9641974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.594+0000 7fe969b28700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe964107e50 con 0x7fe964101dc0 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.594+0000 7fe9637fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 0x7fe9641974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.594+0000 7fe9637fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 0x7fe9641974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39822/0 (socket says 172.21.15.28:39822) 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.594+0000 7fe9637fe700 1 -- 172.21.15.28:0/4254847464 learned_addr learned my addr 172.21.15.28:0/4254847464 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.594+0000 7fe9637fe700 1 -- 172.21.15.28:0/4254847464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe964197a00 con 0x7fe964101dc0 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.594+0000 7fe9637fe700 1 --2- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 0x7fe9641974c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe94c006010 tx=0x7fe94c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.594+0000 7fe960ff9700 1 -- 172.21.15.28:0/4254847464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe94c027070 con 0x7fe964101dc0 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.595+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe964197c90 con 0x7fe964101dc0 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.595+0000 7fe960ff9700 1 -- 172.21.15.28:0/4254847464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe94c00be40 con 0x7fe964101dc0 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.595+0000 7fe960ff9700 1 -- 172.21.15.28:0/4254847464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe94c00f460 con 0x7fe964101dc0 2024-05-09T08:17:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.595+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe96419a930 con 0x7fe964101dc0 2024-05-09T08:17:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.595+0000 7fe960ff9700 1 -- 172.21.15.28:0/4254847464 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fe94c02a070 con 0x7fe964101dc0 2024-05-09T08:17:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.596+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9641085d0 con 0x7fe964101dc0 2024-05-09T08:17:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.596+0000 7fe960ff9700 1 --2- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe9500390f0 0x7fe95003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.596+0000 7fe960ff9700 1 -- 172.21.15.28:0/4254847464 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe94c01e030 con 0x7fe964101dc0 2024-05-09T08:17:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.596+0000 7fe962ffd700 1 --2- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe9500390f0 0x7fe95003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.596+0000 7fe962ffd700 1 --2- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe9500390f0 0x7fe95003b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe954006fd0 tx=0x7fe954006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.601+0000 7fe960ff9700 1 -- 172.21.15.28:0/4254847464 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe94c013070 con 0x7fe964101dc0 2024-05-09T08:17:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.815+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fe96402dc90 con 0x7fe9500390f0 2024-05-09T08:17:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.818+0000 7fe960ff9700 1 -- 172.21.15.28:0/4254847464 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe96402dc90 con 0x7fe9500390f0 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe9500390f0 msgr2=0x7fe95003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 --2- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe9500390f0 0x7fe95003b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe954006fd0 tx=0x7fe954006e40 comp rx=0 tx=0).stop 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 msgr2=0x7fe9641974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 --2- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 0x7fe9641974c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe94c006010 tx=0x7fe94c005050 comp rx=0 tx=0).stop 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 shutdown_connections 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 --2- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe9500390f0 0x7fe95003b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 --2- 172.21.15.28:0/4254847464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe964101dc0 0x7fe9641974c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 >> 172.21.15.28:0/4254847464 conn(0x7fe9640fd320 msgr2=0x7fe9640fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 shutdown_connections 2024-05-09T08:17:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:04.820+0000 7fe969b28700 1 -- 172.21.15.28:0/4254847464 wait complete. 2024-05-09T08:17:05.495 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi165 2024-05-09T08:17:05.495 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:17:05.495 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.conf 2024-05-09T08:17:05.515 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:17:05.515 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:05.572 INFO:tasks.cephadm:Adding host smithi165 to orchestrator... 2024-05-09T08:17:05.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch host add smithi165 2024-05-09T08:17:05.827 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:17:06.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:05 smithi028 ceph-mon[30119]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:17:06.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:05 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:06.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:05 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:06.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:05 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:06.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:05 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:06.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:05 smithi028 ceph-mon[30119]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-09T08:17:06.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:05 smithi028 ceph-mon[30119]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:06.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:05 smithi028 ceph-mon[30119]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:07.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:06 smithi028 ceph-mon[30119]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:17:07.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:06 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:07.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:06 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:07.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:06 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:07.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.296+0000 7f7677464700 1 -- 172.21.15.28:0/4227181125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 msgr2=0x7f7670103370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:07.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.296+0000 7f7677464700 1 --2- 172.21.15.28:0/4227181125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 0x7f7670103370 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f766c009910 tx=0x7f766c009c20 comp rx=0 tx=0).stop 2024-05-09T08:17:07.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.296+0000 7f7677464700 1 -- 172.21.15.28:0/4227181125 shutdown_connections 2024-05-09T08:17:07.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.296+0000 7f7677464700 1 --2- 172.21.15.28:0/4227181125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 0x7f7670103370 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:07.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.296+0000 7f7677464700 1 -- 172.21.15.28:0/4227181125 >> 172.21.15.28:0/4227181125 conn(0x7f76700fc780 msgr2=0x7f76700feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:07.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.296+0000 7f7677464700 1 -- 172.21.15.28:0/4227181125 shutdown_connections 2024-05-09T08:17:07.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.296+0000 7f7677464700 1 -- 172.21.15.28:0/4227181125 wait complete. 2024-05-09T08:17:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.297+0000 7f7677464700 1 Processor -- start 2024-05-09T08:17:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.297+0000 7f7677464700 1 -- start start 2024-05-09T08:17:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f7677464700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 0x7f767019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f7677464700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f767010c410 con 0x7f7670102f90 2024-05-09T08:17:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f7675200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 0x7f767019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f7675200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 0x7f767019d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38900/0 (socket says 172.21.15.28:38900) 2024-05-09T08:17:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f7675200700 1 -- 172.21.15.28:0/1857840001 learned_addr learned my addr 172.21.15.28:0/1857840001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:17:07.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f7675200700 1 -- 172.21.15.28:0/1857840001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767019dbb0 con 0x7f7670102f90 2024-05-09T08:17:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f7675200700 1 --2- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 0x7f767019d670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f766c006010 tx=0x7f766c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f76667fc700 1 -- 172.21.15.28:0/1857840001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f766c026070 con 0x7f7670102f90 2024-05-09T08:17:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f76667fc700 1 -- 172.21.15.28:0/1857840001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f766c00bc30 con 0x7f7670102f90 2024-05-09T08:17:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f767019de40 con 0x7f7670102f90 2024-05-09T08:17:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f76667fc700 1 -- 172.21.15.28:0/1857840001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f766c0039a0 con 0x7f7670102f90 2024-05-09T08:17:07.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.298+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7670197740 con 0x7f7670102f90 2024-05-09T08:17:07.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.299+0000 7f76667fc700 1 -- 172.21.15.28:0/1857840001 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f766c029070 con 0x7f7670102f90 2024-05-09T08:17:07.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.299+0000 7f76667fc700 1 --2- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f765c039150 0x7f765c03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:07.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.299+0000 7f76667fc700 1 -- 172.21.15.28:0/1857840001 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f766c01e070 con 0x7f7670102f90 2024-05-09T08:17:07.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.299+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f767006aa10 con 0x7f7670102f90 2024-05-09T08:17:07.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.300+0000 7f76749ff700 1 --2- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f765c039150 0x7f765c03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:07.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.300+0000 7f76749ff700 1 --2- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f765c039150 0x7f765c03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7660006fd0 tx=0x7f7660006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.305+0000 7f76667fc700 1 -- 172.21.15.28:0/1857840001 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f766c030070 con 0x7f7670102f90 2024-05-09T08:17:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:07.515+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}) -- 0x7f7670063980 con 0x7f765c039150 2024-05-09T08:17:08.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:08 smithi028 ceph-mon[30119]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:17:08.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:08 smithi028 ceph-mon[30119]: Deploying cephadm binary to smithi165 2024-05-09T08:17:10.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:10 smithi028 ceph-mon[30119]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:11.935 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi165' with addr '172.21.15.165' 2024-05-09T08:17:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.933+0000 7f76667fc700 1 -- 172.21.15.28:0/1857840001 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f7670063980 con 0x7f765c039150 2024-05-09T08:17:11.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f765c039150 msgr2=0x7f765c03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 --2- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f765c039150 0x7f765c03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7660006fd0 tx=0x7f7660006e40 comp rx=0 tx=0).stop 2024-05-09T08:17:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 msgr2=0x7f767019d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 --2- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 0x7f767019d670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f766c006010 tx=0x7f766c005030 comp rx=0 tx=0).stop 2024-05-09T08:17:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 shutdown_connections 2024-05-09T08:17:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 --2- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f765c039150 0x7f765c03b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 --2- 172.21.15.28:0/1857840001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670102f90 0x7f767019d670 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 >> 172.21.15.28:0/1857840001 conn(0x7f76700fc780 msgr2=0x7f7670103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 shutdown_connections 2024-05-09T08:17:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:11.936+0000 7f7677464700 1 -- 172.21.15.28:0/1857840001 wait complete. 2024-05-09T08:17:12.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-09T08:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:17:13.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:12 smithi028 ceph-mon[30119]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:13.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:13.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:12 smithi028 ceph-mon[30119]: Added host smithi165 2024-05-09T08:17:13.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:13.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.141+0000 7fe6af890700 1 -- 172.21.15.28:0/2305738551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 msgr2=0x7fe6a8102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.141+0000 7fe6af890700 1 --2- 172.21.15.28:0/2305738551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 0x7fe6a8102370 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009910 tx=0x7fe6a4009c20 comp rx=0 tx=0).stop 2024-05-09T08:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.141+0000 7fe6af890700 1 -- 172.21.15.28:0/2305738551 shutdown_connections 2024-05-09T08:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.141+0000 7fe6af890700 1 --2- 172.21.15.28:0/2305738551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 0x7fe6a8102370 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.141+0000 7fe6af890700 1 -- 172.21.15.28:0/2305738551 >> 172.21.15.28:0/2305738551 conn(0x7fe6a80fd860 msgr2=0x7fe6a80ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.142+0000 7fe6af890700 1 -- 172.21.15.28:0/2305738551 shutdown_connections 2024-05-09T08:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.142+0000 7fe6af890700 1 -- 172.21.15.28:0/2305738551 wait complete. 2024-05-09T08:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.142+0000 7fe6af890700 1 Processor -- start 2024-05-09T08:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.142+0000 7fe6af890700 1 -- start start 2024-05-09T08:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.142+0000 7fe6af890700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 0x7fe6a8074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.142+0000 7fe6af890700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a810c360 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe6ad62c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 0x7fe6a8074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe6ad62c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 0x7fe6a8074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38914/0 (socket says 172.21.15.28:38914) 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe6ad62c700 1 -- 172.21.15.28:0/412815340 learned_addr learned my addr 172.21.15.28:0/412815340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe6ad62c700 1 -- 172.21.15.28:0/412815340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a8076510 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe6ad62c700 1 --2- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 0x7fe6a8074620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4005850 tx=0x7fe6a4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe69e7fc700 1 -- 172.21.15.28:0/412815340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6a4026070 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a8074bc0 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe69e7fc700 1 -- 172.21.15.28:0/412815340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a400bc30 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe69e7fc700 1 -- 172.21.15.28:0/412815340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6a4003a20 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.143+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a8075080 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.144+0000 7fe69e7fc700 1 -- 172.21.15.28:0/412815340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fe6a4029070 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.144+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6a8103470 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.144+0000 7fe69e7fc700 1 --2- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6940390f0 0x7fe69403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.144+0000 7fe69e7fc700 1 -- 172.21.15.28:0/412815340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe6a401e070 con 0x7fe6a8101f90 2024-05-09T08:17:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.144+0000 7fe6ace2b700 1 --2- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6940390f0 0x7fe69403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.145+0000 7fe6ace2b700 1 --2- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6940390f0 0x7fe69403b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe698006fd0 tx=0x7fe698006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.149+0000 7fe69e7fc700 1 -- 172.21.15.28:0/412815340 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe6a4013070 con 0x7fe6a8101f90 2024-05-09T08:17:14.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.362+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe6a8075b50 con 0x7fe6940390f0 2024-05-09T08:17:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.363+0000 7fe69e7fc700 1 -- 172.21.15.28:0/412815340 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fe6a8075b50 con 0x7fe6940390f0 2024-05-09T08:17:14.366 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:17:14.366 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}] 2024-05-09T08:17:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.365+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6940390f0 msgr2=0x7fe69403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.365+0000 7fe6af890700 1 --2- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6940390f0 0x7fe69403b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe698006fd0 tx=0x7fe698006e40 comp rx=0 tx=0).stop 2024-05-09T08:17:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.365+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 msgr2=0x7fe6a8074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.365+0000 7fe6af890700 1 --2- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 0x7fe6a8074620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4005850 tx=0x7fe6a4005030 comp rx=0 tx=0).stop 2024-05-09T08:17:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.365+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 shutdown_connections 2024-05-09T08:17:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.365+0000 7fe6af890700 1 --2- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6940390f0 0x7fe69403b5b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.365+0000 7fe6af890700 1 --2- 172.21.15.28:0/412815340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6a8101f90 0x7fe6a8074620 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.366+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 >> 172.21.15.28:0/412815340 conn(0x7fe6a80fd860 msgr2=0x7fe6a8104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:14.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.366+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 shutdown_connections 2024-05-09T08:17:14.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:14.366+0000 7fe6af890700 1 -- 172.21.15.28:0/412815340 wait complete. 2024-05-09T08:17:14.985 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi184 2024-05-09T08:17:14.985 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:17:14.985 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.conf 2024-05-09T08:17:15.004 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:17:15.004 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:15.063 INFO:tasks.cephadm:Adding host smithi184 to orchestrator... 2024-05-09T08:17:15.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch host add smithi184 2024-05-09T08:17:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:17:15.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:15 smithi028 ceph-mon[30119]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:15.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:15 smithi028 ceph-mon[30119]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:17:15.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.820+0000 7f67ce255700 1 -- 172.21.15.28:0/1932571099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 msgr2=0x7f67c8102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.820+0000 7f67ce255700 1 --2- 172.21.15.28:0/1932571099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 0x7f67c8102370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f67b800b0a0 tx=0x7f67b800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:17:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.820+0000 7f67ce255700 1 -- 172.21.15.28:0/1932571099 shutdown_connections 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.820+0000 7f67ce255700 1 --2- 172.21.15.28:0/1932571099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 0x7f67c8102370 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.820+0000 7f67ce255700 1 -- 172.21.15.28:0/1932571099 >> 172.21.15.28:0/1932571099 conn(0x7f67c80fd860 msgr2=0x7f67c80ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.820+0000 7f67ce255700 1 -- 172.21.15.28:0/1932571099 shutdown_connections 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.820+0000 7f67ce255700 1 -- 172.21.15.28:0/1932571099 wait complete. 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.821+0000 7f67ce255700 1 Processor -- start 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.821+0000 7f67ce255700 1 -- start start 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.821+0000 7f67ce255700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 0x7f67c8074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.821+0000 7f67ce255700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67c810c360 con 0x7f67c8101f90 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.821+0000 7f67c77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 0x7f67c8074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.821+0000 7f67c77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 0x7f67c8074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44522/0 (socket says 172.21.15.28:44522) 2024-05-09T08:17:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.821+0000 7f67c77fe700 1 -- 172.21.15.28:0/2987305497 learned_addr learned my addr 172.21.15.28:0/2987305497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:17:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.821+0000 7f67c77fe700 1 -- 172.21.15.28:0/2987305497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c8076510 con 0x7f67c8101f90 2024-05-09T08:17:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.822+0000 7f67c77fe700 1 --2- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 0x7f67c8074620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f67b8015040 tx=0x7f67b8003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.822+0000 7f67c4ff9700 1 -- 172.21.15.28:0/2987305497 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67b8017040 con 0x7f67c8101f90 2024-05-09T08:17:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.822+0000 7f67c4ff9700 1 -- 172.21.15.28:0/2987305497 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67b8022070 con 0x7f67c8101f90 2024-05-09T08:17:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.822+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c8074bc0 con 0x7f67c8101f90 2024-05-09T08:17:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.822+0000 7f67c4ff9700 1 -- 172.21.15.28:0/2987305497 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67b801d5d0 con 0x7f67c8101f90 2024-05-09T08:17:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.822+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c8075080 con 0x7f67c8101f90 2024-05-09T08:17:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.823+0000 7f67c4ff9700 1 -- 172.21.15.28:0/2987305497 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f67b801d8d0 con 0x7f67c8101f90 2024-05-09T08:17:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.823+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67c8103470 con 0x7f67c8101f90 2024-05-09T08:17:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.823+0000 7f67c4ff9700 1 --2- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67b00390f0 0x7f67b003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.823+0000 7f67c4ff9700 1 -- 172.21.15.28:0/2987305497 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f67b800f050 con 0x7f67c8101f90 2024-05-09T08:17:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.823+0000 7f67c6ffd700 1 --2- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67b00390f0 0x7f67b003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.823+0000 7f67c6ffd700 1 --2- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67b00390f0 0x7f67b003b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f67bc006fd0 tx=0x7f67bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:16.828+0000 7f67c4ff9700 1 -- 172.21.15.28:0/2987305497 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f67b8007e20 con 0x7f67c8101f90 2024-05-09T08:17:17.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:17.039+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}) -- 0x7f67c8075bc0 con 0x7f67b00390f0 2024-05-09T08:17:17.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:17 smithi028 ceph-mon[30119]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:18.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:18 smithi028 ceph-mon[30119]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:17:18.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:18.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:18.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:18.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:18.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:17:18.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:18.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:19.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:19 smithi028 ceph-mon[30119]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:19.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:19 smithi028 ceph-mon[30119]: Deploying cephadm binary to smithi184 2024-05-09T08:17:19.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:19 smithi028 ceph-mon[30119]: Updating smithi165:/etc/ceph/ceph.conf 2024-05-09T08:17:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:19 smithi028 ceph-mon[30119]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:19.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:19 smithi028 ceph-mon[30119]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.254+0000 7f67c4ff9700 1 -- 172.21.15.28:0/2987305497 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f67c8075bc0 con 0x7f67b00390f0 2024-05-09T08:17:21.258 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi184' with addr '172.21.15.184' 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.256+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67b00390f0 msgr2=0x7f67b003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.256+0000 7f67ce255700 1 --2- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67b00390f0 0x7f67b003b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f67bc006fd0 tx=0x7f67bc006e40 comp rx=0 tx=0).stop 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.256+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 msgr2=0x7f67c8074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.256+0000 7f67ce255700 1 --2- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 0x7f67c8074620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f67b8015040 tx=0x7f67b8003d50 comp rx=0 tx=0).stop 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.256+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 shutdown_connections 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.256+0000 7f67ce255700 1 --2- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67b00390f0 0x7f67b003b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.256+0000 7f67ce255700 1 --2- 172.21.15.28:0/2987305497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8101f90 0x7f67c8074620 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.257+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 >> 172.21.15.28:0/2987305497 conn(0x7f67c80fd860 msgr2=0x7f67c8103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.257+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 shutdown_connections 2024-05-09T08:17:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:21.257+0000 7f67ce255700 1 -- 172.21.15.28:0/2987305497 wait complete. 2024-05-09T08:17:21.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:21 smithi028 ceph-mon[30119]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:21.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:21 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:21.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:21 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:21.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:21 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:21.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-09T08:17:22.037 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:17:22.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:22 smithi028 ceph-mon[30119]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:17:22.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:22 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:22.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:22 smithi028 ceph-mon[30119]: Added host smithi184 2024-05-09T08:17:22.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:22 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:22.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:22 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:23.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:23 smithi028 ceph-mon[30119]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:23.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.516+0000 7fa75e90f700 1 -- 172.21.15.28:0/4067823939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 msgr2=0x7fa758102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.516+0000 7fa75e90f700 1 --2- 172.21.15.28:0/4067823939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 0x7fa758102370 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa748009910 tx=0x7fa748009c20 comp rx=0 tx=0).stop 2024-05-09T08:17:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.516+0000 7fa75e90f700 1 -- 172.21.15.28:0/4067823939 shutdown_connections 2024-05-09T08:17:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.516+0000 7fa75e90f700 1 --2- 172.21.15.28:0/4067823939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 0x7fa758102370 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.516+0000 7fa75e90f700 1 -- 172.21.15.28:0/4067823939 >> 172.21.15.28:0/4067823939 conn(0x7fa7580fd860 msgr2=0x7fa7580ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.516+0000 7fa75e90f700 1 -- 172.21.15.28:0/4067823939 shutdown_connections 2024-05-09T08:17:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.516+0000 7fa75e90f700 1 -- 172.21.15.28:0/4067823939 wait complete. 2024-05-09T08:17:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.517+0000 7fa75e90f700 1 Processor -- start 2024-05-09T08:17:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.517+0000 7fa75e90f700 1 -- start start 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.517+0000 7fa75e90f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 0x7fa75819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.517+0000 7fa75e90f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa75810c360 con 0x7fa758101f90 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa757fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 0x7fa75819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa757fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 0x7fa75819d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44528/0 (socket says 172.21.15.28:44528) 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa757fff700 1 -- 172.21.15.28:0/2181675053 learned_addr learned my addr 172.21.15.28:0/2181675053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa757fff700 1 -- 172.21.15.28:0/2181675053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75819dba0 con 0x7fa758101f90 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa757fff700 1 --2- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 0x7fa75819d660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa748005850 tx=0x7fa748005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa7557fa700 1 -- 172.21.15.28:0/2181675053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa748026070 con 0x7fa758101f90 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa7557fa700 1 -- 172.21.15.28:0/2181675053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa74800bbd0 con 0x7fa758101f90 2024-05-09T08:17:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75819de90 con 0x7fa758101f90 2024-05-09T08:17:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa7557fa700 1 -- 172.21.15.28:0/2181675053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa748003970 con 0x7fa758101f90 2024-05-09T08:17:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.518+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa758197900 con 0x7fa758101f90 2024-05-09T08:17:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.519+0000 7fa7557fa700 1 -- 172.21.15.28:0/2181675053 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fa748029070 con 0x7fa758101f90 2024-05-09T08:17:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.519+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa758103470 con 0x7fa758101f90 2024-05-09T08:17:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.519+0000 7fa7557fa700 1 --2- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa740039150 0x7fa74003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.519+0000 7fa7557fa700 1 -- 172.21.15.28:0/2181675053 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa74801e070 con 0x7fa758101f90 2024-05-09T08:17:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.519+0000 7fa7577fe700 1 --2- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa740039150 0x7fa74003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.520+0000 7fa7577fe700 1 --2- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa740039150 0x7fa74003b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa74c006fd0 tx=0x7fa74c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.524+0000 7fa7557fa700 1 -- 172.21.15.28:0/2181675053 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa748013070 con 0x7fa758101f90 2024-05-09T08:17:23.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.736+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa758198c60 con 0x7fa740039150 2024-05-09T08:17:23.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.737+0000 7fa7557fa700 1 -- 172.21.15.28:0/2181675053 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fa758198c60 con 0x7fa740039150 2024-05-09T08:17:23.740 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:17:23.740 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}, {"addr": "172.21.15.184", "hostname": "smithi184", "labels": [], "status": ""}] 2024-05-09T08:17:23.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa740039150 msgr2=0x7fa74003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 --2- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa740039150 0x7fa74003b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa74c006fd0 tx=0x7fa74c006e40 comp rx=0 tx=0).stop 2024-05-09T08:17:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 msgr2=0x7fa75819d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 --2- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 0x7fa75819d660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa748005850 tx=0x7fa748005030 comp rx=0 tx=0).stop 2024-05-09T08:17:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 shutdown_connections 2024-05-09T08:17:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 --2- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa740039150 0x7fa74003b610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 --2- 172.21.15.28:0/2181675053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa758101f90 0x7fa75819d660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 >> 172.21.15.28:0/2181675053 conn(0x7fa7580fd860 msgr2=0x7fa758104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 shutdown_connections 2024-05-09T08:17:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:23.739+0000 7fa75e90f700 1 -- 172.21.15.28:0/2181675053 wait complete. 2024-05-09T08:17:24.404 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-09T08:17:24.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd crush tunables default 2024-05-09T08:17:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:17:25.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:24 smithi028 ceph-mon[30119]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:25.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:24 smithi028 ceph-mon[30119]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:17:25.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:24 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:26.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.271+0000 7f9c023ad700 1 -- 172.21.15.28:0/1929805274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 msgr2=0x7f9bfc1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:26.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.271+0000 7f9c023ad700 1 --2- 172.21.15.28:0/1929805274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 0x7f9bfc1012b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009910 tx=0x7f9bec009c20 comp rx=0 tx=0).stop 2024-05-09T08:17:26.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.271+0000 7f9c023ad700 1 -- 172.21.15.28:0/1929805274 shutdown_connections 2024-05-09T08:17:26.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.271+0000 7f9c023ad700 1 --2- 172.21.15.28:0/1929805274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 0x7f9bfc1012b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:26.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.271+0000 7f9c023ad700 1 -- 172.21.15.28:0/1929805274 >> 172.21.15.28:0/1929805274 conn(0x7f9bfc076780 msgr2=0x7f9bfc076b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:26.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.271+0000 7f9c023ad700 1 -- 172.21.15.28:0/1929805274 shutdown_connections 2024-05-09T08:17:26.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.271+0000 7f9c023ad700 1 -- 172.21.15.28:0/1929805274 wait complete. 2024-05-09T08:17:26.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.272+0000 7f9c023ad700 1 Processor -- start 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.272+0000 7f9c023ad700 1 -- start start 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.272+0000 7f9c023ad700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 0x7f9bfc102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.272+0000 7f9c023ad700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9bfc10b250 con 0x7f9bfc100ed0 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.272+0000 7f9bfbfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 0x7f9bfc102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.272+0000 7f9bfbfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 0x7f9bfc102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37596/0 (socket says 172.21.15.28:37596) 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.272+0000 7f9bfbfff700 1 -- 172.21.15.28:0/645989816 learned_addr learned my addr 172.21.15.28:0/645989816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.273+0000 7f9bfbfff700 1 -- 172.21.15.28:0/645989816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bfc104810 con 0x7f9bfc100ed0 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.273+0000 7f9bfbfff700 1 --2- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 0x7f9bfc102920 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9bec005850 tx=0x7f9bec005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.273+0000 7f9bf9ffb700 1 -- 172.21.15.28:0/645989816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bec026070 con 0x7f9bfc100ed0 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.273+0000 7f9bf9ffb700 1 -- 172.21.15.28:0/645989816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bec00bc30 con 0x7f9bfc100ed0 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.273+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bfc102ec0 con 0x7f9bfc100ed0 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.273+0000 7f9bf9ffb700 1 -- 172.21.15.28:0/645989816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bec003a20 con 0x7f9bfc100ed0 2024-05-09T08:17:26.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.273+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bfc103380 con 0x7f9bfc100ed0 2024-05-09T08:17:26.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.274+0000 7f9bf9ffb700 1 -- 172.21.15.28:0/645989816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9bec029070 con 0x7f9bfc100ed0 2024-05-09T08:17:26.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.274+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bfc102360 con 0x7f9bfc100ed0 2024-05-09T08:17:26.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.274+0000 7f9bf9ffb700 1 --2- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9be8039150 0x7f9be803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:26.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.274+0000 7f9bf9ffb700 1 -- 172.21.15.28:0/645989816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9bec01e070 con 0x7f9bfc100ed0 2024-05-09T08:17:26.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.274+0000 7f9bf3fff700 1 --2- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9be8039150 0x7f9be803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:26.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.275+0000 7f9bf3fff700 1 --2- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9be8039150 0x7f9be803b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9be4006fd0 tx=0x7f9be4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:26.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.279+0000 7f9bf9ffb700 1 -- 172.21.15.28:0/645989816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9bec013070 con 0x7f9bfc100ed0 2024-05-09T08:17:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.492+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f9bfc103e50 con 0x7f9bfc100ed0 2024-05-09T08:17:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.851+0000 7f9bf9ffb700 1 -- 172.21.15.28:0/645989816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f9bec030020 con 0x7f9bfc100ed0 2024-05-09T08:17:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9be8039150 msgr2=0x7f9be803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 --2- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9be8039150 0x7f9be803b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9be4006fd0 tx=0x7f9be4006e40 comp rx=0 tx=0).stop 2024-05-09T08:17:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 msgr2=0x7f9bfc102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 --2- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 0x7f9bfc102920 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9bec005850 tx=0x7f9bec005030 comp rx=0 tx=0).stop 2024-05-09T08:17:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 shutdown_connections 2024-05-09T08:17:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 --2- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9be8039150 0x7f9be803b610 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 --2- 172.21.15.28:0/645989816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bfc100ed0 0x7f9bfc102920 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 >> 172.21.15.28:0/645989816 conn(0x7f9bfc076780 msgr2=0x7f9bfc069740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 shutdown_connections 2024-05-09T08:17:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:26.854+0000 7f9c023ad700 1 -- 172.21.15.28:0/645989816 wait complete. 2024-05-09T08:17:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-05-09T08:17:27.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:26 smithi028 ceph-mon[30119]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:27.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:26 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/645989816' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-09T08:17:27.473 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-05-09T08:17:27.473 INFO:tasks.cephadm:Adding mon.b on smithi165 2024-05-09T08:17:27.474 INFO:tasks.cephadm:Adding mon.c on smithi184 2024-05-09T08:17:27.474 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi165:172.21.15.165=b;smithi184:172.21.15.184=c' 2024-05-09T08:17:27.716 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:28.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/645989816' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-09T08:17:28.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:17:28.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:28.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:28.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:28.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:28.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:17:28.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:28.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:28.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: Updating smithi184:/etc/ceph/ceph.conf 2024-05-09T08:17:28.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:27 smithi028 ceph-mon[30119]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:29.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.049+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/883960310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 msgr2=0x7f3f54101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.049+0000 7f3f5a4b0700 1 --2- 172.21.15.184:0/883960310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 0x7f3f54101270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009910 tx=0x7f3f48009c20 comp rx=0 tx=0).stop 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.049+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/883960310 shutdown_connections 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.049+0000 7f3f5a4b0700 1 --2- 172.21.15.184:0/883960310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 0x7f3f54101270 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.049+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/883960310 >> 172.21.15.184:0/883960310 conn(0x7f3f540fc740 msgr2=0x7f3f540feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.049+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/883960310 shutdown_connections 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.049+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/883960310 wait complete. 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.049+0000 7f3f5a4b0700 1 Processor -- start 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.049+0000 7f3f5a4b0700 1 -- start start 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.050+0000 7f3f5a4b0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 0x7f3f54102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.050+0000 7f3f5a4b0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5410b260 con 0x7f3f54100e90 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.050+0000 7f3f53fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 0x7f3f54102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:29.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.050+0000 7f3f53fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 0x7f3f54102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.184:34944/0 (socket says 172.21.15.184:34944) 2024-05-09T08:17:29.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.050+0000 7f3f53fff700 1 -- 172.21.15.184:0/1172095652 learned_addr learned my addr 172.21.15.184:0/1172095652 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-05-09T08:17:29.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.050+0000 7f3f53fff700 1 -- 172.21.15.184:0/1172095652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f54104820 con 0x7f3f54100e90 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.050+0000 7f3f53fff700 1 --2- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 0x7f3f54102930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f48005850 tx=0x7f3f48005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.051+0000 7f3f517fa700 1 -- 172.21.15.184:0/1172095652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f48026070 con 0x7f3f54100e90 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.051+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f54102ed0 con 0x7f3f54100e90 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.051+0000 7f3f517fa700 1 -- 172.21.15.184:0/1172095652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f4800bc30 con 0x7f3f54100e90 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.051+0000 7f3f517fa700 1 -- 172.21.15.184:0/1172095652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f48003a20 con 0x7f3f54100e90 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.051+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f54103390 con 0x7f3f54100e90 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.051+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f54102370 con 0x7f3f54100e90 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.052+0000 7f3f517fa700 1 -- 172.21.15.184:0/1172095652 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3f48029070 con 0x7f3f54100e90 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.052+0000 7f3f517fa700 1 --2- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3f3c0390f0 0x7f3f3c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.052+0000 7f3f517fa700 1 -- 172.21.15.184:0/1172095652 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3f4801e070 con 0x7f3f54100e90 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.052+0000 7f3f537fe700 1 --2- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3f3c0390f0 0x7f3f3c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:29.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.052+0000 7f3f537fe700 1 --2- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3f3c0390f0 0x7f3f3c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3f44006fd0 tx=0x7f3f44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:29.060 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.058+0000 7f3f517fa700 1 -- 172.21.15.184:0/1172095652 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3f48013070 con 0x7f3f54100e90 2024-05-09T08:17:29.267 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.263+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi165:172.21.15.165=b;smithi184:172.21.15.184=c", "target": ["mon-mgr", ""]}) -- 0x7f3f54103620 con 0x7f3f3c0390f0 2024-05-09T08:17:29.272 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.269+0000 7f3f517fa700 1 -- 172.21.15.184:0/1172095652 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3f54103620 con 0x7f3f3c0390f0 2024-05-09T08:17:29.272 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mon update... 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3f3c0390f0 msgr2=0x7f3f3c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 --2- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3f3c0390f0 0x7f3f3c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3f44006fd0 tx=0x7f3f44006e40 comp rx=0 tx=0).stop 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 msgr2=0x7f3f54102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 --2- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 0x7f3f54102930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f48005850 tx=0x7f3f48005030 comp rx=0 tx=0).stop 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 shutdown_connections 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 --2- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3f3c0390f0 0x7f3f3c03b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 --2- 172.21.15.184:0/1172095652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54100e90 0x7f3f54102930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 >> 172.21.15.184:0/1172095652 conn(0x7f3f540fc740 msgr2=0x7f3f5406a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 shutdown_connections 2024-05-09T08:17:29.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:29.271+0000 7f3f5a4b0700 1 -- 172.21.15.184:0/1172095652 wait complete. 2024-05-09T08:17:29.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:28 smithi028 ceph-mon[30119]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:29.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:28 smithi028 ceph-mon[30119]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:29.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:28 smithi028 ceph-mon[30119]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:17:29.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:28 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:29.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:28 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:29.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:28 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:29.816 DEBUG:teuthology.orchestra.run.smithi165:mon.b> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.b.service 2024-05-09T08:17:29.819 DEBUG:teuthology.orchestra.run.smithi184:mon.c> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.c.service 2024-05-09T08:17:29.821 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T08:17:29.821 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T08:17:30.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi165:172.21.15.165=b;smithi184:172.21.15.184=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:17:30.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi165:172.21.15.165=b;smithi184:172.21.15.184=c;count:3 2024-05-09T08:17:30.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:30.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:30.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:30.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:30.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:30.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:30.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:30.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:30 smithi028 ceph-mon[30119]: Deploying daemon mon.c on smithi184 2024-05-09T08:17:30.904 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:31.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:31 smithi028 ceph-mon[30119]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:33.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 podman[34486]: 2024-05-09 08:17:33.580498342 +0000 UTC m=+0.311276185 container create 0923518f48c71bcc5aa1dbfee2d24b3da45526a7292eaf2424c9b884827aaf10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-c, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T08:17:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:33 smithi028 ceph-mon[30119]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 podman[34486]: 2024-05-09 08:17:33.758007551 +0000 UTC m=+0.488785393 container init 0923518f48c71bcc5aa1dbfee2d24b3da45526a7292eaf2424c9b884827aaf10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 podman[34486]: 2024-05-09 08:17:33.76082021 +0000 UTC m=+0.491598050 container start 0923518f48c71bcc5aa1dbfee2d24b3da45526a7292eaf2424c9b884827aaf10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: set uid:gid to 167:167 (ceph:ceph) 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev), process ceph-mon, pid 2 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pidfile_write: ignore empty --pid-file 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: load: jerasure load: lrc 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: RocksDB version: 7.9.2 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Git sha 0 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Compile date 2024-05-07 19:19:25 2024-05-09T08:17:33.995 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: DB SUMMARY 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: DB Session ID: 0KN8M8JGJUAUK3CKA2JZ 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: CURRENT file: CURRENT 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: IDENTITY file: IDENTITY 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.error_if_exists: 0 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.create_if_missing: 0 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.paranoid_checks: 1 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.env: 0x55ac921f7c60 2024-05-09T08:17:33.996 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.fs: PosixFileSystem 2024-05-09T08:17:33.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.info_log: 0x55ac93b97ac0 2024-05-09T08:17:33.997 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_file_opening_threads: 16 2024-05-09T08:17:33.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.statistics: (nil) 2024-05-09T08:17:33.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.use_fsync: 0 2024-05-09T08:17:33.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_log_file_size: 0 2024-05-09T08:17:33.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-09T08:17:33.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-09T08:17:33.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.keep_log_file_num: 1000 2024-05-09T08:17:33.998 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.recycle_log_file_num: 0 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.allow_fallocate: 1 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.allow_mmap_reads: 0 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.allow_mmap_writes: 0 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.use_direct_reads: 0 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.create_missing_column_families: 0 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.db_log_dir: 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.wal_dir: 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-09T08:17:33.999 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.advise_random_on_open: 1 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.db_write_buffer_size: 0 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.write_buffer_manager: 0x55ac93b6a1e0 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.rate_limiter: (nil) 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.wal_recovery_mode: 2 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.enable_thread_tracking: 0 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.enable_pipelined_write: 0 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.unordered_write: 0 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-09T08:17:34.000 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.row_cache: None 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.wal_filter: None 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.allow_ingest_behind: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.two_write_queues: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.manual_wal_flush: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.wal_compression: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.atomic_flush: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.log_readahead_size: 0 2024-05-09T08:17:34.001 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.best_efforts_recovery: 0 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.allow_data_in_errors: 0 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.db_host_id: __hostname__ 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_background_jobs: 2 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_background_compactions: -1 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_subcompactions: 1 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_total_wal_size: 0 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-09T08:17:34.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_open_files: -1 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bytes_per_sync: 0 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_readahead_size: 0 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_background_flushes: -1 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Compression algorithms supported: 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: kZSTD supported: 0 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: kXpressCompression supported: 0 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: kLZ4HCCompression supported: 1 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: kZlibCompression supported: 1 2024-05-09T08:17:34.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: kSnappyCompression supported: 1 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: kLZ4Compression supported: 1 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: kBZip2Compression supported: 0 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.merge_operator: 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_filter: None 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_filter_factory: None 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.sst_partitioner_factory: None 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-09T08:17:34.004 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ac93b97bc0) 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: cache_index_and_filter_blocks: 1 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: pin_top_level_index_and_filter: 1 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_type: 0 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: data_block_index_type: 0 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_shortening: 1 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: checksum: 4 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: no_block_cache: 0 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache: 0x55ac93b9b090 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_name: BinnedLRUCache 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_options: 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: capacity : 536870912 2024-05-09T08:17:34.005 INFO:journalctl@ceph.mon.c.smithi184.stdout: num_shard_bits : 4 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: strict_capacity_limit : 0 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: high_pri_pool_ratio: 0.000 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_cache_compressed: (nil) 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: persistent_cache: (nil) 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_size: 4096 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_size_deviation: 10 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_restart_interval: 16 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: index_block_restart_interval: 1 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: metadata_block_size: 4096 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: partition_filters: 0 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: use_delta_encoding: 1 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: filter_policy: bloomfilter 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: whole_key_filtering: 1 2024-05-09T08:17:34.006 INFO:journalctl@ceph.mon.c.smithi184.stdout: verify_compression: 0 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout: read_amp_bytes_per_bit: 0 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout: format_version: 5 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout: enable_index_compression: 1 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout: block_align: 0 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout: max_auto_readahead_size: 262144 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout: prepopulate_block_cache: 0 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout: initial_auto_readahead_size: 8192 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout: num_file_reads_for_auto_readahead: 2 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.write_buffer_size: 33554432 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_write_buffer_number: 2 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression: NoCompression 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression: Disabled 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.prefix_extractor: nullptr 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-09T08:17:34.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.num_levels: 7 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression_opts.level: 32767 2024-05-09T08:17:34.008 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression_opts.strategy: 0 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression_opts.enabled: false 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.target_file_size_base: 67108864 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-09T08:17:34.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.arena_block_size: 1048576 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.disable_auto_compactions: 0 2024-05-09T08:17:34.010 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.table_properties_collectors: 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.inplace_update_support: 0 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-09T08:17:34.011 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.bloom_locality: 0 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.max_successive_merges: 0 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.paranoid_file_checks: 0 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.force_consistency_checks: 1 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.report_bg_io_stats: 0 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.ttl: 2592000 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.enable_blob_files: false 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.min_blob_size: 0 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.blob_file_size: 268435456 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-09T08:17:34.012 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.blob_file_starting_level: 0 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 61e75600-8be9-41d4-8e24-346a074ae924 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715242653790241, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715242653790661, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715242653, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "61e75600-8be9-41d4-8e24-346a074ae924", "db_session_id": "0KN8M8JGJUAUK3CKA2JZ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715242653790740, "job": 1, "event": "recovery_finished"} 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-09T08:17:34.013 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ac93c8c000 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: DB pointer 0x55ac93c76000 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** DB Stats ** 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** Compaction Stats [default] ** 2024-05-09T08:17:34.014 INFO:journalctl@ceph.mon.c.smithi184.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** Compaction Stats [default] ** 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:17:34.015 INFO:journalctl@ceph.mon.c.smithi184.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: Block cache BinnedLRUCache@0x55ac93b9b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: using public_addr v2:172.21.15.184:0/0 -> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: starting mon.c rank -1 at public addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] at bind addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:17:34.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(???) e0 preinit fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).mds e1 new map 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout: e1 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout: legacy client fscid: -1 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout: 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout: No filesystems configured 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-09T08:17:34.017 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-09T08:17:34.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:17:34.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:17:34.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:17:34.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:34.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:17:34.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Deploying cephadm binary to smithi184 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Updating smithi165:/etc/ceph/ceph.conf 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.019 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Added host smithi184 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/645989816' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/645989816' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:17:34.020 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Updating smithi184:/etc/ceph/ceph.conf 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.021 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi165:172.21.15.165=b;smithi184:172.21.15.184=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi165:172.21.15.165=b;smithi184:172.21.15.184=c;count:3 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: Deploying daemon mon.c on smithi184 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 ceph-mon[34509]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 bash[34486]: 0923518f48c71bcc5aa1dbfee2d24b3da45526a7292eaf2424c9b884827aaf10 2024-05-09T08:17:34.022 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:33 smithi184 systemd[1]: Started Ceph mon.c for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:17:39.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:39.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:17:39.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: mon.a calling monitor election 2024-05-09T08:17:39.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:39.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: mon.c calling monitor election 2024-05-09T08:17:39.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: monmap epoch 2 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: last_changed 2024-05-09T08:17:33.813050+0000 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: created 2024-05-09T08:15:41.070448+0000 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: min_mon_release 19 (squid) 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: election_strategy: 1 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: fsmap 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: mgrmap e13: x(active, since 49s) 2024-05-09T08:17:39.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: overall HEALTH_OK 2024-05-09T08:17:39.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:39.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:39.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:39.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:17:39.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: mon.a calling monitor election 2024-05-09T08:17:39.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:39.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: mon.c calling monitor election 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: monmap epoch 2 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: last_changed 2024-05-09T08:17:33.813050+0000 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: created 2024-05-09T08:15:41.070448+0000 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: min_mon_release 19 (squid) 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: election_strategy: 1 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-05-09T08:17:39.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: fsmap 2024-05-09T08:17:39.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:17:39.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: mgrmap e13: x(active, since 49s) 2024-05-09T08:17:39.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: overall HEALTH_OK 2024-05-09T08:17:39.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:39.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:40.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.097+0000 7faa93642700 1 -- 172.21.15.184:0/2129427247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa780062a0 msgr2=0x7faa78008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:40.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.097+0000 7faa93642700 1 --2- 172.21.15.184:0/2129427247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa780062a0 0x7faa78008710 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faa8c102df0 tx=0x7faa8800fe90 comp rx=0 tx=0).stop 2024-05-09T08:17:40.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.097+0000 7faa93642700 1 -- 172.21.15.184:0/2129427247 shutdown_connections 2024-05-09T08:17:40.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.097+0000 7faa93642700 1 --2- 172.21.15.184:0/2129427247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa780062a0 0x7faa78008710 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:40.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.097+0000 7faa93642700 1 -- 172.21.15.184:0/2129427247 >> 172.21.15.184:0/2129427247 conn(0x7faa8c0fd860 msgr2=0x7faa8c0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:40.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.097+0000 7faa93642700 1 -- 172.21.15.184:0/2129427247 shutdown_connections 2024-05-09T08:17:40.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.097+0000 7faa93642700 1 -- 172.21.15.184:0/2129427247 wait complete. 2024-05-09T08:17:40.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa93642700 1 Processor -- start 2024-05-09T08:17:40.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa93642700 1 -- start start 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa93642700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa780062a0 0x7faa8c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa93642700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 0x7faa8c074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa93642700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c109da0 con 0x7faa780062a0 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa93642700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c109c20 con 0x7faa8c074b60 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa90bdd700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 0x7faa8c074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa90bdd700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 0x7faa8c074fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:46698/0 (socket says 172.21.15.184:46698) 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa90bdd700 1 -- 172.21.15.184:0/701622715 learned_addr learned my addr 172.21.15.184:0/701622715 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa913de700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa780062a0 0x7faa8c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa90bdd700 1 -- 172.21.15.184:0/701622715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 msgr2=0x7faa8c074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa90bdd700 1 -- 172.21.15.184:0/701622715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 msgr2=0x7faa8c074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa90bdd700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 0x7faa8c074fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.098+0000 7faa90bdd700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 0x7faa8c074fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:17:40.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.099+0000 7faa913de700 1 -- 172.21.15.184:0/701622715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 msgr2=0x7faa8c074fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:17:40.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.099+0000 7faa913de700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 0x7faa8c074fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:40.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.099+0000 7faa913de700 1 -- 172.21.15.184:0/701622715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8c076540 con 0x7faa780062a0 2024-05-09T08:17:40.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.099+0000 7faa913de700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa780062a0 0x7faa8c074620 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faa88010010 tx=0x7faa8801da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:40.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.099+0000 7faa827fc700 1 -- 172.21.15.184:0/701622715 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7faa88005400 con 0x7faa780062a0 2024-05-09T08:17:40.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.099+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8c1a3c70 con 0x7faa780062a0 2024-05-09T08:17:40.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.099+0000 7faa827fc700 1 -- 172.21.15.184:0/701622715 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa880055a0 con 0x7faa780062a0 2024-05-09T08:17:40.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.099+0000 7faa827fc700 1 -- 172.21.15.184:0/701622715 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7faa88011760 con 0x7faa780062a0 2024-05-09T08:17:40.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.099+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8c1a41b0 con 0x7faa780062a0 2024-05-09T08:17:40.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.100+0000 7faa827fc700 1 -- 172.21.15.184:0/701622715 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7faa88011a20 con 0x7faa780062a0 2024-05-09T08:17:40.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.100+0000 7faa827fc700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa70038e90 0x7faa7003b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:40.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.100+0000 7faa827fc700 1 -- 172.21.15.184:0/701622715 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7faa8804f1f0 con 0x7faa780062a0 2024-05-09T08:17:40.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.101+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa8c103470 con 0x7faa780062a0 2024-05-09T08:17:40.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.101+0000 7faa90bdd700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa70038e90 0x7faa7003b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:40.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.101+0000 7faa90bdd700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa70038e90 0x7faa7003b350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009180 tx=0x7faa7c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:40.108 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.106+0000 7faa827fc700 1 -- 172.21.15.184:0/701622715 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faa8801b2d0 con 0x7faa780062a0 2024-05-09T08:17:40.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:40.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:40.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:40.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:39 smithi028 ceph-mon[30119]: Deploying daemon mon.b on smithi165 2024-05-09T08:17:40.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:40.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:40.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:40.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:40.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:39 smithi184 ceph-mon[34509]: Deploying daemon mon.b on smithi165 2024-05-09T08:17:40.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:40.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.389+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7faa8c02db80 con 0x7faa780062a0 2024-05-09T08:17:40.392 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.390+0000 7faa827fc700 1 -- 172.21.15.184:0/701622715 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7faa8802ee00 con 0x7faa780062a0 2024-05-09T08:17:40.392 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:17:40.392 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":2,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","modified":"2024-05-09T08:17:33.813050Z","created":"2024-05-09T08:15:41.070448Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-09T08:17:40.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa70038e90 msgr2=0x7faa7003b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:40.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa70038e90 0x7faa7003b350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009180 tx=0x7faa7c008040 comp rx=0 tx=0).stop 2024-05-09T08:17:40.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa780062a0 msgr2=0x7faa8c074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:40.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa780062a0 0x7faa8c074620 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faa88010010 tx=0x7faa8801da60 comp rx=0 tx=0).stop 2024-05-09T08:17:40.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 shutdown_connections 2024-05-09T08:17:40.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa70038e90 0x7faa7003b350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:40.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa780062a0 0x7faa8c074620 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:40.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 --2- 172.21.15.184:0/701622715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa8c074b60 0x7faa8c074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:40.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 >> 172.21.15.184:0/701622715 conn(0x7faa8c0fd860 msgr2=0x7faa8c0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:40.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 shutdown_connections 2024-05-09T08:17:40.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:40.392+0000 7faa93642700 1 -- 172.21.15.184:0/701622715 wait complete. 2024-05-09T08:17:40.396 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 2 2024-05-09T08:17:40.972 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:40 smithi184 ceph-mon[34509]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:40.972 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:40 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/701622715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:17:41.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:40 smithi028 ceph-mon[30119]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:41.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:40 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/701622715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:17:41.999 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T08:17:42.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T08:17:42.245 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:17:48.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.081+0000 7fbc81263700 1 -- 172.21.15.184:0/744623229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc640035c0 msgr2=0x7fbc64005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:48.137 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.081+0000 7fbc81263700 1 --2- 172.21.15.184:0/744623229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc640035c0 0x7fbc64005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c101d70 tx=0x7fbc6c00fe60 comp rx=0 tx=0).stop 2024-05-09T08:17:48.137 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.082+0000 7fbc81263700 1 -- 172.21.15.184:0/744623229 shutdown_connections 2024-05-09T08:17:48.137 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.082+0000 7fbc81263700 1 --2- 172.21.15.184:0/744623229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc640035c0 0x7fbc64005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:48.138 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.082+0000 7fbc81263700 1 -- 172.21.15.184:0/744623229 >> 172.21.15.184:0/744623229 conn(0x7fbc7c0fd930 msgr2=0x7fbc7c0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:48.138 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.082+0000 7fbc81263700 1 -- 172.21.15.184:0/744623229 shutdown_connections 2024-05-09T08:17:48.138 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.082+0000 7fbc81263700 1 -- 172.21.15.184:0/744623229 wait complete. 2024-05-09T08:17:48.138 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.082+0000 7fbc81263700 1 Processor -- start 2024-05-09T08:17:48.138 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc81263700 1 -- start start 2024-05-09T08:17:48.140 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc81263700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc640035c0 0x7fbc7c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc81263700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 0x7fbc7c074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc81263700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc7c1a3c50 0x7fbc7c1a6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc81263700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c109e80 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc81263700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c109d00 con 0x7fbc640035c0 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc81263700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10a000 con 0x7fbc7c074b60 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7a7fc700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 0x7fbc7c074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7a7fc700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 0x7fbc7c074fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:55874/0 (socket says 172.21.15.184:55874) 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc7c1a3c50 0x7fbc7c1a6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7a7fc700 1 -- 172.21.15.184:0/133150678 learned_addr learned my addr 172.21.15.184:0/133150678 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7affd700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc640035c0 0x7fbc7c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7a7fc700 1 -- 172.21.15.184:0/133150678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 msgr2=0x7fbc7c074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-09T08:17:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7a7fc700 1 -- 172.21.15.184:0/133150678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 msgr2=0x7fbc7c074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7a7fc700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 0x7fbc7c074fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7affd700 1 -- 172.21.15.184:0/133150678 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc640035c0 msgr2=0x7fbc7c074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7affd700 1 -- 172.21.15.184:0/133150678 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc640035c0 msgr2=0x7fbc7c074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7affd700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc640035c0 0x7fbc7c074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7affd700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc640035c0 0x7fbc7c074620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7a7fc700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 0x7fbc7c074fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7b7fe700 1 -- 172.21.15.184:0/133150678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 msgr2=0x7fbc7c074fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7b7fe700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 0x7fbc7c074fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7b7fe700 1 -- 172.21.15.184:0/133150678 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc640035c0 msgr2=0x7fbc7c074620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7b7fe700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc640035c0 0x7fbc7c074620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.083+0000 7fbc7b7fe700 1 -- 172.21.15.184:0/133150678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c1a6770 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.084+0000 7fbc7b7fe700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc7c1a3c50 0x7fbc7c1a6040 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400bd50 tx=0x7fbc7400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.084+0000 7fbc63fff700 1 -- 172.21.15.184:0/133150678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74019070 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.084+0000 7fbc63fff700 1 -- 172.21.15.184:0/133150678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc7400cd10 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.084+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c1a6a60 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.084+0000 7fbc63fff700 1 -- 172.21.15.184:0/133150678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc740127e0 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.084+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c1a6fa0 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.085+0000 7fbc63fff700 1 -- 172.21.15.184:0/133150678 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fbc7401c070 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.085+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c063980 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.085+0000 7fbc63fff700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbc68039580 0x7fbc6803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.085+0000 7fbc63fff700 1 -- 172.21.15.184:0/133150678 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbc7404bc00 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.085+0000 7fbc7affd700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbc68039580 0x7fbc6803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.086+0000 7fbc7affd700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbc68039580 0x7fbc6803ba40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c020040 tx=0x7fbc6c001e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.091+0000 7fbc63fff700 1 -- 172.21.15.184:0/133150678 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbc74052080 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: mon.a calling monitor election 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: mon.c calling monitor election 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: mon.b calling monitor election 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T08:17:48.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: monmap epoch 3 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: last_changed 2024-05-09T08:17:42.794801+0000 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: created 2024-05-09T08:15:41.070448+0000 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: min_mon_release 19 (squid) 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: election_strategy: 1 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: 2: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: fsmap 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: mgrmap e13: x(active, since 58s) 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: overall HEALTH_OK 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:48.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:48.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:48.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:47 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:48.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:17:48.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: mon.a calling monitor election 2024-05-09T08:17:48.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: mon.c calling monitor election 2024-05-09T08:17:48.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:17:48.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:48.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: mon.b calling monitor election 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: monmap epoch 3 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: last_changed 2024-05-09T08:17:42.794801+0000 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: created 2024-05-09T08:15:41.070448+0000 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: min_mon_release 19 (squid) 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: election_strategy: 1 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: 1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.c 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: 2: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: fsmap 2024-05-09T08:17:48.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:17:48.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: mgrmap e13: x(active, since 58s) 2024-05-09T08:17:48.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: overall HEALTH_OK 2024-05-09T08:17:48.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:48.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:48.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:48.362 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:48.373 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.371+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fbc7c02dd40 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.373 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.371+0000 7fbc63fff700 1 -- 172.21.15.184:0/133150678 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7fbc74013c30 con 0x7fbc7c1a3c50 2024-05-09T08:17:48.374 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-05-09T08:17:48.374 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":3,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","modified":"2024-05-09T08:17:42.794801Z","created":"2024-05-09T08:15:41.070448Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-09T08:17:48.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbc68039580 msgr2=0x7fbc6803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:48.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbc68039580 0x7fbc6803ba40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c020040 tx=0x7fbc6c001e70 comp rx=0 tx=0).stop 2024-05-09T08:17:48.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc7c1a3c50 msgr2=0x7fbc7c1a6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:48.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc7c1a3c50 0x7fbc7c1a6040 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400bd50 tx=0x7fbc7400be30 comp rx=0 tx=0).stop 2024-05-09T08:17:48.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 shutdown_connections 2024-05-09T08:17:48.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbc68039580 0x7fbc6803ba40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:48.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc7c1a3c50 0x7fbc7c1a6040 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:48.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc640035c0 0x7fbc7c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:48.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 --2- 172.21.15.184:0/133150678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc7c074b60 0x7fbc7c074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:48.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 >> 172.21.15.184:0/133150678 conn(0x7fbc7c0fd930 msgr2=0x7fbc7c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:48.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 shutdown_connections 2024-05-09T08:17:48.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:48.374+0000 7fbc81263700 1 -- 172.21.15.184:0/133150678 wait complete. 2024-05-09T08:17:48.377 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 3 2024-05-09T08:17:48.950 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:48 smithi184 ceph-mon[34509]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:48.950 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:48.950 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:48 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/133150678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:17:48.951 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:48.980 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-09T08:17:48.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph config generate-minimal-conf 2024-05-09T08:17:49.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:48 smithi165 ceph-mon[34540]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:49.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:49.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:48 smithi165 ceph-mon[34540]: from='client.? 172.21.15.184:0/133150678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:17:49.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:49.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:48 smithi028 ceph-mon[30119]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:49.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:49.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:48 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/133150678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:17:49.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:17:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:17:50.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:50.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:50.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: Updating smithi165:/etc/ceph/ceph.conf 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: Updating smithi184:/etc/ceph/ceph.conf 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:50 smithi028 ceph-mon[30119]: Reconfiguring daemon mon.a on smithi028 2024-05-09T08:17:50.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: Updating smithi165:/etc/ceph/ceph.conf 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: Updating smithi184:/etc/ceph/ceph.conf 2024-05-09T08:17:50.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:50.856 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:50.856 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:50.856 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.856 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.856 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.856 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.856 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:50.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:50 smithi165 ceph-mon[34540]: Reconfiguring daemon mon.a on smithi028 2024-05-09T08:17:50.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:50.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: Updating smithi165:/etc/ceph/ceph.conf 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: Updating smithi184:/etc/ceph/ceph.conf 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:50.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:50 smithi184 ceph-mon[34509]: Reconfiguring daemon mon.a on smithi028 2024-05-09T08:17:50.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.883+0000 7fc33e6cb700 1 -- 172.21.15.28:0/1204845183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 msgr2=0x7fc338101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:50.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.883+0000 7fc33e6cb700 1 --2- 172.21.15.28:0/1204845183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 0x7fc338101270 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc328009910 tx=0x7fc328009c20 comp rx=0 tx=0).stop 2024-05-09T08:17:50.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.884+0000 7fc33e6cb700 1 -- 172.21.15.28:0/1204845183 shutdown_connections 2024-05-09T08:17:50.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.884+0000 7fc33e6cb700 1 --2- 172.21.15.28:0/1204845183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 0x7fc338101270 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:50.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.884+0000 7fc33e6cb700 1 -- 172.21.15.28:0/1204845183 >> 172.21.15.28:0/1204845183 conn(0x7fc3380fc720 msgr2=0x7fc3380feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:50.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.884+0000 7fc33e6cb700 1 -- 172.21.15.28:0/1204845183 shutdown_connections 2024-05-09T08:17:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.884+0000 7fc33e6cb700 1 -- 172.21.15.28:0/1204845183 wait complete. 2024-05-09T08:17:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc33e6cb700 1 Processor -- start 2024-05-09T08:17:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc33e6cb700 1 -- start start 2024-05-09T08:17:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc33e6cb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 0x7fc338198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc33e6cb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc338198e70 0x7fc3381992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc33e6cb700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3381a19a0 0x7fc3381a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:50.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc33e6cb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc33810b220 con 0x7fc338100e90 2024-05-09T08:17:50.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc33e6cb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc33810b0a0 con 0x7fc338198e70 2024-05-09T08:17:50.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc33e6cb700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc33810b3a0 con 0x7fc3381a19a0 2024-05-09T08:17:50.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc337fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 0x7fc338198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:50.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc337fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 0x7fc338198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52368/0 (socket says 172.21.15.28:52368) 2024-05-09T08:17:50.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc337fff700 1 -- 172.21.15.28:0/873191582 learned_addr learned my addr 172.21.15.28:0/873191582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:17:50.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc32ffff700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc338198e70 0x7fc3381992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:50.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc337fff700 1 -- 172.21.15.28:0/873191582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3381a19a0 msgr2=0x7fc3381a3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:50.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.885+0000 7fc33cc68700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3381a19a0 0x7fc3381a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:50.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc337fff700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3381a19a0 0x7fc3381a3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:50.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc337fff700 1 -- 172.21.15.28:0/873191582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc338198e70 msgr2=0x7fc3381992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:50.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc337fff700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc338198e70 0x7fc3381992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc337fff700 1 -- 172.21.15.28:0/873191582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3381a45a0 con 0x7fc338100e90 2024-05-09T08:17:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc33cc68700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3381a19a0 0x7fc3381a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:17:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc32ffff700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc338198e70 0x7fc3381992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:17:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc337fff700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 0x7fc338198930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc32800bdc0 tx=0x7fc32800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc335ffb700 1 -- 172.21.15.28:0/873191582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328026070 con 0x7fc338100e90 2024-05-09T08:17:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc335ffb700 1 -- 172.21.15.28:0/873191582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3280050d0 con 0x7fc338100e90 2024-05-09T08:17:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3381a4890 con 0x7fc338100e90 2024-05-09T08:17:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc335ffb700 1 -- 172.21.15.28:0/873191582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32800fdf0 con 0x7fc338100e90 2024-05-09T08:17:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.886+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3381a4e20 con 0x7fc338100e90 2024-05-09T08:17:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.887+0000 7fc335ffb700 1 -- 172.21.15.28:0/873191582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fc328029070 con 0x7fc338100e90 2024-05-09T08:17:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.887+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc338102370 con 0x7fc338100e90 2024-05-09T08:17:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.887+0000 7fc335ffb700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc324039580 0x7fc32403ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.887+0000 7fc335ffb700 1 -- 172.21.15.28:0/873191582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc328004dc0 con 0x7fc338100e90 2024-05-09T08:17:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.887+0000 7fc32ffff700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc324039580 0x7fc32403ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.888+0000 7fc32ffff700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc324039580 0x7fc32403ba40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3200079b0 tx=0x7fc320008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:50.893+0000 7fc335ffb700 1 -- 172.21.15.28:0/873191582 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc328013070 con 0x7fc338100e90 2024-05-09T08:17:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.107+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc3381a5050 con 0x7fc338100e90 2024-05-09T08:17:51.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.107+0000 7fc335ffb700 1 -- 172.21.15.28:0/873191582 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fc328030070 con 0x7fc338100e90 2024-05-09T08:17:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:17:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-05-09T08:17:51.114 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:17:51.115 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2024-05-09T08:17:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc324039580 msgr2=0x7fc32403ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc324039580 0x7fc32403ba40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3200079b0 tx=0x7fc320008040 comp rx=0 tx=0).stop 2024-05-09T08:17:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 msgr2=0x7fc338198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 0x7fc338198930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc32800bdc0 tx=0x7fc32800bea0 comp rx=0 tx=0).stop 2024-05-09T08:17:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 shutdown_connections 2024-05-09T08:17:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc324039580 0x7fc32403ba40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3381a19a0 0x7fc3381a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc338100e90 0x7fc338198930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 --2- 172.21.15.28:0/873191582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc338198e70 0x7fc3381992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 >> 172.21.15.28:0/873191582 conn(0x7fc3380fc720 msgr2=0x7fc338102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 shutdown_connections 2024-05-09T08:17:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:17:51.110+0000 7fc33e6cb700 1 -- 172.21.15.28:0/873191582 wait complete. 2024-05-09T08:17:51.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:51 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/873191582' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:51.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:51 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/873191582' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:51.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:51 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/873191582' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:52.287 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-09T08:17:52.287 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:17:52.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T08:17:52.316 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:17:52.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:52.384 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:17:52.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T08:17:52.413 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:17:52.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:52.480 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:17:52.480 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T08:17:52.512 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:17:52.512 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:17:52.580 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-05-09T08:17:52.580 INFO:tasks.cephadm:Adding mgr.y on smithi165 2024-05-09T08:17:52.580 INFO:tasks.cephadm:Adding mgr.z on smithi184 2024-05-09T08:17:52.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch apply mgr '3;smithi028=x;smithi165=y;smithi184=z' 2024-05-09T08:17:52.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:52 smithi028 ceph-mon[30119]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:52.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:52 smithi165 ceph-mon[34540]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:52.858 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:17:52.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:52 smithi184 ceph-mon[34509]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:53.981 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:53.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:53.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:53 smithi184 ceph-mon[34509]: Reconfiguring mon.b (monmap changed)... 2024-05-09T08:17:53.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:53.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:17:53.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:53.982 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:53 smithi184 ceph-mon[34509]: Reconfiguring daemon mon.b on smithi165 2024-05-09T08:17:54.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:54.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:54.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:53 smithi165 ceph-mon[34540]: Reconfiguring mon.b (monmap changed)... 2024-05-09T08:17:54.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:54.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:17:54.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:54.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:53 smithi165 ceph-mon[34540]: Reconfiguring daemon mon.b on smithi165 2024-05-09T08:17:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:53 smithi028 ceph-mon[30119]: Reconfiguring mon.b (monmap changed)... 2024-05-09T08:17:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:17:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:54.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:53 smithi028 ceph-mon[30119]: Reconfiguring daemon mon.b on smithi165 2024-05-09T08:17:54.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.322+0000 7fb0e4536700 1 -- 172.21.15.184:0/1579292710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0dc100dc0 msgr2=0x7fb0dc1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:54.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.322+0000 7fb0e4536700 1 --2- 172.21.15.184:0/1579292710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0dc100dc0 0x7fb0dc1011e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009910 tx=0x7fb0d8009c20 comp rx=0 tx=0).stop 2024-05-09T08:17:54.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.322+0000 7fb0e4536700 1 -- 172.21.15.184:0/1579292710 shutdown_connections 2024-05-09T08:17:54.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.322+0000 7fb0e4536700 1 --2- 172.21.15.184:0/1579292710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0dc100dc0 0x7fb0dc1011e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:54.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.322+0000 7fb0e4536700 1 -- 172.21.15.184:0/1579292710 >> 172.21.15.184:0/1579292710 conn(0x7fb0dc0fc360 msgr2=0x7fb0dc0fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:54.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.322+0000 7fb0e4536700 1 -- 172.21.15.184:0/1579292710 shutdown_connections 2024-05-09T08:17:54.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.322+0000 7fb0e4536700 1 -- 172.21.15.184:0/1579292710 wait complete. 2024-05-09T08:17:54.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e4536700 1 Processor -- start 2024-05-09T08:17:54.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e4536700 1 -- start start 2024-05-09T08:17:54.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e4536700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc100dc0 0x7fb0dc196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:54.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e4536700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0dc196b70 0x7fb0dc19bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:54.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e4536700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0dc196ff0 0x7fb0dc19e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:54.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e4536700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc106e50 con 0x7fb0dc196ff0 2024-05-09T08:17:54.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e4536700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc106cd0 con 0x7fb0dc196b70 2024-05-09T08:17:54.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e4536700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc106fd0 con 0x7fb0dc100dc0 2024-05-09T08:17:54.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e22d2700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc100dc0 0x7fb0dc196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:54.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e22d2700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc100dc0 0x7fb0dc196630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:51292/0 (socket says 172.21.15.184:51292) 2024-05-09T08:17:54.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e22d2700 1 -- 172.21.15.184:0/3057509818 learned_addr learned my addr 172.21.15.184:0/3057509818 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-05-09T08:17:54.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e2ad3700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0dc196ff0 0x7fb0dc19e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:54.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e1ad1700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0dc196b70 0x7fb0dc19bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:54.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e22d2700 1 -- 172.21.15.184:0/3057509818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0dc196b70 msgr2=0x7fb0dc19bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:54.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.323+0000 7fb0e22d2700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0dc196b70 0x7fb0dc19bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:54.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0e22d2700 1 -- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0dc196ff0 msgr2=0x7fb0dc19e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:54.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0e22d2700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0dc196ff0 0x7fb0dc19e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:54.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0e22d2700 1 -- 172.21.15.184:0/3057509818 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0dc19ea40 con 0x7fb0dc100dc0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0e2ad3700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0dc196ff0 0x7fb0dc19e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0e22d2700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc100dc0 0x7fb0dc196630 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8004f80 tx=0x7fb0d8004fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0d37fe700 1 -- 172.21.15.184:0/3057509818 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8026070 con 0x7fb0dc100dc0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0dc19ecd0 con 0x7fb0dc100dc0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0d37fe700 1 -- 172.21.15.184:0/3057509818 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d800b750 con 0x7fb0dc100dc0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0d37fe700 1 -- 172.21.15.184:0/3057509818 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d80039b0 con 0x7fb0dc100dc0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.324+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0dc19f1b0 con 0x7fb0dc100dc0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.325+0000 7fb0d37fe700 1 -- 172.21.15.184:0/3057509818 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb0d8029070 con 0x7fb0dc100dc0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.325+0000 7fb0d37fe700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0c8039580 0x7fb0c803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.325+0000 7fb0d37fe700 1 -- 172.21.15.184:0/3057509818 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb0d804ce30 con 0x7fb0dc100dc0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.325+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0dc063980 con 0x7fb0dc100dc0 2024-05-09T08:17:54.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.325+0000 7fb0e1ad1700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0c8039580 0x7fb0c803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:17:54.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.326+0000 7fb0e1ad1700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0c8039580 0x7fb0c803ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009ad0 tx=0x7fb0cc009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:17:54.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.330+0000 7fb0d37fe700 1 -- 172.21.15.184:0/3057509818 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb0d8030020 con 0x7fb0dc100dc0 2024-05-09T08:17:54.542 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.539+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi165=y;smithi184=z", "target": ["mon-mgr", ""]}) -- 0x7fb0dc062760 con 0x7fb0c8039580 2024-05-09T08:17:54.547 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.545+0000 7fb0d37fe700 1 -- 172.21.15.184:0/3057509818 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb0dc062760 con 0x7fb0c8039580 2024-05-09T08:17:54.547 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mgr update... 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0c8039580 msgr2=0x7fb0c803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0c8039580 0x7fb0c803ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009ad0 tx=0x7fb0cc009210 comp rx=0 tx=0).stop 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc100dc0 msgr2=0x7fb0dc196630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc100dc0 0x7fb0dc196630 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8004f80 tx=0x7fb0d8004fb0 comp rx=0 tx=0).stop 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 shutdown_connections 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0c8039580 0x7fb0c803ba40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc100dc0 0x7fb0dc196630 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0dc196ff0 0x7fb0dc19e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 --2- 172.21.15.184:0/3057509818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0dc196b70 0x7fb0dc19bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:17:54.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 >> 172.21.15.184:0/3057509818 conn(0x7fb0dc0fc360 msgr2=0x7fb0dc0fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:17:54.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.547+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 shutdown_connections 2024-05-09T08:17:54.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:17:54.548+0000 7fb0e4536700 1 -- 172.21.15.184:0/3057509818 wait complete. 2024-05-09T08:17:54.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:54 smithi184 ceph-mon[34509]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:54.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:54 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:54.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:54 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:54.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:54 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:54.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:54 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:17:54.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:54 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:54.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:54 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:55.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:54 smithi165 ceph-mon[34540]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:55.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:54 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:55.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:54 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:55.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:54 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:55.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:54 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:17:55.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:54 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:55.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:54 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:55.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:54 smithi028 ceph-mon[30119]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:55.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:54 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:55.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:54 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:55.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:54 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:17:55.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:54 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:17:55.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:54 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:55.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:54 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:55.310 DEBUG:teuthology.orchestra.run.smithi165:mgr.y> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.y.service 2024-05-09T08:17:55.313 DEBUG:teuthology.orchestra.run.smithi184:mgr.z> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.z.service 2024-05-09T08:17:55.315 INFO:tasks.cephadm:Deploying OSDs... 2024-05-09T08:17:55.315 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:17:55.316 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T08:17:55.336 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T08:17:55.337 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-05-09T08:17:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T08:17:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:55.397 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-09T08:17:55.397 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:55.397 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:55.397 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-09 08:17:00.194412126 +0000 2024-05-09T08:17:55.397 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-09 08:11:53.283081336 +0000 2024-05-09T08:17:55.397 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-09 08:11:53.283081336 +0000 2024-05-09T08:17:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-09 08:11:53.283081336 +0000 2024-05-09T08:17:55.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T08:17:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-05-09T08:17:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-05-09T08:17:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000129554 s, 4.0 MB/s 2024-05-09T08:17:55.467 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T08:17:55.528 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-05-09T08:17:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T08:17:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:55.589 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-09T08:17:55.589 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:55.589 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:55.590 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-09 08:17:00.223411298 +0000 2024-05-09T08:17:55.590 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-09 08:11:53.637071248 +0000 2024-05-09T08:17:55.590 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-09 08:11:53.637071248 +0000 2024-05-09T08:17:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-09 08:11:53.637071248 +0000 2024-05-09T08:17:55.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T08:17:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-05-09T08:17:55.658 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-05-09T08:17:55.658 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000137824 s, 3.7 MB/s 2024-05-09T08:17:55.659 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T08:17:55.718 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-05-09T08:17:55.777 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T08:17:55.777 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:55.777 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-09T08:17:55.777 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:55.778 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:55.778 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-09 08:17:00.249410556 +0000 2024-05-09T08:17:55.778 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-09 08:11:53.978061531 +0000 2024-05-09T08:17:55.778 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-09 08:11:53.978061531 +0000 2024-05-09T08:17:55.778 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-09 08:11:53.978061531 +0000 2024-05-09T08:17:55.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T08:17:55.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-05-09T08:17:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-05-09T08:17:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000140985 s, 3.6 MB/s 2024-05-09T08:17:55.846 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T08:17:55.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:55 smithi184 ceph-mon[34509]: Reconfiguring mon.c (monmap changed)... 2024-05-09T08:17:55.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:55 smithi184 ceph-mon[34509]: Reconfiguring daemon mon.c on smithi184 2024-05-09T08:17:55.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:55 smithi184 ceph-mon[34509]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi165=y;smithi184=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:17:55.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:55 smithi184 ceph-mon[34509]: Saving service mgr spec with placement smithi028=x;smithi165=y;smithi184=z;count:3 2024-05-09T08:17:55.904 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-05-09T08:17:55.962 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T08:17:55.962 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:55.962 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T08:17:55.963 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:55.963 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:55.963 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-09 08:17:00.275409814 +0000 2024-05-09T08:17:55.963 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-09 08:11:54.306052184 +0000 2024-05-09T08:17:55.963 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-09 08:11:54.306052184 +0000 2024-05-09T08:17:55.963 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-09 08:11:54.306052184 +0000 2024-05-09T08:17:55.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T08:17:56.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:55 smithi028 ceph-mon[30119]: Reconfiguring mon.c (monmap changed)... 2024-05-09T08:17:56.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:55 smithi028 ceph-mon[30119]: Reconfiguring daemon mon.c on smithi184 2024-05-09T08:17:56.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:55 smithi028 ceph-mon[30119]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi165=y;smithi184=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:17:56.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:55 smithi028 ceph-mon[30119]: Saving service mgr spec with placement smithi028=x;smithi165=y;smithi184=z;count:3 2024-05-09T08:17:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-05-09T08:17:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-05-09T08:17:56.031 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000120705 s, 4.2 MB/s 2024-05-09T08:17:56.031 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T08:17:56.088 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-05-09T08:17:56.088 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T08:17:56.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:55 smithi165 ceph-mon[34540]: Reconfiguring mon.c (monmap changed)... 2024-05-09T08:17:56.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:55 smithi165 ceph-mon[34540]: Reconfiguring daemon mon.c on smithi184 2024-05-09T08:17:56.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:55 smithi165 ceph-mon[34540]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi165=y;smithi184=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:17:56.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:55 smithi165 ceph-mon[34540]: Saving service mgr spec with placement smithi028=x;smithi165=y;smithi184=z;count:3 2024-05-09T08:17:56.107 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T08:17:56.107 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-05-09T08:17:56.165 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T08:17:56.165 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:56.165 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-09T08:17:56.165 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:56.165 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:56.165 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-05-09 08:17:42.219748140 +0000 2024-05-09T08:17:56.165 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-05-09 08:12:18.585585641 +0000 2024-05-09T08:17:56.165 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-05-09 08:12:18.585585641 +0000 2024-05-09T08:17:56.165 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-05-09 08:12:18.585585641 +0000 2024-05-09T08:17:56.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T08:17:56.231 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-05-09T08:17:56.232 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-05-09T08:17:56.232 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000220773 s, 2.3 MB/s 2024-05-09T08:17:56.233 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T08:17:56.290 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-05-09T08:17:56.346 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T08:17:56.346 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:56.346 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-05-09T08:17:56.347 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:56.347 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:56.347 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-05-09 08:17:42.220748125 +0000 2024-05-09T08:17:56.347 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-05-09 08:12:18.928580460 +0000 2024-05-09T08:17:56.347 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-05-09 08:12:18.928580460 +0000 2024-05-09T08:17:56.347 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-05-09 08:12:18.928580460 +0000 2024-05-09T08:17:56.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T08:17:56.411 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-05-09T08:17:56.411 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-05-09T08:17:56.411 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000208886 s, 2.5 MB/s 2024-05-09T08:17:56.412 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T08:17:56.470 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-05-09T08:17:56.525 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T08:17:56.526 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:56.526 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-09T08:17:56.526 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:56.526 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:56.526 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-05-09 08:17:42.220748125 +0000 2024-05-09T08:17:56.526 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-05-09 08:12:19.273575247 +0000 2024-05-09T08:17:56.526 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-05-09 08:12:19.273575247 +0000 2024-05-09T08:17:56.526 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-05-09 08:12:19.273575247 +0000 2024-05-09T08:17:56.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T08:17:56.589 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-05-09T08:17:56.589 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-05-09T08:17:56.589 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000203133 s, 2.5 MB/s 2024-05-09T08:17:56.590 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T08:17:56.646 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-05-09T08:17:56.702 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T08:17:56.702 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:56.702 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-09T08:17:56.702 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:56.702 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:56.702 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-05-09 08:17:42.220748125 +0000 2024-05-09T08:17:56.703 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-05-09 08:12:19.611570141 +0000 2024-05-09T08:17:56.703 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-05-09 08:12:19.611570141 +0000 2024-05-09T08:17:56.703 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-05-09 08:12:19.611570141 +0000 2024-05-09T08:17:56.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T08:17:56.764 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-05-09T08:17:56.764 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-05-09T08:17:56.764 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000166118 s, 3.1 MB/s 2024-05-09T08:17:56.765 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T08:17:56.823 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-05-09T08:17:56.823 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T08:17:56.842 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T08:17:56.842 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-05-09T08:17:56.900 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T08:17:56.900 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-09T08:17:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-05-09 08:17:32.993515135 +0000 2024-05-09T08:17:56.900 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-05-09 08:12:01.561367873 +0000 2024-05-09T08:17:56.901 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-05-09 08:12:01.561367873 +0000 2024-05-09T08:17:56.901 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-05-09 08:12:01.561367873 +0000 2024-05-09T08:17:56.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T08:17:56.966 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-05-09T08:17:56.967 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-05-09T08:17:56.967 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000283755 s, 1.8 MB/s 2024-05-09T08:17:56.968 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T08:17:57.025 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-05-09T08:17:57.081 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T08:17:57.081 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:57.081 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-05-09T08:17:57.081 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:57.081 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:57.081 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-05-09 08:17:32.993515135 +0000 2024-05-09T08:17:57.081 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-05-09 08:12:01.895361340 +0000 2024-05-09T08:17:57.081 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-05-09 08:12:01.895361340 +0000 2024-05-09T08:17:57.081 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-05-09 08:12:01.895361340 +0000 2024-05-09T08:17:57.082 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T08:17:57.144 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-05-09T08:17:57.144 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-05-09T08:17:57.144 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000179416 s, 2.9 MB/s 2024-05-09T08:17:57.145 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T08:17:57.200 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-05-09T08:17:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T08:17:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:57.256 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-09T08:17:57.256 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:57.256 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:57.256 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-05-09 08:17:32.994515115 +0000 2024-05-09T08:17:57.256 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-05-09 08:12:02.250354396 +0000 2024-05-09T08:17:57.256 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-05-09 08:12:02.250354396 +0000 2024-05-09T08:17:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-05-09 08:12:02.250354396 +0000 2024-05-09T08:17:57.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T08:17:57.318 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-05-09T08:17:57.319 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-05-09T08:17:57.319 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000217736 s, 2.4 MB/s 2024-05-09T08:17:57.320 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T08:17:57.376 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-05-09T08:17:57.431 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T08:17:57.431 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:17:57.431 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-09T08:17:57.431 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:17:57.431 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:17:57.431 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-05-09 08:17:32.994515115 +0000 2024-05-09T08:17:57.431 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-05-09 08:12:02.579347960 +0000 2024-05-09T08:17:57.431 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-05-09 08:12:02.579347960 +0000 2024-05-09T08:17:57.431 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-05-09 08:12:02.579347960 +0000 2024-05-09T08:17:57.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T08:17:57.493 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-05-09T08:17:57.493 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-05-09T08:17:57.493 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000191209 s, 2.7 MB/s 2024-05-09T08:17:57.495 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T08:17:57.550 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-05-09T08:17:57.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T08:17:57.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:57.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:57.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:57.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:57.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:57.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:57.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:57.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:17:57.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:17:57.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:17:57.567 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:57.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:57.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:57.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:57.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:17:57.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:17:57.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:17:57.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:57.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:57.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:57.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:57.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:17:57.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:57.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:17:57.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:17:57.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:17:57.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:17:57.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:17:57.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:17:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:17:58.566 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:58 smithi184 ceph-mon[34509]: Deploying daemon mgr.z on smithi184 2024-05-09T08:17:58.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:58 smithi165 ceph-mon[34540]: Deploying daemon mgr.z on smithi184 2024-05-09T08:17:58.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:58 smithi028 ceph-mon[30119]: Deploying daemon mgr.z on smithi184 2024-05-09T08:17:58.859 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:58 smithi184 systemd[1]: Starting Ceph mgr.z for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:17:59.289 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:58 smithi184 podman[36228]: 2024-05-09 08:17:58.896764998 +0000 UTC m=+0.018278816 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:17:59.290 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:58 smithi184 podman[36228]: 2024-05-09 08:17:58.994279484 +0000 UTC m=+0.115793301 container create 4a3ae0841ccb7dd0005aca57ccddd7d69bb7a5ede942e27995840bf116131609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-09T08:17:59.290 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:59 smithi184 podman[36228]: 2024-05-09 08:17:59.196398744 +0000 UTC m=+0.317912561 container init 4a3ae0841ccb7dd0005aca57ccddd7d69bb7a5ede942e27995840bf116131609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-09T08:17:59.290 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:59 smithi184 podman[36228]: 2024-05-09 08:17:59.199011216 +0000 UTC m=+0.320525037 container start 4a3ae0841ccb7dd0005aca57ccddd7d69bb7a5ede942e27995840bf116131609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-96805dc) 2024-05-09T08:17:59.290 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:59 smithi184 ceph-mgr[36247]: -- 172.21.15.184:0/2835109912 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8bbaac5a0 con 0x55f8bba80800 2024-05-09T08:17:59.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:17:59 smithi028 ceph-mon[30119]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:59.543 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:17:59 smithi184 ceph-mon[34509]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:59.544 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:59 smithi184 bash[36228]: 4a3ae0841ccb7dd0005aca57ccddd7d69bb7a5ede942e27995840bf116131609 2024-05-09T08:17:59.544 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:59 smithi184 systemd[1]: Started Ceph mgr.z for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:17:59.544 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:17:59.352+0000 7fa6c2006200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:17:59.544 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:17:59.541+0000 7fa6c2006200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:17:59.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:17:59 smithi165 ceph-mon[34540]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:17:59.859 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:17:59.641+0000 7fa6c2006200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:17:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T08:17:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T08:17:59.932 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-05-09T08:17:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-05-09T08:17:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00955785 s, 1.1 GB/s 2024-05-09T08:17:59.932 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 2024-05-09T08:17:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-05-09T08:18:00.359 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:17:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:17:59.945+0000 7fa6c2006200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:18:00.359 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:00.029+0000 7fa6c2006200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:18:00.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-05-09T08:18:00.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:00.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:00.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:00.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:18:00.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:18:00.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:18:00.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:00.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:00 smithi028 ceph-mon[30119]: Deploying daemon mgr.y on smithi165 2024-05-09T08:18:00.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:00.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:00.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:00.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:18:00.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:18:00.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:18:00.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:00.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:00 smithi165 ceph-mon[34540]: Deploying daemon mgr.y on smithi165 2024-05-09T08:18:00.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:00.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:00.684 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:00.685 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:18:00.685 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:18:00.685 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:18:00.685 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:00.685 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:00 smithi184 ceph-mon[34509]: Deploying daemon mgr.y on smithi165 2024-05-09T08:18:00.778 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:18:01.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:01 smithi028 ceph-mon[30119]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:01.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:01 smithi165 ceph-mon[34540]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:01.583 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:01 smithi165 systemd[1]: Starting Ceph mgr.y for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:18:01.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:01 smithi184 ceph-mon[34509]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:01.605 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:01.259+0000 7fa6c2006200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:18:01.606 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:01.428+0000 7fa6c2006200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:18:01.606 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:01.602+0000 7fa6c2006200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:18:01.854 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:01 smithi165 podman[35908]: 2024-05-09 08:18:01.55684508 +0000 UTC m=+0.017889744 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:18:01.854 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:01 smithi165 podman[35908]: 2024-05-09 08:18:01.689274178 +0000 UTC m=+0.150318861 container create 7f78607ac683604e10947988d9d02102fdb6b0ec84038cb8e7be8776ab4cdf42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:18:02.191 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:01.890+0000 7fa6c2006200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:18:02.192 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:01.981+0000 7fa6c2006200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:18:02.192 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:02.188+0000 7fa6c2006200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:18:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.190+0000 7f3b79774700 1 -- 172.21.15.28:0/1435586209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b74102e90 msgr2=0x7f3b74105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.190+0000 7f3b79774700 1 --2- 172.21.15.28:0/1435586209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b74102e90 0x7f3b74105280 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400b0a0 tx=0x7f3b6400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:18:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.190+0000 7f3b79774700 1 -- 172.21.15.28:0/1435586209 shutdown_connections 2024-05-09T08:18:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.190+0000 7f3b79774700 1 --2- 172.21.15.28:0/1435586209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b74102e90 0x7f3b74105280 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.190+0000 7f3b79774700 1 --2- 172.21.15.28:0/1435586209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b741080f0 0x7f3b7410a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.190+0000 7f3b79774700 1 --2- 172.21.15.28:0/1435586209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b741057c0 0x7f3b74107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.190+0000 7f3b79774700 1 -- 172.21.15.28:0/1435586209 >> 172.21.15.28:0/1435586209 conn(0x7f3b740fc840 msgr2=0x7f3b740fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:18:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.190+0000 7f3b79774700 1 -- 172.21.15.28:0/1435586209 shutdown_connections 2024-05-09T08:18:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.190+0000 7f3b79774700 1 -- 172.21.15.28:0/1435586209 wait complete. 2024-05-09T08:18:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.191+0000 7f3b79774700 1 Processor -- start 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.191+0000 7f3b79774700 1 -- start start 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.191+0000 7f3b79774700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b74102e90 0x7f3b741985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.191+0000 7f3b79774700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b741057c0 0x7f3b74198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.191+0000 7f3b79774700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b741080f0 0x7f3b7419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.191+0000 7f3b79774700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7410e000 con 0x7f3b741080f0 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.191+0000 7f3b79774700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7410de80 con 0x7f3b74102e90 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.191+0000 7f3b79774700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7410e180 con 0x7f3b741057c0 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b72ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b74102e90 0x7f3b741985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b72ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b74102e90 0x7f3b741985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:57130/0 (socket says 172.21.15.28:57130) 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b72ffd700 1 -- 172.21.15.28:0/2506805172 learned_addr learned my addr 172.21.15.28:0/2506805172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b727fc700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b741057c0 0x7f3b74198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b737fe700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b741080f0 0x7f3b7419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b72ffd700 1 -- 172.21.15.28:0/2506805172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b741057c0 msgr2=0x7f3b74198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b72ffd700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b741057c0 0x7f3b74198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b72ffd700 1 -- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b741080f0 msgr2=0x7f3b7419fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b72ffd700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b741080f0 0x7f3b7419fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b72ffd700 1 -- 172.21.15.28:0/2506805172 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b741a0210 con 0x7f3b74102e90 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b72ffd700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b74102e90 0x7f3b741985b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400b070 tx=0x7f3b64004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b5bfff700 1 -- 172.21.15.28:0/2506805172 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64021070 con 0x7f3b74102e90 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b5bfff700 1 -- 172.21.15.28:0/2506805172 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b64007b80 con 0x7f3b74102e90 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b741a04a0 con 0x7f3b74102e90 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b5bfff700 1 -- 172.21.15.28:0/2506805172 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b640052a0 con 0x7f3b74102e90 2024-05-09T08:18:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.192+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b741a0950 con 0x7f3b74102e90 2024-05-09T08:18:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.193+0000 7f3b5bfff700 1 -- 172.21.15.28:0/2506805172 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3b64024070 con 0x7f3b74102e90 2024-05-09T08:18:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.194+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b74069010 con 0x7f3b74102e90 2024-05-09T08:18:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.194+0000 7f3b5bfff700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3b5c039330 0x7f3b5c03b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.194+0000 7f3b5bfff700 1 -- 172.21.15.28:0/2506805172 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3b6400f020 con 0x7f3b74102e90 2024-05-09T08:18:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.194+0000 7f3b727fc700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3b5c039330 0x7f3b5c03b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.194+0000 7f3b727fc700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3b5c039330 0x7f3b5c03b7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009680 tx=0x7f3b68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:18:02.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.199+0000 7f3b5bfff700 1 -- 172.21.15.28:0/2506805172 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3b6402b020 con 0x7f3b74102e90 2024-05-09T08:18:02.309 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:01 smithi165 podman[35908]: 2024-05-09 08:18:01.958382425 +0000 UTC m=+0.419427068 container init 7f78607ac683604e10947988d9d02102fdb6b0ec84038cb8e7be8776ab4cdf42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-09T08:18:02.310 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:01 smithi165 podman[35908]: 2024-05-09 08:18:01.961502007 +0000 UTC m=+0.422546647 container start 7f78607ac683604e10947988d9d02102fdb6b0ec84038cb8e7be8776ab4cdf42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-09T08:18:02.310 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:01 smithi165 ceph-mgr[35928]: -- 172.21.15.165:0/475380303 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b3fca6a5a0 con 0x55b3fca3ec00 2024-05-09T08:18:02.310 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:02 smithi165 bash[35908]: 7f78607ac683604e10947988d9d02102fdb6b0ec84038cb8e7be8776ab4cdf42 2024-05-09T08:18:02.310 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:02 smithi165 systemd[1]: Started Ceph mgr.y for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:18:02.310 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:02 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:02.114+0000 7f77ff44e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:18:02.310 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:02 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:02.307+0000 7f77ff44e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:18:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:02.417+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3b74062760 con 0x7f3b5c039330 2024-05-09T08:18:02.604 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:02 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:02.417+0000 7f77ff44e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:18:02.609 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:02.388+0000 7fa6c2006200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:18:03.065 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:02 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:02.748+0000 7f77ff44e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:18:03.065 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:02 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:02.843+0000 7f77ff44e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:18:03.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:03 smithi028 ceph-mon[30119]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:03.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:03.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:18:03.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:18:03.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:03.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:03 smithi165 ceph-mon[34540]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:03.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:03.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:18:03.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:18:03.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:03.359 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:03.039+0000 7fa6c2006200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:18:03.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:03 smithi184 ceph-mon[34509]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:03.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:03.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:03.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:18:03.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:18:03.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:03.795 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:03.486+0000 7fa6c2006200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:18:03.795 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:03.589+0000 7fa6c2006200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:18:03.795 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:03.792+0000 7fa6c2006200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:18:04.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:18:04.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:04.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:04.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:18:04.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:18:04.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:04.354 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:04.090+0000 7f77ff44e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:18:04.354 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:04.261+0000 7f77ff44e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:18:04.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:18:04.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:04.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:04.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:18:04.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:18:04.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:04.410 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:04.316+0000 7fa6c2006200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:18:04.410 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:04.406+0000 7fa6c2006200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:18:04.410 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:18:04.410 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.410 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.410 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:04.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:04.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:04.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:18:04.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:18:04.411 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:04.723 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:04.433+0000 7f77ff44e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:18:04.723 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:04.720+0000 7f77ff44e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:18:04.784 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:04.649+0000 7fa6c2006200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:18:04.784 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:04.781+0000 7fa6c2006200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:18:05.019 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:04.812+0000 7f77ff44e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:18:05.019 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:05 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:05.016+0000 7f77ff44e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:18:05.109 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:04.876+0000 7fa6c2006200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:18:05.109 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:04.990+0000 7fa6c2006200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:18:05.354 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:05 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:05.208+0000 7f77ff44e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:18:05.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:05 smithi165 ceph-mon[34540]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:05.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:05 smithi165 ceph-mon[34540]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T08:18:05.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:05 smithi165 ceph-mon[34540]: Reconfiguring daemon mgr.x on smithi028 2024-05-09T08:18:05.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:05 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:05.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:05 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3863176211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb0b1568-830d-4e49-afa7-96ea4d64ce19"}]: dispatch 2024-05-09T08:18:05.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:05 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3863176211' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb0b1568-830d-4e49-afa7-96ea4d64ce19"}]': finished 2024-05-09T08:18:05.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:05 smithi165 ceph-mon[34540]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T08:18:05.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:05 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:05.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:05 smithi028 ceph-mon[30119]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:05.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:05 smithi028 ceph-mon[30119]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T08:18:05.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:05 smithi028 ceph-mon[30119]: Reconfiguring daemon mgr.x on smithi028 2024-05-09T08:18:05.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:05 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:05.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:05 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3863176211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb0b1568-830d-4e49-afa7-96ea4d64ce19"}]: dispatch 2024-05-09T08:18:05.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:05 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3863176211' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb0b1568-830d-4e49-afa7-96ea4d64ce19"}]': finished 2024-05-09T08:18:05.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:05 smithi028 ceph-mon[30119]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T08:18:05.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:05 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:05.609 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:05.296+0000 7fa6c2006200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:18:05.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:05 smithi184 ceph-mon[34509]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:05.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:05 smithi184 ceph-mon[34509]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T08:18:05.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:05 smithi184 ceph-mon[34509]: Reconfiguring daemon mgr.x on smithi028 2024-05-09T08:18:05.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:05 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:05.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:05 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3863176211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb0b1568-830d-4e49-afa7-96ea4d64ce19"}]: dispatch 2024-05-09T08:18:05.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:05 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3863176211' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb0b1568-830d-4e49-afa7-96ea4d64ce19"}]': finished 2024-05-09T08:18:05.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:05 smithi184 ceph-mon[34509]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T08:18:05.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:05 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:06.104 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:05 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:05.832+0000 7f77ff44e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:18:06.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:06 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3140090970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:18:06.360 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:05.897+0000 7fa6c2006200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:18:06.360 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:18:06 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:18:06.112+0000 7fa6c2006200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:18:06.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3140090970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:18:06.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:06 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3140090970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:18:06.551 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:06 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:06.253+0000 7f77ff44e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:18:06.551 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:06 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:06.349+0000 7f77ff44e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:18:06.551 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:06 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:06.548+0000 7f77ff44e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:18:07.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:07.126+0000 7f3b5bfff700 1 -- 172.21.15.28:0/2506805172 <== mon.2 v2:172.21.15.165:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7f3b64026450 con 0x7f3b74102e90 2024-05-09T08:18:07.354 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:07 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:07.089+0000 7f77ff44e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:18:07.354 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:07 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:07.182+0000 7f77ff44e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:18:07.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:07 smithi165 ceph-mon[34540]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:07.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:07 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:18:07.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:07 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:18:07.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:07 smithi165 ceph-mon[34540]: Standby manager daemon z started 2024-05-09T08:18:07.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:07 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:18:07.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:07 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:18:07.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:07 smithi028 ceph-mon[30119]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:07.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:07 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:18:07.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:07 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:18:07.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:07 smithi028 ceph-mon[30119]: Standby manager daemon z started 2024-05-09T08:18:07.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:07 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:18:07.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:07 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:18:07.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:07 smithi184 ceph-mon[34509]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:07.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:07 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:18:07.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:07 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:18:07.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:07 smithi184 ceph-mon[34509]: Standby manager daemon z started 2024-05-09T08:18:07.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:07 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:18:07.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:07 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.184:0/3243910877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:18:07.765 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:07 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:07.420+0000 7f77ff44e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:18:07.765 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:07 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:07.553+0000 7f77ff44e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:18:07.765 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:07 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:07.649+0000 7f77ff44e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:18:07.765 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:07 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:07.763+0000 7f77ff44e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:18:08.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:08 smithi165 ceph-mon[34540]: mgrmap e14: x(active, since 77s), standbys: z 2024-05-09T08:18:08.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:08 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:18:08.354 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:08 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:08.065+0000 7f77ff44e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:18:08.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:08 smithi028 ceph-mon[30119]: mgrmap e14: x(active, since 77s), standbys: z 2024-05-09T08:18:08.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:08 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:18:08.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:08 smithi184 ceph-mon[34509]: mgrmap e14: x(active, since 77s), standbys: z 2024-05-09T08:18:08.616 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:08 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:18:09.104 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:08 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:08.663+0000 7f77ff44e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:18:09.104 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:18:08 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:18:08.876+0000 7f77ff44e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:18:09.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:09.136+0000 7f3b5bfff700 1 -- 172.21.15.28:0/2506805172 <== mon.2 v2:172.21.15.165:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f3b64011390 con 0x7f3b74102e90 2024-05-09T08:18:09.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:09 smithi028 ceph-mon[30119]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:09.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:09 smithi028 ceph-mon[30119]: Standby manager daemon y started 2024-05-09T08:18:09.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:09 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:18:09.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:09 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:18:09.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:09 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:18:09.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:09 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:18:09.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:09 smithi165 ceph-mon[34540]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:09.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:09 smithi165 ceph-mon[34540]: Standby manager daemon y started 2024-05-09T08:18:09.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:09 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:18:09.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:09 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:18:09.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:09 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:18:09.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:09 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:18:09.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:09 smithi184 ceph-mon[34509]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:09.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:09 smithi184 ceph-mon[34509]: Standby manager daemon y started 2024-05-09T08:18:09.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:09 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:18:09.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:09 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:18:09.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:09 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:18:09.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:09 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.165:0/1078631383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:18:10.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:10 smithi028 ceph-mon[30119]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-09T08:18:10.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:10 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:18:10.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:10 smithi165 ceph-mon[34540]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-09T08:18:10.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:10 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:18:10.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:10 smithi184 ceph-mon[34509]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-09T08:18:10.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:10 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:18:11.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:11 smithi165 ceph-mon[34540]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:11.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:11.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:11.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:11.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:11.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:11.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:11.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:11 smithi028 ceph-mon[30119]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:11.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:11.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:11.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:11.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:11.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:11.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:11.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:11 smithi184 ceph-mon[34509]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:11.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:11.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:11.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:11.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:11.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:11.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:13.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:13 smithi028 ceph-mon[30119]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:13.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:13 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T08:18:13.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:13 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:13 smithi165 ceph-mon[34540]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:13 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T08:18:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:13 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:13.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:13 smithi184 ceph-mon[34509]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:13.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:13 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T08:18:13.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:13 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:14.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:14 smithi028 ceph-mon[30119]: Deploying daemon osd.0 on smithi028 2024-05-09T08:18:14.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:14 smithi165 ceph-mon[34540]: Deploying daemon osd.0 on smithi028 2024-05-09T08:18:14.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:14 smithi184 ceph-mon[34509]: Deploying daemon osd.0 on smithi028 2024-05-09T08:18:15.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:15 smithi028 ceph-mon[30119]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:15.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:15 smithi165 ceph-mon[34540]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:15.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:15 smithi184 ceph-mon[34509]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:17 smithi028 ceph-mon[30119]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:17.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:17 smithi165 ceph-mon[34540]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:17.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:17 smithi184 ceph-mon[34509]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:19.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:19 smithi028 ceph-mon[30119]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:19.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:19 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:19.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:19 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:19.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:19 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:19.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:19 smithi165 ceph-mon[34540]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:19.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:19 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:19.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:19 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:19.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:19 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:19.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:19 smithi184 ceph-mon[34509]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:19.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:19 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:19.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:19 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:19.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:19 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:21.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:21 smithi028 ceph-mon[30119]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.525+0000 7f3b5bfff700 1 -- 172.21.15.28:0/2506805172 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3b74062760 con 0x7f3b5c039330 2024-05-09T08:18:21.528 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3b5c039330 msgr2=0x7f3b5c03b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3b5c039330 0x7f3b5c03b7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009680 tx=0x7f3b68009260 comp rx=0 tx=0).stop 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b74102e90 msgr2=0x7f3b741985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b74102e90 0x7f3b741985b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400b070 tx=0x7f3b64004c80 comp rx=0 tx=0).stop 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 shutdown_connections 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3b5c039330 0x7f3b5c03b7f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b741080f0 0x7f3b7419fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b74102e90 0x7f3b741985b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 --2- 172.21.15.28:0/2506805172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b741057c0 0x7f3b74198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 >> 172.21.15.28:0/2506805172 conn(0x7f3b740fc840 msgr2=0x7f3b741037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:18:21.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 shutdown_connections 2024-05-09T08:18:21.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:18:21.528+0000 7f3b79774700 1 -- 172.21.15.28:0/2506805172 wait complete. 2024-05-09T08:18:21.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:21 smithi165 ceph-mon[34540]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:21.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:21 smithi184 ceph-mon[34509]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:22.217 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.0.service 2024-05-09T08:18:22.259 INFO:tasks.cephadm:Deploying osd.1 on smithi165 with /dev/vg_nvme/lv_4... 2024-05-09T08:18:22.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T08:18:22.502 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:18:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:22 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:22 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:22 smithi165 ceph-mon[34540]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:22 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:22 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:22 smithi165 ceph-mon[34540]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T08:18:22.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:22 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:22 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:22 smithi028 ceph-mon[30119]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:22.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:22 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:22 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:22 smithi028 ceph-mon[30119]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T08:18:22.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:22 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:22 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:22 smithi184 ceph-mon[34509]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:22.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:22 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:22 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:22.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:22 smithi184 ceph-mon[34509]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T08:18:23.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:23 smithi165 ceph-mon[34540]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T08:18:23.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:23 smithi165 ceph-mon[34540]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T08:18:23.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:23 smithi165 ceph-mon[34540]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:18:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:23.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:23 smithi184 ceph-mon[34509]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T08:18:23.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:23 smithi184 ceph-mon[34509]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T08:18:23.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:23 smithi184 ceph-mon[34509]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:18:23.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:23.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:23 smithi028 ceph-mon[30119]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T08:18:23.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:23 smithi028 ceph-mon[30119]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T08:18:23.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:23 smithi028 ceph-mon[30119]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:18:23.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:24.185 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:18:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0[42216]: 2024-05-09T08:18:24.079+0000 7f735aff5700 -1 osd.0 0 waiting for initial osdmap 2024-05-09T08:18:24.185 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:18:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0[42216]: 2024-05-09T08:18:24.082+0000 7f73525df700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:18:24.254 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T08:18:24.254 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T08:18:24.254 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-05-09T08:18:24.255 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-05-09T08:18:24.255 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114032 s, 920 MB/s 2024-05-09T08:18:24.255 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-05-09T08:18:24.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:24 smithi165 ceph-mon[34540]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:24.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:24 smithi165 ceph-mon[34540]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:18:24.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:24 smithi165 ceph-mon[34540]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T08:18:24.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:24 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:24.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:24 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:24.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:24 smithi165 ceph-mon[34540]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' 2024-05-09T08:18:24.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:24 smithi184 ceph-mon[34509]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:24.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:24 smithi184 ceph-mon[34509]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:18:24.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:24 smithi184 ceph-mon[34509]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T08:18:24.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:24 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:24.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:24 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:24.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:24 smithi184 ceph-mon[34509]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' 2024-05-09T08:18:24.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:24 smithi028 ceph-mon[30119]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:24.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:24 smithi028 ceph-mon[30119]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:18:24.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:24 smithi028 ceph-mon[30119]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T08:18:24.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:24 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:24.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:24 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:24.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:24 smithi028 ceph-mon[30119]: from='osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731]' entity='osd.0' 2024-05-09T08:18:24.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_4 2024-05-09T08:18:25.152 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:18:25.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:18:25.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:18:25.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:25.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] boot 2024-05-09T08:18:25.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T08:18:25.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:25.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi028 2024-05-09T08:18:25.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:18:25.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi028 to 17093M 2024-05-09T08:18:25.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:25.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:25.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:25.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:18:25.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:18:25.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:25.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] boot 2024-05-09T08:18:25.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T08:18:25.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:25.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi028 2024-05-09T08:18:25.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:18:25.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi028 to 17093M 2024-05-09T08:18:25.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:25.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:25.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:25.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:18:25.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:18:25.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:25.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: osd.0 [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] boot 2024-05-09T08:18:25.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi028 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi028 to 17093M 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:25.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:26.694 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.689+0000 7fa12df18700 1 -- 172.21.15.165:0/527027151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 msgr2=0x7fa128105260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:26.694 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.689+0000 7fa12df18700 1 --2- 172.21.15.165:0/527027151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 0x7fa128105260 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa11800b0a0 tx=0x7fa11800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:18:26.694 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.689+0000 7fa12df18700 1 -- 172.21.15.165:0/527027151 shutdown_connections 2024-05-09T08:18:26.694 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.689+0000 7fa12df18700 1 --2- 172.21.15.165:0/527027151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1281080d0 0x7fa12810a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:26.694 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.689+0000 7fa12df18700 1 --2- 172.21.15.165:0/527027151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 0x7fa128105260 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.689+0000 7fa12df18700 1 --2- 172.21.15.165:0/527027151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1281057a0 0x7fa128107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.689+0000 7fa12df18700 1 -- 172.21.15.165:0/527027151 >> 172.21.15.165:0/527027151 conn(0x7fa1280fc820 msgr2=0x7fa1280fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.689+0000 7fa12df18700 1 -- 172.21.15.165:0/527027151 shutdown_connections 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.689+0000 7fa12df18700 1 -- 172.21.15.165:0/527027151 wait complete. 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa12df18700 1 Processor -- start 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa12df18700 1 -- start start 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa12df18700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 0x7fa1281985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa12df18700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1281057a0 0x7fa128198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa12df18700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1281080d0 0x7fa12819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:26.695 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa12df18700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa12810df90 con 0x7fa1281080d0 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa12df18700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa12810de10 con 0x7fa128102e70 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa12df18700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa12810e110 con 0x7fa1281057a0 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 0x7fa1281985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 0x7fa1281985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:40168/0 (socket says 172.21.15.165:40168) 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.690+0000 7fa1277fe700 1 -- 172.21.15.165:0/1510225382 learned_addr learned my addr 172.21.15.165:0/1510225382 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa127fff700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1281080d0 0x7fa12819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa126ffd700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1281057a0 0x7fa128198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa1277fe700 1 -- 172.21.15.165:0/1510225382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1281057a0 msgr2=0x7fa128198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa1277fe700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1281057a0 0x7fa128198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa1277fe700 1 -- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1281080d0 msgr2=0x7fa12819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa1277fe700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1281080d0 0x7fa12819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa1277fe700 1 -- 172.21.15.165:0/1510225382 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1281a0240 con 0x7fa128102e70 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa1277fe700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 0x7fa1281985e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa118004cf0 tx=0x7fa118004d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:18:26.696 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa124ff9700 1 -- 172.21.15.165:0/1510225382 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118021070 con 0x7fa128102e70 2024-05-09T08:18:26.697 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa124ff9700 1 -- 172.21.15.165:0/1510225382 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa118007b80 con 0x7fa128102e70 2024-05-09T08:18:26.697 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1281a04d0 con 0x7fa128102e70 2024-05-09T08:18:26.697 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa124ff9700 1 -- 172.21.15.165:0/1510225382 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118005330 con 0x7fa128102e70 2024-05-09T08:18:26.697 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.691+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1281a0980 con 0x7fa128102e70 2024-05-09T08:18:26.697 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.692+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa128069010 con 0x7fa128102e70 2024-05-09T08:18:26.698 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.693+0000 7fa124ff9700 1 -- 172.21.15.165:0/1510225382 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fa118024070 con 0x7fa128102e70 2024-05-09T08:18:26.698 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.693+0000 7fa124ff9700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1100a2b90 0x7fa1100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:26.698 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.694+0000 7fa124ff9700 1 -- 172.21.15.165:0/1510225382 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fa11800f020 con 0x7fa128102e70 2024-05-09T08:18:26.698 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.694+0000 7fa126ffd700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1100a2b90 0x7fa1100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:26.698 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.694+0000 7fa126ffd700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1100a2b90 0x7fa1100a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009bf0 tx=0x7fa11c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:18:26.700 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.698+0000 7fa124ff9700 1 -- 172.21.15.165:0/1510225382 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa11802b020 con 0x7fa128102e70 2024-05-09T08:18:26.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:26 smithi165 ceph-mon[34540]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:26.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:26 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:26.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:26 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:26.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:26 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:26.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:26 smithi165 ceph-mon[34540]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T08:18:26.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:26 smithi184 ceph-mon[34509]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:26.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:26 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:26.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:26 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:26.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:26 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:26.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:26 smithi184 ceph-mon[34509]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T08:18:26.903 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:26.901+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa128062820 con 0x7fa1100a2b90 2024-05-09T08:18:26.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:26 smithi028 ceph-mon[30119]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:18:26.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:26 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:26.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:26 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:26.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:26 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:26.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:26 smithi028 ceph-mon[30119]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T08:18:27.755 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:27 smithi165 ceph-mon[34540]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:18:27.756 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:18:27.756 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:18:27.756 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:27.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:27 smithi028 ceph-mon[30119]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:18:27.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:18:27.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:18:27.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:27.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:27 smithi184 ceph-mon[34509]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:18:27.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:18:27.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:18:27.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:28.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:28 smithi165 ceph-mon[34540]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:28.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:28 smithi184 ceph-mon[34509]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:28.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:28 smithi028 ceph-mon[30119]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:29.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:29 smithi165 ceph-mon[34540]: from='client.? 172.21.15.165:0/3432508013' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a57c6ac-92fa-4d2c-9ce8-ec3906cab422"}]: dispatch 2024-05-09T08:18:29.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:29 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a57c6ac-92fa-4d2c-9ce8-ec3906cab422"}]: dispatch 2024-05-09T08:18:29.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:29 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a57c6ac-92fa-4d2c-9ce8-ec3906cab422"}]': finished 2024-05-09T08:18:29.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:29 smithi165 ceph-mon[34540]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T08:18:29.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:29.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:29 smithi184 ceph-mon[34509]: from='client.? 172.21.15.165:0/3432508013' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a57c6ac-92fa-4d2c-9ce8-ec3906cab422"}]: dispatch 2024-05-09T08:18:29.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:29 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a57c6ac-92fa-4d2c-9ce8-ec3906cab422"}]: dispatch 2024-05-09T08:18:29.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:29 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a57c6ac-92fa-4d2c-9ce8-ec3906cab422"}]': finished 2024-05-09T08:18:29.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:29 smithi184 ceph-mon[34509]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T08:18:29.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:29.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:29 smithi028 ceph-mon[30119]: from='client.? 172.21.15.165:0/3432508013' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a57c6ac-92fa-4d2c-9ce8-ec3906cab422"}]: dispatch 2024-05-09T08:18:29.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:29 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a57c6ac-92fa-4d2c-9ce8-ec3906cab422"}]: dispatch 2024-05-09T08:18:29.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:29 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a57c6ac-92fa-4d2c-9ce8-ec3906cab422"}]': finished 2024-05-09T08:18:29.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:29 smithi028 ceph-mon[30119]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T08:18:29.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:30.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:30 smithi184 ceph-mon[34509]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:30.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:30 smithi184 ceph-mon[34509]: from='client.? 172.21.15.165:0/2347584487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:18:30.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:30 smithi028 ceph-mon[30119]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:30.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:30 smithi028 ceph-mon[30119]: from='client.? 172.21.15.165:0/2347584487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:18:31.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:30 smithi165 ceph-mon[34540]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:31.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:30 smithi165 ceph-mon[34540]: from='client.? 172.21.15.165:0/2347584487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:18:32.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:32 smithi184 ceph-mon[34509]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:32.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:32 smithi028 ceph-mon[30119]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:33.077 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:32 smithi165 ceph-mon[34540]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:34.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:34 smithi165 ceph-mon[34540]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:34.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:34 smithi028 ceph-mon[30119]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:35.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:34 smithi184 ceph-mon[34509]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:36.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:36 smithi165 ceph-mon[34540]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:36.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T08:18:36.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:36.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:36 smithi165 ceph-mon[34540]: Deploying daemon osd.1 on smithi165 2024-05-09T08:18:36.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:36 smithi028 ceph-mon[30119]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:36.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T08:18:36.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:36.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:36 smithi028 ceph-mon[30119]: Deploying daemon osd.1 on smithi165 2024-05-09T08:18:37.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:36 smithi184 ceph-mon[34509]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:37.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T08:18:37.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:37.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:36 smithi184 ceph-mon[34509]: Deploying daemon osd.1 on smithi165 2024-05-09T08:18:38.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:38 smithi165 ceph-mon[34540]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:38.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:38 smithi028 ceph-mon[30119]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:39.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:38 smithi184 ceph-mon[34509]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:40.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:40 smithi165 ceph-mon[34540]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:40 smithi028 ceph-mon[30119]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:41.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:40 smithi184 ceph-mon[34509]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:42.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:42.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:42.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:42.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:42 smithi165 ceph-mon[34540]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:42.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:42.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:42.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:42.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:42 smithi184 ceph-mon[34509]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:42.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:42.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:42.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:42.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:42 smithi028 ceph-mon[30119]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:44.969 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.966+0000 7fa124ff9700 1 -- 172.21.15.165:0/1510225382 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa128062820 con 0x7fa1100a2b90 2024-05-09T08:18:44.970 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 1 on host 'smithi165' 2024-05-09T08:18:44.971 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1100a2b90 msgr2=0x7fa1100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:44.971 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1100a2b90 0x7fa1100a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009bf0 tx=0x7fa11c009260 comp rx=0 tx=0).stop 2024-05-09T08:18:44.971 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 msgr2=0x7fa1281985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:44.971 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 0x7fa1281985e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa118004cf0 tx=0x7fa118004d20 comp rx=0 tx=0).stop 2024-05-09T08:18:44.972 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 shutdown_connections 2024-05-09T08:18:44.972 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1100a2b90 0x7fa1100a5050 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:44.972 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1281080d0 0x7fa12819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:44.972 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa128102e70 0x7fa1281985e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:44.972 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 --2- 172.21.15.165:0/1510225382 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1281057a0 0x7fa128198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:44.972 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 >> 172.21.15.165:0/1510225382 conn(0x7fa1280fc820 msgr2=0x7fa128103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:18:44.972 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.968+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 shutdown_connections 2024-05-09T08:18:44.972 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:18:44.969+0000 7fa12df18700 1 -- 172.21.15.165:0/1510225382 wait complete. 2024-05-09T08:18:45.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:44 smithi028 ceph-mon[30119]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:45.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:45.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:45.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:45.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:45.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:45.271 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:44 smithi165 ceph-mon[34540]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:45.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:45.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:45.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:45.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:45.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:45.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:44 smithi184 ceph-mon[34509]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:45.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:45.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:45.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:45.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:45.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:45.582 DEBUG:teuthology.orchestra.run.smithi165:osd.1> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.1.service 2024-05-09T08:18:45.585 INFO:tasks.cephadm:Deploying osd.2 on smithi184 with /dev/vg_nvme/lv_4... 2024-05-09T08:18:45.585 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T08:18:45.830 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:18:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:45 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:45 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:45 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:45 smithi165 ceph-mon[34540]: from='osd.1 [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:18:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:45 smithi165 ceph-mon[34540]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:18:46.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:45 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:46.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:45 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:46.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:45 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:46.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:45 smithi028 ceph-mon[30119]: from='osd.1 [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:18:46.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:45 smithi028 ceph-mon[30119]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:18:46.233 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:45 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:46.233 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:45 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:18:46.233 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:45 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:46.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:45 smithi184 ceph-mon[34509]: from='osd.1 [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:18:46.234 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:45 smithi184 ceph-mon[34509]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:18:46.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:46 smithi184 ceph-mon[34509]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:46.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:46 smithi184 ceph-mon[34509]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T08:18:46.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:46 smithi184 ceph-mon[34509]: from='osd.1 [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:18:46.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:46 smithi184 ceph-mon[34509]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T08:18:46.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:46 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:46.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:46 smithi184 ceph-mon[34509]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:18:47.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:46 smithi165 ceph-mon[34540]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:47.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:46 smithi165 ceph-mon[34540]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T08:18:47.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:46 smithi165 ceph-mon[34540]: from='osd.1 [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:18:47.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:46 smithi165 ceph-mon[34540]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T08:18:47.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:46 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:47.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:46 smithi165 ceph-mon[34540]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:18:47.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:46 smithi028 ceph-mon[30119]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:47.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:46 smithi028 ceph-mon[30119]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T08:18:47.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:46 smithi028 ceph-mon[30119]: from='osd.1 [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:18:47.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:46 smithi028 ceph-mon[30119]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T08:18:47.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:46 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:47.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:46 smithi028 ceph-mon[30119]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:18:47.532 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T08:18:47.532 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T08:18:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-05-09T08:18:47.532 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-05-09T08:18:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123893 s, 846 MB/s 2024-05-09T08:18:47.533 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-05-09T08:18:47.604 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:18:47 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1[39124]: 2024-05-09T08:18:47.432+0000 7f994252a700 -1 osd.1 0 waiting for initial osdmap 2024-05-09T08:18:47.604 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:18:47 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1[39124]: 2024-05-09T08:18:47.437+0000 7f9939b14700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:18:48.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:47 smithi165 ceph-mon[34540]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:18:48.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:47 smithi165 ceph-mon[34540]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T08:18:48.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:47 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:48.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:47 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:48.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:47 smithi165 ceph-mon[34540]: from='osd.1 ' entity='osd.1' 2024-05-09T08:18:48.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_4 2024-05-09T08:18:48.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:47 smithi184 ceph-mon[34509]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:18:48.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:47 smithi184 ceph-mon[34509]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T08:18:48.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:48.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:48.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:47 smithi184 ceph-mon[34509]: from='osd.1 ' entity='osd.1' 2024-05-09T08:18:48.426 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:18:48.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:47 smithi028 ceph-mon[30119]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:18:48.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:47 smithi028 ceph-mon[30119]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T08:18:48.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:48.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:48.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:47 smithi028 ceph-mon[30119]: from='osd.1 ' entity='osd.1' 2024-05-09T08:18:49.074 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:18:49.074 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:18:49.074 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:49.074 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:49.074 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: osd.1 [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] boot 2024-05-09T08:18:49.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T08:18:49.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:49.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:18:49.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:49.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:49.075 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:18:49.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:18:49.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:49.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:49.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: osd.1 [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] boot 2024-05-09T08:18:49.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T08:18:49.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:49.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:18:49.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:49.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:49.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:18:49.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:18:49.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:18:49.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:49.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: osd.1 [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] boot 2024-05-09T08:18:49.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T08:18:49.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:18:49.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:18:49.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:49.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:18:49.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:18:49.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/547009235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc100a10 msgr2=0x7ff4dc100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:49.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/547009235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc100a10 0x7ff4dc100e30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc009990 tx=0x7ff4cc009ca0 comp rx=0 tx=0).stop 2024-05-09T08:18:49.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/547009235 shutdown_connections 2024-05-09T08:18:49.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/547009235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc100a10 0x7ff4dc100e30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:49.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/547009235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4dc1025b0 0x7ff4dc108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:49.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/547009235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc101c10 0x7ff4dc102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:49.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/547009235 >> 172.21.15.184:0/547009235 conn(0x7ff4dc0fbf90 msgr2=0x7ff4dc0fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/547009235 shutdown_connections 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/547009235 wait complete. 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.803+0000 7ff4e2d7a700 1 Processor -- start 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e2d7a700 1 -- start start 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e2d7a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc100a10 0x7ff4dc1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e2d7a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4dc101c10 0x7ff4dc198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e2d7a700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc1025b0 0x7ff4dc19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e2d7a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4dc10cea0 con 0x7ff4dc100a10 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e2d7a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff4dc10cd20 con 0x7ff4dc101c10 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e2d7a700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff4dc10d020 con 0x7ff4dc1025b0 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e1317700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc1025b0 0x7ff4dc19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e1317700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc1025b0 0x7ff4dc19fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:38996/0 (socket says 172.21.15.184:38996) 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e1317700 1 -- 172.21.15.184:0/3746549379 learned_addr learned my addr 172.21.15.184:0/3746549379 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-05-09T08:18:49.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e0b16700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc100a10 0x7ff4dc1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:49.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4dbfff700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4dc101c10 0x7ff4dc198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:49.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e1317700 1 -- 172.21.15.184:0/3746549379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4dc101c10 msgr2=0x7ff4dc198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:49.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e1317700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4dc101c10 0x7ff4dc198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:49.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e1317700 1 -- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc100a10 msgr2=0x7ff4dc1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:18:49.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e1317700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc100a10 0x7ff4dc1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:18:49.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e1317700 1 -- 172.21.15.184:0/3746549379 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4dc1a0230 con 0x7ff4dc1025b0 2024-05-09T08:18:49.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e1317700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc1025b0 0x7ff4dc19fb90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4007f10 tx=0x7ff4d400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:18:49.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4d9ffb700 1 -- 172.21.15.184:0/3746549379 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4004bb0 con 0x7ff4dc1025b0 2024-05-09T08:18:49.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4d9ffb700 1 -- 172.21.15.184:0/3746549379 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d4004d50 con 0x7ff4dc1025b0 2024-05-09T08:18:49.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4dc1a0520 con 0x7ff4dc1025b0 2024-05-09T08:18:49.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.804+0000 7ff4d9ffb700 1 -- 172.21.15.184:0/3746549379 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4005a60 con 0x7ff4dc1025b0 2024-05-09T08:18:49.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.805+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4dc1a0a10 con 0x7ff4dc1025b0 2024-05-09T08:18:49.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.806+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4dc069010 con 0x7ff4dc1025b0 2024-05-09T08:18:49.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.807+0000 7ff4d9ffb700 1 -- 172.21.15.184:0/3746549379 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7ff4d401a020 con 0x7ff4dc1025b0 2024-05-09T08:18:49.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.807+0000 7ff4d9ffb700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff4c40a2b90 0x7ff4c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:18:49.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.808+0000 7ff4d9ffb700 1 -- 172.21.15.184:0/3746549379 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7ff4d40ccfd0 con 0x7ff4dc1025b0 2024-05-09T08:18:49.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.808+0000 7ff4e0b16700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff4c40a2b90 0x7ff4c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:18:49.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.808+0000 7ff4e0b16700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff4c40a2b90 0x7ff4c40a5050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc000c00 tx=0x7ff4cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:18:49.814 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:49.812+0000 7ff4d9ffb700 1 -- 172.21.15.184:0/3746549379 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff4d4010070 con 0x7ff4dc1025b0 2024-05-09T08:18:50.018 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:18:50.016+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff4dc0fe170 con 0x7ff4c40a2b90 2024-05-09T08:18:50.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:49 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi165 2024-05-09T08:18:50.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:49 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi165 to 17088M 2024-05-09T08:18:50.018 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:49 smithi184 ceph-mon[34509]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T08:18:50.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:49 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi165 2024-05-09T08:18:50.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:49 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi165 to 17088M 2024-05-09T08:18:50.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:49 smithi165 ceph-mon[34540]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T08:18:50.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:49 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi165 2024-05-09T08:18:50.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:49 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi165 to 17088M 2024-05-09T08:18:50.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:49 smithi028 ceph-mon[30119]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T08:18:51.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:50 smithi184 ceph-mon[34509]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:51.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:50 smithi184 ceph-mon[34509]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:18:51.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:18:51.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:18:51.273 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:51.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:50 smithi165 ceph-mon[34540]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:51.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:50 smithi165 ceph-mon[34540]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:18:51.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:18:51.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:18:51.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:51.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:50 smithi028 ceph-mon[30119]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:51.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:50 smithi028 ceph-mon[30119]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:18:51.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:18:51.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:18:51.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:18:53.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:52 smithi165 ceph-mon[34540]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:53.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:52 smithi165 ceph-mon[34540]: from='client.? 172.21.15.184:0/3165359700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9ab7ba3-b176-4560-a3d4-80a438a4efda"}]: dispatch 2024-05-09T08:18:53.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:52 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9ab7ba3-b176-4560-a3d4-80a438a4efda"}]: dispatch 2024-05-09T08:18:53.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:52 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9ab7ba3-b176-4560-a3d4-80a438a4efda"}]': finished 2024-05-09T08:18:53.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:52 smithi165 ceph-mon[34540]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T08:18:53.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:52 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:18:53.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:52 smithi184 ceph-mon[34509]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:53.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:52 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/3165359700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9ab7ba3-b176-4560-a3d4-80a438a4efda"}]: dispatch 2024-05-09T08:18:53.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:52 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9ab7ba3-b176-4560-a3d4-80a438a4efda"}]: dispatch 2024-05-09T08:18:53.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:52 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9ab7ba3-b176-4560-a3d4-80a438a4efda"}]': finished 2024-05-09T08:18:53.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:52 smithi184 ceph-mon[34509]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T08:18:53.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:52 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:18:53.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:52 smithi028 ceph-mon[30119]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:53.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:52 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/3165359700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9ab7ba3-b176-4560-a3d4-80a438a4efda"}]: dispatch 2024-05-09T08:18:53.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:52 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9ab7ba3-b176-4560-a3d4-80a438a4efda"}]: dispatch 2024-05-09T08:18:53.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:52 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9ab7ba3-b176-4560-a3d4-80a438a4efda"}]': finished 2024-05-09T08:18:53.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:52 smithi028 ceph-mon[30119]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T08:18:53.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:52 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:18:54.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:53 smithi165 ceph-mon[34540]: from='client.? 172.21.15.184:0/784470257' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:18:54.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:53 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/784470257' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:18:54.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:53 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/784470257' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:18:55.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:54 smithi165 ceph-mon[34540]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:55.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:54 smithi184 ceph-mon[34509]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:54 smithi028 ceph-mon[30119]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:56.975 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:56 smithi184 ceph-mon[34509]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:57.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:56 smithi028 ceph-mon[30119]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:57.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:56 smithi165 ceph-mon[34540]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:58 smithi184 ceph-mon[34509]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:59.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:58 smithi165 ceph-mon[34540]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:18:59.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:58 smithi028 ceph-mon[30119]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:00.255 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:59 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T08:19:00.255 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:18:59 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:00.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:59 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T08:19:00.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:18:59 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:00.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:59 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T08:19:00.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:18:59 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:01.333 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:00 smithi184 ceph-mon[34509]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:01.333 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:00 smithi184 ceph-mon[34509]: Deploying daemon osd.2 on smithi184 2024-05-09T08:19:01.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:00 smithi165 ceph-mon[34540]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:01.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:00 smithi165 ceph-mon[34540]: Deploying daemon osd.2 on smithi184 2024-05-09T08:19:01.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:00 smithi028 ceph-mon[30119]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:01.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:00 smithi028 ceph-mon[30119]: Deploying daemon osd.2 on smithi184 2024-05-09T08:19:03.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:02 smithi184 ceph-mon[34509]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:03.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:02 smithi165 ceph-mon[34540]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:03.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:02 smithi028 ceph-mon[30119]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:05.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:04 smithi184 ceph-mon[34509]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:05.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:05.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:05.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:05.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:04 smithi165 ceph-mon[34540]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:05.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:05.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:05.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:04 smithi028 ceph-mon[30119]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:07.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:06 smithi165 ceph-mon[34540]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:07.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:06 smithi184 ceph-mon[34509]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:07.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:06 smithi028 ceph-mon[30119]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:08.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:08 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:08.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:08 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:08.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:08 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:08.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:08 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:08.439 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:08 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:08.439 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.435+0000 7ff4d9ffb700 1 -- 172.21.15.184:0/3746549379 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff4dc0fe170 con 0x7ff4c40a2b90 2024-05-09T08:19:08.439 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 2 on host 'smithi184' 2024-05-09T08:19:08.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff4c40a2b90 msgr2=0x7ff4c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:08.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff4c40a2b90 0x7ff4c40a5050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc000c00 tx=0x7ff4cc005fb0 comp rx=0 tx=0).stop 2024-05-09T08:19:08.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc1025b0 msgr2=0x7ff4dc19fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:08.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc1025b0 0x7ff4dc19fb90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4007f10 tx=0x7ff4d400d3b0 comp rx=0 tx=0).stop 2024-05-09T08:19:08.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 shutdown_connections 2024-05-09T08:19:08.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff4c40a2b90 0x7ff4c40a5050 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4dc1025b0 0x7ff4dc19fb90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc100a10 0x7ff4dc1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 --2- 172.21.15.184:0/3746549379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4dc101c10 0x7ff4dc198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 >> 172.21.15.184:0/3746549379 conn(0x7ff4dc0fbf90 msgr2=0x7ff4dc0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:19:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 shutdown_connections 2024-05-09T08:19:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:19:08.438+0000 7ff4e2d7a700 1 -- 172.21.15.184:0/3746549379 wait complete. 2024-05-09T08:19:08.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:08 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:08.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:08 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:08.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:08 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:08.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:08 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:08.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:08 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:08.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:08 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:08.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:08 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:08.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:08 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:08.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:08 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:08.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:08 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:09.017 DEBUG:teuthology.orchestra.run.smithi184:osd.2> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.2.service 2024-05-09T08:19:09.059 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-05-09T08:19:09.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T08:19:09.320 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:19:09.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:09 smithi184 ceph-mon[34509]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:09.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:09 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:09.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:09 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:09.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:09 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:09.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:09 smithi184 ceph-mon[34509]: from='osd.2 [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:19:09.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:09 smithi184 ceph-mon[34509]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:19:09.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:09 smithi028 ceph-mon[30119]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:09.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:09 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:09.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:09 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:09.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:09 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:09.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:09 smithi028 ceph-mon[30119]: from='osd.2 [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:19:09.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:09 smithi028 ceph-mon[30119]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:19:09.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:09 smithi165 ceph-mon[34540]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:09.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:09 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:09.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:09 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:09.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:09 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:09.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:09 smithi165 ceph-mon[34540]: from='osd.2 [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:19:09.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:09 smithi165 ceph-mon[34540]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:19:10.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:10 smithi028 ceph-mon[30119]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T08:19:10.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:10 smithi028 ceph-mon[30119]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T08:19:10.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:10 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:10.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:10 smithi028 ceph-mon[30119]: from='osd.2 [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:19:10.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:10 smithi028 ceph-mon[30119]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:19:10.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:10 smithi165 ceph-mon[34540]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T08:19:10.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:10 smithi165 ceph-mon[34540]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T08:19:10.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:10 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:10.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:10 smithi165 ceph-mon[34540]: from='osd.2 [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:19:10.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:10 smithi165 ceph-mon[34540]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:19:10.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:10 smithi184 ceph-mon[34509]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T08:19:10.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:10 smithi184 ceph-mon[34509]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T08:19:10.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:10 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:10.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:10 smithi184 ceph-mon[34509]: from='osd.2 [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:19:10.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:10 smithi184 ceph-mon[34509]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:19:10.860 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2[39523]: 2024-05-09T08:19:10.430+0000 7f843891c700 -1 osd.2 0 waiting for initial osdmap 2024-05-09T08:19:10.860 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2[39523]: 2024-05-09T08:19:10.434+0000 7f843470f700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:19:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T08:19:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T08:19:11.343 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-05-09T08:19:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-05-09T08:19:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0105296 s, 996 MB/s 2024-05-09T08:19:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-05-09T08:19:11.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:11 smithi184 ceph-mon[34509]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:11.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:11 smithi184 ceph-mon[34509]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:19:11.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:11 smithi184 ceph-mon[34509]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T08:19:11.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:11.570 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:11.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:11 smithi028 ceph-mon[30119]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:11.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:11 smithi028 ceph-mon[30119]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:19:11.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:11 smithi028 ceph-mon[30119]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T08:19:11.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:11.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:11.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:11 smithi165 ceph-mon[34540]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:11.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:11 smithi165 ceph-mon[34540]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:19:11.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:11 smithi165 ceph-mon[34540]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T08:19:11.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:11.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:12.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-05-09T08:19:12.282 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: osd.2 [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] boot 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi184 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:19:12.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi184 to 17088M 2024-05-09T08:19:12.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:12.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:12.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: osd.2 [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] boot 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi184 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:19:12.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi184 to 17088M 2024-05-09T08:19:12.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:12.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:12.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: osd.2 [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] boot 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi184 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:19:12.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi184 to 17088M 2024-05-09T08:19:12.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:12.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:12.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:12.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:13 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T08:19:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:13 smithi028 ceph-mon[30119]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T08:19:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:13 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:19:13.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:13 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T08:19:13.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:13 smithi165 ceph-mon[34540]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T08:19:13.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:13 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:19:13.861 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:13 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T08:19:13.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:13 smithi184 ceph-mon[34509]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T08:19:13.862 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:13 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:19:13.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.928+0000 7fd80b8d3700 1 -- 172.21.15.28:0/2703887606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804102e20 msgr2=0x7fd8041096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.928+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/2703887606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804102e20 0x7fd8041096d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00b0d0 tx=0x7fd7fc00b3e0 comp rx=0 tx=0).stop 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.928+0000 7fd80b8d3700 1 -- 172.21.15.28:0/2703887606 shutdown_connections 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.928+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/2703887606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd804101280 0x7fd8041016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.928+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/2703887606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804102e20 0x7fd8041096d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.928+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/2703887606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd804102480 0x7fd8041028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.928+0000 7fd80b8d3700 1 -- 172.21.15.28:0/2703887606 >> 172.21.15.28:0/2703887606 conn(0x7fd8040fc820 msgr2=0x7fd8040fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.928+0000 7fd80b8d3700 1 -- 172.21.15.28:0/2703887606 shutdown_connections 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.928+0000 7fd80b8d3700 1 -- 172.21.15.28:0/2703887606 wait complete. 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80b8d3700 1 Processor -- start 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80b8d3700 1 -- start start 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80b8d3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804101280 0x7fd80419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:19:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80b8d3700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd804102480 0x7fd80419ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80b8d3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd804102e20 0x7fd8041a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80b8d3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd80410cee0 con 0x7fd804101280 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80b8d3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd80410cd60 con 0x7fd804102e20 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80b8d3700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd80410d060 con 0x7fd804102480 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80966f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804101280 0x7fd80419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.929+0000 7fd80966f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804101280 0x7fd80419a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51964/0 (socket says 172.21.15.28:51964) 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd80966f700 1 -- 172.21.15.28:0/1704997857 learned_addr learned my addr 172.21.15.28:0/1704997857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd808e6e700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd804102480 0x7fd80419ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd809e70700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd804102e20 0x7fd8041a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd80966f700 1 -- 172.21.15.28:0/1704997857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd804102480 msgr2=0x7fd80419ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:13.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd80966f700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd804102480 0x7fd80419ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd80966f700 1 -- 172.21.15.28:0/1704997857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd804102e20 msgr2=0x7fd8041a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd80966f700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd804102e20 0x7fd8041a1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd80966f700 1 -- 172.21.15.28:0/1704997857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8041a2470 con 0x7fd804101280 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd808e6e700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd804102480 0x7fd80419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd80966f700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804101280 0x7fd80419a810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd80000b530 tx=0x7fd80000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd7f67fc700 1 -- 172.21.15.28:0/1704997857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800015020 con 0x7fd804101280 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd7f67fc700 1 -- 172.21.15.28:0/1704997857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd800011d10 con 0x7fd804101280 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8041a2760 con 0x7fd804101280 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.930+0000 7fd7f67fc700 1 -- 172.21.15.28:0/1704997857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80000f2e0 con 0x7fd804101280 2024-05-09T08:19:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.931+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8041a2d70 con 0x7fd804101280 2024-05-09T08:19:13.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.932+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd804069010 con 0x7fd804101280 2024-05-09T08:19:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.933+0000 7fd7f67fc700 1 -- 172.21.15.28:0/1704997857 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fd800018020 con 0x7fd804101280 2024-05-09T08:19:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.933+0000 7fd7f67fc700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:19:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.934+0000 7fd808e6e700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:19:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.934+0000 7fd7f67fc700 1 -- 172.21.15.28:0/1704997857 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7fd8000cd6a0 con 0x7fd804101280 2024-05-09T08:19:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.934+0000 7fd808e6e700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8005950 tx=0x7fd7f8005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:19:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:13.937+0000 7fd7f67fc700 1 -- 172.21.15.28:0/1704997857 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd80001f020 con 0x7fd804101280 2024-05-09T08:19:14.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:14.159+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd8040fe9d0 con 0x7fd7f00a2c10 2024-05-09T08:19:14.434 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:19:13 smithi028 sudo[46428]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:19:14.434 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:19:13 smithi028 sudo[46428]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:14.435 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:19:13 smithi028 sudo[46428]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:14.435 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:19:14 smithi028 sudo[46428]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:14.460+0000 7fd7f67fc700 1 -- 172.21.15.28:0/1704997857 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd800097a20 con 0x7fd804101280 2024-05-09T08:19:14.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:19:14.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:14 smithi165 ceph-mon[34540]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T08:19:14.723 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:14 smithi165 ceph-mon[34540]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:14.723 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:14 smithi165 ceph-mon[34540]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:19:14.723 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:19:14.723 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:19:14.723 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:19:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:14 smithi028 ceph-mon[30119]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T08:19:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:14 smithi028 ceph-mon[30119]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:14 smithi028 ceph-mon[30119]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:19:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:19:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:19:14.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:14.812 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:19:14 smithi028 sudo[46498]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:19:14.812 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:19:14 smithi028 sudo[46498]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:14.812 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:19:14 smithi028 sudo[46498]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:14.812 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:19:14 smithi028 sudo[46498]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:19:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:14 smithi184 ceph-mon[34509]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T08:19:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:14 smithi184 ceph-mon[34509]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:14 smithi184 ceph-mon[34509]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:19:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:19:14.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:19:14.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:14.976 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:19:14 smithi165 sudo[42825]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:19:14.976 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:19:14 smithi165 sudo[42825]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:14.977 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:19:14 smithi165 sudo[42825]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:14.977 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:19:14 smithi165 sudo[42825]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:15.353 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:19:14 smithi165 sudo[42828]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:19:15.353 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:19:14 smithi165 sudo[42828]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:15.353 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:19:14 smithi165 sudo[42828]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:15.354 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:19:14 smithi165 sudo[42828]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:15.499 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:15 smithi184 sudo[43573]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:19:15.500 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:15 smithi184 sudo[43573]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:15.500 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:15 smithi184 sudo[43573]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:15.500 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:15 smithi184 sudo[43573]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:15.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:15 smithi184 ceph-mon[34509]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T08:19:15.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:15 smithi184 ceph-mon[34509]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T08:19:15.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:15.500 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:15.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:15 smithi028 ceph-mon[30119]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T08:19:15.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:15 smithi028 ceph-mon[30119]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T08:19:15.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:15.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:15.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:15 smithi165 ceph-mon[34540]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T08:19:15.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:15 smithi165 ceph-mon[34540]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T08:19:15.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:15.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:15.859 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:15 smithi184 sudo[43576]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:19:15.859 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:15 smithi184 sudo[43576]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:15.859 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:15 smithi184 sudo[43576]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:15.859 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:19:15 smithi184 sudo[43576]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:16.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:15 smithi028 sudo[46595]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:19:16.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:15 smithi028 sudo[46595]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:16.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:15 smithi028 sudo[46595]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:16.130 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:15 smithi028 sudo[46595]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:16.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 sudo[46605]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:19:16.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 sudo[46605]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:16.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 sudo[46605]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:16.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 sudo[46605]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:16.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 sudo[43579]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:19:16.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 sudo[43579]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:16.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 sudo[43579]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:16.923 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 sudo[43579]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:17.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:19:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:19:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:19:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:19:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:19:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:19:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:19:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:19:17.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 sudo[43582]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:19:17.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 sudo[43582]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:17.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 sudo[43582]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:17.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 sudo[43582]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:19:17.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:19:17.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 sudo[42831]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:19:17.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 sudo[42831]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:17.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 sudo[42831]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:17.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 sudo[42831]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:17.416 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 sudo[42834]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:19:17.416 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 sudo[42834]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:19:17.416 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 sudo[42834]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:19:17.416 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 sudo[42834]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:19:17.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:17.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:19:17.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:17.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:19:17.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:19:17.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:19:17.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:19:17.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:19:17.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:19:17.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/448237631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00f5fefc-7634-4f7a-b543-8b3d86be1a5d"}]: dispatch 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/448237631' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00f5fefc-7634-4f7a-b543-8b3d86be1a5d"}]': finished 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T08:19:18.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:17 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:18.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/448237631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00f5fefc-7634-4f7a-b543-8b3d86be1a5d"}]: dispatch 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/448237631' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00f5fefc-7634-4f7a-b543-8b3d86be1a5d"}]': finished 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T08:19:18.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:18.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:19:18.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:19:18.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:19:18.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:19:18.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:19:18.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:19:18.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/448237631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00f5fefc-7634-4f7a-b543-8b3d86be1a5d"}]: dispatch 2024-05-09T08:19:18.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:19:18.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/448237631' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00f5fefc-7634-4f7a-b543-8b3d86be1a5d"}]': finished 2024-05-09T08:19:18.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T08:19:18.360 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:17 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:19.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:18 smithi028 ceph-mon[30119]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:19.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:18 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3995205718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:19:19.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:18 smithi165 ceph-mon[34540]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:19.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:18 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3995205718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:19:19.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:18 smithi184 ceph-mon[34509]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:19.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:18 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3995205718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:19:21.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:20 smithi165 ceph-mon[34540]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 7.8 KiB/s rd, 357 KiB/s wr, 24 op/s 2024-05-09T08:19:21.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:20 smithi184 ceph-mon[34509]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 7.8 KiB/s rd, 357 KiB/s wr, 24 op/s 2024-05-09T08:19:21.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:20 smithi028 ceph-mon[30119]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 7.8 KiB/s rd, 357 KiB/s wr, 24 op/s 2024-05-09T08:19:23.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:22 smithi028 ceph-mon[30119]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 256 KiB/s wr, 17 op/s 2024-05-09T08:19:23.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:22 smithi165 ceph-mon[34540]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 256 KiB/s wr, 17 op/s 2024-05-09T08:19:23.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:22 smithi184 ceph-mon[34509]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 256 KiB/s wr, 17 op/s 2024-05-09T08:19:25.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:24 smithi165 ceph-mon[34540]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 4.4 KiB/s rd, 200 KiB/s wr, 13 op/s 2024-05-09T08:19:25.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:24 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T08:19:25.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:24 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:25.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:24 smithi028 ceph-mon[30119]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 4.4 KiB/s rd, 200 KiB/s wr, 13 op/s 2024-05-09T08:19:25.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:24 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T08:19:25.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:24 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:25.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:24 smithi184 ceph-mon[34509]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 4.4 KiB/s rd, 200 KiB/s wr, 13 op/s 2024-05-09T08:19:25.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:24 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T08:19:25.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:24 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:26.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:25 smithi165 ceph-mon[34540]: Deploying daemon osd.3 on smithi028 2024-05-09T08:19:26.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:25 smithi184 ceph-mon[34509]: Deploying daemon osd.3 on smithi028 2024-05-09T08:19:26.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:25 smithi028 ceph-mon[30119]: Deploying daemon osd.3 on smithi028 2024-05-09T08:19:27.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:26 smithi165 ceph-mon[34540]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-05-09T08:19:27.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:26 smithi184 ceph-mon[34509]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-05-09T08:19:27.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:26 smithi028 ceph-mon[30119]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-05-09T08:19:29.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:28 smithi028 ceph-mon[30119]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 180 KiB/s wr, 12 op/s 2024-05-09T08:19:29.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:28 smithi165 ceph-mon[34540]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 180 KiB/s wr, 12 op/s 2024-05-09T08:19:29.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:28 smithi184 ceph-mon[34509]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 180 KiB/s wr, 12 op/s 2024-05-09T08:19:31.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:30 smithi028 ceph-mon[30119]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-05-09T08:19:31.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:31.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:31.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:30 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:31.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:30 smithi165 ceph-mon[34540]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-05-09T08:19:31.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:30 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:31.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:30 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:31.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:30 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:31.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:30 smithi184 ceph-mon[34509]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-05-09T08:19:31.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:30 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:31.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:30 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:31.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:30 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:33.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:32 smithi028 ceph-mon[30119]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:33.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:32 smithi165 ceph-mon[34540]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:33.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:32 smithi184 ceph-mon[34509]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:34.134 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-05-09T08:19:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.133+0000 7fd7f67fc700 1 -- 172.21.15.28:0/1704997857 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd8040fe9d0 con 0x7fd7f00a2c10 2024-05-09T08:19:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd7f00a2c10 msgr2=0x7fd7f00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8005950 tx=0x7fd7f8005f50 comp rx=0 tx=0).stop 2024-05-09T08:19:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804101280 msgr2=0x7fd80419a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804101280 0x7fd80419a810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd80000b530 tx=0x7fd80000b840 comp rx=0 tx=0).stop 2024-05-09T08:19:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 shutdown_connections 2024-05-09T08:19:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd804102e20 0x7fd8041a1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804101280 0x7fd80419a810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 --2- 172.21.15.28:0/1704997857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd804102480 0x7fd80419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.135+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 >> 172.21.15.28:0/1704997857 conn(0x7fd8040fc820 msgr2=0x7fd8040fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:19:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.136+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 shutdown_connections 2024-05-09T08:19:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:19:34.136+0000 7fd80b8d3700 1 -- 172.21.15.28:0/1704997857 wait complete. 2024-05-09T08:19:34.777 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.3.service 2024-05-09T08:19:34.780 INFO:tasks.cephadm:Deploying osd.4 on smithi165 with /dev/vg_nvme/lv_3... 2024-05-09T08:19:34.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T08:19:35.029 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:19:35.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:35 smithi165 ceph-mon[34540]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:35.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:35 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:35 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:35 smithi165 ceph-mon[34540]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T08:19:35.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:35 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:35 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:35 smithi028 ceph-mon[30119]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:35 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:35 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:35 smithi028 ceph-mon[30119]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T08:19:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:35 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:35 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:35 smithi184 ceph-mon[34509]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:35.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:35 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:35 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:35 smithi184 ceph-mon[34509]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T08:19:35.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:35 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:35.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:35 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:36.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:36 smithi165 ceph-mon[34540]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T08:19:36.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:36 smithi165 ceph-mon[34540]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T08:19:36.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:36 smithi165 ceph-mon[34540]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:19:36.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:36.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:36 smithi028 ceph-mon[30119]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T08:19:36.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:36 smithi028 ceph-mon[30119]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T08:19:36.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:36 smithi028 ceph-mon[30119]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:19:36.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:36.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:36 smithi184 ceph-mon[34509]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T08:19:36.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:36 smithi184 ceph-mon[34509]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T08:19:36.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:36 smithi184 ceph-mon[34509]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:19:36.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:36.802 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:19:36 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:19:36.568+0000 7f00cbf2e700 -1 osd.3 0 waiting for initial osdmap 2024-05-09T08:19:36.802 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:19:36 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:19:36.573+0000 7f00c3d19700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:19:37.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:37 smithi165 ceph-mon[34540]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:37.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:37 smithi165 ceph-mon[34540]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:19:37.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:37 smithi165 ceph-mon[34540]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T08:19:37.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:37.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:37.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:37 smithi165 ceph-mon[34540]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' 2024-05-09T08:19:37.406 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T08:19:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T08:19:37.407 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-05-09T08:19:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-05-09T08:19:37.407 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114683 s, 914 MB/s 2024-05-09T08:19:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-05-09T08:19:37.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:37 smithi028 ceph-mon[30119]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:37.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:37 smithi028 ceph-mon[30119]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:19:37.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:37 smithi028 ceph-mon[30119]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T08:19:37.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:37.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:37.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:37 smithi028 ceph-mon[30119]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' 2024-05-09T08:19:37.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:37 smithi184 ceph-mon[34509]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:19:37.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:37 smithi184 ceph-mon[34509]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:19:37.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:37 smithi184 ceph-mon[34509]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T08:19:37.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:37.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:37.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:37 smithi184 ceph-mon[34509]: from='osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147]' entity='osd.3' 2024-05-09T08:19:37.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_3 2024-05-09T08:19:38.219 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:19:38.335 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:38 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:19:38.335 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:38 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:19:38.335 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:38 smithi165 ceph-mon[34540]: osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] boot 2024-05-09T08:19:38.335 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:38 smithi165 ceph-mon[34540]: osdmap e25: 4 total, 4 up, 4 in 2024-05-09T08:19:38.335 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:38 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:38 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:19:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:38 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:19:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:38 smithi028 ceph-mon[30119]: osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] boot 2024-05-09T08:19:38.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:38 smithi028 ceph-mon[30119]: osdmap e25: 4 total, 4 up, 4 in 2024-05-09T08:19:38.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:38.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:38 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:19:38.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:38 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:19:38.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:38 smithi184 ceph-mon[34509]: osd.3 [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] boot 2024-05-09T08:19:38.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:38 smithi184 ceph-mon[34509]: osdmap e25: 4 total, 4 up, 4 in 2024-05-09T08:19:38.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:19:39.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:39.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T08:19:39.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:39.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:39.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:39.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:39.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T08:19:39.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:19:39.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:19:39.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:39.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:39.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:39.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:39.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:39.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:39.865 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.861+0000 7fad14732700 1 -- 172.21.15.165:0/675456979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 msgr2=0x7fad0c10a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:39.865 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.861+0000 7fad14732700 1 --2- 172.21.15.165:0/675456979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 0x7fad0c10a7e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fad0400b0d0 tx=0x7fad0400b3e0 comp rx=0 tx=0).stop 2024-05-09T08:19:39.865 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 -- 172.21.15.165:0/675456979 shutdown_connections 2024-05-09T08:19:39.865 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 --2- 172.21.15.165:0/675456979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 0x7fad0c10a7e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 --2- 172.21.15.165:0/675456979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c102390 0x7fad0c1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 --2- 172.21.15.165:0/675456979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad0c103590 0x7fad0c1039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 -- 172.21.15.165:0/675456979 >> 172.21.15.165:0/675456979 conn(0x7fad0c0fd930 msgr2=0x7fad0c0ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 -- 172.21.15.165:0/675456979 shutdown_connections 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 -- 172.21.15.165:0/675456979 wait complete. 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 Processor -- start 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 -- start start 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c102390 0x7fad0c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad0c103590 0x7fad0c198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 0x7fad0c19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c10df80 con 0x7fad0c102390 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c10de00 con 0x7fad0c103f30 2024-05-09T08:19:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.862+0000 7fad14732700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c10e100 con 0x7fad0c103590 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad12ccf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 0x7fad0c19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad12ccf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 0x7fad0c19faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:41142/0 (socket says 172.21.15.165:41142) 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad12ccf700 1 -- 172.21.15.165:0/3629517035 learned_addr learned my addr 172.21.15.165:0/3629517035 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad124ce700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c102390 0x7fad0c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad11ccd700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad0c103590 0x7fad0c198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad12ccf700 1 -- 172.21.15.165:0/3629517035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad0c103590 msgr2=0x7fad0c198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad12ccf700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad0c103590 0x7fad0c198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad12ccf700 1 -- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c102390 msgr2=0x7fad0c1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad12ccf700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c102390 0x7fad0c1984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad12ccf700 1 -- 172.21.15.165:0/3629517035 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c1a0140 con 0x7fad0c103f30 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad12ccf700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 0x7fad0c19faa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fad04006070 tx=0x7fad04007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad037fe700 1 -- 172.21.15.165:0/3629517035 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04021020 con 0x7fad0c103f30 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad037fe700 1 -- 172.21.15.165:0/3629517035 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad0401d950 con 0x7fad0c103f30 2024-05-09T08:19:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0c1a03d0 con 0x7fad0c103f30 2024-05-09T08:19:39.868 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad037fe700 1 -- 172.21.15.165:0/3629517035 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04005430 con 0x7fad0c103f30 2024-05-09T08:19:39.868 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.863+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0c1a08c0 con 0x7fad0c103f30 2024-05-09T08:19:39.869 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.864+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad0c069010 con 0x7fad0c103f30 2024-05-09T08:19:39.869 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.865+0000 7fad037fe700 1 -- 172.21.15.165:0/3629517035 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fad04024020 con 0x7fad0c103f30 2024-05-09T08:19:39.869 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.866+0000 7fad037fe700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7facf80a2c20 0x7facf80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:19:39.869 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.866+0000 7fad037fe700 1 -- 172.21.15.165:0/3629517035 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7fad040cf4e0 con 0x7fad0c103f30 2024-05-09T08:19:39.869 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.866+0000 7fad124ce700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7facf80a2c20 0x7facf80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:19:39.869 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.866+0000 7fad124ce700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7facf80a2c20 0x7facf80a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad0c1033f0 tx=0x7fad0800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:19:39.871 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:39.869+0000 7fad037fe700 1 -- 172.21.15.165:0/3629517035 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fad0402b020 con 0x7fad0c103f30 2024-05-09T08:19:40.074 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:40.073+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fad0c062820 con 0x7facf80a2c20 2024-05-09T08:19:40.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:40 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi028 2024-05-09T08:19:40.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:40 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi028 to 8546M 2024-05-09T08:19:40.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:40 smithi165 ceph-mon[34540]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T08:19:40.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:19:40.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:19:40.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:40.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:40 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi028 2024-05-09T08:19:40.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:40 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi028 to 8546M 2024-05-09T08:19:40.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:40 smithi184 ceph-mon[34509]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T08:19:40.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:19:40.432 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:19:40.433 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:40.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:40 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi028 2024-05-09T08:19:40.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:40 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi028 to 8546M 2024-05-09T08:19:40.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:40 smithi028 ceph-mon[30119]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T08:19:40.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:19:40.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:19:40.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:41.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:41 smithi165 ceph-mon[34540]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:41.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:41 smithi165 ceph-mon[34540]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:19:41.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:41 smithi165 ceph-mon[34540]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T08:19:41.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:41 smithi028 ceph-mon[30119]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:41.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:41 smithi028 ceph-mon[30119]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:19:41.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:41 smithi028 ceph-mon[30119]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T08:19:41.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:41 smithi184 ceph-mon[34509]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:41.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:41 smithi184 ceph-mon[34509]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:19:41.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:41 smithi184 ceph-mon[34509]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T08:19:42.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:42 smithi165 ceph-mon[34540]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T08:19:42.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:42 smithi028 ceph-mon[30119]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T08:19:42.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:42 smithi184 ceph-mon[34509]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T08:19:43.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:43 smithi028 ceph-mon[30119]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:43.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:43 smithi028 ceph-mon[30119]: from='client.? 172.21.15.165:0/3181503018' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac8fdba-d8a4-453d-9091-121d932d6904"}]: dispatch 2024-05-09T08:19:43.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:43 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac8fdba-d8a4-453d-9091-121d932d6904"}]: dispatch 2024-05-09T08:19:43.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:43 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cac8fdba-d8a4-453d-9091-121d932d6904"}]': finished 2024-05-09T08:19:43.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:43 smithi028 ceph-mon[30119]: osdmap e30: 5 total, 4 up, 5 in 2024-05-09T08:19:43.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:43 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:19:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:43 smithi165 ceph-mon[34540]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:43 smithi165 ceph-mon[34540]: from='client.? 172.21.15.165:0/3181503018' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac8fdba-d8a4-453d-9091-121d932d6904"}]: dispatch 2024-05-09T08:19:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:43 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac8fdba-d8a4-453d-9091-121d932d6904"}]: dispatch 2024-05-09T08:19:43.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:43 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cac8fdba-d8a4-453d-9091-121d932d6904"}]': finished 2024-05-09T08:19:43.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:43 smithi165 ceph-mon[34540]: osdmap e30: 5 total, 4 up, 5 in 2024-05-09T08:19:43.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:43 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:19:43.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:43 smithi184 ceph-mon[34509]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:43.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:43 smithi184 ceph-mon[34509]: from='client.? 172.21.15.165:0/3181503018' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac8fdba-d8a4-453d-9091-121d932d6904"}]: dispatch 2024-05-09T08:19:43.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:43 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac8fdba-d8a4-453d-9091-121d932d6904"}]: dispatch 2024-05-09T08:19:43.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:43 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cac8fdba-d8a4-453d-9091-121d932d6904"}]': finished 2024-05-09T08:19:43.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:43 smithi184 ceph-mon[34509]: osdmap e30: 5 total, 4 up, 5 in 2024-05-09T08:19:43.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:43 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:19:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:44 smithi028 ceph-mon[30119]: from='client.? 172.21.15.165:0/3653341497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:19:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:44 smithi165 ceph-mon[34540]: from='client.? 172.21.15.165:0/3653341497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:19:44.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:44 smithi184 ceph-mon[34509]: from='client.? 172.21.15.165:0/3653341497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:19:45.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:45 smithi028 ceph-mon[30119]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:45.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:45 smithi165 ceph-mon[34540]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:45.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:45 smithi184 ceph-mon[34509]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:47.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:47 smithi165 ceph-mon[34540]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:19:47.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:47 smithi028 ceph-mon[30119]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:19:47.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:47 smithi184 ceph-mon[34509]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:19:49.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:49 smithi028 ceph-mon[30119]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-09T08:19:49.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:49 smithi165 ceph-mon[34540]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-09T08:19:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:49 smithi184 ceph-mon[34509]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-09T08:19:50.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T08:19:50.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:50.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T08:19:50.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:50.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T08:19:50.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:51.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:51 smithi165 ceph-mon[34540]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-09T08:19:51.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:51 smithi165 ceph-mon[34540]: Deploying daemon osd.4 on smithi165 2024-05-09T08:19:51.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:51 smithi184 ceph-mon[34509]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-09T08:19:51.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:51 smithi184 ceph-mon[34509]: Deploying daemon osd.4 on smithi165 2024-05-09T08:19:51.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:51 smithi028 ceph-mon[30119]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-09T08:19:51.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:51 smithi028 ceph-mon[30119]: Deploying daemon osd.4 on smithi165 2024-05-09T08:19:53.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:53 smithi165 ceph-mon[34540]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T08:19:53.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:53 smithi184 ceph-mon[34509]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T08:19:53.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:53 smithi028 ceph-mon[30119]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T08:19:55.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:55 smithi165 ceph-mon[34540]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-05-09T08:19:55.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:55 smithi184 ceph-mon[34509]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-05-09T08:19:55.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:55 smithi028 ceph-mon[30119]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-05-09T08:19:56.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:56 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:56.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:56 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:56.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:56 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:56.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:56 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:56.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:56 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:56.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:56 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:56.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:56 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:56.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:56 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:56.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:56 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:19:57.585 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:57 smithi028 ceph-mon[30119]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:19:57.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:57 smithi165 ceph-mon[34540]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:19:57.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:57 smithi184 ceph-mon[34509]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:19:58.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:58 smithi165 ceph-mon[34540]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:58.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:58.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:58.744 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:58.744 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:58.744 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:58.794 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.791+0000 7fad037fe700 1 -- 172.21.15.165:0/3629517035 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fad0c062820 con 0x7facf80a2c20 2024-05-09T08:19:58.794 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 4 on host 'smithi165' 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7facf80a2c20 msgr2=0x7facf80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7facf80a2c20 0x7facf80a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad0c1033f0 tx=0x7fad0800b410 comp rx=0 tx=0).stop 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 msgr2=0x7fad0c19faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 0x7fad0c19faa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fad04006070 tx=0x7fad04007d10 comp rx=0 tx=0).stop 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 shutdown_connections 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7facf80a2c20 0x7facf80a50e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0c103f30 0x7fad0c19faa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c102390 0x7fad0c1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 --2- 172.21.15.165:0/3629517035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad0c103590 0x7fad0c198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:19:58.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.793+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 >> 172.21.15.165:0/3629517035 conn(0x7fad0c0fd930 msgr2=0x7fad0c1067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:19:58.797 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.794+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 shutdown_connections 2024-05-09T08:19:58.797 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:19:58.794+0000 7fad14732700 1 -- 172.21.15.165:0/3629517035 wait complete. 2024-05-09T08:19:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:58 smithi184 ceph-mon[34509]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:58 smithi028 ceph-mon[30119]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:19:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:19:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:19:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:19:59.410 DEBUG:teuthology.orchestra.run.smithi165:osd.4> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.4.service 2024-05-09T08:19:59.413 INFO:tasks.cephadm:Deploying osd.5 on smithi184 with /dev/vg_nvme/lv_3... 2024-05-09T08:19:59.414 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T08:19:59.658 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:20:00.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:59 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:00.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:59 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:00.060 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:59 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:00.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:59 smithi184 ceph-mon[34509]: from='osd.4 [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:20:00.061 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:19:59 smithi184 ceph-mon[34509]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:20:00.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:59 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:00.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:59 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:00.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:59 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:00.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:59 smithi165 ceph-mon[34540]: from='osd.4 [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:20:00.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:19:59 smithi165 ceph-mon[34540]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:20:00.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:59 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:00.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:59 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:00.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:59 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:00.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:59 smithi028 ceph-mon[30119]: from='osd.4 [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:20:00.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:19:59 smithi028 ceph-mon[30119]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:20:00.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:00 smithi184 ceph-mon[34509]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:20:00.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:00 smithi184 ceph-mon[34509]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T08:20:00.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:00 smithi184 ceph-mon[34509]: from='osd.4 [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:20:00.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:00 smithi184 ceph-mon[34509]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T08:20:00.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:00.953 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:00 smithi184 ceph-mon[34509]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:20:00.954 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:00 smithi184 ceph-mon[34509]: overall HEALTH_OK 2024-05-09T08:20:01.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:00 smithi165 ceph-mon[34540]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:20:01.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:00 smithi165 ceph-mon[34540]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T08:20:01.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:00 smithi165 ceph-mon[34540]: from='osd.4 [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:20:01.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:00 smithi165 ceph-mon[34540]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T08:20:01.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:01.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:00 smithi165 ceph-mon[34540]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:20:01.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:00 smithi165 ceph-mon[34540]: overall HEALTH_OK 2024-05-09T08:20:01.104 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:20:00 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4[45640]: 2024-05-09T08:20:00.789+0000 7fd2623cf700 -1 osd.4 0 waiting for initial osdmap 2024-05-09T08:20:01.104 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:20:00 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4[45640]: 2024-05-09T08:20:00.795+0000 7fd25e1c2700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:20:01.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:00 smithi028 ceph-mon[30119]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:20:01.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:00 smithi028 ceph-mon[30119]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T08:20:01.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:00 smithi028 ceph-mon[30119]: from='osd.4 [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:20:01.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:00 smithi028 ceph-mon[30119]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T08:20:01.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:01.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:00 smithi028 ceph-mon[30119]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:20:01.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:00 smithi028 ceph-mon[30119]: overall HEALTH_OK 2024-05-09T08:20:01.675 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T08:20:01.675 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T08:20:01.675 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-05-09T08:20:01.675 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-05-09T08:20:01.675 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124978 s, 839 MB/s 2024-05-09T08:20:01.675 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-05-09T08:20:01.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:01 smithi165 ceph-mon[34540]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:20:01.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:01 smithi165 ceph-mon[34540]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T08:20:01.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:01 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:01.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:01 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:01.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:01 smithi184 ceph-mon[34509]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:20:01.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:01 smithi184 ceph-mon[34509]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T08:20:01.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:01 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:01.979 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:01 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:02.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:01 smithi028 ceph-mon[30119]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:20:02.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:01 smithi028 ceph-mon[30119]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T08:20:02.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:01 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:02.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:01 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:02.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_3 2024-05-09T08:20:02.534 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:20:02.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:20:02.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:20:02.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:20:02.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: osd.4 [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] boot 2024-05-09T08:20:02.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: osdmap e33: 5 total, 5 up, 5 in 2024-05-09T08:20:02.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:02.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi165 2024-05-09T08:20:02.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:02.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:02.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:02.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:02.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi165 to 8544M 2024-05-09T08:20:02.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:02.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:02.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:02.860 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:20:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:20:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:20:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: osd.4 [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] boot 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: osdmap e33: 5 total, 5 up, 5 in 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi165 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi165 to 8544M 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:03.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: osd.4 [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] boot 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: osdmap e33: 5 total, 5 up, 5 in 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi165 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi165 to 8544M 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:03.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:03.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:03.938 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:03 smithi184 ceph-mon[34509]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T08:20:04.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:03 smithi165 ceph-mon[34540]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T08:20:04.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:03 smithi028 ceph-mon[30119]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T08:20:04.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.182+0000 7f3db39b6700 1 -- 172.21.15.184:0/1727432110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac101270 msgr2=0x7f3dac101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:04.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.182+0000 7f3db39b6700 1 --2- 172.21.15.184:0/1727432110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac101270 0x7f3dac101690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009990 tx=0x7f3da8009ca0 comp rx=0 tx=0).stop 2024-05-09T08:20:04.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.183+0000 7f3db39b6700 1 -- 172.21.15.184:0/1727432110 shutdown_connections 2024-05-09T08:20:04.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.183+0000 7f3db39b6700 1 --2- 172.21.15.184:0/1727432110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac101270 0x7f3dac101690 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:04.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.183+0000 7f3db39b6700 1 --2- 172.21.15.184:0/1727432110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dac102e10 0x7f3dac1096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:04.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.183+0000 7f3db39b6700 1 --2- 172.21.15.184:0/1727432110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dac102470 0x7f3dac1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:04.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.183+0000 7f3db39b6700 1 -- 172.21.15.184:0/1727432110 >> 172.21.15.184:0/1727432110 conn(0x7f3dac0fc810 msgr2=0x7f3dac0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:20:04.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.183+0000 7f3db39b6700 1 -- 172.21.15.184:0/1727432110 shutdown_connections 2024-05-09T08:20:04.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.183+0000 7f3db39b6700 1 -- 172.21.15.184:0/1727432110 wait complete. 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.183+0000 7f3db39b6700 1 Processor -- start 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.183+0000 7f3db39b6700 1 -- start start 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db39b6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dac101270 0x7f3dac1973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db39b6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dac102470 0x7f3dac197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db39b6700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac102e10 0x7f3dac19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db39b6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac10ce30 con 0x7f3dac102470 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db39b6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac10ccb0 con 0x7f3dac101270 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db39b6700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac10cfb0 con 0x7f3dac102e10 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1f53700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac102e10 0x7f3dac19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1f53700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac102e10 0x7f3dac19e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:35200/0 (socket says 172.21.15.184:35200) 2024-05-09T08:20:04.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1f53700 1 -- 172.21.15.184:0/2579846850 learned_addr learned my addr 172.21.15.184:0/2579846850 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db0f51700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dac102470 0x7f3dac197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1752700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dac101270 0x7f3dac1973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1f53700 1 -- 172.21.15.184:0/2579846850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dac101270 msgr2=0x7f3dac1973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1f53700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dac101270 0x7f3dac1973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1f53700 1 -- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dac102470 msgr2=0x7f3dac197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1f53700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dac102470 0x7f3dac197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1f53700 1 -- 172.21.15.184:0/2579846850 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac19f050 con 0x7f3dac102e10 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db1f53700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac102e10 0x7f3dac19e9b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3da4007d40 tx=0x7f3da400a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3da27fc700 1 -- 172.21.15.184:0/2579846850 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4016020 con 0x7f3dac102e10 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3da27fc700 1 -- 172.21.15.184:0/2579846850 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da400acb0 con 0x7f3dac102e10 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dac19f340 con 0x7f3dac102e10 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.184+0000 7f3da27fc700 1 -- 172.21.15.184:0/2579846850 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da40115e0 con 0x7f3dac102e10 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.185+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dac19f830 con 0x7f3dac102e10 2024-05-09T08:20:04.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.185+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dac069010 con 0x7f3dac102e10 2024-05-09T08:20:04.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.186+0000 7f3da27fc700 1 -- 172.21.15.184:0/2579846850 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3da4019020 con 0x7f3dac102e10 2024-05-09T08:20:04.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.187+0000 7f3da27fc700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d980a2c20 0x7f3d980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:04.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.187+0000 7f3da27fc700 1 -- 172.21.15.184:0/2579846850 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f3da40cded0 con 0x7f3dac102e10 2024-05-09T08:20:04.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.187+0000 7f3db1752700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d980a2c20 0x7f3d980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:04.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.188+0000 7f3db1752700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d980a2c20 0x7f3d980a50e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3da8003890 tx=0x7f3da8003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:20:04.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.194+0000 7f3da27fc700 1 -- 172.21.15.184:0/2579846850 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3da4020020 con 0x7f3dac102e10 2024-05-09T08:20:04.398 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:04.395+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3dac062760 con 0x7f3d980a2c20 2024-05-09T08:20:05.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:04 smithi165 ceph-mon[34540]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:05.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:04 smithi165 ceph-mon[34540]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:20:05.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:20:05.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:20:05.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:04 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:05.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:04 smithi184 ceph-mon[34509]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:05.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:04 smithi184 ceph-mon[34509]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:20:05.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:20:05.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:20:05.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:04 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:05.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:04 smithi028 ceph-mon[30119]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:05.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:04 smithi028 ceph-mon[30119]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:20:05.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:20:05.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:20:05.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:04 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:07.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:06 smithi165 ceph-mon[34540]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:07.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:06 smithi184 ceph-mon[34509]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:07.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:06 smithi028 ceph-mon[30119]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:08.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.184:0/117639454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072f3379-6c55-40cd-9898-639846b9369f"}]: dispatch 2024-05-09T08:20:08.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:07 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072f3379-6c55-40cd-9898-639846b9369f"}]: dispatch 2024-05-09T08:20:08.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:07 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "072f3379-6c55-40cd-9898-639846b9369f"}]': finished 2024-05-09T08:20:08.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:07 smithi165 ceph-mon[34540]: osdmap e35: 6 total, 5 up, 6 in 2024-05-09T08:20:08.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:07 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:08.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.184:0/3032767488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:20:08.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/117639454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072f3379-6c55-40cd-9898-639846b9369f"}]: dispatch 2024-05-09T08:20:08.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:07 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072f3379-6c55-40cd-9898-639846b9369f"}]: dispatch 2024-05-09T08:20:08.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:07 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "072f3379-6c55-40cd-9898-639846b9369f"}]': finished 2024-05-09T08:20:08.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:07 smithi184 ceph-mon[34509]: osdmap e35: 6 total, 5 up, 6 in 2024-05-09T08:20:08.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:07 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:08.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/3032767488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:20:08.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/117639454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072f3379-6c55-40cd-9898-639846b9369f"}]: dispatch 2024-05-09T08:20:08.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:07 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072f3379-6c55-40cd-9898-639846b9369f"}]: dispatch 2024-05-09T08:20:08.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:07 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "072f3379-6c55-40cd-9898-639846b9369f"}]': finished 2024-05-09T08:20:08.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:07 smithi028 ceph-mon[30119]: osdmap e35: 6 total, 5 up, 6 in 2024-05-09T08:20:08.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:07 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:08.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/3032767488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:20:09.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:08 smithi165 ceph-mon[34540]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:09.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:08 smithi184 ceph-mon[34509]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:09.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:08 smithi028 ceph-mon[30119]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:11.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:10 smithi184 ceph-mon[34509]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:11.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:10 smithi165 ceph-mon[34540]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:11.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:10 smithi028 ceph-mon[30119]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:13.199 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:12 smithi184 ceph-mon[34509]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:12 smithi165 ceph-mon[34540]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:13.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:12 smithi028 ceph-mon[30119]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:15.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:14 smithi184 ceph-mon[34509]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:15.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T08:20:15.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:15.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:14 smithi184 ceph-mon[34509]: Deploying daemon osd.5 on smithi184 2024-05-09T08:20:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:14 smithi165 ceph-mon[34540]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T08:20:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:14 smithi165 ceph-mon[34540]: Deploying daemon osd.5 on smithi184 2024-05-09T08:20:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:14 smithi028 ceph-mon[30119]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T08:20:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:14 smithi028 ceph-mon[30119]: Deploying daemon osd.5 on smithi184 2024-05-09T08:20:17.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:16 smithi184 ceph-mon[34509]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:17.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:16 smithi165 ceph-mon[34540]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:17.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:16 smithi028 ceph-mon[30119]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:19.151 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:18 smithi184 ceph-mon[34509]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:19.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:18 smithi165 ceph-mon[34540]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:19.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:18 smithi028 ceph-mon[30119]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:21.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:20 smithi184 ceph-mon[34509]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:21.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:20 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:21.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:20 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:21.003 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:20 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:21.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:20 smithi165 ceph-mon[34540]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:21.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:20 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:21.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:20 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:21.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:20 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:21.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:20 smithi028 ceph-mon[30119]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:21.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:20 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:21.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:20 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:21.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:20 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:22.762 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:22 smithi184 ceph-mon[34509]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:22.821 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.820+0000 7f3da27fc700 1 -- 172.21.15.184:0/2579846850 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3dac062760 con 0x7f3d980a2c20 2024-05-09T08:20:22.822 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 5 on host 'smithi184' 2024-05-09T08:20:22.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.823+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d980a2c20 msgr2=0x7f3d980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:22.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.823+0000 7f3db39b6700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d980a2c20 0x7f3d980a50e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3da8003890 tx=0x7f3da8003960 comp rx=0 tx=0).stop 2024-05-09T08:20:22.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.823+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac102e10 msgr2=0x7f3dac19e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:22.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.823+0000 7f3db39b6700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac102e10 0x7f3dac19e9b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3da4007d40 tx=0x7f3da400a340 comp rx=0 tx=0).stop 2024-05-09T08:20:22.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.824+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 shutdown_connections 2024-05-09T08:20:22.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.824+0000 7f3db39b6700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d980a2c20 0x7f3d980a50e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:22.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.824+0000 7f3db39b6700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dac102e10 0x7f3dac19e9b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:22.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.824+0000 7f3db39b6700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dac101270 0x7f3dac1973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:22.826 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.824+0000 7f3db39b6700 1 --2- 172.21.15.184:0/2579846850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dac102470 0x7f3dac197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:22.826 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.824+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 >> 172.21.15.184:0/2579846850 conn(0x7f3dac0fc810 msgr2=0x7f3dac105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:20:22.826 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.824+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 shutdown_connections 2024-05-09T08:20:22.826 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:20:22.824+0000 7f3db39b6700 1 -- 172.21.15.184:0/2579846850 wait complete. 2024-05-09T08:20:23.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:22 smithi165 ceph-mon[34540]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:23.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:22 smithi028 ceph-mon[30119]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:23.678 DEBUG:teuthology.orchestra.run.smithi184:osd.5> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.5.service 2024-05-09T08:20:23.680 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-05-09T08:20:23.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T08:20:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:23.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:23.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:23 smithi184 ceph-mon[34509]: from='osd.5 [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:20:23.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:23 smithi184 ceph-mon[34509]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:20:23.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:23.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:20:24.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:24.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:24.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:23 smithi028 ceph-mon[30119]: from='osd.5 [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:20:24.080 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:23 smithi028 ceph-mon[30119]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:20:24.080 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:24.080 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:24.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:24.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:24.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:23 smithi165 ceph-mon[34540]: from='osd.5 [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:20:24.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:23 smithi165 ceph-mon[34540]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:20:24.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:24.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:25.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:24 smithi028 ceph-mon[30119]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:25.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:24 smithi028 ceph-mon[30119]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T08:20:25.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:24 smithi028 ceph-mon[30119]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T08:20:25.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:24 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:25.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:24 smithi028 ceph-mon[30119]: from='osd.5 [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:20:25.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:24 smithi028 ceph-mon[30119]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:20:25.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:24 smithi165 ceph-mon[34540]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:25.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:24 smithi165 ceph-mon[34540]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T08:20:25.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:24 smithi165 ceph-mon[34540]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T08:20:25.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:24 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:25.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:24 smithi165 ceph-mon[34540]: from='osd.5 [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:20:25.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:24 smithi165 ceph-mon[34540]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:20:25.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:24 smithi184 ceph-mon[34509]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:25.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:24 smithi184 ceph-mon[34509]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T08:20:25.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:24 smithi184 ceph-mon[34509]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T08:20:25.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:24 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:25.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:24 smithi184 ceph-mon[34509]: from='osd.5 [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:20:25.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:24 smithi184 ceph-mon[34509]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:20:25.506 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:20:25 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:20:25.229+0000 7ff8d388c700 -1 osd.5 0 waiting for initial osdmap 2024-05-09T08:20:25.506 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:20:25 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:20:25.235+0000 7ff8cb677700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:20:26.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:25 smithi028 ceph-mon[30119]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:20:26.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:25 smithi028 ceph-mon[30119]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T08:20:26.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:26.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:26.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:25 smithi028 ceph-mon[30119]: from='osd.5 ' entity='osd.5' 2024-05-09T08:20:26.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:25 smithi028 ceph-mon[30119]: osd.5 [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] boot 2024-05-09T08:20:26.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:25 smithi028 ceph-mon[30119]: osdmap e38: 6 total, 6 up, 6 in 2024-05-09T08:20:26.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:26.095 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T08:20:26.095 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T08:20:26.095 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-05-09T08:20:26.095 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-05-09T08:20:26.096 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0095788 s, 1.1 GB/s 2024-05-09T08:20:26.096 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-05-09T08:20:26.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:25 smithi165 ceph-mon[34540]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:20:26.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:25 smithi165 ceph-mon[34540]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T08:20:26.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:26.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:26.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:25 smithi165 ceph-mon[34540]: from='osd.5 ' entity='osd.5' 2024-05-09T08:20:26.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:25 smithi165 ceph-mon[34540]: osd.5 [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] boot 2024-05-09T08:20:26.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:25 smithi165 ceph-mon[34540]: osdmap e38: 6 total, 6 up, 6 in 2024-05-09T08:20:26.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:26.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:25 smithi184 ceph-mon[34509]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:20:26.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:25 smithi184 ceph-mon[34509]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T08:20:26.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:26.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:26.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:25 smithi184 ceph-mon[34509]: from='osd.5 ' entity='osd.5' 2024-05-09T08:20:26.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:25 smithi184 ceph-mon[34509]: osd.5 [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] boot 2024-05-09T08:20:26.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:25 smithi184 ceph-mon[34509]: osdmap e38: 6 total, 6 up, 6 in 2024-05-09T08:20:26.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:20:26.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-05-09T08:20:27.179 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:27.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:27.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:27.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:27.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:27 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:20:27.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:20:27.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:27.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T08:20:27.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:27.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:27 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:20:27.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:27.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:27 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:28.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:28 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi184 2024-05-09T08:20:28.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:28 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi184 to 8544M 2024-05-09T08:20:28.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:28 smithi028 ceph-mon[30119]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T08:20:28.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:28 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi184 2024-05-09T08:20:28.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:28 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi184 to 8544M 2024-05-09T08:20:28.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:28 smithi165 ceph-mon[34540]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T08:20:28.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:28 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi184 2024-05-09T08:20:28.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:28 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi184 to 8544M 2024-05-09T08:20:28.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:28 smithi184 ceph-mon[34509]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T08:20:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.069+0000 7f185f40f700 1 -- 172.21.15.28:0/3839912330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858076350 msgr2=0x7f18580747b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.069+0000 7f185f40f700 1 --2- 172.21.15.28:0/3839912330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858076350 0x7f18580747b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1854009990 tx=0x7f1854009ca0 comp rx=0 tx=0).stop 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.069+0000 7f185f40f700 1 -- 172.21.15.28:0/3839912330 shutdown_connections 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.069+0000 7f185f40f700 1 --2- 172.21.15.28:0/3839912330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f185810a270 0x7f185810c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.069+0000 7f185f40f700 1 --2- 172.21.15.28:0/3839912330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858076350 0x7f18580747b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.069+0000 7f185f40f700 1 --2- 172.21.15.28:0/3839912330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1858074d80 0x7f1858075200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.069+0000 7f185f40f700 1 -- 172.21.15.28:0/3839912330 >> 172.21.15.28:0/3839912330 conn(0x7f18580fd940 msgr2=0x7f18580ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.069+0000 7f185f40f700 1 -- 172.21.15.28:0/3839912330 shutdown_connections 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.069+0000 7f185f40f700 1 -- 172.21.15.28:0/3839912330 wait complete. 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185f40f700 1 Processor -- start 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185f40f700 1 -- start start 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185f40f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858074d80 0x7f185819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185f40f700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1858076350 0x7f185819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185f40f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f185810a270 0x7f18581a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185f40f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110190 con 0x7f1858074d80 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185f40f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110010 con 0x7f185810a270 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185f40f700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110310 con 0x7f1858076350 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185d1ab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858074d80 0x7f185819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185d1ab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858074d80 0x7f185819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41940/0 (socket says 172.21.15.28:41940) 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185d1ab700 1 -- 172.21.15.28:0/473832817 learned_addr learned my addr 172.21.15.28:0/473832817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.070+0000 7f185d9ac700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f185810a270 0x7f18581a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185c9aa700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1858076350 0x7f185819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185d1ab700 1 -- 172.21.15.28:0/473832817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1858076350 msgr2=0x7f185819ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185d1ab700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1858076350 0x7f185819ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185d1ab700 1 -- 172.21.15.28:0/473832817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f185810a270 msgr2=0x7f18581a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185d1ab700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f185810a270 0x7f18581a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185d1ab700 1 -- 172.21.15.28:0/473832817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18581a2450 con 0x7f1858074d80 2024-05-09T08:20:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185d9ac700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f185810a270 0x7f18581a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:20:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185c9aa700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1858076350 0x7f185819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:20:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185d1ab700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858074d80 0x7f185819a7f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1854000c00 tx=0x7f18540047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:20:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f184e7fc700 1 -- 172.21.15.28:0/473832817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1854003d30 con 0x7f1858074d80 2024-05-09T08:20:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f184e7fc700 1 -- 172.21.15.28:0/473832817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1854004830 con 0x7f1858074d80 2024-05-09T08:20:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18581a26e0 con 0x7f1858074d80 2024-05-09T08:20:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f184e7fc700 1 -- 172.21.15.28:0/473832817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1854005420 con 0x7f1858074d80 2024-05-09T08:20:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.071+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18581a2ad0 con 0x7f1858074d80 2024-05-09T08:20:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.072+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1858069010 con 0x7f1858074d80 2024-05-09T08:20:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.073+0000 7f184e7fc700 1 -- 172.21.15.28:0/473832817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1854004cc0 con 0x7f1858074d80 2024-05-09T08:20:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.074+0000 7f184e7fc700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f18440a2c20 0x7f18440a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.074+0000 7f185c9aa700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f18440a2c20 0x7f18440a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.074+0000 7f184e7fc700 1 -- 172.21.15.28:0/473832817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4533+0+0 (secure 0 0 0) 0x7f18540cf6c0 con 0x7f1858074d80 2024-05-09T08:20:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.074+0000 7f185c9aa700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f18440a2c20 0x7f18440a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1848005fd0 tx=0x7f1848005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:20:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.077+0000 7f184e7fc700 1 -- 172.21.15.28:0/473832817 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1854013070 con 0x7f1858074d80 2024-05-09T08:20:29.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:29.291+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1858062760 con 0x7f18440a2c20 2024-05-09T08:20:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:29 smithi165 ceph-mon[34540]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:29 smithi165 ceph-mon[34540]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T08:20:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:20:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:20:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:29 smithi184 ceph-mon[34509]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:29 smithi184 ceph-mon[34509]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T08:20:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:20:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:20:29.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:29.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:29 smithi028 ceph-mon[30119]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:29.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:29 smithi028 ceph-mon[30119]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T08:20:29.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:20:29.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:20:29.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:30.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:30 smithi028 ceph-mon[30119]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:20:30.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:30 smithi028 ceph-mon[30119]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T08:20:30.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:30 smithi165 ceph-mon[34540]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:20:30.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:30 smithi165 ceph-mon[34540]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T08:20:30.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:30 smithi184 ceph-mon[34509]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:20:30.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:30 smithi184 ceph-mon[34509]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T08:20:31.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:31 smithi028 ceph-mon[30119]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T08:20:31.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:31 smithi165 ceph-mon[34540]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T08:20:31.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:31 smithi184 ceph-mon[34509]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T08:20:32.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:32 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/798259613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19"}]: dispatch 2024-05-09T08:20:32.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:32 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/798259613' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19"}]': finished 2024-05-09T08:20:32.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:32 smithi184 ceph-mon[34509]: osdmap e43: 7 total, 6 up, 7 in 2024-05-09T08:20:32.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:32.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:32 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/798259613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19"}]: dispatch 2024-05-09T08:20:32.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:32 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/798259613' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19"}]': finished 2024-05-09T08:20:32.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:32 smithi028 ceph-mon[30119]: osdmap e43: 7 total, 6 up, 7 in 2024-05-09T08:20:32.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:32.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:32 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/798259613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19"}]: dispatch 2024-05-09T08:20:32.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:32 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/798259613' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19"}]': finished 2024-05-09T08:20:32.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:32 smithi165 ceph-mon[34540]: osdmap e43: 7 total, 6 up, 7 in 2024-05-09T08:20:32.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:33.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:33 smithi184 ceph-mon[34509]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-09T08:20:33.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:33 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/90241539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:20:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:33 smithi028 ceph-mon[30119]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-09T08:20:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:33 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/90241539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:20:33.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:33 smithi165 ceph-mon[34540]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-09T08:20:33.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:33 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/90241539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:20:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:35 smithi028 ceph-mon[30119]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:20:35.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:35 smithi165 ceph-mon[34540]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:20:35.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:35 smithi184 ceph-mon[34509]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:20:37.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:37 smithi028 ceph-mon[30119]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-09T08:20:37.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:37 smithi165 ceph-mon[34540]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-09T08:20:37.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:37 smithi184 ceph-mon[34509]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-09T08:20:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:39 smithi028 ceph-mon[30119]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:39.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:39 smithi165 ceph-mon[34540]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:39.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:39 smithi184 ceph-mon[34509]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:40.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T08:20:40.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:40.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T08:20:40.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:40.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T08:20:40.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:41.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:41 smithi028 ceph-mon[30119]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:41.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:41 smithi028 ceph-mon[30119]: Deploying daemon osd.6 on smithi028 2024-05-09T08:20:41.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:41 smithi165 ceph-mon[34540]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:41.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:41 smithi165 ceph-mon[34540]: Deploying daemon osd.6 on smithi028 2024-05-09T08:20:41.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:41 smithi184 ceph-mon[34509]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:41.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:41 smithi184 ceph-mon[34509]: Deploying daemon osd.6 on smithi028 2024-05-09T08:20:43.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:43 smithi028 ceph-mon[30119]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:43 smithi165 ceph-mon[34540]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:43.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:43 smithi184 ceph-mon[34509]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:45.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:45 smithi028 ceph-mon[30119]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:45.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:45 smithi165 ceph-mon[34540]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:45.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:45 smithi184 ceph-mon[34509]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:47.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:46 smithi028 ceph-mon[30119]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:47.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:46 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:47.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:46 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:47.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:46 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:47.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:46 smithi165 ceph-mon[34540]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:47.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:46 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:47.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:46 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:47.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:46 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:47.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:46 smithi184 ceph-mon[34509]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:47.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:46 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:47.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:46 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:47.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:46 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:49.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:48 smithi028 ceph-mon[30119]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:49.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:48 smithi165 ceph-mon[34540]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:49.356 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-05-09T08:20:49.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.354+0000 7f184e7fc700 1 -- 172.21.15.28:0/473832817 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1858062760 con 0x7f18440a2c20 2024-05-09T08:20:49.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:48 smithi184 ceph-mon[34509]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f18440a2c20 msgr2=0x7f18440a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f18440a2c20 0x7f18440a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1848005fd0 tx=0x7f1848005f00 comp rx=0 tx=0).stop 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858074d80 msgr2=0x7f185819a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858074d80 0x7f185819a7f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1854000c00 tx=0x7f18540047b0 comp rx=0 tx=0).stop 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 shutdown_connections 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f18440a2c20 0x7f18440a50e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f185810a270 0x7f18581a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1858074d80 0x7f185819a7f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 --2- 172.21.15.28:0/473832817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1858076350 0x7f185819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 >> 172.21.15.28:0/473832817 conn(0x7f18580fd940 msgr2=0x7f1858106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:20:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 shutdown_connections 2024-05-09T08:20:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:20:49.357+0000 7f185f40f700 1 -- 172.21.15.28:0/473832817 wait complete. 2024-05-09T08:20:50.196 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.6.service 2024-05-09T08:20:50.198 INFO:tasks.cephadm:Deploying osd.7 on smithi165 with /dev/vg_nvme/lv_2... 2024-05-09T08:20:50.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T08:20:50.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:50 smithi028 ceph-mon[30119]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T08:20:50.448 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:20:50.569 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:50 smithi165 ceph-mon[34540]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T08:20:50.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:50.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:50 smithi184 ceph-mon[34509]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T08:20:51.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:51 smithi165 ceph-mon[34540]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:51.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:51 smithi165 ceph-mon[34540]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T08:20:51.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:51 smithi165 ceph-mon[34540]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T08:20:51.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:51 smithi165 ceph-mon[34540]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:20:51.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:51.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:51 smithi184 ceph-mon[34509]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:51.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:51 smithi184 ceph-mon[34509]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T08:20:51.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:51 smithi184 ceph-mon[34509]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T08:20:51.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:51 smithi184 ceph-mon[34509]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:20:51.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:51.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:51 smithi028 ceph-mon[30119]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:51.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:51 smithi028 ceph-mon[30119]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T08:20:51.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:51 smithi028 ceph-mon[30119]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T08:20:51.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:51 smithi028 ceph-mon[30119]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:20:51.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:51.867 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:20:51 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:20:51.702+0000 7f9cb6dc5700 -1 osd.6 0 waiting for initial osdmap 2024-05-09T08:20:51.867 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:20:51 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:20:51.710+0000 7f9cb2bb8700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:20:52.650 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T08:20:52.650 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T08:20:52.650 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-05-09T08:20:52.650 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-05-09T08:20:52.650 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011917 s, 880 MB/s 2024-05-09T08:20:52.650 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-05-09T08:20:52.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:52 smithi165 ceph-mon[34540]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:52.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:52 smithi165 ceph-mon[34540]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:20:52.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:52 smithi165 ceph-mon[34540]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T08:20:52.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:52 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:52.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:52 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:52.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:52 smithi028 ceph-mon[30119]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:52.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:52 smithi028 ceph-mon[30119]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:20:52.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:52 smithi028 ceph-mon[30119]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T08:20:52.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:52 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:52.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:52 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:53.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:52 smithi184 ceph-mon[34509]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:20:53.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:52 smithi184 ceph-mon[34509]: from='osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:20:53.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:52 smithi184 ceph-mon[34509]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T08:20:53.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:52 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:53.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:52 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:53.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_2 2024-05-09T08:20:53.608 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:20:53.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:20:53.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:20:53.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] boot 2024-05-09T08:20:53.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: osdmap e46: 7 total, 7 up, 7 in 2024-05-09T08:20:53.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:53.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi028 2024-05-09T08:20:53.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi028 to 5697M 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:53.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:53 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] boot 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: osdmap e46: 7 total, 7 up, 7 in 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi028 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi028 to 5697M 2024-05-09T08:20:54.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:54.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:54.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:54.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:54.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:54.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:53 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:20:54.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:20:54.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: osd.6 [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] boot 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: osdmap e46: 7 total, 7 up, 7 in 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi028 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi028 to 5697M 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:20:54.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:54.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:20:54.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:53 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:20:55.064 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:54 smithi165 ceph-mon[34540]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:20:55.064 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:54 smithi165 ceph-mon[34540]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T08:20:55.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:54 smithi184 ceph-mon[34509]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:20:55.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:54 smithi184 ceph-mon[34509]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T08:20:55.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:54 smithi028 ceph-mon[30119]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:20:55.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:54 smithi028 ceph-mon[30119]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2463867493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 msgr2=0x7f6cb4101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2463867493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 0x7f6cb4101690 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00099c0 tx=0x7f6cb0009cd0 comp rx=0 tx=0).stop 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2463867493 shutdown_connections 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2463867493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb4102e10 0x7f6cb41096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2463867493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 0x7f6cb4101690 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2463867493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cb4102470 0x7f6cb41028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2463867493 >> 172.21.15.165:0/2463867493 conn(0x7f6cb40fc810 msgr2=0x7f6cb40fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2463867493 shutdown_connections 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2463867493 wait complete. 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 Processor -- start 2024-05-09T08:20:55.306 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.302+0000 7f6cbba4d700 1 -- start start 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cbba4d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 0x7f6cb4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cbba4d700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cb4102470 0x7f6cb4198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cbba4d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb4102e10 0x7f6cb419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cbba4d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb410df90 con 0x7f6cb4102e10 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cbba4d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb410de10 con 0x7f6cb4101270 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cbba4d700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb410e110 con 0x7f6cb4102470 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb97e9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 0x7f6cb4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb97e9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 0x7f6cb4198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:56198/0 (socket says 172.21.15.165:56198) 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb97e9700 1 -- 172.21.15.165:0/2921655471 learned_addr learned my addr 172.21.15.165:0/2921655471 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb8fe8700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cb4102470 0x7f6cb4198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb9fea700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb4102e10 0x7f6cb419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb97e9700 1 -- 172.21.15.165:0/2921655471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cb4102470 msgr2=0x7f6cb4198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:55.307 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb97e9700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cb4102470 0x7f6cb4198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb97e9700 1 -- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb4102e10 msgr2=0x7f6cb419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:20:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb97e9700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb4102e10 0x7f6cb419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:20:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb97e9700 1 -- 172.21.15.165:0/2921655471 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb41a01f0 con 0x7f6cb4101270 2024-05-09T08:20:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cb97e9700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 0x7f6cb4198590 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6cb000c010 tx=0x7f6cb0009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:20:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6caa7fc700 1 -- 172.21.15.165:0/2921655471 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0027070 con 0x7f6cb4101270 2024-05-09T08:20:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6caa7fc700 1 -- 172.21.15.165:0/2921655471 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cb0005940 con 0x7f6cb4101270 2024-05-09T08:20:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb41a0480 con 0x7f6cb4101270 2024-05-09T08:20:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6caa7fc700 1 -- 172.21.15.165:0/2921655471 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0022d90 con 0x7f6cb4101270 2024-05-09T08:20:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.303+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb41a0960 con 0x7f6cb4101270 2024-05-09T08:20:55.309 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.304+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cb4069010 con 0x7f6cb4101270 2024-05-09T08:20:55.309 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.305+0000 7f6caa7fc700 1 -- 172.21.15.165:0/2921655471 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6cb002a070 con 0x7f6cb4101270 2024-05-09T08:20:55.309 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.306+0000 7f6caa7fc700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6ca00a2c20 0x7f6ca00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:20:55.309 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.306+0000 7f6caa7fc700 1 -- 172.21.15.165:0/2921655471 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f6cb00cf5a0 con 0x7f6cb4101270 2024-05-09T08:20:55.309 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.306+0000 7f6cb8fe8700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6ca00a2c20 0x7f6ca00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:20:55.309 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.307+0000 7f6cb8fe8700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6ca00a2c20 0x7f6ca00a50e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4004630 tx=0x7f6ca400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:20:55.311 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.309+0000 7f6caa7fc700 1 -- 172.21.15.165:0/2921655471 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6cb0031070 con 0x7f6cb4101270 2024-05-09T08:20:55.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:20:55.516+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6cb4062760 con 0x7f6ca00a2c20 2024-05-09T08:20:55.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:55 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:20:55.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:55 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:20:55.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:55 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:56.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:55 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:20:56.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:55 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:20:56.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:55 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:56.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:55 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:20:56.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:55 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:20:56.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:55 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:20:57.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:56 smithi028 ceph-mon[30119]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:20:57.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:56 smithi028 ceph-mon[30119]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:20:57.086 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:56 smithi165 ceph-mon[34540]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:20:57.087 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:56 smithi165 ceph-mon[34540]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:20:57.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:56 smithi184 ceph-mon[34509]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:20:57.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:56 smithi184 ceph-mon[34509]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:20:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:58 smithi165 ceph-mon[34540]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:20:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:58 smithi165 ceph-mon[34540]: from='client.? 172.21.15.165:0/2175457655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e"}]: dispatch 2024-05-09T08:20:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:58 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e"}]: dispatch 2024-05-09T08:20:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:58 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e"}]': finished 2024-05-09T08:20:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:58 smithi165 ceph-mon[34540]: osdmap e48: 8 total, 7 up, 8 in 2024-05-09T08:20:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:20:59.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:58 smithi184 ceph-mon[34509]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:20:59.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:58 smithi184 ceph-mon[34509]: from='client.? 172.21.15.165:0/2175457655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e"}]: dispatch 2024-05-09T08:20:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:58 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e"}]: dispatch 2024-05-09T08:20:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:58 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e"}]': finished 2024-05-09T08:20:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:58 smithi184 ceph-mon[34509]: osdmap e48: 8 total, 7 up, 8 in 2024-05-09T08:20:59.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:20:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:58 smithi028 ceph-mon[30119]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:20:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:58 smithi028 ceph-mon[30119]: from='client.? 172.21.15.165:0/2175457655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e"}]: dispatch 2024-05-09T08:20:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:58 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e"}]: dispatch 2024-05-09T08:20:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:58 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e"}]': finished 2024-05-09T08:20:59.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:58 smithi028 ceph-mon[30119]: osdmap e48: 8 total, 7 up, 8 in 2024-05-09T08:20:59.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:00.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:20:59 smithi184 ceph-mon[34509]: from='client.? 172.21.15.165:0/1402271143' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:21:00.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:20:59 smithi165 ceph-mon[34540]: from='client.? 172.21.15.165:0/1402271143' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:21:00.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:20:59 smithi028 ceph-mon[30119]: from='client.? 172.21.15.165:0/1402271143' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:21:01.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:00 smithi165 ceph-mon[34540]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:01.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:00 smithi184 ceph-mon[34509]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:01.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:00 smithi028 ceph-mon[30119]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:03.088 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:02 smithi165 ceph-mon[34540]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:02 smithi184 ceph-mon[34509]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:02 smithi028 ceph-mon[30119]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:05.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:04 smithi165 ceph-mon[34540]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:05.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:04 smithi184 ceph-mon[34509]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:05.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:04 smithi028 ceph-mon[30119]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:06.077 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:05 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T08:21:06.077 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:05 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:06.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:05 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T08:21:06.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:05 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:06.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:05 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T08:21:06.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:05 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:07.066 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:06 smithi165 ceph-mon[34540]: Deploying daemon osd.7 on smithi165 2024-05-09T08:21:07.067 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:06 smithi165 ceph-mon[34540]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:07.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:06 smithi184 ceph-mon[34509]: Deploying daemon osd.7 on smithi165 2024-05-09T08:21:07.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:06 smithi184 ceph-mon[34509]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:07.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:06 smithi028 ceph-mon[30119]: Deploying daemon osd.7 on smithi165 2024-05-09T08:21:07.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:06 smithi028 ceph-mon[30119]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:09.077 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:08 smithi165 ceph-mon[34540]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:09.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:08 smithi184 ceph-mon[34509]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:09.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:08 smithi028 ceph-mon[30119]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:11.061 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:10 smithi165 ceph-mon[34540]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:11.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:10 smithi184 ceph-mon[34509]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:11.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:10 smithi028 ceph-mon[30119]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:12.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:12.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:12.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:12.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:12.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:12.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:12.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:12.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:12.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:13.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:12 smithi165 ceph-mon[34540]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:12 smithi184 ceph-mon[34509]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:12 smithi028 ceph-mon[30119]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:14.790 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.785+0000 7f6caa7fc700 1 -- 172.21.15.165:0/2921655471 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6cb4062760 con 0x7f6ca00a2c20 2024-05-09T08:21:14.791 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 7 on host 'smithi165' 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6ca00a2c20 msgr2=0x7f6ca00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6ca00a2c20 0x7f6ca00a50e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4004630 tx=0x7f6ca400b3c0 comp rx=0 tx=0).stop 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 msgr2=0x7f6cb4198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 0x7f6cb4198590 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6cb000c010 tx=0x7f6cb0009510 comp rx=0 tx=0).stop 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 shutdown_connections 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6ca00a2c20 0x7f6ca00a50e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb4102e10 0x7f6cb419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cb4101270 0x7f6cb4198590 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 --2- 172.21.15.165:0/2921655471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cb4102470 0x7f6cb4198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 >> 172.21.15.165:0/2921655471 conn(0x7f6cb40fc810 msgr2=0x7f6cb4105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:21:14.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 shutdown_connections 2024-05-09T08:21:14.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:21:14.787+0000 7f6cbba4d700 1 -- 172.21.15.165:0/2921655471 wait complete. 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='osd.7 [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:21:15.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:15 smithi165 ceph-mon[34540]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:21:15.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:15.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:15.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:15.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:15.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='osd.7 [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:21:15.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:15 smithi184 ceph-mon[34509]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:21:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:15.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:15.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='osd.7 [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:21:15.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:15 smithi028 ceph-mon[30119]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:21:15.460 DEBUG:teuthology.orchestra.run.smithi165:osd.7> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.7.service 2024-05-09T08:21:15.462 INFO:tasks.cephadm:Deploying osd.8 on smithi184 with /dev/vg_nvme/lv_2... 2024-05-09T08:21:15.462 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T08:21:15.710 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:21:17.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:16 smithi165 ceph-mon[34540]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:17.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:16 smithi165 ceph-mon[34540]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T08:21:17.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:16 smithi165 ceph-mon[34540]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T08:21:17.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:16 smithi165 ceph-mon[34540]: from='osd.7 [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:21:17.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:16 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:17.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:16 smithi165 ceph-mon[34540]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:21:17.016 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:21:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:21:16.784+0000 7f162166d700 -1 osd.7 0 waiting for initial osdmap 2024-05-09T08:21:17.016 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:21:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:21:16.793+0000 7f161d460700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:21:17.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:16 smithi184 ceph-mon[34509]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:17.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:16 smithi184 ceph-mon[34509]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T08:21:17.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:16 smithi184 ceph-mon[34509]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T08:21:17.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:16 smithi184 ceph-mon[34509]: from='osd.7 [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:21:17.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:17.045 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:16 smithi184 ceph-mon[34509]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:21:17.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:16 smithi028 ceph-mon[30119]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:17.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:16 smithi028 ceph-mon[30119]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T08:21:17.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:16 smithi028 ceph-mon[30119]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T08:21:17.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:16 smithi028 ceph-mon[30119]: from='osd.7 [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:21:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:17.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:16 smithi028 ceph-mon[30119]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:21:17.737 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T08:21:17.738 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T08:21:17.738 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-05-09T08:21:17.738 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-05-09T08:21:17.738 INFO:teuthology.orchestra.run.smithi184.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0144946 s, 723 MB/s 2024-05-09T08:21:17.738 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-05-09T08:21:18.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:17 smithi184 ceph-mon[34509]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:21:18.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:17 smithi184 ceph-mon[34509]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T08:21:18.043 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:17 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:18.044 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:17 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:18.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:17 smithi165 ceph-mon[34540]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:21:18.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:17 smithi165 ceph-mon[34540]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T08:21:18.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:18.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:18.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:17 smithi028 ceph-mon[30119]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:21:18.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:17 smithi028 ceph-mon[30119]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T08:21:18.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:17 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:18.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:17 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:18.374 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_2 2024-05-09T08:21:18.621 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:21:19.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:21:19.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:21:19.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:19.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: osd.7 [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] boot 2024-05-09T08:21:19.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: osdmap e51: 8 total, 8 up, 8 in 2024-05-09T08:21:19.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:19.015 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:19.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:19.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:19.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:19.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:19.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.016 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:18 smithi184 ceph-mon[34509]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T08:21:19.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:21:19.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:21:19.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:19.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: osd.7 [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] boot 2024-05-09T08:21:19.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: osdmap e51: 8 total, 8 up, 8 in 2024-05-09T08:21:19.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:19.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:19.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:19.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:19.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:19.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:19.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:18 smithi165 ceph-mon[34540]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T08:21:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:21:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:21:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:21:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: osd.7 [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] boot 2024-05-09T08:21:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: osdmap e51: 8 total, 8 up, 8 in 2024-05-09T08:21:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:21:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:19.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:19.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:19.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:19.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:19.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:19.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:18 smithi028 ceph-mon[30119]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T08:21:19.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:19 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi165 2024-05-09T08:21:19.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:19 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi165 to 5696M 2024-05-09T08:21:19.970 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:19 smithi184 ceph-mon[34509]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T08:21:20.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:19 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi165 2024-05-09T08:21:20.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:19 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi165 to 5696M 2024-05-09T08:21:20.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:19 smithi165 ceph-mon[34540]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T08:21:20.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:19 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi165 2024-05-09T08:21:20.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:19 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi165 to 5696M 2024-05-09T08:21:20.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:19 smithi028 ceph-mon[30119]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T08:21:20.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.182+0000 7ff6735c1700 1 -- 172.21.15.184:0/483303763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c103f40 msgr2=0x7ff66c10a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:20.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.182+0000 7ff6735c1700 1 --2- 172.21.15.184:0/483303763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c103f40 0x7ff66c10a7f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff66400b0a0 tx=0x7ff66400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:21:20.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.182+0000 7ff6735c1700 1 -- 172.21.15.184:0/483303763 shutdown_connections 2024-05-09T08:21:20.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.182+0000 7ff6735c1700 1 --2- 172.21.15.184:0/483303763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c103f40 0x7ff66c10a7f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:20.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.182+0000 7ff6735c1700 1 --2- 172.21.15.184:0/483303763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66c1023a0 0x7ff66c1027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:20.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.182+0000 7ff6735c1700 1 --2- 172.21.15.184:0/483303763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c1035a0 0x7ff66c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.182+0000 7ff6735c1700 1 -- 172.21.15.184:0/483303763 >> 172.21.15.184:0/483303763 conn(0x7ff66c0fd940 msgr2=0x7ff66c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 -- 172.21.15.184:0/483303763 shutdown_connections 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 -- 172.21.15.184:0/483303763 wait complete. 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 Processor -- start 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 -- start start 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c1023a0 0x7ff66c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66c1035a0 0x7ff66c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c103f40 0x7ff66c19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c10dfe0 con 0x7ff66c103f40 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c10de60 con 0x7ff66c1035a0 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff6735c1700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c10e160 con 0x7ff66c1023a0 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff67135d700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c1023a0 0x7ff66c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff67135d700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c1023a0 0x7ff66c198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:39140/0 (socket says 172.21.15.184:39140) 2024-05-09T08:21:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.183+0000 7ff67135d700 1 -- 172.21.15.184:0/2251585797 learned_addr learned my addr 172.21.15.184:0/2251585797 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff671b5e700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c103f40 0x7ff66c19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff670b5c700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66c1035a0 0x7ff66c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff67135d700 1 -- 172.21.15.184:0/2251585797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66c1035a0 msgr2=0x7ff66c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff67135d700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66c1035a0 0x7ff66c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff67135d700 1 -- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c103f40 msgr2=0x7ff66c19fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff67135d700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c103f40 0x7ff66c19fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff67135d700 1 -- 172.21.15.184:0/2251585797 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66c1a01c0 con 0x7ff66c1023a0 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff67135d700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c1023a0 0x7ff66c198560 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c9a0 tx=0x7ff66800cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff6627fc700 1 -- 172.21.15.184:0/2251585797 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff668004220 con 0x7ff66c1023a0 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff6627fc700 1 -- 172.21.15.184:0/2251585797 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff668007d40 con 0x7ff66c1023a0 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff66c1a04b0 con 0x7ff66c1023a0 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff6627fc700 1 -- 172.21.15.184:0/2251585797 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66800f450 con 0x7ff66c1023a0 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.184+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff66c1a0a70 con 0x7ff66c1023a0 2024-05-09T08:21:20.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.185+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff66c069010 con 0x7ff66c1023a0 2024-05-09T08:21:20.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.186+0000 7ff6627fc700 1 -- 172.21.15.184:0/2251585797 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff668007860 con 0x7ff66c1023a0 2024-05-09T08:21:20.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.187+0000 7ff6627fc700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff6580a2c20 0x7ff6580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:21:20.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.187+0000 7ff6627fc700 1 -- 172.21.15.184:0/2251585797 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7ff6680ce4b0 con 0x7ff66c1023a0 2024-05-09T08:21:20.190 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.187+0000 7ff670b5c700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff6580a2c20 0x7ff6580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:21:20.190 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.188+0000 7ff670b5c700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff6580a2c20 0x7ff6580a50e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff65c006fd0 tx=0x7ff65c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:21:20.192 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.192+0000 7ff6627fc700 1 -- 172.21.15.184:0/2251585797 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff668014030 con 0x7ff66c1023a0 2024-05-09T08:21:20.398 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:20.397+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff66c062760 con 0x7ff6580a2c20 2024-05-09T08:21:21.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:21 smithi165 ceph-mon[34540]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:21.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:21 smithi165 ceph-mon[34540]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T08:21:21.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:21 smithi165 ceph-mon[34540]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:21:21.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:21 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:21:21.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:21 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:21:21.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:21 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:21.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:21 smithi184 ceph-mon[34509]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:21.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:21 smithi184 ceph-mon[34509]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T08:21:21.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:21 smithi184 ceph-mon[34509]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:21:21.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:21 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:21:21.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:21 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:21:21.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:21 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:21.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:21 smithi028 ceph-mon[30119]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:21.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:21 smithi028 ceph-mon[30119]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T08:21:21.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:21 smithi028 ceph-mon[30119]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:21:21.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:21 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:21:21.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:21 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:21:21.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:21 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:22.417 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:22 smithi184 ceph-mon[34509]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T08:21:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:22 smithi165 ceph-mon[34540]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T08:21:22.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:22 smithi028 ceph-mon[30119]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T08:21:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:23 smithi165 ceph-mon[34540]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:23 smithi165 ceph-mon[34540]: from='client.? 172.21.15.184:0/3059063164' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12434642-450b-4e7b-a86a-4f32e76d2286"}]: dispatch 2024-05-09T08:21:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:23 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12434642-450b-4e7b-a86a-4f32e76d2286"}]: dispatch 2024-05-09T08:21:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:23 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12434642-450b-4e7b-a86a-4f32e76d2286"}]': finished 2024-05-09T08:21:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:23 smithi165 ceph-mon[34540]: osdmap e56: 9 total, 8 up, 9 in 2024-05-09T08:21:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:23 smithi184 ceph-mon[34509]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:23 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/3059063164' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12434642-450b-4e7b-a86a-4f32e76d2286"}]: dispatch 2024-05-09T08:21:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:23 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12434642-450b-4e7b-a86a-4f32e76d2286"}]: dispatch 2024-05-09T08:21:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:23 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12434642-450b-4e7b-a86a-4f32e76d2286"}]': finished 2024-05-09T08:21:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:23 smithi184 ceph-mon[34509]: osdmap e56: 9 total, 8 up, 9 in 2024-05-09T08:21:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:23.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:23 smithi028 ceph-mon[30119]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:23.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:23 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/3059063164' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12434642-450b-4e7b-a86a-4f32e76d2286"}]: dispatch 2024-05-09T08:21:23.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:23 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12434642-450b-4e7b-a86a-4f32e76d2286"}]: dispatch 2024-05-09T08:21:23.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:23 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12434642-450b-4e7b-a86a-4f32e76d2286"}]': finished 2024-05-09T08:21:23.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:23 smithi028 ceph-mon[30119]: osdmap e56: 9 total, 8 up, 9 in 2024-05-09T08:21:23.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:24.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:24 smithi165 ceph-mon[34540]: from='client.? 172.21.15.184:0/1750900713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:21:24.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:24 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/1750900713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:21:24.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:24 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/1750900713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:21:25.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:25 smithi165 ceph-mon[34540]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:25.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:25 smithi184 ceph-mon[34509]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:25.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:25 smithi028 ceph-mon[30119]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:27.548 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:27 smithi184 ceph-mon[34509]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:27.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:27 smithi165 ceph-mon[34540]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:27.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:27 smithi028 ceph-mon[30119]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:29.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:29 smithi184 ceph-mon[34509]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:29.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:29 smithi028 ceph-mon[30119]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:29.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:29 smithi165 ceph-mon[34540]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:31 smithi184 ceph-mon[34509]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:31 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T08:21:31.550 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:31 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:31.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:31 smithi165 ceph-mon[34540]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:31.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:31 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T08:21:31.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:31 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:31.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:31 smithi028 ceph-mon[30119]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:31.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:31 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T08:21:31.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:31 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:32.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:32 smithi184 ceph-mon[34509]: Deploying daemon osd.8 on smithi184 2024-05-09T08:21:32.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:32 smithi028 ceph-mon[30119]: Deploying daemon osd.8 on smithi184 2024-05-09T08:21:32.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:32 smithi165 ceph-mon[34540]: Deploying daemon osd.8 on smithi184 2024-05-09T08:21:33.579 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:33 smithi184 ceph-mon[34509]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:33.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:33 smithi028 ceph-mon[30119]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:33.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:33 smithi165 ceph-mon[34540]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:35.581 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:35 smithi184 ceph-mon[34509]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:35 smithi028 ceph-mon[30119]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:35.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:35 smithi165 ceph-mon[34540]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:37.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:37 smithi184 ceph-mon[34509]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:37.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:37.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:37.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:37.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:37 smithi165 ceph-mon[34540]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:37.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:37.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:37.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:37.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:37 smithi028 ceph-mon[30119]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:37.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:37.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:37.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:39.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:39 smithi184 ceph-mon[34509]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:39.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:39.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:39.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:39.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:39.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:39 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:39.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:39 smithi165 ceph-mon[34540]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:39.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:39.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:39.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:39.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:39.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:39 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:39 smithi028 ceph-mon[30119]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:39 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:40.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.117+0000 7ff6627fc700 1 -- 172.21.15.184:0/2251585797 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff66c062760 con 0x7ff6580a2c20 2024-05-09T08:21:40.120 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 8 on host 'smithi184' 2024-05-09T08:21:40.121 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff6580a2c20 msgr2=0x7ff6580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:40.121 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff6580a2c20 0x7ff6580a50e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff65c006fd0 tx=0x7ff65c008040 comp rx=0 tx=0).stop 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c1023a0 msgr2=0x7ff66c198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c1023a0 0x7ff66c198560 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c9a0 tx=0x7ff66800cd60 comp rx=0 tx=0).stop 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 shutdown_connections 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff6580a2c20 0x7ff6580a50e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff66c1023a0 0x7ff66c198560 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c103f40 0x7ff66c19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 --2- 172.21.15.184:0/2251585797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66c1035a0 0x7ff66c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.119+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 >> 172.21.15.184:0/2251585797 conn(0x7ff66c0fd940 msgr2=0x7ff66c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.120+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 shutdown_connections 2024-05-09T08:21:40.122 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:21:40.120+0000 7ff6735c1700 1 -- 172.21.15.184:0/2251585797 wait complete. 2024-05-09T08:21:40.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:40 smithi184 ceph-mon[34509]: from='osd.8 [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:21:40.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:40 smithi184 ceph-mon[34509]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:21:40.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:40.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:40.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:40.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:40 smithi165 ceph-mon[34540]: from='osd.8 [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:21:40.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:40 smithi165 ceph-mon[34540]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:21:40.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:40.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:40.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:40.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:40 smithi028 ceph-mon[30119]: from='osd.8 [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:21:40.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:40 smithi028 ceph-mon[30119]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:21:40.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:40.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:40.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:21:40.836 DEBUG:teuthology.orchestra.run.smithi184:osd.8> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.8.service 2024-05-09T08:21:40.838 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-05-09T08:21:40.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T08:21:41.097 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:21:41.318 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:41 smithi184 ceph-mon[34509]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:41.318 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:41 smithi184 ceph-mon[34509]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T08:21:41.318 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:41 smithi184 ceph-mon[34509]: from='osd.8 [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:21:41.318 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:41 smithi184 ceph-mon[34509]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T08:21:41.318 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:41 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:41.318 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:41 smithi184 ceph-mon[34509]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:21:41.319 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:21:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:21:41.199+0000 7f4747b6b700 -1 osd.8 0 waiting for initial osdmap 2024-05-09T08:21:41.319 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:21:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:21:41.208+0000 7f474395e700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:21:41.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:41 smithi028 ceph-mon[30119]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:41.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:41 smithi028 ceph-mon[30119]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T08:21:41.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:41 smithi028 ceph-mon[30119]: from='osd.8 [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:21:41.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:41 smithi028 ceph-mon[30119]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T08:21:41.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:41 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:41.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:41 smithi028 ceph-mon[30119]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:21:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:41 smithi165 ceph-mon[34540]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:41 smithi165 ceph-mon[34540]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T08:21:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:41 smithi165 ceph-mon[34540]: from='osd.8 [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:21:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:41 smithi165 ceph-mon[34540]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T08:21:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:41 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:41 smithi165 ceph-mon[34540]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:21:42.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:42 smithi184 ceph-mon[34509]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:21:42.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:42 smithi184 ceph-mon[34509]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T08:21:42.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:42.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:42.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:42 smithi165 ceph-mon[34540]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:21:42.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:42 smithi165 ceph-mon[34540]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T08:21:42.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:42.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:42.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:42 smithi028 ceph-mon[30119]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:21:42.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:42 smithi028 ceph-mon[30119]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T08:21:42.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:42.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:43.331 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T08:21:43.331 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T08:21:43.332 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-05-09T08:21:43.332 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-05-09T08:21:43.332 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0099089 s, 1.1 GB/s 2024-05-09T08:21:43.332 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-05-09T08:21:43.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:43 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:21:43.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:43 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:21:43.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:43 smithi184 ceph-mon[34509]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:43.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:43 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:43.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:43 smithi184 ceph-mon[34509]: osd.8 [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] boot 2024-05-09T08:21:43.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:43 smithi184 ceph-mon[34509]: osdmap e59: 9 total, 9 up, 9 in 2024-05-09T08:21:43.497 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:43 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:43 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:21:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:43 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:21:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:43 smithi165 ceph-mon[34540]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:43 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:43 smithi165 ceph-mon[34540]: osd.8 [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] boot 2024-05-09T08:21:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:43 smithi165 ceph-mon[34540]: osdmap e59: 9 total, 9 up, 9 in 2024-05-09T08:21:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:43 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:43.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:43 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:21:43.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:43 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:21:43.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:43 smithi028 ceph-mon[30119]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:21:43.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:43 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:43.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:43 smithi028 ceph-mon[30119]: osd.8 [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] boot 2024-05-09T08:21:43.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:43 smithi028 ceph-mon[30119]: osdmap e59: 9 total, 9 up, 9 in 2024-05-09T08:21:43.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:43 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:21:44.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-05-09T08:21:44.407 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:21:44.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T08:21:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:44.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:44 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T08:21:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:44.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:44 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T08:21:44.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:44.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:44.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:21:44.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:44.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:44.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:21:44.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:44 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:21:45.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:45 smithi028 ceph-mon[30119]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:21:45.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:45 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi184 2024-05-09T08:21:45.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:45 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi184 to 5696M 2024-05-09T08:21:45.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:45 smithi028 ceph-mon[30119]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T08:21:45.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:45 smithi165 ceph-mon[34540]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:21:45.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:45 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi184 2024-05-09T08:21:45.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:45 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi184 to 5696M 2024-05-09T08:21:45.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:45 smithi165 ceph-mon[34540]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T08:21:45.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:45 smithi184 ceph-mon[34509]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:21:45.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:45 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi184 2024-05-09T08:21:45.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:45 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi184 to 5696M 2024-05-09T08:21:45.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:45 smithi184 ceph-mon[34509]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T08:21:46.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.133+0000 7f6c6b573700 1 -- 172.21.15.28:0/3057685541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64103570 msgr2=0x7f6c64105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.133+0000 7f6c6b573700 1 --2- 172.21.15.28:0/3057685541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64103570 0x7f6c64105a00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009990 tx=0x7f6c54009ca0 comp rx=0 tx=0).stop 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.133+0000 7f6c6b573700 1 -- 172.21.15.28:0/3057685541 shutdown_connections 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.133+0000 7f6c6b573700 1 --2- 172.21.15.28:0/3057685541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c64100c00 0x7f6c64103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.133+0000 7f6c6b573700 1 --2- 172.21.15.28:0/3057685541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c64105f40 0x7f6c641063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.133+0000 7f6c6b573700 1 --2- 172.21.15.28:0/3057685541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64103570 0x7f6c64105a00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.133+0000 7f6c6b573700 1 -- 172.21.15.28:0/3057685541 >> 172.21.15.28:0/3057685541 conn(0x7f6c640fc820 msgr2=0x7f6c640fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.133+0000 7f6c6b573700 1 -- 172.21.15.28:0/3057685541 shutdown_connections 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.133+0000 7f6c6b573700 1 -- 172.21.15.28:0/3057685541 wait complete. 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c6b573700 1 Processor -- start 2024-05-09T08:21:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c6b573700 1 -- start start 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c6b573700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c64100c00 0x7f6c6419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c6b573700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c64103570 0x7f6c6419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c6b573700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64105f40 0x7f6c641a6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c6b573700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641144c0 con 0x7f6c64105f40 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c6b573700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6c64114340 con 0x7f6c64100c00 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c6b573700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6c64114640 con 0x7f6c64103570 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c69b10700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64105f40 0x7f6c641a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c69b10700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64105f40 0x7f6c641a6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44984/0 (socket says 172.21.15.28:44984) 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.134+0000 7f6c69b10700 1 -- 172.21.15.28:0/2242659377 learned_addr learned my addr 172.21.15.28:0/2242659377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c6930f700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c64100c00 0x7f6c6419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c68b0e700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c64103570 0x7f6c6419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:21:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c69b10700 1 -- 172.21.15.28:0/2242659377 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c64103570 msgr2=0x7f6c6419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c69b10700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c64103570 0x7f6c6419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c69b10700 1 -- 172.21.15.28:0/2242659377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c64100c00 msgr2=0x7f6c6419ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c69b10700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c64100c00 0x7f6c6419ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c69b10700 1 -- 172.21.15.28:0/2242659377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c641a66d0 con 0x7f6c64105f40 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c6930f700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c64100c00 0x7f6c6419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c69b10700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64105f40 0x7f6c641a6030 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c007f10 tx=0x7f6c5c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c5a7fc700 1 -- 172.21.15.28:0/2242659377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c004bb0 con 0x7f6c64105f40 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c5a7fc700 1 -- 172.21.15.28:0/2242659377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c5c004d50 con 0x7f6c64105f40 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c641a69c0 con 0x7f6c64105f40 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c5a7fc700 1 -- 172.21.15.28:0/2242659377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c005a60 con 0x7f6c64105f40 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.135+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c641a6f50 con 0x7f6c64105f40 2024-05-09T08:21:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.136+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c64069010 con 0x7f6c64105f40 2024-05-09T08:21:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.138+0000 7f6c5a7fc700 1 -- 172.21.15.28:0/2242659377 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6c5c01a020 con 0x7f6c64105f40 2024-05-09T08:21:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.139+0000 7f6c5a7fc700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6c500a2ca0 0x7f6c500a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:21:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.139+0000 7f6c6930f700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6c500a2ca0 0x7f6c500a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:21:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.139+0000 7f6c5a7fc700 1 -- 172.21.15.28:0/2242659377 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f6c5c0ce4b0 con 0x7f6c64105f40 2024-05-09T08:21:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.139+0000 7f6c6930f700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6c500a2ca0 0x7f6c500a5160 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6c60005fd0 tx=0x7f6c60005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:21:46.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.142+0000 7f6c5a7fc700 1 -- 172.21.15.28:0/2242659377 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6c5c010070 con 0x7f6c64105f40 2024-05-09T08:21:46.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:21:46.362+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6c64062760 con 0x7f6c500a2ca0 2024-05-09T08:21:46.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:46 smithi028 ceph-mon[30119]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T08:21:46.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:46 smithi165 ceph-mon[34540]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T08:21:46.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:46 smithi184 ceph-mon[34509]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T08:21:47.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:47 smithi028 ceph-mon[30119]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:21:47.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:47 smithi028 ceph-mon[30119]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T08:21:47.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:47 smithi028 ceph-mon[30119]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:21:47.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:21:47.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:21:47.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:47 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:47.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:47 smithi165 ceph-mon[34540]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:21:47.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:47 smithi165 ceph-mon[34540]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T08:21:47.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:47 smithi165 ceph-mon[34540]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:21:47.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:47 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:21:47.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:47 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:21:47.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:47 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:47.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:47 smithi184 ceph-mon[34509]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:21:47.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:47 smithi184 ceph-mon[34509]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T08:21:47.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:47 smithi184 ceph-mon[34509]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:21:47.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:21:47.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:21:47.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:47 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:49.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:49 smithi028 ceph-mon[30119]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:21:49.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:49 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2541408334' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00768b2e-17b7-4ba7-b857-f83ea092f65d"}]: dispatch 2024-05-09T08:21:49.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:49 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2541408334' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00768b2e-17b7-4ba7-b857-f83ea092f65d"}]': finished 2024-05-09T08:21:49.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:49 smithi028 ceph-mon[30119]: osdmap e64: 10 total, 9 up, 10 in 2024-05-09T08:21:49.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:49 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:21:49.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:49 smithi165 ceph-mon[34540]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:21:49.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:49 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2541408334' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00768b2e-17b7-4ba7-b857-f83ea092f65d"}]: dispatch 2024-05-09T08:21:49.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:49 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2541408334' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00768b2e-17b7-4ba7-b857-f83ea092f65d"}]': finished 2024-05-09T08:21:49.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:49 smithi165 ceph-mon[34540]: osdmap e64: 10 total, 9 up, 10 in 2024-05-09T08:21:49.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:49 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:21:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:49 smithi184 ceph-mon[34509]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:21:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:49 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2541408334' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00768b2e-17b7-4ba7-b857-f83ea092f65d"}]: dispatch 2024-05-09T08:21:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:49 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2541408334' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00768b2e-17b7-4ba7-b857-f83ea092f65d"}]': finished 2024-05-09T08:21:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:49 smithi184 ceph-mon[34509]: osdmap e64: 10 total, 9 up, 10 in 2024-05-09T08:21:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:49 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:21:50.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:50 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3806622421' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:21:50.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:50 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3806622421' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:21:50.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:50 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3806622421' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:21:51.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:51 smithi165 ceph-mon[34540]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-05-09T08:21:51.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:51 smithi184 ceph-mon[34509]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-05-09T08:21:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:51 smithi028 ceph-mon[30119]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-05-09T08:21:53.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:53 smithi165 ceph-mon[34540]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-05-09T08:21:53.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:53 smithi184 ceph-mon[34509]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-05-09T08:21:53.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:53 smithi028 ceph-mon[30119]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-05-09T08:21:55.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:55 smithi165 ceph-mon[34540]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:21:55.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:55 smithi184 ceph-mon[34509]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:21:55.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:55 smithi028 ceph-mon[30119]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:21:57.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:57 smithi028 ceph-mon[30119]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:21:57.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T08:21:57.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:57 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:57.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:57 smithi165 ceph-mon[34540]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:21:57.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T08:21:57.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:57 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:57.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:57 smithi184 ceph-mon[34509]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:21:57.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T08:21:57.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:57 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:21:58.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:58 smithi028 ceph-mon[30119]: Deploying daemon osd.9 on smithi028 2024-05-09T08:21:58.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:58 smithi165 ceph-mon[34540]: Deploying daemon osd.9 on smithi028 2024-05-09T08:21:58.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:58 smithi184 ceph-mon[34509]: Deploying daemon osd.9 on smithi028 2024-05-09T08:21:59.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:21:59 smithi028 ceph-mon[30119]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T08:21:59.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:21:59 smithi165 ceph-mon[34540]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T08:21:59.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:21:59 smithi184 ceph-mon[34509]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T08:22:01.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:01 smithi028 ceph-mon[30119]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:01.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:01 smithi165 ceph-mon[34540]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:01.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:01 smithi184 ceph-mon[34509]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:03.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:03 smithi165 ceph-mon[34540]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:03.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:03.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:03.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:03 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:03.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:03 smithi184 ceph-mon[34509]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:03.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:03.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:03.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:03 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:03.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:03 smithi028 ceph-mon[30119]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:03.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:03.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:03.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:03 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:05 smithi028 ceph-mon[30119]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:05.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:05 smithi165 ceph-mon[34540]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:05.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:05 smithi184 ceph-mon[34509]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:06.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.725+0000 7f6c5a7fc700 1 -- 172.21.15.28:0/2242659377 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6c64062760 con 0x7f6c500a2ca0 2024-05-09T08:22:06.727 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-05-09T08:22:06.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6c500a2ca0 msgr2=0x7f6c500a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:06.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6c500a2ca0 0x7f6c500a5160 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6c60005fd0 tx=0x7f6c60005e30 comp rx=0 tx=0).stop 2024-05-09T08:22:06.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64105f40 msgr2=0x7f6c641a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:06.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64105f40 0x7f6c641a6030 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c007f10 tx=0x7f6c5c00d3b0 comp rx=0 tx=0).stop 2024-05-09T08:22:06.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 shutdown_connections 2024-05-09T08:22:06.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6c500a2ca0 0x7f6c500a5160 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:06.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c64105f40 0x7f6c641a6030 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:06.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c64100c00 0x7f6c6419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:06.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 --2- 172.21.15.28:0/2242659377 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c64103570 0x7f6c6419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:06.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 >> 172.21.15.28:0/2242659377 conn(0x7f6c640fc820 msgr2=0x7f6c641018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:22:06.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 shutdown_connections 2024-05-09T08:22:06.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:22:06.727+0000 7f6c6b573700 1 -- 172.21.15.28:0/2242659377 wait complete. 2024-05-09T08:22:07.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:07 smithi028 ceph-mon[30119]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:07.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:07 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:07.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:07 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:07.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:07 smithi028 ceph-mon[30119]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T08:22:07.386 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.9.service 2024-05-09T08:22:07.389 INFO:tasks.cephadm:Deploying osd.10 on smithi165 with /dev/vg_nvme/lv_1... 2024-05-09T08:22:07.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T08:22:07.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:07 smithi165 ceph-mon[34540]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:07.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:07 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:07.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:07 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:07.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:07 smithi165 ceph-mon[34540]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T08:22:07.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:07 smithi184 ceph-mon[34509]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:07.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:07 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:07.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:07 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:07.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:07 smithi184 ceph-mon[34509]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T08:22:07.638 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:22:08.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:08 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:08.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:08 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:08.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:08 smithi165 ceph-mon[34540]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:08.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:08 smithi165 ceph-mon[34540]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T08:22:08.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:08 smithi165 ceph-mon[34540]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T08:22:08.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:08 smithi165 ceph-mon[34540]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:22:08.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:08 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:08.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:08 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:08.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:08 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:08.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:08 smithi184 ceph-mon[34509]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:08.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:08 smithi184 ceph-mon[34509]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T08:22:08.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:08 smithi184 ceph-mon[34509]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T08:22:08.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:08 smithi184 ceph-mon[34509]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:22:08.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:08 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:08.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:08 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:08.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:08 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:08.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:08 smithi028 ceph-mon[30119]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:08.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:08 smithi028 ceph-mon[30119]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T08:22:08.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:08 smithi028 ceph-mon[30119]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T08:22:08.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:08 smithi028 ceph-mon[30119]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-09T08:22:08.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:08 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:09.434 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:22:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:22:09.138+0000 7f2f65dbd700 -1 osd.9 0 waiting for initial osdmap 2024-05-09T08:22:09.434 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:22:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:22:09.150+0000 7f2f5dba8700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:22:09.634 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T08:22:09.635 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T08:22:09.635 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-05-09T08:22:09.635 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-05-09T08:22:09.635 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131706 s, 796 MB/s 2024-05-09T08:22:09.635 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-05-09T08:22:09.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:09 smithi028 ceph-mon[30119]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:22:09.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:09 smithi028 ceph-mon[30119]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T08:22:09.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:09 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:09.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:09 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:09.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:09 smithi028 ceph-mon[30119]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' 2024-05-09T08:22:09.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:09 smithi165 ceph-mon[34540]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:22:09.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:09 smithi165 ceph-mon[34540]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T08:22:09.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:09 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:09.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:09 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:09.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:09 smithi165 ceph-mon[34540]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' 2024-05-09T08:22:10.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:09 smithi184 ceph-mon[34509]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-09T08:22:10.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:09 smithi184 ceph-mon[34509]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T08:22:10.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:09 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:10.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:09 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:10.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:09 smithi184 ceph-mon[34509]: from='osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237]' entity='osd.9' 2024-05-09T08:22:10.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_1 2024-05-09T08:22:10.541 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:22:10.848 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:10 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:22:10.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:10 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:22:10.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:10 smithi028 ceph-mon[30119]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:10.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:10 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:10.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:10 smithi028 ceph-mon[30119]: osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] boot 2024-05-09T08:22:10.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:10 smithi028 ceph-mon[30119]: osdmap e67: 10 total, 10 up, 10 in 2024-05-09T08:22:10.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:10 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:10.922 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:10 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:22:10.923 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:10 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:22:10.923 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:10 smithi165 ceph-mon[34540]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:10.923 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:10 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:10.923 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:10 smithi165 ceph-mon[34540]: osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] boot 2024-05-09T08:22:10.923 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:10 smithi165 ceph-mon[34540]: osdmap e67: 10 total, 10 up, 10 in 2024-05-09T08:22:10.923 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:10 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:11.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:10 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:22:11.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:10 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:22:11.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:10 smithi184 ceph-mon[34509]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:22:11.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:10 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:11.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:10 smithi184 ceph-mon[34509]: osd.9 [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] boot 2024-05-09T08:22:11.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:10 smithi184 ceph-mon[34509]: osdmap e67: 10 total, 10 up, 10 in 2024-05-09T08:22:11.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:10 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:22:12.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.259+0000 7f91284ca700 1 -- 172.21.15.165:0/1133746158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201057c0 msgr2=0x7f9120107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:12.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.259+0000 7f91284ca700 1 --2- 172.21.15.165:0/1133746158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201057c0 0x7f9120107bb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9110009990 tx=0x7f9110009ca0 comp rx=0 tx=0).stop 2024-05-09T08:22:12.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.259+0000 7f91284ca700 1 -- 172.21.15.165:0/1133746158 shutdown_connections 2024-05-09T08:22:12.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.259+0000 7f91284ca700 1 --2- 172.21.15.165:0/1133746158 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91201080f0 0x7f912010a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:12.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.259+0000 7f91284ca700 1 --2- 172.21.15.165:0/1133746158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9120102e90 0x7f9120105280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:12.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.259+0000 7f91284ca700 1 --2- 172.21.15.165:0/1133746158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201057c0 0x7f9120107bb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:12.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.259+0000 7f91284ca700 1 -- 172.21.15.165:0/1133746158 >> 172.21.15.165:0/1133746158 conn(0x7f91200fc820 msgr2=0x7f91200fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:22:12.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.259+0000 7f91284ca700 1 -- 172.21.15.165:0/1133746158 shutdown_connections 2024-05-09T08:22:12.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.259+0000 7f91284ca700 1 -- 172.21.15.165:0/1133746158 wait complete. 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f91284ca700 1 Processor -- start 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f91284ca700 1 -- start start 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f91284ca700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9120102e90 0x7f9120198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f91284ca700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91201057c0 0x7f9120198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f91284ca700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201080f0 0x7f912019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f91284ca700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f912010df50 con 0x7f91201057c0 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f91284ca700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f912010ddd0 con 0x7f91201080f0 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f91284ca700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f912010e0d0 con 0x7f9120102e90 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126a67700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201080f0 0x7f912019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126a67700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201080f0 0x7f912019fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:50640/0 (socket says 172.21.15.165:50640) 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126a67700 1 -- 172.21.15.165:0/911599381 learned_addr learned my addr 172.21.15.165:0/911599381 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-05-09T08:22:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126266700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9120102e90 0x7f9120198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9125a65700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91201057c0 0x7f9120198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126a67700 1 -- 172.21.15.165:0/911599381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9120102e90 msgr2=0x7f9120198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126a67700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9120102e90 0x7f9120198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126a67700 1 -- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91201057c0 msgr2=0x7f9120198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126a67700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91201057c0 0x7f9120198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126a67700 1 -- 172.21.15.165:0/911599381 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91201a01d0 con 0x7f91201080f0 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9125a65700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91201057c0 0x7f9120198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.260+0000 7f9126a67700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201080f0 0x7f912019fb30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f911800d950 tx=0x7f911800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.261+0000 7f91177fe700 1 -- 172.21.15.165:0/911599381 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118017070 con 0x7f91201080f0 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.261+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91201a04c0 con 0x7f91201080f0 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.261+0000 7f91177fe700 1 -- 172.21.15.165:0/911599381 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9118005020 con 0x7f91201080f0 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.261+0000 7f91177fe700 1 -- 172.21.15.165:0/911599381 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118009ab0 con 0x7f91201080f0 2024-05-09T08:22:12.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.261+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91201a09b0 con 0x7f91201080f0 2024-05-09T08:22:12.266 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.261+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9120069010 con 0x7f91201080f0 2024-05-09T08:22:12.267 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.263+0000 7f91177fe700 1 -- 172.21.15.165:0/911599381 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f91180054d0 con 0x7f91201080f0 2024-05-09T08:22:12.267 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.263+0000 7f91177fe700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f910c0a2c20 0x7f910c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:12.267 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.263+0000 7f91177fe700 1 -- 172.21.15.165:0/911599381 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f91180ce410 con 0x7f91201080f0 2024-05-09T08:22:12.267 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.264+0000 7f9126266700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f910c0a2c20 0x7f910c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:12.267 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.264+0000 7f9126266700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f910c0a2c20 0x7f910c0a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f911c009550 tx=0x7f911c0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:22:12.269 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.267+0000 7f91177fe700 1 -- 172.21.15.165:0/911599381 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f911809bb70 con 0x7f91201080f0 2024-05-09T08:22:12.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi028 2024-05-09T08:22:12.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi028 to 4273M 2024-05-09T08:22:12.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:12.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:12.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:12.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:12.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:12.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:12 smithi165 ceph-mon[34540]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T08:22:12.474 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:12.472+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9120062820 con 0x7f910c0a2c20 2024-05-09T08:22:12.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi028 2024-05-09T08:22:12.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi028 to 4273M 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:12 smithi184 ceph-mon[34509]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi028 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi028 to 4273M 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:12.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:12 smithi028 ceph-mon[30119]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T08:22:13.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:13 smithi165 ceph-mon[34540]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:13.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:13 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:22:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:13 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:22:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:13 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:13.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:13 smithi028 ceph-mon[30119]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:13 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:22:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:13 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:22:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:13 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:13.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:13 smithi184 ceph-mon[34509]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:13.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:13 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:22:13.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:13 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:22:13.709 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:13 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:14.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:14 smithi028 ceph-mon[30119]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:22:14.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:14 smithi165 ceph-mon[34540]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:22:14.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:14 smithi184 ceph-mon[34509]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:22:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:15 smithi028 ceph-mon[30119]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:15 smithi028 ceph-mon[30119]: from='client.? 172.21.15.165:0/2519506083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111970a1-1719-4b83-bedf-de211b2088c7"}]: dispatch 2024-05-09T08:22:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:15 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111970a1-1719-4b83-bedf-de211b2088c7"}]: dispatch 2024-05-09T08:22:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:15 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "111970a1-1719-4b83-bedf-de211b2088c7"}]': finished 2024-05-09T08:22:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:15 smithi028 ceph-mon[30119]: osdmap e69: 11 total, 10 up, 11 in 2024-05-09T08:22:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:15 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:15.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:15 smithi165 ceph-mon[34540]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:15.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:15 smithi165 ceph-mon[34540]: from='client.? 172.21.15.165:0/2519506083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111970a1-1719-4b83-bedf-de211b2088c7"}]: dispatch 2024-05-09T08:22:15.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:15 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111970a1-1719-4b83-bedf-de211b2088c7"}]: dispatch 2024-05-09T08:22:15.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:15 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "111970a1-1719-4b83-bedf-de211b2088c7"}]': finished 2024-05-09T08:22:15.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:15 smithi165 ceph-mon[34540]: osdmap e69: 11 total, 10 up, 11 in 2024-05-09T08:22:15.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:15 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:15.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:15 smithi184 ceph-mon[34509]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:15.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:15 smithi184 ceph-mon[34509]: from='client.? 172.21.15.165:0/2519506083' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111970a1-1719-4b83-bedf-de211b2088c7"}]: dispatch 2024-05-09T08:22:15.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:15 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "111970a1-1719-4b83-bedf-de211b2088c7"}]: dispatch 2024-05-09T08:22:15.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:15 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "111970a1-1719-4b83-bedf-de211b2088c7"}]': finished 2024-05-09T08:22:15.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:15 smithi184 ceph-mon[34509]: osdmap e69: 11 total, 10 up, 11 in 2024-05-09T08:22:15.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:15 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:16.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:16 smithi028 ceph-mon[30119]: from='client.? 172.21.15.165:0/3565025095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:22:16.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:16 smithi165 ceph-mon[34540]: from='client.? 172.21.15.165:0/3565025095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:22:16.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:16 smithi184 ceph-mon[34509]: from='client.? 172.21.15.165:0/3565025095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:22:17.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:17 smithi028 ceph-mon[30119]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:17.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:17 smithi165 ceph-mon[34540]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:17.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:17 smithi184 ceph-mon[34509]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:19.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:19 smithi165 ceph-mon[34540]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:19.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:19 smithi028 ceph-mon[30119]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:19.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:19 smithi184 ceph-mon[34509]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:21.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:21 smithi028 ceph-mon[30119]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:21.689 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:21 smithi165 ceph-mon[34540]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:21.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:21 smithi184 ceph-mon[34509]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:23.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:23 smithi028 ceph-mon[30119]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:23.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T08:22:23.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:23.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:23 smithi165 ceph-mon[34540]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:23.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T08:22:23.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:23 smithi184 ceph-mon[34509]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T08:22:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:24.691 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:24 smithi165 ceph-mon[34540]: Deploying daemon osd.10 on smithi165 2024-05-09T08:22:24.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:24 smithi184 ceph-mon[34509]: Deploying daemon osd.10 on smithi165 2024-05-09T08:22:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:24 smithi028 ceph-mon[30119]: Deploying daemon osd.10 on smithi165 2024-05-09T08:22:25.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:25 smithi165 ceph-mon[34540]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:25.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:25 smithi184 ceph-mon[34509]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:25.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:25 smithi028 ceph-mon[30119]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:27.559 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:27 smithi165 ceph-mon[34540]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:27.736 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:27 smithi028 ceph-mon[30119]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:27.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:27 smithi184 ceph-mon[34509]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:29.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:29 smithi165 ceph-mon[34540]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:29.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:29.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:29.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:29.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:29 smithi184 ceph-mon[34509]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:29.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:29.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:29.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:29.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:29 smithi028 ceph-mon[30119]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:29.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:29.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:29.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:31.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:31 smithi165 ceph-mon[34540]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:31.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:31 smithi184 ceph-mon[34509]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:31.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:31 smithi028 ceph-mon[30119]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:32.293 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.292+0000 7f91177fe700 1 -- 172.21.15.165:0/911599381 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9120062820 con 0x7f910c0a2c20 2024-05-09T08:22:32.293 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 10 on host 'smithi165' 2024-05-09T08:22:32.295 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f910c0a2c20 msgr2=0x7f910c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:32.295 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f910c0a2c20 0x7f910c0a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f911c009550 tx=0x7f911c0091b0 comp rx=0 tx=0).stop 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201080f0 msgr2=0x7f912019fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201080f0 0x7f912019fb30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f911800d950 tx=0x7f911800dc60 comp rx=0 tx=0).stop 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 shutdown_connections 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f910c0a2c20 0x7f910c0a50e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9120102e90 0x7f9120198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91201080f0 0x7f912019fb30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 --2- 172.21.15.165:0/911599381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91201057c0 0x7f9120198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 >> 172.21.15.165:0/911599381 conn(0x7f91200fc820 msgr2=0x7f91201038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 shutdown_connections 2024-05-09T08:22:32.296 INFO:teuthology.orchestra.run.smithi165.stderr:2024-05-09T08:22:32.295+0000 7f91284ca700 1 -- 172.21.15.165:0/911599381 wait complete. 2024-05-09T08:22:32.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:32.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:32.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:32.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='osd.10 [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:22:32.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:32 smithi165 ceph-mon[34540]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:22:32.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:32.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:32.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:32.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='osd.10 [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:22:32.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:32 smithi184 ceph-mon[34509]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:22:32.933 DEBUG:teuthology.orchestra.run.smithi165:osd.10> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.10.service 2024-05-09T08:22:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:32.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:32.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='osd.10 [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:22:32.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:32 smithi028 ceph-mon[30119]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:22:32.937 INFO:tasks.cephadm:Deploying osd.11 on smithi184 with /dev/vg_nvme/lv_1... 2024-05-09T08:22:32.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T08:22:33.183 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:22:33.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:33 smithi165 ceph-mon[34540]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:33.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:33 smithi165 ceph-mon[34540]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T08:22:33.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:33 smithi165 ceph-mon[34540]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T08:22:33.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:33 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:33.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:33 smithi165 ceph-mon[34540]: from='osd.10 [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:22:33.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:33 smithi165 ceph-mon[34540]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:22:33.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:33 smithi184 ceph-mon[34509]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:33.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:33 smithi184 ceph-mon[34509]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T08:22:33.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:33 smithi184 ceph-mon[34509]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T08:22:33.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:33 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:33.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:33 smithi184 ceph-mon[34509]: from='osd.10 [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:22:33.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:33 smithi184 ceph-mon[34509]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:22:33.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:33 smithi028 ceph-mon[30119]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:33.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:33 smithi028 ceph-mon[30119]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T08:22:33.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:33 smithi028 ceph-mon[30119]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T08:22:33.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:33 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:33.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:33 smithi028 ceph-mon[30119]: from='osd.10 [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:22:33.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:33 smithi028 ceph-mon[30119]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-05-09T08:22:35.103 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:22:34 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:22:34.735+0000 7f95fc53e700 -1 osd.10 0 waiting for initial osdmap 2024-05-09T08:22:35.103 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:22:34 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:22:34.750+0000 7f95f4329700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:22:35.272 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T08:22:35.272 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T08:22:35.272 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-05-09T08:22:35.272 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-05-09T08:22:35.272 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113177 s, 926 MB/s 2024-05-09T08:22:35.272 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-05-09T08:22:35.583 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:35 smithi184 ceph-mon[34509]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:35.583 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:35 smithi184 ceph-mon[34509]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:22:35.583 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:35 smithi184 ceph-mon[34509]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T08:22:35.584 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:35 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:35.584 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:35 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:35.584 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:35 smithi184 ceph-mon[34509]: from='osd.10 ' entity='osd.10' 2024-05-09T08:22:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:35 smithi165 ceph-mon[34540]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:35 smithi165 ceph-mon[34540]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:22:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:35 smithi165 ceph-mon[34540]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T08:22:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:35 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:35.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:35 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:35.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:35 smithi165 ceph-mon[34540]: from='osd.10 ' entity='osd.10' 2024-05-09T08:22:35.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:35 smithi028 ceph-mon[30119]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:35 smithi028 ceph-mon[30119]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-05-09T08:22:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:35 smithi028 ceph-mon[30119]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T08:22:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:35 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:35 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:35 smithi028 ceph-mon[30119]: from='osd.10 ' entity='osd.10' 2024-05-09T08:22:35.908 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_1 2024-05-09T08:22:36.149 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:22:36.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:22:36.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:22:36.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: osd.10 [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] boot 2024-05-09T08:22:36.589 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: osdmap e72: 11 total, 11 up, 11 in 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:36.590 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:36 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: osd.10 [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] boot 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: osdmap e72: 11 total, 11 up, 11 in 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:36.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:36.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:36 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: osd.10 [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] boot 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: osdmap e72: 11 total, 11 up, 11 in 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:22:36.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:36.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:36.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:36.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:36 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:37.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:37 smithi184 ceph-mon[34509]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:37.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:37 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi165 2024-05-09T08:22:37.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:37 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi165 to 4272M 2024-05-09T08:22:37.552 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:37 smithi184 ceph-mon[34509]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T08:22:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:37 smithi165 ceph-mon[34540]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:37 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi165 2024-05-09T08:22:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:37 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi165 to 4272M 2024-05-09T08:22:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:37 smithi165 ceph-mon[34540]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T08:22:37.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:37 smithi028 ceph-mon[30119]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:22:37.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:37 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi165 2024-05-09T08:22:37.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:37 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi165 to 4272M 2024-05-09T08:22:37.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:37 smithi028 ceph-mon[30119]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T08:22:37.784 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.782+0000 7fb2360a8700 1 -- 172.21.15.184:0/2950371198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230102380 msgr2=0x7fb2301027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:37.784 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.782+0000 7fb2360a8700 1 --2- 172.21.15.184:0/2950371198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230102380 0x7fb2301027a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb220009990 tx=0x7fb220009ca0 comp rx=0 tx=0).stop 2024-05-09T08:22:37.784 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.782+0000 7fb2360a8700 1 -- 172.21.15.184:0/2950371198 shutdown_connections 2024-05-09T08:22:37.784 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.782+0000 7fb2360a8700 1 --2- 172.21.15.184:0/2950371198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230102380 0x7fb2301027a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:37.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.782+0000 7fb2360a8700 1 --2- 172.21.15.184:0/2950371198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb230103f20 0x7fb23010a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:37.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.782+0000 7fb2360a8700 1 --2- 172.21.15.184:0/2950371198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb230103580 0x7fb2301039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:37.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.782+0000 7fb2360a8700 1 -- 172.21.15.184:0/2950371198 >> 172.21.15.184:0/2950371198 conn(0x7fb2300fd920 msgr2=0x7fb2300ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:22:37.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 -- 172.21.15.184:0/2950371198 shutdown_connections 2024-05-09T08:22:37.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 -- 172.21.15.184:0/2950371198 wait complete. 2024-05-09T08:22:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 Processor -- start 2024-05-09T08:22:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 -- start start 2024-05-09T08:22:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb230102380 0x7fb230198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb230103580 0x7fb230198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230103f20 0x7fb23019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb23010e060 con 0x7fb230103580 2024-05-09T08:22:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb23010dee0 con 0x7fb230102380 2024-05-09T08:22:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.783+0000 7fb2360a8700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb23010e1e0 con 0x7fb230103f20 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22ffff700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230103f20 0x7fb23019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22ffff700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230103f20 0x7fb23019fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:39584/0 (socket says 172.21.15.184:39584) 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22ffff700 1 -- 172.21.15.184:0/318956156 learned_addr learned my addr 172.21.15.184:0/318956156 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22f7fe700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb230102380 0x7fb230198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22ffff700 1 -- 172.21.15.184:0/318956156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb230102380 msgr2=0x7fb230198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22ffff700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb230102380 0x7fb230198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22ffff700 1 -- 172.21.15.184:0/318956156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb230103580 msgr2=0x7fb230198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22ffff700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb230103580 0x7fb230198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22ffff700 1 -- 172.21.15.184:0/318956156 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2301a03b0 con 0x7fb230103f20 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22f7fe700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb230102380 0x7fb230198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22ffff700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230103f20 0x7fb23019fd10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb22800b7b0 tx=0x7fb22800bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22cff9700 1 -- 172.21.15.184:0/318956156 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22800f840 con 0x7fb230103f20 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22cff9700 1 -- 172.21.15.184:0/318956156 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb228017400 con 0x7fb230103f20 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2301a06a0 con 0x7fb230103f20 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb22cff9700 1 -- 172.21.15.184:0/318956156 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22800d630 con 0x7fb230103f20 2024-05-09T08:22:37.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.784+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2301a0c10 con 0x7fb230103f20 2024-05-09T08:22:37.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.785+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb230069010 con 0x7fb230103f20 2024-05-09T08:22:37.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.786+0000 7fb22cff9700 1 -- 172.21.15.184:0/318956156 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb2280175b0 con 0x7fb230103f20 2024-05-09T08:22:37.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.787+0000 7fb22cff9700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb2180a2c20 0x7fb2180a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:37.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.787+0000 7fb22cff9700 1 -- 172.21.15.184:0/318956156 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7fb2280ce840 con 0x7fb230103f20 2024-05-09T08:22:37.790 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.787+0000 7fb22f7fe700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb2180a2c20 0x7fb2180a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:37.790 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.788+0000 7fb22f7fe700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb2180a2c20 0x7fb2180a50e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb220006010 tx=0x7fb22000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:22:37.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.791+0000 7fb22cff9700 1 -- 172.21.15.184:0/318956156 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb228013030 con 0x7fb230103f20 2024-05-09T08:22:37.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:37.996+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb230062760 con 0x7fb2180a2c20 2024-05-09T08:22:38.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:38 smithi184 ceph-mon[34509]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T08:22:38.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:22:38.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:22:38.614 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:38.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:38 smithi028 ceph-mon[30119]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T08:22:38.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:22:38.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:22:38.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:38.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:38 smithi165 ceph-mon[34540]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T08:22:38.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:38 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:22:38.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:38 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:22:38.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:38 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:39.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:39 smithi028 ceph-mon[30119]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:39 smithi028 ceph-mon[30119]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:22:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:39 smithi028 ceph-mon[30119]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T08:22:39.688 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:39 smithi184 ceph-mon[34509]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:39.688 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:39 smithi184 ceph-mon[34509]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:22:39.688 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:39 smithi184 ceph-mon[34509]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T08:22:39.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:39 smithi165 ceph-mon[34540]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:39.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:39 smithi165 ceph-mon[34540]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:22:39.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:39 smithi165 ceph-mon[34540]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T08:22:41.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:41 smithi184 ceph-mon[34509]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-05-09T08:22:41.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:41 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/2459099245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c95c2ce7-e296-4064-b913-311b7b9dcbd6"}]: dispatch 2024-05-09T08:22:41.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:41 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c95c2ce7-e296-4064-b913-311b7b9dcbd6"}]: dispatch 2024-05-09T08:22:41.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:41 smithi184 ceph-mon[34509]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c95c2ce7-e296-4064-b913-311b7b9dcbd6"}]': finished 2024-05-09T08:22:41.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:41 smithi184 ceph-mon[34509]: osdmap e77: 12 total, 11 up, 12 in 2024-05-09T08:22:41.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:41 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:22:41.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:41 smithi028 ceph-mon[30119]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-05-09T08:22:41.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:41 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/2459099245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c95c2ce7-e296-4064-b913-311b7b9dcbd6"}]: dispatch 2024-05-09T08:22:41.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:41 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c95c2ce7-e296-4064-b913-311b7b9dcbd6"}]: dispatch 2024-05-09T08:22:41.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:41 smithi028 ceph-mon[30119]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c95c2ce7-e296-4064-b913-311b7b9dcbd6"}]': finished 2024-05-09T08:22:41.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:41 smithi028 ceph-mon[30119]: osdmap e77: 12 total, 11 up, 12 in 2024-05-09T08:22:41.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:41 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:22:41.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:41 smithi165 ceph-mon[34540]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-05-09T08:22:41.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:41 smithi165 ceph-mon[34540]: from='client.? 172.21.15.184:0/2459099245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c95c2ce7-e296-4064-b913-311b7b9dcbd6"}]: dispatch 2024-05-09T08:22:41.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:41 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c95c2ce7-e296-4064-b913-311b7b9dcbd6"}]: dispatch 2024-05-09T08:22:41.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:41 smithi165 ceph-mon[34540]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c95c2ce7-e296-4064-b913-311b7b9dcbd6"}]': finished 2024-05-09T08:22:41.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:41 smithi165 ceph-mon[34540]: osdmap e77: 12 total, 11 up, 12 in 2024-05-09T08:22:41.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:41 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:22:42.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:42 smithi028 ceph-mon[30119]: from='client.? 172.21.15.184:0/3213602302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:22:42.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:42 smithi165 ceph-mon[34540]: from='client.? 172.21.15.184:0/3213602302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:22:42.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:42 smithi184 ceph-mon[34509]: from='client.? 172.21.15.184:0/3213602302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:22:43.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:43 smithi028 ceph-mon[30119]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-09T08:22:43.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:43 smithi165 ceph-mon[34540]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-09T08:22:43.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:43 smithi184 ceph-mon[34509]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-09T08:22:45.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:45 smithi184 ceph-mon[34509]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-09T08:22:45.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:45 smithi028 ceph-mon[30119]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-09T08:22:45.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:45 smithi165 ceph-mon[34540]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-09T08:22:47.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:47 smithi184 ceph-mon[34509]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:22:47.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:47 smithi028 ceph-mon[30119]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:22:47.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:47 smithi165 ceph-mon[34540]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:22:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:49 smithi184 ceph-mon[34509]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:49 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T08:22:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:49 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:49.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:49 smithi184 ceph-mon[34509]: Deploying daemon osd.11 on smithi184 2024-05-09T08:22:49.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:49 smithi028 ceph-mon[30119]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:49.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:49 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T08:22:49.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:49 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:49.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:49 smithi028 ceph-mon[30119]: Deploying daemon osd.11 on smithi184 2024-05-09T08:22:49.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:49 smithi165 ceph-mon[34540]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:49.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:49 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T08:22:49.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:49 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:49.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:49 smithi165 ceph-mon[34540]: Deploying daemon osd.11 on smithi184 2024-05-09T08:22:51.662 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:51 smithi184 ceph-mon[34509]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:51.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:51 smithi028 ceph-mon[30119]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:51 smithi165 ceph-mon[34540]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:53.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:53 smithi028 ceph-mon[30119]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:53.710 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:53 smithi184 ceph-mon[34509]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:53 smithi165 ceph-mon[34540]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:55.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:55 smithi184 ceph-mon[34509]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:55.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:55 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:55.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:55 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:55.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:55 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:55.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:55 smithi028 ceph-mon[30119]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:55.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:55 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:55 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:55 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:55.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:55 smithi165 ceph-mon[34540]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:55.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:55 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:55.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:55 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:55.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:55 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:57.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:57 smithi184 ceph-mon[34509]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:57.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:57 smithi028 ceph-mon[30119]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:57.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:57 smithi165 ceph-mon[34540]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:58.322 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 11 on host 'smithi184' 2024-05-09T08:22:58.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.321+0000 7fb22cff9700 1 -- 172.21.15.184:0/318956156 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb230062760 con 0x7fb2180a2c20 2024-05-09T08:22:58.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb2180a2c20 msgr2=0x7fb2180a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:58.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb2180a2c20 0x7fb2180a50e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb220006010 tx=0x7fb22000b540 comp rx=0 tx=0).stop 2024-05-09T08:22:58.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230103f20 msgr2=0x7fb23019fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:58.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230103f20 0x7fb23019fd10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb22800b7b0 tx=0x7fb22800bb70 comp rx=0 tx=0).stop 2024-05-09T08:22:58.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 shutdown_connections 2024-05-09T08:22:58.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb2180a2c20 0x7fb2180a50e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:58.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb230103f20 0x7fb23019fd10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:58.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb230102380 0x7fb230198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:58.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 --2- 172.21.15.184:0/318956156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb230103580 0x7fb230198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:58.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.323+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 >> 172.21.15.184:0/318956156 conn(0x7fb2300fd920 msgr2=0x7fb2301069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:22:58.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.324+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 shutdown_connections 2024-05-09T08:22:58.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:22:58.324+0000 7fb2360a8700 1 -- 172.21.15.184:0/318956156 wait complete. 2024-05-09T08:22:58.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:58.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:58.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='osd.11 [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:22:58.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:22:58.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:58 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:58.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='osd.11 [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:22:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:22:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:58 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:58.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:22:58.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:22:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='osd.11 [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:22:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:22:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:22:58.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:58 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:22:59.026 DEBUG:teuthology.orchestra.run.smithi184:osd.11> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.11.service 2024-05-09T08:22:59.028 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-09T08:22:59.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd stat -f json 2024-05-09T08:22:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:22:59.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:59 smithi184 ceph-mon[34509]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:59.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:59 smithi184 ceph-mon[34509]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T08:22:59.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:59 smithi184 ceph-mon[34509]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T08:22:59.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:59 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:22:59.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:59 smithi184 ceph-mon[34509]: from='osd.11 [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:22:59.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:22:59 smithi184 ceph-mon[34509]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:59 smithi028 ceph-mon[30119]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:59.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:59 smithi028 ceph-mon[30119]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T08:22:59.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:59 smithi028 ceph-mon[30119]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T08:22:59.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:59 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:22:59.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:59 smithi028 ceph-mon[30119]: from='osd.11 [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:22:59.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:22:59 smithi028 ceph-mon[30119]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:22:59.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:59 smithi165 ceph-mon[34540]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:22:59.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:59 smithi165 ceph-mon[34540]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T08:22:59.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:59 smithi165 ceph-mon[34540]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T08:22:59.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:59 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:22:59.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:59 smithi165 ceph-mon[34540]: from='osd.11 [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:22:59.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:22:59 smithi165 ceph-mon[34540]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-05-09T08:23:00.108 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:22:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:22:59.842+0000 7f23f01af700 -1 osd.11 0 waiting for initial osdmap 2024-05-09T08:23:00.108 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:22:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:22:59.855+0000 7f23e7f9a700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:23:00.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:00 smithi028 ceph-mon[30119]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:23:00.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:00 smithi028 ceph-mon[30119]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T08:23:00.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:00.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:00 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:00.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:00 smithi028 ceph-mon[30119]: from='osd.11 ' entity='osd.11' 2024-05-09T08:23:00.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:00 smithi184 ceph-mon[34509]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:23:00.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:00 smithi184 ceph-mon[34509]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T08:23:00.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:00.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:00 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:00.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:00 smithi184 ceph-mon[34509]: from='osd.11 ' entity='osd.11' 2024-05-09T08:23:00.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:00 smithi165 ceph-mon[34540]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-05-09T08:23:00.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:00 smithi165 ceph-mon[34540]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T08:23:00.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:00.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:00 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:00.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:00 smithi165 ceph-mon[34540]: from='osd.11 ' entity='osd.11' 2024-05-09T08:23:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.127+0000 7fba02231700 1 -- 172.21.15.28:0/1707060272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1080d0 msgr2=0x7fb9fc10a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.127+0000 7fba02231700 1 --2- 172.21.15.28:0/1707060272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1080d0 0x7fb9fc10a4e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400b0a0 tx=0x7fb9f400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.127+0000 7fba02231700 1 -- 172.21.15.28:0/1707060272 shutdown_connections 2024-05-09T08:23:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.127+0000 7fba02231700 1 --2- 172.21.15.28:0/1707060272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1080d0 0x7fb9fc10a4e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.127+0000 7fba02231700 1 --2- 172.21.15.28:0/1707060272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9fc102e70 0x7fb9fc105260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.127+0000 7fba02231700 1 --2- 172.21.15.28:0/1707060272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9fc1057a0 0x7fb9fc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.127+0000 7fba02231700 1 -- 172.21.15.28:0/1707060272 >> 172.21.15.28:0/1707060272 conn(0x7fb9fc0fc820 msgr2=0x7fb9fc0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.127+0000 7fba02231700 1 -- 172.21.15.28:0/1707060272 shutdown_connections 2024-05-09T08:23:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.127+0000 7fba02231700 1 -- 172.21.15.28:0/1707060272 wait complete. 2024-05-09T08:23:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.128+0000 7fba02231700 1 Processor -- start 2024-05-09T08:23:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.128+0000 7fba02231700 1 -- start start 2024-05-09T08:23:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.128+0000 7fba02231700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9fc102e70 0x7fb9fc1027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.128+0000 7fba02231700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1057a0 0x7fb9fc100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.128+0000 7fba02231700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9fc1080d0 0x7fb9fc101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fba02231700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc10df40 con 0x7fb9fc1057a0 2024-05-09T08:23:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fba02231700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc10ddc0 con 0x7fb9fc102e70 2024-05-09T08:23:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fba02231700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc10e0c0 con 0x7fb9fc1080d0 2024-05-09T08:23:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f2ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1057a0 0x7fb9fc100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f2ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1057a0 0x7fb9fc100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46254/0 (socket says 172.21.15.28:46254) 2024-05-09T08:23:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f2ffd700 1 -- 172.21.15.28:0/1066915466 learned_addr learned my addr 172.21.15.28:0/1066915466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9fb7fe700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9fc102e70 0x7fb9fc1027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9fbfff700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9fc1080d0 0x7fb9fc101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f2ffd700 1 -- 172.21.15.28:0/1066915466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9fc1080d0 msgr2=0x7fb9fc101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f2ffd700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9fc1080d0 0x7fb9fc101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f2ffd700 1 -- 172.21.15.28:0/1066915466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9fc102e70 msgr2=0x7fb9fc1027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f2ffd700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9fc102e70 0x7fb9fc1027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f2ffd700 1 -- 172.21.15.28:0/1066915466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9fc101c10 con 0x7fb9fc1057a0 2024-05-09T08:23:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f2ffd700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1057a0 0x7fb9fc100e20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400e9a0 tx=0x7fb9e400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f97fa700 1 -- 172.21.15.28:0/1066915466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e400ca10 con 0x7fb9fc1057a0 2024-05-09T08:23:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f97fa700 1 -- 172.21.15.28:0/1066915466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9e400cbb0 con 0x7fb9fc1057a0 2024-05-09T08:23:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.129+0000 7fb9f97fa700 1 -- 172.21.15.28:0/1066915466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4018680 con 0x7fb9fc1057a0 2024-05-09T08:23:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.130+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9fc1a2660 con 0x7fb9fc1057a0 2024-05-09T08:23:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.130+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9fc1a2b20 con 0x7fb9fc1057a0 2024-05-09T08:23:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.131+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9fc069010 con 0x7fb9fc1057a0 2024-05-09T08:23:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.132+0000 7fb9f97fa700 1 -- 172.21.15.28:0/1066915466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb9e4010430 con 0x7fb9fc1057a0 2024-05-09T08:23:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.133+0000 7fb9f97fa700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9e80a2c20 0x7fb9e80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.133+0000 7fb9fb7fe700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9e80a2c20 0x7fb9e80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.133+0000 7fb9f97fa700 1 -- 172.21.15.28:0/1066915466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7fb9e40ced50 con 0x7fb9fc1057a0 2024-05-09T08:23:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.133+0000 7fb9fb7fe700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9e80a2c20 0x7fb9e80a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec007c60 tx=0x7fb9ec0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.137+0000 7fb9f97fa700 1 -- 172.21.15.28:0/1066915466 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb9e40d2030 con 0x7fb9fc1057a0 2024-05-09T08:23:01.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.344+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb9fc02db80 con 0x7fb9fc1057a0 2024-05-09T08:23:01.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.345+0000 7fb9f97fa700 1 -- 172.21.15.28:0/1066915466 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7fb9e409c150 con 0x7fb9fc1057a0 2024-05-09T08:23:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9e80a2c20 msgr2=0x7fb9e80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9e80a2c20 0x7fb9e80a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec007c60 tx=0x7fb9ec0073d0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1057a0 msgr2=0x7fb9fc100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1057a0 0x7fb9fc100e20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400e9a0 tx=0x7fb9e400ecb0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 shutdown_connections 2024-05-09T08:23:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9e80a2c20 0x7fb9e80a50e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9fc1080d0 0x7fb9fc101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9fc102e70 0x7fb9fc1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 --2- 172.21.15.28:0/1066915466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9fc1057a0 0x7fb9fc100e20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 >> 172.21.15.28:0/1066915466 conn(0x7fb9fc0fc820 msgr2=0x7fb9fc1037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 shutdown_connections 2024-05-09T08:23:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:01.347+0000 7fba02231700 1 -- 172.21.15.28:0/1066915466 wait complete. 2024-05-09T08:23:01.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:01 smithi184 ceph-mon[34509]: purged_snaps scrub starts 2024-05-09T08:23:01.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:01 smithi184 ceph-mon[34509]: purged_snaps scrub ok 2024-05-09T08:23:01.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:01 smithi184 ceph-mon[34509]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:23:01.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:01 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:01.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:01 smithi184 ceph-mon[34509]: osd.11 [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] boot 2024-05-09T08:23:01.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:01 smithi184 ceph-mon[34509]: osdmap e80: 12 total, 12 up, 12 in 2024-05-09T08:23:01.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:01 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:01.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:01 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1066915466' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T08:23:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:01 smithi028 ceph-mon[30119]: purged_snaps scrub starts 2024-05-09T08:23:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:01 smithi028 ceph-mon[30119]: purged_snaps scrub ok 2024-05-09T08:23:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:01 smithi028 ceph-mon[30119]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:23:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:01 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:01 smithi028 ceph-mon[30119]: osd.11 [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] boot 2024-05-09T08:23:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:01 smithi028 ceph-mon[30119]: osdmap e80: 12 total, 12 up, 12 in 2024-05-09T08:23:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:01 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:01 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1066915466' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T08:23:01.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:01 smithi165 ceph-mon[34540]: purged_snaps scrub starts 2024-05-09T08:23:01.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:01 smithi165 ceph-mon[34540]: purged_snaps scrub ok 2024-05-09T08:23:01.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:01 smithi165 ceph-mon[34540]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:23:01.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:01 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:01.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:01 smithi165 ceph-mon[34540]: osd.11 [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] boot 2024-05-09T08:23:01.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:01 smithi165 ceph-mon[34540]: osdmap e80: 12 total, 12 up, 12 in 2024-05-09T08:23:01.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:01 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:23:01.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:01 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1066915466' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T08:23:02.137 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1715242980,"num_in_osds":12,"osd_in_since":1715242960,"num_remapped_pgs":0} 2024-05-09T08:23:02.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T08:23:02.385 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:23:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T08:23:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: Detected new or changed devices on smithi184 2024-05-09T08:23:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:02 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: Detected new or changed devices on smithi184 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:03.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:03.109 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:02 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:03.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T08:23:03.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: Detected new or changed devices on smithi184 2024-05-09T08:23:03.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:23:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:03.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:02 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:03.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:03 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi184 to 4272M 2024-05-09T08:23:04.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:03 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi184 to 4272M 2024-05-09T08:23:04.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:03 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi184 to 4272M 2024-05-09T08:23:04.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.199+0000 7f59e4b6e700 1 -- 172.21.15.28:0/1276137273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0101220 msgr2=0x7f59e01016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:04.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.199+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/1276137273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0101220 0x7f59e01016a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009990 tx=0x7f59d4009ca0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.199+0000 7f59e4b6e700 1 -- 172.21.15.28:0/1276137273 shutdown_connections 2024-05-09T08:23:04.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.199+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/1276137273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e0101be0 0x7f59e010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.199+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/1276137273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e0107290 0x7f59e0107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.199+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/1276137273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0101220 0x7f59e01016a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.199+0000 7f59e4b6e700 1 -- 172.21.15.28:0/1276137273 >> 172.21.15.28:0/1276137273 conn(0x7f59e00781d0 msgr2=0x7f59e00785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:04.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.199+0000 7f59e4b6e700 1 -- 172.21.15.28:0/1276137273 shutdown_connections 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.199+0000 7f59e4b6e700 1 -- 172.21.15.28:0/1276137273 wait complete. 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59e4b6e700 1 Processor -- start 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59e4b6e700 1 -- start start 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59e4b6e700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e0101220 0x7f59e019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59e4b6e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e0101be0 0x7f59e019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59e4b6e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0107290 0x7f59e01988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59e4b6e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0111500 con 0x7f59e0107290 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59e4b6e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0111380 con 0x7f59e0101be0 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59e4b6e700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0111680 con 0x7f59e0101220 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59ded9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0107290 0x7f59e01988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59ded9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0107290 0x7f59e01988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46278/0 (socket says 172.21.15.28:46278) 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59ded9d700 1 -- 172.21.15.28:0/922075227 learned_addr learned my addr 172.21.15.28:0/922075227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.200+0000 7f59ddd9b700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e0101be0 0x7f59e019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59de59c700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e0101220 0x7f59e019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59ddd9b700 1 -- 172.21.15.28:0/922075227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e0101220 msgr2=0x7f59e019e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59ddd9b700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e0101220 0x7f59e019e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59ddd9b700 1 -- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0107290 msgr2=0x7f59e01988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59ddd9b700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0107290 0x7f59e01988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59ddd9b700 1 -- 172.21.15.28:0/922075227 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e0199040 con 0x7f59e0101be0 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59ded9d700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0107290 0x7f59e01988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59ddd9b700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e0101be0 0x7f59e019ec60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f59d4006010 tx=0x7f59d400ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59cf7fe700 1 -- 172.21.15.28:0/922075227 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4027070 con 0x7f59e0101be0 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59cf7fe700 1 -- 172.21.15.28:0/922075227 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59d4003d70 con 0x7f59e0101be0 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e01992d0 con 0x7f59e0101be0 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59cf7fe700 1 -- 172.21.15.28:0/922075227 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4022e00 con 0x7f59e0101be0 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.201+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e01a55f0 con 0x7f59e0101be0 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.202+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e0102940 con 0x7f59e0101be0 2024-05-09T08:23:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.203+0000 7f59cf7fe700 1 -- 172.21.15.28:0/922075227 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f59d402a070 con 0x7f59e0101be0 2024-05-09T08:23:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.204+0000 7f59cf7fe700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f59c80a2c20 0x7f59c80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.204+0000 7f59de59c700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f59c80a2c20 0x7f59c80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.204+0000 7f59cf7fe700 1 -- 172.21.15.28:0/922075227 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f59d40d01c0 con 0x7f59e0101be0 2024-05-09T08:23:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.204+0000 7f59de59c700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f59c80a2c20 0x7f59c80a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f59e0199a20 tx=0x7f59d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.207+0000 7f59cf7fe700 1 -- 172.21.15.28:0/922075227 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f59d4013070 con 0x7f59e0101be0 2024-05-09T08:23:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.410+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59e0069010 con 0x7f59e0101be0 2024-05-09T08:23:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.411+0000 7f59cf7fe700 1 -- 172.21.15.28:0/922075227 <== mon.2 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18468 (secure 0 0 0) 0x7f59e0069010 con 0x7f59e0101be0 2024-05-09T08:23:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:04.413 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:01.846859+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:23:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f59c80a2c20 msgr2=0x7f59c80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f59c80a2c20 0x7f59c80a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f59e0199a20 tx=0x7f59d0009260 comp rx=0 tx=0).stop 2024-05-09T08:23:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e0101be0 msgr2=0x7f59e019ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e0101be0 0x7f59e019ec60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f59d4006010 tx=0x7f59d400ba90 comp rx=0 tx=0).stop 2024-05-09T08:23:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 shutdown_connections 2024-05-09T08:23:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f59c80a2c20 0x7f59c80a50e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59e0101220 0x7f59e019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59e0107290 0x7f59e01988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 --2- 172.21.15.28:0/922075227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e0101be0 0x7f59e019ec60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 >> 172.21.15.28:0/922075227 conn(0x7f59e00781d0 msgr2=0x7f59e0105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 shutdown_connections 2024-05-09T08:23:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:04.414+0000 7f59e4b6e700 1 -- 172.21.15.28:0/922075227 wait complete. 2024-05-09T08:23:05.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:04 smithi028 ceph-mon[30119]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:05.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/922075227' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:05.081 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T08:23:05.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-09T08:23:05.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:04 smithi184 ceph-mon[34509]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:05.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/922075227' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:05.334 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:23:05.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:04 smithi165 ceph-mon[34540]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:05.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/922075227' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.049+0000 7f6a309fb700 1 -- 172.21.15.28:0/3231606024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c1083c0 msgr2=0x7f6a2c1087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.049+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3231606024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c1083c0 0x7f6a2c1087a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009990 tx=0x7f6a1c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.049+0000 7f6a309fb700 1 -- 172.21.15.28:0/3231606024 shutdown_connections 2024-05-09T08:23:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.049+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3231606024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a2c06b390 0x7f6a2c10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.049+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3231606024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c1083c0 0x7f6a2c1087a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.049+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3231606024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c06a9f0 0x7f6a2c06ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.049+0000 7f6a309fb700 1 -- 172.21.15.28:0/3231606024 >> 172.21.15.28:0/3231606024 conn(0x7f6a2c0781d0 msgr2=0x7f6a2c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.049+0000 7f6a309fb700 1 -- 172.21.15.28:0/3231606024 shutdown_connections 2024-05-09T08:23:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.049+0000 7f6a309fb700 1 -- 172.21.15.28:0/3231606024 wait complete. 2024-05-09T08:23:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a309fb700 1 Processor -- start 2024-05-09T08:23:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a309fb700 1 -- start start 2024-05-09T08:23:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a309fb700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a2c06a9f0 0x7f6a2c1016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a309fb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c06b390 0x7f6a2c101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a309fb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c1083c0 0x7f6a2c102150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a309fb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c1126d0 con 0x7f6a2c06b390 2024-05-09T08:23:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a309fb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c112550 con 0x7f6a2c1083c0 2024-05-09T08:23:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a309fb700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c112850 con 0x7f6a2c06a9f0 2024-05-09T08:23:07.053 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:06 smithi028 ceph-mon[30119]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:07.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a29d9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c06b390 0x7f6a2c101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:07.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a29d9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c06b390 0x7f6a2c101c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44662/0 (socket says 172.21.15.28:44662) 2024-05-09T08:23:07.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a29d9b700 1 -- 172.21.15.28:0/3040572838 learned_addr learned my addr 172.21.15.28:0/3040572838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:07.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.050+0000 7f6a2a59c700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a2c06a9f0 0x7f6a2c1016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:07.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a2ad9d700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c1083c0 0x7f6a2c102150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:07.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a29d9b700 1 -- 172.21.15.28:0/3040572838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a2c06a9f0 msgr2=0x7f6a2c1016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:07.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a29d9b700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a2c06a9f0 0x7f6a2c1016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a29d9b700 1 -- 172.21.15.28:0/3040572838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c1083c0 msgr2=0x7f6a2c102150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:07.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a29d9b700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c1083c0 0x7f6a2c102150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a29d9b700 1 -- 172.21.15.28:0/3040572838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2c1a7080 con 0x7f6a2c06b390 2024-05-09T08:23:07.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a2ad9d700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c1083c0 0x7f6a2c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:07.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a29d9b700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c06b390 0x7f6a2c101c10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c920 tx=0x7f6a2000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:07.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a1b7fe700 1 -- 172.21.15.28:0/3040572838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20007b00 con 0x7f6a2c06b390 2024-05-09T08:23:07.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a1b7fe700 1 -- 172.21.15.28:0/3040572838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a20004d10 con 0x7f6a2c06b390 2024-05-09T08:23:07.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a2c1a7370 con 0x7f6a2c06b390 2024-05-09T08:23:07.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a1b7fe700 1 -- 172.21.15.28:0/3040572838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20005680 con 0x7f6a2c06b390 2024-05-09T08:23:07.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.051+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a2c1a7900 con 0x7f6a2c06b390 2024-05-09T08:23:07.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.052+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a2c103a70 con 0x7f6a2c06b390 2024-05-09T08:23:07.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.054+0000 7f6a1b7fe700 1 -- 172.21.15.28:0/3040572838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6a20004750 con 0x7f6a2c06b390 2024-05-09T08:23:07.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.055+0000 7f6a1b7fe700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6a140a2ca0 0x7f6a140a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:07.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.055+0000 7f6a2a59c700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6a140a2ca0 0x7f6a140a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:07.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.055+0000 7f6a1b7fe700 1 -- 172.21.15.28:0/3040572838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6a200ceac0 con 0x7f6a2c06b390 2024-05-09T08:23:07.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.056+0000 7f6a2a59c700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6a140a2ca0 0x7f6a140a5160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009750 tx=0x7f6a1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:07.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.058+0000 7f6a1b7fe700 1 -- 172.21.15.28:0/3040572838 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6a20014030 con 0x7f6a2c06b390 2024-05-09T08:23:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.262+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6a2c02db80 con 0x7f6a2c06b390 2024-05-09T08:23:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.262+0000 7f6a1b7fe700 1 -- 172.21.15.28:0/3040572838 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f6a2009bf40 con 0x7f6a2c06b390 2024-05-09T08:23:07.263 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6a140a2ca0 msgr2=0x7f6a140a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6a140a2ca0 0x7f6a140a5160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009750 tx=0x7f6a1c005fd0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c06b390 msgr2=0x7f6a2c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c06b390 0x7f6a2c101c10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c920 tx=0x7f6a2000cc30 comp rx=0 tx=0).stop 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 shutdown_connections 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6a140a2ca0 0x7f6a140a5160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a2c06a9f0 0x7f6a2c1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c1083c0 0x7f6a2c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 --2- 172.21.15.28:0/3040572838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a2c06b390 0x7f6a2c101c10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 >> 172.21.15.28:0/3040572838 conn(0x7f6a2c0781d0 msgr2=0x7f6a2c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 shutdown_connections 2024-05-09T08:23:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:07.264+0000 7f6a309fb700 1 -- 172.21.15.28:0/3040572838 wait complete. 2024-05-09T08:23:07.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:06 smithi165 ceph-mon[34540]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:07.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:06 smithi184 ceph-mon[34509]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:07.932 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-05-09T08:23:07.933 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-05-09T08:23:07.933 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-05-09T08:23:07.933 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-05-09T08:23:07.933 INFO:tasks.cephadm:Adding mds.b on smithi165 2024-05-09T08:23:07.933 INFO:tasks.cephadm:Adding mds.e on smithi165 2024-05-09T08:23:07.933 INFO:tasks.cephadm:Adding mds.h on smithi165 2024-05-09T08:23:07.934 INFO:tasks.cephadm:Adding mds.k on smithi165 2024-05-09T08:23:07.934 INFO:tasks.cephadm:Adding mds.c on smithi184 2024-05-09T08:23:07.934 INFO:tasks.cephadm:Adding mds.f on smithi184 2024-05-09T08:23:07.934 INFO:tasks.cephadm:Adding mds.i on smithi184 2024-05-09T08:23:07.934 INFO:tasks.cephadm:Adding mds.l on smithi184 2024-05-09T08:23:07.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi165=b;smithi165=e;smithi165=h;smithi165=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l' 2024-05-09T08:23:08.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3040572838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:23:08.191 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:23:08.316 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3040572838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:23:08.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3040572838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:23:09.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:08 smithi184 ceph-mon[34509]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:09.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:08 smithi028 ceph-mon[30119]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:09.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:08 smithi165 ceph-mon[34540]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:09.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.824+0000 7fd6f98ce700 1 -- 172.21.15.184:0/1980157197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 msgr2=0x7fd6f410a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:09.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.824+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/1980157197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 0x7fd6f410a7f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00b0a0 tx=0x7fd6ec00b3b0 comp rx=0 tx=0).stop 2024-05-09T08:23:09.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 -- 172.21.15.184:0/1980157197 shutdown_connections 2024-05-09T08:23:09.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/1980157197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 0x7fd6f410a7f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:09.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/1980157197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41023a0 0x7fd6f41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:09.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/1980157197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f41035a0 0x7fd6f4103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:09.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 -- 172.21.15.184:0/1980157197 >> 172.21.15.184:0/1980157197 conn(0x7fd6f40fd940 msgr2=0x7fd6f40ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 -- 172.21.15.184:0/1980157197 shutdown_connections 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 -- 172.21.15.184:0/1980157197 wait complete. 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 Processor -- start 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 -- start start 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f41023a0 0x7fd6f4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41035a0 0x7fd6f41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 0x7fd6f419fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f410df70 con 0x7fd6f41023a0 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f410ddf0 con 0x7fd6f41035a0 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.825+0000 7fd6f98ce700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f410e0f0 con 0x7fd6f4103f40 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f37fe700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 0x7fd6f419fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:09.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f37fe700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 0x7fd6f419fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:42884/0 (socket says 172.21.15.184:42884) 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f37fe700 1 -- 172.21.15.184:0/624726872 learned_addr learned my addr 172.21.15.184:0/624726872 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f2ffd700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f41023a0 0x7fd6f4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f27fc700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41035a0 0x7fd6f41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f37fe700 1 -- 172.21.15.184:0/624726872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41035a0 msgr2=0x7fd6f41989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f37fe700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41035a0 0x7fd6f41989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f37fe700 1 -- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f41023a0 msgr2=0x7fd6f4198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f37fe700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f41023a0 0x7fd6f4198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f37fe700 1 -- 172.21.15.184:0/624726872 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f41a00c0 con 0x7fd6f4103f40 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f37fe700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 0x7fd6f419fa20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009800 tx=0x7fd6ec00bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f88cc700 1 -- 172.21.15.184:0/624726872 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec021070 con 0x7fd6f4103f40 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f41a0350 con 0x7fd6f4103f40 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f88cc700 1 -- 172.21.15.184:0/624726872 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6ec007cc0 con 0x7fd6f4103f40 2024-05-09T08:23:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f88cc700 1 -- 172.21.15.184:0/624726872 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec01b7f0 con 0x7fd6f4103f40 2024-05-09T08:23:09.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.826+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f41a0800 con 0x7fd6f4103f40 2024-05-09T08:23:09.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.827+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f4069010 con 0x7fd6f4103f40 2024-05-09T08:23:09.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.828+0000 7fd6f88cc700 1 -- 172.21.15.184:0/624726872 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6ec024070 con 0x7fd6f4103f40 2024-05-09T08:23:09.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.829+0000 7fd6f88cc700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6dc0a2ca0 0x7fd6dc0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:09.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.829+0000 7fd6f88cc700 1 -- 172.21.15.184:0/624726872 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd6ec00f020 con 0x7fd6f4103f40 2024-05-09T08:23:09.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.829+0000 7fd6f2ffd700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6dc0a2ca0 0x7fd6dc0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:09.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.830+0000 7fd6f2ffd700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6dc0a2ca0 0x7fd6dc0a5160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4103400 tx=0x7fd6e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:09.833 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:09.833+0000 7fd6f88cc700 1 -- 172.21.15.184:0/624726872 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd6ec02b020 con 0x7fd6f4103f40 2024-05-09T08:23:10.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.041+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi165=b;smithi165=e;smithi165=h;smithi165=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l", "target": ["mon-mgr", ""]}) -- 0x7fd6f4062a20 con 0x7fd6dc0a2ca0 2024-05-09T08:23:10.050 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mds.all update... 2024-05-09T08:23:10.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.049+0000 7fd6f88cc700 1 -- 172.21.15.184:0/624726872 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd6f4062a20 con 0x7fd6dc0a2ca0 2024-05-09T08:23:10.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6dc0a2ca0 msgr2=0x7fd6dc0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:10.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6dc0a2ca0 0x7fd6dc0a5160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4103400 tx=0x7fd6e400b410 comp rx=0 tx=0).stop 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 msgr2=0x7fd6f419fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 0x7fd6f419fa20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009800 tx=0x7fd6ec00bd70 comp rx=0 tx=0).stop 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 shutdown_connections 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6dc0a2ca0 0x7fd6dc0a5160 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6f4103f40 0x7fd6f419fa20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f41023a0 0x7fd6f4198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 --2- 172.21.15.184:0/624726872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6f41035a0 0x7fd6f41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 >> 172.21.15.184:0/624726872 conn(0x7fd6f40fd940 msgr2=0x7fd6f41068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 shutdown_connections 2024-05-09T08:23:10.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-05-09T08:23:10.052+0000 7fd6f98ce700 1 -- 172.21.15.184:0/624726872 wait complete. 2024-05-09T08:23:10.749 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.a.service 2024-05-09T08:23:10.752 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.d.service 2024-05-09T08:23:10.795 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.g.service 2024-05-09T08:23:10.838 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.j.service 2024-05-09T08:23:10.881 DEBUG:teuthology.orchestra.run.smithi165:mds.b> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.b.service 2024-05-09T08:23:10.883 DEBUG:teuthology.orchestra.run.smithi165:mds.e> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service 2024-05-09T08:23:10.926 DEBUG:teuthology.orchestra.run.smithi165:mds.h> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.h.service 2024-05-09T08:23:10.969 DEBUG:teuthology.orchestra.run.smithi165:mds.k> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.k.service 2024-05-09T08:23:11.013 DEBUG:teuthology.orchestra.run.smithi184:mds.c> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.c.service 2024-05-09T08:23:11.016 DEBUG:teuthology.orchestra.run.smithi184:mds.f> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.f.service 2024-05-09T08:23:11.060 DEBUG:teuthology.orchestra.run.smithi184:mds.i> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.i.service 2024-05-09T08:23:11.102 DEBUG:teuthology.orchestra.run.smithi184:mds.l> sudo journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.l.service 2024-05-09T08:23:11.145 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-09T08:23:11.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:23:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.259+0000 7f5c55f22640 1 -- 172.21.15.28:0/3175441667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 msgr2=0x7f5c50101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.259+0000 7f5c55f22640 1 --2- 172.21.15.28:0/3175441667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 0x7f5c50101610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009900 tx=0x7f5c4402a410 comp rx=0 tx=0).stop 2024-05-09T08:23:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.259+0000 7f5c55f22640 1 -- 172.21.15.28:0/3175441667 shutdown_connections 2024-05-09T08:23:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.259+0000 7f5c55f22640 1 --2- 172.21.15.28:0/3175441667 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c50101b50 0x7f5c501083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.259+0000 7f5c55f22640 1 --2- 172.21.15.28:0/3175441667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 0x7f5c50101610 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.259+0000 7f5c55f22640 1 --2- 172.21.15.28:0/3175441667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c500fffa0 0x7f5c501003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.259+0000 7f5c55f22640 1 -- 172.21.15.28:0/3175441667 >> 172.21.15.28:0/3175441667 conn(0x7f5c500fb510 msgr2=0x7f5c500fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.259+0000 7f5c55f22640 1 -- 172.21.15.28:0/3175441667 shutdown_connections 2024-05-09T08:23:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.259+0000 7f5c55f22640 1 -- 172.21.15.28:0/3175441667 wait complete. 2024-05-09T08:23:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.260+0000 7f5c55f22640 1 Processor -- start 2024-05-09T08:23:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.260+0000 7f5c55f22640 1 -- start start 2024-05-09T08:23:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.260+0000 7f5c55f22640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c500fffa0 0x7f5c50197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.260+0000 7f5c55f22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 0x7f5c501980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.260+0000 7f5c55f22640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c50101b50 0x7f5c5019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.260+0000 7f5c55f22640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5010b3b0 con 0x7f5c50101190 2024-05-09T08:23:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.260+0000 7f5c55f22640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5010b230 con 0x7f5c50101b50 2024-05-09T08:23:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.260+0000 7f5c55f22640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5010b530 con 0x7f5c500fffa0 2024-05-09T08:23:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 0x7f5c501980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 0x7f5c501980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44686/0 (socket says 172.21.15.28:44686) 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4effd640 1 -- 172.21.15.28:0/2945137423 learned_addr learned my addr 172.21.15.28:0/2945137423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4f7fe640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c500fffa0 0x7f5c50197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4ffff640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c50101b50 0x7f5c5019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4effd640 1 -- 172.21.15.28:0/2945137423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c500fffa0 msgr2=0x7f5c50197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4effd640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c500fffa0 0x7f5c50197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4effd640 1 -- 172.21.15.28:0/2945137423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c50101b50 msgr2=0x7f5c5019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4effd640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c50101b50 0x7f5c5019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4effd640 1 -- 172.21.15.28:0/2945137423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5019f7c0 con 0x7f5c50101190 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4effd640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 0x7f5c501980e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009230 tx=0x7f5c44004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4cff9640 1 -- 172.21.15.28:0/2945137423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4403f030 con 0x7f5c50101190 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4cff9640 1 -- 172.21.15.28:0/2945137423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c44003ce0 con 0x7f5c50101190 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c5019fa50 con 0x7f5c50101190 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c4cff9640 1 -- 172.21.15.28:0/2945137423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44035070 con 0x7f5c50101190 2024-05-09T08:23:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.261+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c5019ff40 con 0x7f5c50101190 2024-05-09T08:23:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.262+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c501072c0 con 0x7f5c50101190 2024-05-09T08:23:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.264+0000 7f5c4cff9640 1 -- 172.21.15.28:0/2945137423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5c44042030 con 0x7f5c50101190 2024-05-09T08:23:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.264+0000 7f5c4cff9640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5c380a2b90 0x7f5c380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.265+0000 7f5c4f7fe640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5c380a2b90 0x7f5c380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.265+0000 7f5c4cff9640 1 -- 172.21.15.28:0/2945137423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5c440ef170 con 0x7f5c50101190 2024-05-09T08:23:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.265+0000 7f5c4f7fe640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5c380a2b90 0x7f5c380a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5c40005fd0 tx=0x7f5c4000b510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.269+0000 7f5c4cff9640 1 -- 172.21.15.28:0/2945137423 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5c440bbf20 con 0x7f5c50101190 2024-05-09T08:23:11.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:11.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi165=b;smithi165=e;smithi165=h;smithi165=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:11.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi165=b;smithi165=e;smithi165=h;smithi165=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l;count:12 2024-05-09T08:23:11.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:11.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:11.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:11.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:11.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:11.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:11.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:11.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:11 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi165=b;smithi165=e;smithi165=h;smithi165=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi165=b;smithi165=e;smithi165=h;smithi165=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l;count:12 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:11.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:11 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:11.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:11.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi165=b;smithi165=e;smithi165=h;smithi165=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:11.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi165=b;smithi165=e;smithi165=h;smithi165=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l;count:12 2024-05-09T08:23:11.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:11.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:11.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:11.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:11.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:11.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:11.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:11.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:11 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.454+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c50062430 con 0x7f5c50101190 2024-05-09T08:23:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.455+0000 7f5c4cff9640 1 -- 172.21.15.28:0/2945137423 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18468 (secure 0 0 0) 0x7f5c440c0220 con 0x7f5c50101190 2024-05-09T08:23:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:11.457 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:01.846859+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5c380a2b90 msgr2=0x7f5c380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5c380a2b90 0x7f5c380a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5c40005fd0 tx=0x7f5c4000b510 comp rx=0 tx=0).stop 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 msgr2=0x7f5c501980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 0x7f5c501980e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009230 tx=0x7f5c44004360 comp rx=0 tx=0).stop 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 shutdown_connections 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5c380a2b90 0x7f5c380a5050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c50101b50 0x7f5c5019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c50101190 0x7f5c501980e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 --2- 172.21.15.28:0/2945137423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c500fffa0 0x7f5c50197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 >> 172.21.15.28:0/2945137423 conn(0x7f5c500fb510 msgr2=0x7f5c50104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 shutdown_connections 2024-05-09T08:23:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.457+0000 7f5c55f22640 1 -- 172.21.15.28:0/2945137423 wait complete. 2024-05-09T08:23:11.475 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T08:23:11.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:23:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 -- 172.21.15.28:0/3087224084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 msgr2=0x7f86a0102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 --2- 172.21.15.28:0/3087224084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 0x7f86a0102ef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8694009900 tx=0x7f869402a410 comp rx=0 tx=0).stop 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 -- 172.21.15.28:0/3087224084 shutdown_connections 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 --2- 172.21.15.28:0/3087224084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86a0103430 0x7f86a0109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 --2- 172.21.15.28:0/3087224084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 0x7f86a0102ef0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 --2- 172.21.15.28:0/3087224084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86a0101880 0x7f86a0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 -- 172.21.15.28:0/3087224084 >> 172.21.15.28:0/3087224084 conn(0x7f86a00fce10 msgr2=0x7f86a00ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 -- 172.21.15.28:0/3087224084 shutdown_connections 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 -- 172.21.15.28:0/3087224084 wait complete. 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 Processor -- start 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 -- start start 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.583+0000 7f86a7083640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86a0101880 0x7f86a0197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f86a7083640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 0x7f86a01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f86a7083640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86a0103430 0x7f86a019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f86a7083640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86a010d560 con 0x7f86a0102a70 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f86a7083640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f86a010d3e0 con 0x7f86a0101880 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f86a7083640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f86a010d6e0 con 0x7f86a0103430 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 0x7f86a01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 0x7f86a01980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44708/0 (socket says 172.21.15.28:44708) 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869ffff640 1 -- 172.21.15.28:0/3509511729 learned_addr learned my addr 172.21.15.28:0/3509511729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f86a5620640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86a0103430 0x7f86a019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f86a4e1f640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86a0101880 0x7f86a0197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869ffff640 1 -- 172.21.15.28:0/3509511729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86a0103430 msgr2=0x7f86a019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869ffff640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86a0103430 0x7f86a019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869ffff640 1 -- 172.21.15.28:0/3509511729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86a0101880 msgr2=0x7f86a0197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869ffff640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86a0101880 0x7f86a0197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869ffff640 1 -- 172.21.15.28:0/3509511729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a019f7c0 con 0x7f86a0102a70 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f86a4e1f640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86a0101880 0x7f86a0197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869ffff640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 0x7f86a01980e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8694009230 tx=0x7f8694004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869dffb640 1 -- 172.21.15.28:0/3509511729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869403f030 con 0x7f86a0102a70 2024-05-09T08:23:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869dffb640 1 -- 172.21.15.28:0/3509511729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8694003ce0 con 0x7f86a0102a70 2024-05-09T08:23:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a019fa50 con 0x7f86a0102a70 2024-05-09T08:23:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.584+0000 7f869dffb640 1 -- 172.21.15.28:0/3509511729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694035070 con 0x7f86a0102a70 2024-05-09T08:23:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.585+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a019ff90 con 0x7f86a0102a70 2024-05-09T08:23:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.585+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a0108c20 con 0x7f86a0102a70 2024-05-09T08:23:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.587+0000 7f869dffb640 1 -- 172.21.15.28:0/3509511729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8694042030 con 0x7f86a0102a70 2024-05-09T08:23:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.587+0000 7f869dffb640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f86880a2c10 0x7f86880a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.587+0000 7f86a4e1f640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f86880a2c10 0x7f86880a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.587+0000 7f869dffb640 1 -- 172.21.15.28:0/3509511729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f86940ef760 con 0x7f86a0102a70 2024-05-09T08:23:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.588+0000 7f86a4e1f640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f86880a2c10 0x7f86880a50d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86a01028d0 tx=0x7f8690005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:11.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.591+0000 7f869dffb640 1 -- 172.21.15.28:0/3509511729 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f86940bcb60 con 0x7f86a0102a70 2024-05-09T08:23:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.776+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f86a002d350 con 0x7f86a0102a70 2024-05-09T08:23:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.777+0000 7f869dffb640 1 -- 172.21.15.28:0/3509511729 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f86940bc460 con 0x7f86a0102a70 2024-05-09T08:23:11.778 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:23:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f86880a2c10 msgr2=0x7f86880a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f86880a2c10 0x7f86880a50d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86a01028d0 tx=0x7f8690005db0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 msgr2=0x7f86a01980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 0x7f86a01980e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8694009230 tx=0x7f8694004430 comp rx=0 tx=0).stop 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 shutdown_connections 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f86880a2c10 0x7f86880a50d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86a0103430 0x7f86a019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0102a70 0x7f86a01980e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 --2- 172.21.15.28:0/3509511729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86a0101880 0x7f86a0197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 >> 172.21.15.28:0/3509511729 conn(0x7f86a00fce10 msgr2=0x7f86a0105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 shutdown_connections 2024-05-09T08:23:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.779+0000 7f86a7083640 1 -- 172.21.15.28:0/3509511729 wait complete. 2024-05-09T08:23:11.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:11.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.896+0000 7fd0a2c15640 1 -- 172.21.15.28:0/1236021377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 msgr2=0x7fd09c101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.896+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/1236021377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 0x7fd09c101e00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd090009900 tx=0x7fd09002a410 comp rx=0 tx=0).stop 2024-05-09T08:23:11.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.896+0000 7fd0a2c15640 1 -- 172.21.15.28:0/1236021377 shutdown_connections 2024-05-09T08:23:11.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.896+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/1236021377 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c102340 0x7fd09c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.896+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/1236021377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 0x7fd09c101e00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.896+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/1236021377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd09c100790 0x7fd09c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.896+0000 7fd0a2c15640 1 -- 172.21.15.28:0/1236021377 >> 172.21.15.28:0/1236021377 conn(0x7fd09c0fbd20 msgr2=0x7fd09c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.896+0000 7fd0a2c15640 1 -- 172.21.15.28:0/1236021377 shutdown_connections 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.896+0000 7fd0a2c15640 1 -- 172.21.15.28:0/1236021377 wait complete. 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd0a2c15640 1 Processor -- start 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd0a2c15640 1 -- start start 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd0a2c15640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c100790 0x7fd09c104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd0a2c15640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 0x7fd09c102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd0a2c15640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd09c102340 0x7fd09c1032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd0a2c15640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c10c470 con 0x7fd09c101980 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd0a2c15640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c10c2f0 con 0x7fd09c102340 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd0a2c15640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c10c5f0 con 0x7fd09c100790 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 0x7fd09c102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd09bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 0x7fd09c102d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44732/0 (socket says 172.21.15.28:44732) 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.897+0000 7fd09bfff640 1 -- 172.21.15.28:0/2640458113 learned_addr learned my addr 172.21.15.28:0/2640458113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd0a09b1640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c100790 0x7fd09c104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd0a11b2640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd09c102340 0x7fd09c1032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd09bfff640 1 -- 172.21.15.28:0/2640458113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c100790 msgr2=0x7fd09c104720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd09bfff640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c100790 0x7fd09c104720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd09bfff640 1 -- 172.21.15.28:0/2640458113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd09c102340 msgr2=0x7fd09c1032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd09bfff640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd09c102340 0x7fd09c1032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd09bfff640 1 -- 172.21.15.28:0/2640458113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c103b30 con 0x7fd09c101980 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd0a09b1640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c100790 0x7fd09c104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd09bfff640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 0x7fd09c102d60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd090009230 tx=0x7fd090004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd099ffb640 1 -- 172.21.15.28:0/2640458113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09003f030 con 0x7fd09c101980 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd099ffb640 1 -- 172.21.15.28:0/2640458113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd090003ce0 con 0x7fd09c101980 2024-05-09T08:23:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09c1a1ca0 con 0x7fd09c101980 2024-05-09T08:23:11.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd099ffb640 1 -- 172.21.15.28:0/2640458113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd090035070 con 0x7fd09c101980 2024-05-09T08:23:11.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.898+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09c1a21a0 con 0x7fd09c101980 2024-05-09T08:23:11.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.899+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd09c1a23d0 con 0x7fd09c101980 2024-05-09T08:23:11.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.900+0000 7fd099ffb640 1 -- 172.21.15.28:0/2640458113 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd090042030 con 0x7fd09c101980 2024-05-09T08:23:11.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.901+0000 7fd099ffb640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0840a2b90 0x7fd0840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:11.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.901+0000 7fd099ffb640 1 -- 172.21.15.28:0/2640458113 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd0900ef660 con 0x7fd09c101980 2024-05-09T08:23:11.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.901+0000 7fd0a09b1640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0840a2b90 0x7fd0840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:11.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.901+0000 7fd0a09b1640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0840a2b90 0x7fd0840a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0058c0 tx=0x7fd08c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:11.905+0000 7fd099ffb640 1 -- 172.21.15.28:0/2640458113 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd0900bc9c0 con 0x7fd09c101980 2024-05-09T08:23:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.131+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd09c1a26c0 con 0x7fd09c101980 2024-05-09T08:23:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.132+0000 7fd099ffb640 1 -- 172.21.15.28:0/2640458113 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fd0900bc2c0 con 0x7fd09c101980 2024-05-09T08:23:12.133 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:12.134 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-05-09T08:23:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 1 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0840a2b90 msgr2=0x7fd0840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0840a2b90 0x7fd0840a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0058c0 tx=0x7fd08c005850 comp rx=0 tx=0).stop 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 msgr2=0x7fd09c102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 0x7fd09c102d60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd090009230 tx=0x7fd090004360 comp rx=0 tx=0).stop 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 shutdown_connections 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0840a2b90 0x7fd0840a5050 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd09c102340 0x7fd09c1032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09c101980 0x7fd09c102d60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 --2- 172.21.15.28:0/2640458113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c100790 0x7fd09c104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 >> 172.21.15.28:0/2640458113 conn(0x7fd09c0fbd20 msgr2=0x7fd09c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 shutdown_connections 2024-05-09T08:23:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:12.134+0000 7fd0a2c15640 1 -- 172.21.15.28:0/2640458113 wait complete. 2024-05-09T08:23:12.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:12 smithi165 ceph-mon[34540]: Deploying daemon mds.l on smithi184 2024-05-09T08:23:12.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:12 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2945137423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:12.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:12 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3509511729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:23:12.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:12 smithi184 ceph-mon[34509]: Deploying daemon mds.l on smithi184 2024-05-09T08:23:12.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:12 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2945137423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:12.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:12 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3509511729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:23:12.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:12 smithi028 ceph-mon[30119]: Deploying daemon mds.l on smithi184 2024-05-09T08:23:12.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:12 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2945137423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:12.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:12 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3509511729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:23:13.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:13 smithi184 ceph-mon[34509]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:13.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:13 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2640458113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:13 smithi165 ceph-mon[34540]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:13 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2640458113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:13 smithi028 ceph-mon[30119]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:13 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2640458113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:13.530 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:23:13 smithi184 podman[64443]: 2024-05-09 08:23:13.506719378 +0000 UTC m=+0.358882669 container start e8df3acc9d5d430db04757cf259344a3f00a4995bc00f57baed7ff392edb930c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-09T08:23:13.858 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:23:13 smithi184 ceph-mds[64462]: -- 172.21.15.184:0/943234213 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d4808da40 con 0x560d48060800 2024-05-09T08:23:13.858 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:23:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l[64458]: starting mds.l at 2024-05-09T08:23:13.858 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:23:13 smithi184 bash[64443]: e8df3acc9d5d430db04757cf259344a3f00a4995bc00f57baed7ff392edb930c 2024-05-09T08:23:13.858 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:23:13 smithi184 systemd[1]: Started Ceph mds.l for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:14.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.265+0000 7f6d315ab640 1 -- 172.21.15.28:0/874941039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c1075d0 msgr2=0x7f6d2c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.265+0000 7f6d315ab640 1 --2- 172.21.15.28:0/874941039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c1075d0 0x7f6d2c1099c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009ca0 tx=0x7f6d2402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.265+0000 7f6d315ab640 1 -- 172.21.15.28:0/874941039 shutdown_connections 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.265+0000 7f6d315ab640 1 --2- 172.21.15.28:0/874941039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c1075d0 0x7f6d2c1099c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.265+0000 7f6d315ab640 1 --2- 172.21.15.28:0/874941039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2c104ca0 0x7f6d2c107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.265+0000 7f6d315ab640 1 --2- 172.21.15.28:0/874941039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d2c102370 0x7f6d2c104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.265+0000 7f6d315ab640 1 -- 172.21.15.28:0/874941039 >> 172.21.15.28:0/874941039 conn(0x7f6d2c0fbd20 msgr2=0x7f6d2c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.265+0000 7f6d315ab640 1 -- 172.21.15.28:0/874941039 shutdown_connections 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.265+0000 7f6d315ab640 1 -- 172.21.15.28:0/874941039 wait complete. 2024-05-09T08:23:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d315ab640 1 Processor -- start 2024-05-09T08:23:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d315ab640 1 -- start start 2024-05-09T08:23:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d315ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c102370 0x7f6d2c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d315ab640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2c104ca0 0x7f6d2c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d315ab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d2c1075d0 0x7f6d2c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d315ab640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c10d530 con 0x7f6d2c102370 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d315ab640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c10d3b0 con 0x7f6d2c104ca0 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d315ab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c10d6b0 con 0x7f6d2c1075d0 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c102370 0x7f6d2c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c102370 0x7f6d2c197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44756/0 (socket says 172.21.15.28:44756) 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.266+0000 7f6d2affd640 1 -- 172.21.15.28:0/2224519668 learned_addr learned my addr 172.21.15.28:0/2224519668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d2b7fe640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d2c1075d0 0x7f6d2c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d2a7fc640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2c104ca0 0x7f6d2c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d2affd640 1 -- 172.21.15.28:0/2224519668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d2c1075d0 msgr2=0x7f6d2c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d2affd640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d2c1075d0 0x7f6d2c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d2affd640 1 -- 172.21.15.28:0/2224519668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2c104ca0 msgr2=0x7f6d2c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d2affd640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2c104ca0 0x7f6d2c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d2affd640 1 -- 172.21.15.28:0/2224519668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2c19f790 con 0x7f6d2c102370 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d2b7fe640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d2c1075d0 0x7f6d2c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d2affd640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c102370 0x7f6d2c197b70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00d770 tx=0x7f6d1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d13fff640 1 -- 172.21.15.28:0/2224519668 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c017070 con 0x7f6d2c102370 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d13fff640 1 -- 172.21.15.28:0/2224519668 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d1c005500 con 0x7f6d2c102370 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2c19fa80 con 0x7f6d2c102370 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d13fff640 1 -- 172.21.15.28:0/2224519668 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c004e00 con 0x7f6d2c102370 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.267+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2c19ff80 con 0x7f6d2c102370 2024-05-09T08:23:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.268+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d2c0fd810 con 0x7f6d2c102370 2024-05-09T08:23:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.269+0000 7f6d13fff640 1 -- 172.21.15.28:0/2224519668 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6d1c004fa0 con 0x7f6d2c102370 2024-05-09T08:23:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.270+0000 7f6d13fff640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6d140a2b90 0x7f6d140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.270+0000 7f6d2a7fc640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6d140a2b90 0x7f6d140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.270+0000 7f6d13fff640 1 -- 172.21.15.28:0/2224519668 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6d1c0ce6d0 con 0x7f6d2c102370 2024-05-09T08:23:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.271+0000 7f6d2a7fc640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6d140a2b90 0x7f6d140a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c199080 tx=0x7f6d20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.274+0000 7f6d13fff640 1 -- 172.21.15.28:0/2224519668 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d1c09bad0 con 0x7f6d2c102370 2024-05-09T08:23:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.498+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d2c02d420 con 0x7f6d2c102370 2024-05-09T08:23:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.498+0000 7f6d13fff640 1 -- 172.21.15.28:0/2224519668 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f6d1c09b3d0 con 0x7f6d2c102370 2024-05-09T08:23:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:14.500 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:23:14.500 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-05-09T08:23:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6d140a2b90 msgr2=0x7f6d140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6d140a2b90 0x7f6d140a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c199080 tx=0x7f6d20005f90 comp rx=0 tx=0).stop 2024-05-09T08:23:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c102370 msgr2=0x7f6d2c197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c102370 0x7f6d2c197b70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00d770 tx=0x7f6d1c00dc30 comp rx=0 tx=0).stop 2024-05-09T08:23:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 shutdown_connections 2024-05-09T08:23:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6d140a2b90 0x7f6d140a5050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d2c1075d0 0x7f6d2c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2c104ca0 0x7f6d2c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 --2- 172.21.15.28:0/2224519668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d2c102370 0x7f6d2c197b70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 >> 172.21.15.28:0/2224519668 conn(0x7f6d2c0fbd20 msgr2=0x7f6d2c102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 shutdown_connections 2024-05-09T08:23:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:14.500+0000 7f6d315ab640 1 -- 172.21.15.28:0/2224519668 wait complete. 2024-05-09T08:23:14.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6832/1853687224,v1:172.21.15.184:6833/1853687224] up:boot 2024-05-09T08:23:14.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: fsmap 1 up:standby 2024-05-09T08:23:14.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:23:14.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:14.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:14.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:14.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:14.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:14.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:14.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:14.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: Deploying daemon mds.k on smithi165 2024-05-09T08:23:14.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:14 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2224519668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:14.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6832/1853687224,v1:172.21.15.184:6833/1853687224] up:boot 2024-05-09T08:23:14.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: fsmap 1 up:standby 2024-05-09T08:23:14.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:23:14.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:14.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:14.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:14.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: Deploying daemon mds.k on smithi165 2024-05-09T08:23:14.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:14 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2224519668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:14.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6832/1853687224,v1:172.21.15.184:6833/1853687224] up:boot 2024-05-09T08:23:14.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: fsmap 1 up:standby 2024-05-09T08:23:14.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:23:14.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:14.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:14.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:14.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:14.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:14.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:14.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:14.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: Deploying daemon mds.k on smithi165 2024-05-09T08:23:14.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:14 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2224519668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:16.083 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:23:15 smithi165 systemd[1]: Starting Ceph mds.k for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:23:16.335 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:23:16 smithi165 podman[63645]: 2024-05-09 08:23:15.985823058 +0000 UTC m=+0.017920540 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:23:16.335 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:23:16 smithi165 podman[63645]: 2024-05-09 08:23:16.106020768 +0000 UTC m=+0.138118295 container create b45a185d7e34eded395dd00a51a82d988f4aa6877bfbad5169cc023479d8da13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-09T08:23:16.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:16.602 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:23:16 smithi165 podman[63645]: 2024-05-09 08:23:16.450031694 +0000 UTC m=+0.482129176 container init b45a185d7e34eded395dd00a51a82d988f4aa6877bfbad5169cc023479d8da13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-96805dc) 2024-05-09T08:23:16.602 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:23:16 smithi165 podman[63645]: 2024-05-09 08:23:16.452758074 +0000 UTC m=+0.484855556 container start b45a185d7e34eded395dd00a51a82d988f4aa6877bfbad5169cc023479d8da13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-96805dc) 2024-05-09T08:23:16.602 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:23:16 smithi165 ceph-mds[63664]: -- 172.21.15.165:0/2813632620 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56349dd0fa40 con 0x56349dce2800 2024-05-09T08:23:16.602 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:23:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k[63660]: starting mds.k at 2024-05-09T08:23:16.603 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:23:16 smithi165 bash[63645]: b45a185d7e34eded395dd00a51a82d988f4aa6877bfbad5169cc023479d8da13 2024-05-09T08:23:16.603 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:23:16 smithi165 systemd[1]: Started Ceph mds.k for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:16.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.633+0000 7f1abbb9e640 1 -- 172.21.15.28:0/436224878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41000e0 msgr2=0x7f1ab4100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:16.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.633+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/436224878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41000e0 0x7f1ab4100500 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009900 tx=0x7f1ab002a410 comp rx=0 tx=0).stop 2024-05-09T08:23:16.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.633+0000 7f1abbb9e640 1 -- 172.21.15.28:0/436224878 shutdown_connections 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.633+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/436224878 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab41013e0 0x7f1ab4101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.633+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/436224878 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4100a40 0x7f1ab4100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.633+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/436224878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41000e0 0x7f1ab4100500 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.633+0000 7f1abbb9e640 1 -- 172.21.15.28:0/436224878 >> 172.21.15.28:0/436224878 conn(0x7f1ab40fbce0 msgr2=0x7f1ab40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.633+0000 7f1abbb9e640 1 -- 172.21.15.28:0/436224878 shutdown_connections 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.633+0000 7f1abbb9e640 1 -- 172.21.15.28:0/436224878 wait complete. 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1abbb9e640 1 Processor -- start 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1abbb9e640 1 -- start start 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1abbb9e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab41000e0 0x7f1ab4199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1abbb9e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab4100a40 0x7f1ab419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1abbb9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41013e0 0x7f1ab41a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1abbb9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab410f6d0 con 0x7f1ab41013e0 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1abbb9e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab410f550 con 0x7f1ab41000e0 2024-05-09T08:23:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1abbb9e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab410f850 con 0x7f1ab4100a40 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1aba13b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41013e0 0x7f1ab41a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1aba13b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41013e0 0x7f1ab41a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36790/0 (socket says 172.21.15.28:36790) 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1aba13b640 1 -- 172.21.15.28:0/4250876072 learned_addr learned my addr 172.21.15.28:0/4250876072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1ab993a640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab41000e0 0x7f1ab4199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1ab9139640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab4100a40 0x7f1ab419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1aba13b640 1 -- 172.21.15.28:0/4250876072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab4100a40 msgr2=0x7f1ab419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1aba13b640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab4100a40 0x7f1ab419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1aba13b640 1 -- 172.21.15.28:0/4250876072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab41000e0 msgr2=0x7f1ab4199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1aba13b640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab41000e0 0x7f1ab4199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1aba13b640 1 -- 172.21.15.28:0/4250876072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab41a1960 con 0x7f1ab41013e0 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1ab9139640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab4100a40 0x7f1ab419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.634+0000 7f1aba13b640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41013e0 0x7f1ab41a12f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00eb10 tx=0x7f1aac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.635+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4250876072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac004b20 con 0x7f1ab41013e0 2024-05-09T08:23:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.635+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab41a1c50 con 0x7f1ab41013e0 2024-05-09T08:23:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.635+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4250876072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1aac004cc0 con 0x7f1ab41013e0 2024-05-09T08:23:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.635+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4250876072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac010760 con 0x7f1ab41013e0 2024-05-09T08:23:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.635+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab41a21a0 con 0x7f1ab41013e0 2024-05-09T08:23:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.636+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab410afd0 con 0x7f1ab41013e0 2024-05-09T08:23:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.637+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4250876072 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1aac005000 con 0x7f1ab41013e0 2024-05-09T08:23:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.638+0000 7f1aa6ffd640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aa00a2b90 0x7f1aa00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.638+0000 7f1ab993a640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aa00a2b90 0x7f1aa00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.638+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4250876072 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1aac0cf650 con 0x7f1ab41013e0 2024-05-09T08:23:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.638+0000 7f1ab993a640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aa00a2b90 0x7f1aa00a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00314e0 tx=0x7f1ab0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:16.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.641+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4250876072 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1aac09ca50 con 0x7f1ab41013e0 2024-05-09T08:23:16.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.871+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ab402d410 con 0x7f1ab41013e0 2024-05-09T08:23:16.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.871+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4250876072 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f1aac09c350 con 0x7f1ab41013e0 2024-05-09T08:23:16.873 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:16.873 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:23:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-05-09T08:23:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aa00a2b90 msgr2=0x7f1aa00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aa00a2b90 0x7f1aa00a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00314e0 tx=0x7f1ab0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41013e0 msgr2=0x7f1ab41a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41013e0 0x7f1ab41a12f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00eb10 tx=0x7f1aac00c2d0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 shutdown_connections 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aa00a2b90 0x7f1aa00a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab41013e0 0x7f1ab41a12f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ab4100a40 0x7f1ab419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 --2- 172.21.15.28:0/4250876072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab41000e0 0x7f1ab4199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 >> 172.21.15.28:0/4250876072 conn(0x7f1ab40fbce0 msgr2=0x7f1ab40fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 shutdown_connections 2024-05-09T08:23:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:16.873+0000 7f1abbb9e640 1 -- 172.21.15.28:0/4250876072 wait complete. 2024-05-09T08:23:16.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:16 smithi028 ceph-mon[30119]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:16.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:16.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:16.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:16.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:16.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:16.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:16 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:16 smithi165 ceph-mon[34540]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:16 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:16 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:17.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:16 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:17.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:16 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:17.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:16 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:17.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:16 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:16 smithi184 ceph-mon[34509]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:16 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:17.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:17 smithi165 ceph-mon[34540]: Deploying daemon mds.j on smithi028 2024-05-09T08:23:17.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:17 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6832/3903056410,v1:172.21.15.165:6833/3903056410] up:boot 2024-05-09T08:23:17.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:17 smithi165 ceph-mon[34540]: fsmap 2 up:standby 2024-05-09T08:23:17.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:17 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:23:17.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:17 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4250876072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:17.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:17 smithi028 ceph-mon[30119]: Deploying daemon mds.j on smithi028 2024-05-09T08:23:17.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:17 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6832/3903056410,v1:172.21.15.165:6833/3903056410] up:boot 2024-05-09T08:23:17.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:17 smithi028 ceph-mon[30119]: fsmap 2 up:standby 2024-05-09T08:23:17.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:17 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:23:17.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4250876072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:17 smithi184 ceph-mon[34509]: Deploying daemon mds.j on smithi028 2024-05-09T08:23:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:17 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6832/3903056410,v1:172.21.15.165:6833/3903056410] up:boot 2024-05-09T08:23:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:17 smithi184 ceph-mon[34509]: fsmap 2 up:standby 2024-05-09T08:23:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:17 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:23:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:17 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4250876072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:18.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.014+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/4093087400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14100780 msgr2=0x7f4a14100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.014+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/4093087400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14100780 0x7f4a14100ba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009900 tx=0x7f4a0402a410 comp rx=0 tx=0).stop 2024-05-09T08:23:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.014+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/4093087400 shutdown_connections 2024-05-09T08:23:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.014+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/4093087400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14102330 0x7f4a14108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.014+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/4093087400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a14101970 0x7f4a14101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.014+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/4093087400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14100780 0x7f4a14100ba0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.014+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/4093087400 >> 172.21.15.28:0/4093087400 conn(0x7f4a140fbd30 msgr2=0x7f4a140fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.015+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/4093087400 shutdown_connections 2024-05-09T08:23:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.015+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/4093087400 wait complete. 2024-05-09T08:23:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.015+0000 7f4a1a7b7640 1 Processor -- start 2024-05-09T08:23:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.015+0000 7f4a1a7b7640 1 -- start start 2024-05-09T08:23:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a1a7b7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a14100780 0x7f4a14197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a1a7b7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14101970 0x7f4a141980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a1a7b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14102330 0x7f4a1419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a1a7b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1410c3f0 con 0x7f4a14102330 2024-05-09T08:23:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a1a7b7640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1410c270 con 0x7f4a14101970 2024-05-09T08:23:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a1a7b7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1410c570 con 0x7f4a14100780 2024-05-09T08:23:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a18d54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14102330 0x7f4a1419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a18d54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14102330 0x7f4a1419f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36806/0 (socket says 172.21.15.28:36806) 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a18d54640 1 -- 172.21.15.28:0/2872844153 learned_addr learned my addr 172.21.15.28:0/2872844153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a137fe640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14101970 0x7f4a141980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a13fff640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a14100780 0x7f4a14197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a18d54640 1 -- 172.21.15.28:0/2872844153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a14100780 msgr2=0x7f4a14197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a18d54640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a14100780 0x7f4a14197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a18d54640 1 -- 172.21.15.28:0/2872844153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14101970 msgr2=0x7f4a141980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a18d54640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14101970 0x7f4a141980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a18d54640 1 -- 172.21.15.28:0/2872844153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1419f7b0 con 0x7f4a14102330 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a137fe640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14101970 0x7f4a141980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.016+0000 7f4a18d54640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14102330 0x7f4a1419f140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00eb10 tx=0x7f4a0c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.017+0000 7f4a117fa640 1 -- 172.21.15.28:0/2872844153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c004b20 con 0x7f4a14102330 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.017+0000 7f4a117fa640 1 -- 172.21.15.28:0/2872844153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a0c004cc0 con 0x7f4a14102330 2024-05-09T08:23:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.017+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a14102bf0 con 0x7f4a14102330 2024-05-09T08:23:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.017+0000 7f4a117fa640 1 -- 172.21.15.28:0/2872844153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c005650 con 0x7f4a14102330 2024-05-09T08:23:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.017+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a141030f0 con 0x7f4a14102330 2024-05-09T08:23:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.018+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a14103350 con 0x7f4a14102330 2024-05-09T08:23:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.019+0000 7f4a117fa640 1 -- 172.21.15.28:0/2872844153 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4a0c0058d0 con 0x7f4a14102330 2024-05-09T08:23:19.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.020+0000 7f4a117fa640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49fc0a2b90 0x7f49fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:19.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.020+0000 7f4a13fff640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49fc0a2b90 0x7f49fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:19.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.020+0000 7f4a117fa640 1 -- 172.21.15.28:0/2872844153 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4a0c0d31e0 con 0x7f4a14102330 2024-05-09T08:23:19.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.020+0000 7f4a13fff640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49fc0a2b90 0x7f49fc0a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4a04005ab0 tx=0x7f4a04000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:19.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.026+0000 7f4a117fa640 1 -- 172.21.15.28:0/2872844153 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a0c09ff70 con 0x7f4a14102330 2024-05-09T08:23:19.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:18 smithi165 ceph-mon[34540]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:19.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:18 smithi184 ceph-mon[34509]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:19.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:18 smithi028 ceph-mon[30119]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.262+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a14103b20 con 0x7f4a14102330 2024-05-09T08:23:19.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.263+0000 7f4a117fa640 1 -- 172.21.15.28:0/2872844153 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f4a0c0a4270 con 0x7f4a14102330 2024-05-09T08:23:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:19.264 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:23:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49fc0a2b90 msgr2=0x7f49fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49fc0a2b90 0x7f49fc0a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4a04005ab0 tx=0x7f4a04000bc0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14102330 msgr2=0x7f4a1419f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14102330 0x7f4a1419f140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00eb10 tx=0x7f4a0c00c2d0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 shutdown_connections 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49fc0a2b90 0x7f49fc0a5050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14102330 0x7f4a1419f140 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14101970 0x7f4a141980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 --2- 172.21.15.28:0/2872844153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a14100780 0x7f4a14197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 >> 172.21.15.28:0/2872844153 conn(0x7f4a140fbd30 msgr2=0x7f4a140fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 shutdown_connections 2024-05-09T08:23:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:19.265+0000 7f4a1a7b7640 1 -- 172.21.15.28:0/2872844153 wait complete. 2024-05-09T08:23:19.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:19 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2872844153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:19.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:19 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:19.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:19 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:19 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:19 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:19 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:19 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:20.093 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:19 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2872844153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:20.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:19 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:20.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:19 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:20.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:19 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:20.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:19 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:20.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:19 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:20.094 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:19 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:20.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:19 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2872844153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:20.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:19 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:20.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:19 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:20.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:19 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:20.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:19 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:20.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:19 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:20.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:19 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:20.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:20 smithi028 ceph-mon[30119]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:20.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:20 smithi028 ceph-mon[30119]: Deploying daemon mds.d on smithi028 2024-05-09T08:23:20.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:20 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.28:6834/523365778,v1:172.21.15.28:6835/523365778] up:boot 2024-05-09T08:23:20.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:20 smithi028 ceph-mon[30119]: fsmap 3 up:standby 2024-05-09T08:23:20.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:20 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:23:21.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:20 smithi165 ceph-mon[34540]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:21.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:20 smithi165 ceph-mon[34540]: Deploying daemon mds.d on smithi028 2024-05-09T08:23:21.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:20 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.28:6834/523365778,v1:172.21.15.28:6835/523365778] up:boot 2024-05-09T08:23:21.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:20 smithi165 ceph-mon[34540]: fsmap 3 up:standby 2024-05-09T08:23:21.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:20 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:23:21.112 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:20 smithi184 ceph-mon[34509]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:21.112 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:20 smithi184 ceph-mon[34509]: Deploying daemon mds.d on smithi028 2024-05-09T08:23:21.112 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:20 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.28:6834/523365778,v1:172.21.15.28:6835/523365778] up:boot 2024-05-09T08:23:21.112 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:20 smithi184 ceph-mon[34509]: fsmap 3 up:standby 2024-05-09T08:23:21.112 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:20 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:23:21.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.401+0000 7f1f536cd640 1 -- 172.21.15.28:0/3805427686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c100770 msgr2=0x7f1f4c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.401+0000 7f1f536cd640 1 --2- 172.21.15.28:0/3805427686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c100770 0x7f1f4c100b90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009900 tx=0x7f1f4802a410 comp rx=0 tx=0).stop 2024-05-09T08:23:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.401+0000 7f1f536cd640 1 -- 172.21.15.28:0/3805427686 shutdown_connections 2024-05-09T08:23:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.401+0000 7f1f536cd640 1 --2- 172.21.15.28:0/3805427686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f4c102320 0x7f1f4c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.401+0000 7f1f536cd640 1 --2- 172.21.15.28:0/3805427686 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4c101960 0x7f1f4c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.401+0000 7f1f536cd640 1 --2- 172.21.15.28:0/3805427686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c100770 0x7f1f4c100b90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.401+0000 7f1f536cd640 1 -- 172.21.15.28:0/3805427686 >> 172.21.15.28:0/3805427686 conn(0x7f1f4c0fbce0 msgr2=0x7f1f4c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.401+0000 7f1f536cd640 1 -- 172.21.15.28:0/3805427686 shutdown_connections 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.401+0000 7f1f536cd640 1 -- 172.21.15.28:0/3805427686 wait complete. 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f536cd640 1 Processor -- start 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f536cd640 1 -- start start 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f536cd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4c100770 0x7f1f4c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f536cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c101960 0x7f1f4c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f536cd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f4c102320 0x7f1f4c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f536cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4c10c3c0 con 0x7f1f4c101960 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f536cd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4c10c240 con 0x7f1f4c102320 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f536cd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4c10c540 con 0x7f1f4c100770 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f50c68640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c101960 0x7f1f4c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f50c68640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c101960 0x7f1f4c1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36850/0 (socket says 172.21.15.28:36850) 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.402+0000 7f1f50c68640 1 -- 172.21.15.28:0/2558630620 learned_addr learned my addr 172.21.15.28:0/2558630620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f51469640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4c100770 0x7f1f4c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f51c6a640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f4c102320 0x7f1f4c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f50c68640 1 -- 172.21.15.28:0/2558630620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4c100770 msgr2=0x7f1f4c197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f50c68640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4c100770 0x7f1f4c197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f50c68640 1 -- 172.21.15.28:0/2558630620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f4c102320 msgr2=0x7f1f4c19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f50c68640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f4c102320 0x7f1f4c19f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f50c68640 1 -- 172.21.15.28:0/2558630620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4c19f7b0 con 0x7f1f4c101960 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f51469640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4c100770 0x7f1f4c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f50c68640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c101960 0x7f1f4c1980d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00d950 tx=0x7f1f3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f427fc640 1 -- 172.21.15.28:0/2558630620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c017030 con 0x7f1f4c101960 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f427fc640 1 -- 172.21.15.28:0/2558630620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3c004020 con 0x7f1f4c101960 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f4c19faa0 con 0x7f1f4c101960 2024-05-09T08:23:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f427fc640 1 -- 172.21.15.28:0/2558630620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c0047a0 con 0x7f1f4c101960 2024-05-09T08:23:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.403+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f4c19fff0 con 0x7f1f4c101960 2024-05-09T08:23:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.404+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f4c1a0280 con 0x7f1f4c101960 2024-05-09T08:23:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.405+0000 7f1f427fc640 1 -- 172.21.15.28:0/2558630620 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1f3c0041d0 con 0x7f1f4c101960 2024-05-09T08:23:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.406+0000 7f1f427fc640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1f380a2b90 0x7f1f380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.406+0000 7f1f51469640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1f380a2b90 0x7f1f380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.406+0000 7f1f427fc640 1 -- 172.21.15.28:0/2558630620 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1f3c0ce760 con 0x7f1f4c101960 2024-05-09T08:23:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.407+0000 7f1f51469640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1f380a2b90 0x7f1f380a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f48000c00 tx=0x7f1f48031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:21.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.410+0000 7f1f427fc640 1 -- 172.21.15.28:0/2558630620 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1f3c09bac0 con 0x7f1f4c101960 2024-05-09T08:23:21.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.647+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f4c1a0450 con 0x7f1f4c101960 2024-05-09T08:23:21.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.648+0000 7f1f427fc640 1 -- 172.21.15.28:0/2558630620 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f1f3c09b3c0 con 0x7f1f4c101960 2024-05-09T08:23:21.650 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:21.650 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:23:21.650 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1f380a2b90 msgr2=0x7f1f380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1f380a2b90 0x7f1f380a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f48000c00 tx=0x7f1f48031400 comp rx=0 tx=0).stop 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c101960 msgr2=0x7f1f4c1980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c101960 0x7f1f4c1980d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00d950 tx=0x7f1f3c00de10 comp rx=0 tx=0).stop 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 shutdown_connections 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1f380a2b90 0x7f1f380a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f4c102320 0x7f1f4c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f4c101960 0x7f1f4c1980d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 --2- 172.21.15.28:0/2558630620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4c100770 0x7f1f4c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.650+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 >> 172.21.15.28:0/2558630620 conn(0x7f1f4c0fbce0 msgr2=0x7f1f4c0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.651+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 shutdown_connections 2024-05-09T08:23:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:21.651+0000 7f1f536cd640 1 -- 172.21.15.28:0/2558630620 wait complete. 2024-05-09T08:23:21.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:21 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2558630620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:21.934 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:23:21 smithi028 systemd[1]: Starting Ceph mds.d for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:23:22.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:21 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2558630620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:22.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:21 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2558630620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:22.312 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:23:22 smithi028 podman[68078]: 2024-05-09 08:23:21.961048074 +0000 UTC m=+0.018564900 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:23:22.312 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:23:22 smithi028 podman[68078]: 2024-05-09 08:23:22.067444849 +0000 UTC m=+0.124961658 container create de5d67530b5945c7057680633d64799de036c1ce31af84fd23aeea69969966b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-09T08:23:22.312 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:23:22 smithi028 podman[68078]: 2024-05-09 08:23:22.312098849 +0000 UTC m=+0.369615656 container init de5d67530b5945c7057680633d64799de036c1ce31af84fd23aeea69969966b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-09T08:23:22.683 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:23:22 smithi028 podman[68078]: 2024-05-09 08:23:22.315089136 +0000 UTC m=+0.372605946 container start de5d67530b5945c7057680633d64799de036c1ce31af84fd23aeea69969966b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:23:22.683 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:23:22 smithi028 ceph-mds[68098]: -- 172.21.15.28:0/1951127787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558a07c83a40 con 0x558a07c56c00 2024-05-09T08:23:22.683 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:23:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[68094]: starting mds.d at 2024-05-09T08:23:22.683 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:23:22 smithi028 bash[68078]: de5d67530b5945c7057680633d64799de036c1ce31af84fd23aeea69969966b7 2024-05-09T08:23:22.684 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:23:22 smithi028 systemd[1]: Started Ceph mds.d for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:23.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.785+0000 7f2e6569e640 1 -- 172.21.15.28:0/1501471555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e60102a90 msgr2=0x7f2e60102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.785+0000 7f2e6569e640 1 --2- 172.21.15.28:0/1501471555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e60102a90 0x7f2e60102f10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2e54009900 tx=0x7f2e5402a410 comp rx=0 tx=0).stop 2024-05-09T08:23:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.785+0000 7f2e6569e640 1 -- 172.21.15.28:0/1501471555 shutdown_connections 2024-05-09T08:23:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.785+0000 7f2e6569e640 1 --2- 172.21.15.28:0/1501471555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e60103450 0x7f2e60109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.785+0000 7f2e6569e640 1 --2- 172.21.15.28:0/1501471555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e60102a90 0x7f2e60102f10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.785+0000 7f2e6569e640 1 --2- 172.21.15.28:0/1501471555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e601018a0 0x7f2e60101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.785+0000 7f2e6569e640 1 -- 172.21.15.28:0/1501471555 >> 172.21.15.28:0/1501471555 conn(0x7f2e600fce30 msgr2=0x7f2e600ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.785+0000 7f2e6569e640 1 -- 172.21.15.28:0/1501471555 shutdown_connections 2024-05-09T08:23:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.785+0000 7f2e6569e640 1 -- 172.21.15.28:0/1501471555 wait complete. 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.786+0000 7f2e6569e640 1 Processor -- start 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.786+0000 7f2e6569e640 1 -- start start 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.786+0000 7f2e6569e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e601018a0 0x7f2e600755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.786+0000 7f2e6569e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e60102a90 0x7f2e60073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.786+0000 7f2e6569e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e60103450 0x7f2e60074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.786+0000 7f2e6569e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6010d500 con 0x7f2e601018a0 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.786+0000 7f2e6569e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6010d380 con 0x7f2e60102a90 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.786+0000 7f2e6569e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6010d680 con 0x7f2e60103450 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.786+0000 7f2e5effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e601018a0 0x7f2e600755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e601018a0 0x7f2e600755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36868/0 (socket says 172.21.15.28:36868) 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5effd640 1 -- 172.21.15.28:0/3899049881 learned_addr learned my addr 172.21.15.28:0/3899049881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5e7fc640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e60102a90 0x7f2e60073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5f7fe640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e60103450 0x7f2e60074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5effd640 1 -- 172.21.15.28:0/3899049881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e60103450 msgr2=0x7f2e60074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5effd640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e60103450 0x7f2e60074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5effd640 1 -- 172.21.15.28:0/3899049881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e60102a90 msgr2=0x7f2e60073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5effd640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e60102a90 0x7f2e60073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5effd640 1 -- 172.21.15.28:0/3899049881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e600749f0 con 0x7f2e601018a0 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5f7fe640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e60103450 0x7f2e60074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e5effd640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e601018a0 0x7f2e600755e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000c970 tx=0x7f2e5000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e47fff640 1 -- 172.21.15.28:0/3899049881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50004220 con 0x7f2e601018a0 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e47fff640 1 -- 172.21.15.28:0/3899049881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e500043c0 con 0x7f2e601018a0 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e601a1cd0 con 0x7f2e601018a0 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e47fff640 1 -- 172.21.15.28:0/3899049881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5000f450 con 0x7f2e601018a0 2024-05-09T08:23:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.787+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e601a21a0 con 0x7f2e601018a0 2024-05-09T08:23:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.788+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e60068aa0 con 0x7f2e601018a0 2024-05-09T08:23:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.789+0000 7f2e47fff640 1 -- 172.21.15.28:0/3899049881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2e500079b0 con 0x7f2e601018a0 2024-05-09T08:23:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.790+0000 7f2e47fff640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e480a2b90 0x7f2e480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.790+0000 7f2e5e7fc640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e480a2b90 0x7f2e480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.790+0000 7f2e47fff640 1 -- 172.21.15.28:0/3899049881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2e500cea50 con 0x7f2e601018a0 2024-05-09T08:23:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.791+0000 7f2e5e7fc640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e480a2b90 0x7f2e480a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2e540314e0 tx=0x7f2e54000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:23.794+0000 7f2e47fff640 1 -- 172.21.15.28:0/3899049881 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2e50014030 con 0x7f2e601018a0 2024-05-09T08:23:23.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:23.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:23.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:23.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:23.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:23.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:23.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:23.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.28:6836/3978374239,v1:172.21.15.28:6837/3978374239] up:boot 2024-05-09T08:23:23.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: fsmap 4 up:standby 2024-05-09T08:23:23.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.28:6836/3978374239,v1:172.21.15.28:6837/3978374239] up:boot 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: fsmap 4 up:standby 2024-05-09T08:23:23.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:23:23.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:23.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:23.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:23.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:23.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:23.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:23.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:23.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.28:6836/3978374239,v1:172.21.15.28:6837/3978374239] up:boot 2024-05-09T08:23:23.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: fsmap 4 up:standby 2024-05-09T08:23:23.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:23:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.022+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e60108bc0 con 0x7f2e601018a0 2024-05-09T08:23:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.023+0000 7f2e47fff640 1 -- 172.21.15.28:0/3899049881 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f2e5009bdb0 con 0x7f2e601018a0 2024-05-09T08:23:24.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:24.024 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:23:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-05-09T08:23:24.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.024+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e480a2b90 msgr2=0x7f2e480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:24.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.024+0000 7f2e6569e640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e480a2b90 0x7f2e480a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2e540314e0 tx=0x7f2e54000bc0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.024+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e601018a0 msgr2=0x7f2e600755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:24.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.024+0000 7f2e6569e640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e601018a0 0x7f2e600755e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000c970 tx=0x7f2e5000ce30 comp rx=0 tx=0).stop 2024-05-09T08:23:24.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.025+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 shutdown_connections 2024-05-09T08:23:24.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.025+0000 7f2e6569e640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e480a2b90 0x7f2e480a5050 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.025+0000 7f2e6569e640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e60103450 0x7f2e60074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.025+0000 7f2e6569e640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e60102a90 0x7f2e60073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.025+0000 7f2e6569e640 1 --2- 172.21.15.28:0/3899049881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e601018a0 0x7f2e600755e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.025+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 >> 172.21.15.28:0/3899049881 conn(0x7f2e600fce30 msgr2=0x7f2e60105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:24.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.025+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 shutdown_connections 2024-05-09T08:23:24.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:24.025+0000 7f2e6569e640 1 -- 172.21.15.28:0/3899049881 wait complete. 2024-05-09T08:23:24.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:24 smithi165 ceph-mon[34540]: Deploying daemon mds.e on smithi165 2024-05-09T08:23:24.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:24 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3899049881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:24.853 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:23:24 smithi165 systemd[1]: Starting Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:23:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:24 smithi184 ceph-mon[34509]: Deploying daemon mds.e on smithi165 2024-05-09T08:23:24.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:24 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3899049881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:24 smithi028 ceph-mon[30119]: Deploying daemon mds.e on smithi165 2024-05-09T08:23:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:24 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3899049881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:25.220 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:23:24 smithi165 podman[63963]: 2024-05-09 08:23:24.829506922 +0000 UTC m=+0.018098383 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:23:25.220 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:23:24 smithi165 podman[63963]: 2024-05-09 08:23:24.950961047 +0000 UTC m=+0.139552532 container create 5a02348fcbf90a5d78f3ecc7c42eec7864d8721fe8df9c06683b34486e040720 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5) 2024-05-09T08:23:25.220 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:23:25 smithi165 podman[63963]: 2024-05-09 08:23:25.219857977 +0000 UTC m=+0.408449420 container init 5a02348fcbf90a5d78f3ecc7c42eec7864d8721fe8df9c06683b34486e040720 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, org.label-schema.build-date=20240507, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-09T08:23:25.471 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:23:25 smithi165 podman[63963]: 2024-05-09 08:23:25.222473282 +0000 UTC m=+0.411064726 container start 5a02348fcbf90a5d78f3ecc7c42eec7864d8721fe8df9c06683b34486e040720 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-09T08:23:25.471 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:23:25 smithi165 ceph-mds[63982]: -- 172.21.15.165:0/1445040002 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5608e48cba40 con 0x5608e489e800 2024-05-09T08:23:25.471 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:23:25 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[63978]: starting mds.e at 2024-05-09T08:23:25.471 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:23:25 smithi165 bash[63963]: 5a02348fcbf90a5d78f3ecc7c42eec7864d8721fe8df9c06683b34486e040720 2024-05-09T08:23:25.471 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:23:25 smithi165 systemd[1]: Started Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:25.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:25 smithi165 ceph-mon[34540]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:25.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:25.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:25.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:25.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:25.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:25.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:25 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:25.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:25 smithi165 ceph-mon[34540]: Deploying daemon mds.f on smithi184 2024-05-09T08:23:25.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:25 smithi184 ceph-mon[34509]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:25.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:25.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:25.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:25.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:25.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:25.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:25 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:25.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:25 smithi184 ceph-mon[34509]: Deploying daemon mds.f on smithi184 2024-05-09T08:23:25.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:25 smithi028 ceph-mon[30119]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:25.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:25.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:25.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:25.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:25.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:25.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:25 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:25.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:25 smithi028 ceph-mon[30119]: Deploying daemon mds.f on smithi184 2024-05-09T08:23:26.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 -- 172.21.15.28:0/1136683155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 msgr2=0x7f68e0101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 --2- 172.21.15.28:0/1136683155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 0x7f68e0101de0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009900 tx=0x7f68d402a410 comp rx=0 tx=0).stop 2024-05-09T08:23:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 -- 172.21.15.28:0/1136683155 shutdown_connections 2024-05-09T08:23:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 --2- 172.21.15.28:0/1136683155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68e0102320 0x7f68e0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 --2- 172.21.15.28:0/1136683155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 0x7f68e0101de0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 --2- 172.21.15.28:0/1136683155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e0100770 0x7f68e0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 -- 172.21.15.28:0/1136683155 >> 172.21.15.28:0/1136683155 conn(0x7f68e00fbce0 msgr2=0x7f68e00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 -- 172.21.15.28:0/1136683155 shutdown_connections 2024-05-09T08:23:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 -- 172.21.15.28:0/1136683155 wait complete. 2024-05-09T08:23:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.155+0000 7f68e5e12640 1 Processor -- start 2024-05-09T08:23:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68e5e12640 1 -- start start 2024-05-09T08:23:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68e5e12640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e0100770 0x7f68e0069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68e5e12640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 0x7f68e019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68e5e12640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68e0102320 0x7f68e019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68e5e12640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68e010c440 con 0x7f68e0101960 2024-05-09T08:23:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68e5e12640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f68e010c2c0 con 0x7f68e0102320 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68e5e12640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68e010c5c0 con 0x7f68e0100770 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 0x7f68e019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 0x7f68e019c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60336/0 (socket says 172.21.15.28:60336) 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68deffd640 1 -- 172.21.15.28:0/2696774219 learned_addr learned my addr 172.21.15.28:0/2696774219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68df7fe640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e0100770 0x7f68e0069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68dffff640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68e0102320 0x7f68e019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68deffd640 1 -- 172.21.15.28:0/2696774219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e0100770 msgr2=0x7f68e0069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68deffd640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e0100770 0x7f68e0069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68deffd640 1 -- 172.21.15.28:0/2696774219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68e0102320 msgr2=0x7f68e019ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68deffd640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68e0102320 0x7f68e019ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68deffd640 1 -- 172.21.15.28:0/2696774219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e019f830 con 0x7f68e0101960 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68dffff640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68e0102320 0x7f68e019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.156+0000 7f68deffd640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 0x7f68e019c900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f68d4031440 tx=0x7f68d4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.157+0000 7f68dcff9640 1 -- 172.21.15.28:0/2696774219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d403f030 con 0x7f68e0101960 2024-05-09T08:23:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.157+0000 7f68dcff9640 1 -- 172.21.15.28:0/2696774219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68d4003ce0 con 0x7f68e0101960 2024-05-09T08:23:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.157+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e019fa60 con 0x7f68e0101960 2024-05-09T08:23:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.157+0000 7f68dcff9640 1 -- 172.21.15.28:0/2696774219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4035070 con 0x7f68e0101960 2024-05-09T08:23:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.157+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e019ffa0 con 0x7f68e0101960 2024-05-09T08:23:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.158+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e0107a90 con 0x7f68e0101960 2024-05-09T08:23:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.159+0000 7f68dcff9640 1 -- 172.21.15.28:0/2696774219 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f68d4042030 con 0x7f68e0101960 2024-05-09T08:23:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.160+0000 7f68dcff9640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f68c80a2c10 0x7f68c80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.160+0000 7f68df7fe640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f68c80a2c10 0x7f68c80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.160+0000 7f68dcff9640 1 -- 172.21.15.28:0/2696774219 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f68d40ef5a0 con 0x7f68e0101960 2024-05-09T08:23:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.160+0000 7f68df7fe640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f68c80a2c10 0x7f68c80a50d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f68e01017c0 tx=0x7f68d0009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:26.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.163+0000 7f68dcff9640 1 -- 172.21.15.28:0/2696774219 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68d40bc900 con 0x7f68e0101960 2024-05-09T08:23:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.389+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68e002d380 con 0x7f68e0101960 2024-05-09T08:23:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.390+0000 7f68dcff9640 1 -- 172.21.15.28:0/2696774219 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7f68d40bc200 con 0x7f68e0101960 2024-05-09T08:23:26.391 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:26.391 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-09T08:23:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 6 2024-05-09T08:23:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f68c80a2c10 msgr2=0x7f68c80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f68c80a2c10 0x7f68c80a50d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f68e01017c0 tx=0x7f68d0009680 comp rx=0 tx=0).stop 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 msgr2=0x7f68e019c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 0x7f68e019c900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f68d4031440 tx=0x7f68d4032ce0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 shutdown_connections 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f68c80a2c10 0x7f68c80a50d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68e0102320 0x7f68e019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e0101960 0x7f68e019c900 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 --2- 172.21.15.28:0/2696774219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68e0100770 0x7f68e0069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 >> 172.21.15.28:0/2696774219 conn(0x7f68e00fbce0 msgr2=0x7f68e0104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 shutdown_connections 2024-05-09T08:23:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:26.392+0000 7f68e5e12640 1 -- 172.21.15.28:0/2696774219 wait complete. 2024-05-09T08:23:26.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:26 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6834/3736338591,v1:172.21.15.165:6835/3736338591] up:boot 2024-05-09T08:23:26.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:26 smithi184 ceph-mon[34509]: fsmap 5 up:standby 2024-05-09T08:23:26.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:26 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:23:26.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:26 smithi184 ceph-mon[34509]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:26.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:26 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2696774219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:26 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6834/3736338591,v1:172.21.15.165:6835/3736338591] up:boot 2024-05-09T08:23:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:26 smithi028 ceph-mon[30119]: fsmap 5 up:standby 2024-05-09T08:23:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:26 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:23:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:26 smithi028 ceph-mon[30119]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:26 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2696774219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:26.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:26 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6834/3736338591,v1:172.21.15.165:6835/3736338591] up:boot 2024-05-09T08:23:26.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:26 smithi165 ceph-mon[34540]: fsmap 5 up:standby 2024-05-09T08:23:26.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:26 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:23:26.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:26 smithi165 ceph-mon[34540]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:26.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:26 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2696774219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:27.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:23:27 smithi184 systemd[1]: Starting Ceph mds.f for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:23:27.992 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:23:27 smithi184 podman[64754]: 2024-05-09 08:23:27.631797726 +0000 UTC m=+0.018613896 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:23:27.992 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:23:27 smithi184 podman[64754]: 2024-05-09 08:23:27.755952666 +0000 UTC m=+0.142768809 container create ac05c3efc91d257756575b8fc73362196f0995a10b670254834d0cbfe43f2742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-09T08:23:27.992 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:23:27 smithi184 podman[64754]: 2024-05-09 08:23:27.992011617 +0000 UTC m=+0.378827781 container init ac05c3efc91d257756575b8fc73362196f0995a10b670254834d0cbfe43f2742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-09T08:23:28.358 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:23:27 smithi184 podman[64754]: 2024-05-09 08:23:27.994597428 +0000 UTC m=+0.381413574 container start ac05c3efc91d257756575b8fc73362196f0995a10b670254834d0cbfe43f2742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-09T08:23:28.358 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:23:28 smithi184 ceph-mds[64773]: -- 172.21.15.184:0/1223688891 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555da0e93a40 con 0x555da0e66800 2024-05-09T08:23:28.358 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:23:28 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: starting mds.f at 2024-05-09T08:23:28.358 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:23:28 smithi184 bash[64754]: ac05c3efc91d257756575b8fc73362196f0995a10b670254834d0cbfe43f2742 2024-05-09T08:23:28.358 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:23:28 smithi184 systemd[1]: Started Ceph mds.f for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:28.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.523+0000 7fac2ebdc640 1 -- 172.21.15.28:0/3981130822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101980 msgr2=0x7fac28101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.523+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/3981130822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101980 0x7fac28101e00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fac18009ca0 tx=0x7fac1802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.523+0000 7fac2ebdc640 1 -- 172.21.15.28:0/3981130822 shutdown_connections 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.523+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/3981130822 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac28102340 0x7fac28108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.523+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/3981130822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101980 0x7fac28101e00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.523+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/3981130822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac28100790 0x7fac28100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.523+0000 7fac2ebdc640 1 -- 172.21.15.28:0/3981130822 >> 172.21.15.28:0/3981130822 conn(0x7fac280fbd20 msgr2=0x7fac280fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.523+0000 7fac2ebdc640 1 -- 172.21.15.28:0/3981130822 shutdown_connections 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.523+0000 7fac2ebdc640 1 -- 172.21.15.28:0/3981130822 wait complete. 2024-05-09T08:23:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2ebdc640 1 Processor -- start 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2ebdc640 1 -- start start 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2ebdc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac28100790 0x7fac28197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2ebdc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac28101980 0x7fac281980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2ebdc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28102340 0x7fac2819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2ebdc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac2810c410 con 0x7fac28102340 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2ebdc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fac2810c290 con 0x7fac28100790 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2ebdc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac2810c590 con 0x7fac28101980 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2d179640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28102340 0x7fac2819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2d179640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28102340 0x7fac2819f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60362/0 (socket says 172.21.15.28:60362) 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2d179640 1 -- 172.21.15.28:0/265874903 learned_addr learned my addr 172.21.15.28:0/265874903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac27fff640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac28101980 0x7fac281980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.524+0000 7fac2c978640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac28100790 0x7fac28197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac2d179640 1 -- 172.21.15.28:0/265874903 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac28101980 msgr2=0x7fac281980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac2d179640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac28101980 0x7fac281980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac2d179640 1 -- 172.21.15.28:0/265874903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac28100790 msgr2=0x7fac28197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac2d179640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac28100790 0x7fac28197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac2d179640 1 -- 172.21.15.28:0/265874903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2819f7c0 con 0x7fac28102340 2024-05-09T08:23:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac2d179640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28102340 0x7fac2819f150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fac2000e970 tx=0x7fac2000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac25ffb640 1 -- 172.21.15.28:0/265874903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2000ccc0 con 0x7fac28102340 2024-05-09T08:23:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac25ffb640 1 -- 172.21.15.28:0/265874903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac2000ce60 con 0x7fac28102340 2024-05-09T08:23:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2819fab0 con 0x7fac28102340 2024-05-09T08:23:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac25ffb640 1 -- 172.21.15.28:0/265874903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac20010490 con 0x7fac28102340 2024-05-09T08:23:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.525+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac281a0020 con 0x7fac28102340 2024-05-09T08:23:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.526+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac281a0250 con 0x7fac28102340 2024-05-09T08:23:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.527+0000 7fac25ffb640 1 -- 172.21.15.28:0/265874903 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fac20005020 con 0x7fac28102340 2024-05-09T08:23:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.528+0000 7fac25ffb640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fac140a2c10 0x7fac140a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.528+0000 7fac2c978640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fac140a2c10 0x7fac140a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.528+0000 7fac25ffb640 1 -- 172.21.15.28:0/265874903 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fac200ceff0 con 0x7fac28102340 2024-05-09T08:23:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.529+0000 7fac2c978640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fac140a2c10 0x7fac140a50d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac100058c0 tx=0x7fac10005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:28.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.532+0000 7fac25ffb640 1 -- 172.21.15.28:0/265874903 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fac2009c3f0 con 0x7fac28102340 2024-05-09T08:23:28.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.758+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac281a04c0 con 0x7fac28102340 2024-05-09T08:23:28.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.758+0000 7fac25ffb640 1 -- 172.21.15.28:0/265874903 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7fac2009bcf0 con 0x7fac28102340 2024-05-09T08:23:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:28.764 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-05-09T08:23:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-05-09T08:23:28.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fac140a2c10 msgr2=0x7fac140a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:28.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fac140a2c10 0x7fac140a50d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac100058c0 tx=0x7fac10005850 comp rx=0 tx=0).stop 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28102340 msgr2=0x7fac2819f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28102340 0x7fac2819f150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fac2000e970 tx=0x7fac2000ed70 comp rx=0 tx=0).stop 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 shutdown_connections 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fac140a2c10 0x7fac140a50d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28102340 0x7fac2819f150 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac28101980 0x7fac281980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 --2- 172.21.15.28:0/265874903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac28100790 0x7fac28197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.760+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 >> 172.21.15.28:0/265874903 conn(0x7fac280fbd20 msgr2=0x7fac280fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.761+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 shutdown_connections 2024-05-09T08:23:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:28.761+0000 7fac2ebdc640 1 -- 172.21.15.28:0/265874903 wait complete. 2024-05-09T08:23:29.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:29.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:29.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:29.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:29.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:29.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:29.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:29.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: Deploying daemon mds.c on smithi184 2024-05-09T08:23:29.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] up:boot 2024-05-09T08:23:29.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: fsmap 6 up:standby 2024-05-09T08:23:29.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:23:29.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:29 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/265874903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:29.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: Deploying daemon mds.c on smithi184 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] up:boot 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: fsmap 6 up:standby 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:23:29.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:29 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/265874903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:29.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: Deploying daemon mds.c on smithi184 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] up:boot 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: fsmap 6 up:standby 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:23:29.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:29 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/265874903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:30.358 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:23:30 smithi184 systemd[1]: Starting Ceph mds.c for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:23:30.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:30.858 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:23:30 smithi184 podman[65055]: 2024-05-09 08:23:30.344682389 +0000 UTC m=+0.017880108 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:23:30.858 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:23:30 smithi184 podman[65055]: 2024-05-09 08:23:30.462058966 +0000 UTC m=+0.135256672 container create 16b205826f39c8598752120a1df1dc78987f74579ca206c84e797ed771ada3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, ceph=True, GIT_CLEAN=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:23:30.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 -- 172.21.15.28:0/1634431771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 msgr2=0x7f52a0100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:30.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 --2- 172.21.15.28:0/1634431771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 0x7f52a0100bb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f529c009900 tx=0x7f529c02a410 comp rx=0 tx=0).stop 2024-05-09T08:23:30.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 -- 172.21.15.28:0/1634431771 shutdown_connections 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 --2- 172.21.15.28:0/1634431771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0102340 0x7f52a0108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 --2- 172.21.15.28:0/1634431771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52a0101980 0x7f52a0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 --2- 172.21.15.28:0/1634431771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 0x7f52a0100bb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 -- 172.21.15.28:0/1634431771 >> 172.21.15.28:0/1634431771 conn(0x7f52a00fbd20 msgr2=0x7f52a00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 -- 172.21.15.28:0/1634431771 shutdown_connections 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 -- 172.21.15.28:0/1634431771 wait complete. 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 Processor -- start 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.893+0000 7f52a7b53640 1 -- start start 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a7b53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 0x7f52a0196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a7b53640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52a0101980 0x7f52a0197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a7b53640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0102340 0x7f52a019e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a7b53640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52a010c410 con 0x7f52a0100790 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a7b53640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f52a010c290 con 0x7f52a0101980 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a7b53640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52a010c590 con 0x7f52a0102340 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a58ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 0x7f52a0196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a58ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 0x7f52a0196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60374/0 (socket says 172.21.15.28:60374) 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a58ef640 1 -- 172.21.15.28:0/3432052399 learned_addr learned my addr 172.21.15.28:0/3432052399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a60f0640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0102340 0x7f52a019e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a50ee640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52a0101980 0x7f52a0197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a58ef640 1 -- 172.21.15.28:0/3432052399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0102340 msgr2=0x7f52a019e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a58ef640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0102340 0x7f52a019e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a58ef640 1 -- 172.21.15.28:0/3432052399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52a0101980 msgr2=0x7f52a0197000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a58ef640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52a0101980 0x7f52a0197000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a58ef640 1 -- 172.21.15.28:0/3432052399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a019e690 con 0x7f52a0100790 2024-05-09T08:23:30.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f52a58ef640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 0x7f52a0196ac0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f529c031610 tx=0x7f529c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:30.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.894+0000 7f5296ffd640 1 -- 172.21.15.28:0/3432052399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c03f030 con 0x7f52a0100790 2024-05-09T08:23:30.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.895+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a019e920 con 0x7f52a0100790 2024-05-09T08:23:30.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.895+0000 7f5296ffd640 1 -- 172.21.15.28:0/3432052399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f529c003ce0 con 0x7f52a0100790 2024-05-09T08:23:30.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.895+0000 7f5296ffd640 1 -- 172.21.15.28:0/3432052399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c035070 con 0x7f52a0100790 2024-05-09T08:23:30.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.895+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a019ee10 con 0x7f52a0100790 2024-05-09T08:23:30.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.896+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a0107b30 con 0x7f52a0100790 2024-05-09T08:23:30.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.897+0000 7f5296ffd640 1 -- 172.21.15.28:0/3432052399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f529c042030 con 0x7f52a0100790 2024-05-09T08:23:30.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.897+0000 7f5296ffd640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f528c0a2b90 0x7f528c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:30.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.898+0000 7f52a50ee640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f528c0a2b90 0x7f528c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:30.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.898+0000 7f5296ffd640 1 -- 172.21.15.28:0/3432052399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f529c0eee10 con 0x7f52a0100790 2024-05-09T08:23:30.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.898+0000 7f52a50ee640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f528c0a2b90 0x7f528c0a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5290005fd0 tx=0x7f5290005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:30.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:30.902+0000 7f5296ffd640 1 -- 172.21.15.28:0/3432052399 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f529c0bc210 con 0x7f52a0100790 2024-05-09T08:23:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.129+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52a002d1c0 con 0x7f52a0100790 2024-05-09T08:23:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.129+0000 7f5296ffd640 1 -- 172.21.15.28:0/3432052399 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f529c0bbb10 con 0x7f52a0100790 2024-05-09T08:23:31.130 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:31.130 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-05-09T08:23:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-05-09T08:23:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f528c0a2b90 msgr2=0x7f528c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f528c0a2b90 0x7f528c0a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5290005fd0 tx=0x7f5290005850 comp rx=0 tx=0).stop 2024-05-09T08:23:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 msgr2=0x7f52a0196ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 0x7f52a0196ac0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f529c031610 tx=0x7f529c0043b0 comp rx=0 tx=0).stop 2024-05-09T08:23:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 shutdown_connections 2024-05-09T08:23:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f528c0a2b90 0x7f528c0a5050 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0102340 0x7f52a019e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52a0101980 0x7f52a0197000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 --2- 172.21.15.28:0/3432052399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a0100790 0x7f52a0196ac0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 >> 172.21.15.28:0/3432052399 conn(0x7f52a00fbd20 msgr2=0x7f52a0104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.131+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 shutdown_connections 2024-05-09T08:23:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:31.132+0000 7f52a7b53640 1 -- 172.21.15.28:0/3432052399 wait complete. 2024-05-09T08:23:31.145 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:23:30 smithi184 podman[65055]: 2024-05-09 08:23:30.876321875 +0000 UTC m=+0.549519581 container init 16b205826f39c8598752120a1df1dc78987f74579ca206c84e797ed771ada3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-09T08:23:31.146 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:23:30 smithi184 podman[65055]: 2024-05-09 08:23:30.879108197 +0000 UTC m=+0.552305904 container start 16b205826f39c8598752120a1df1dc78987f74579ca206c84e797ed771ada3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-05-09T08:23:31.146 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:23:30 smithi184 ceph-mds[65075]: -- 172.21.15.184:0/1042418775 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619e9303a40 con 0x5619e92d6c00 2024-05-09T08:23:31.146 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:23:30 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: starting mds.c at 2024-05-09T08:23:31.146 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:23:30 smithi184 bash[65055]: 16b205826f39c8598752120a1df1dc78987f74579ca206c84e797ed771ada3cc 2024-05-09T08:23:31.146 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:23:30 smithi184 systemd[1]: Started Ceph mds.c for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:31.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:31 smithi165 ceph-mon[34540]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:31.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:31 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:31.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:31 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:31.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:31 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:31.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:31 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:31.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:31 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:31.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:31 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:31.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:31 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3432052399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:31.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:31 smithi028 ceph-mon[30119]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:31.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:31 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:31.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:31 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:31.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:31 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:31.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:31 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:31.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:31 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:31.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:31 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:31.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:31 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3432052399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:31.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:31 smithi184 ceph-mon[34509]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:31.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:31 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:31.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:31 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:31.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:31 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:31.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:31 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:31.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:31 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:31.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:31 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:31.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:31 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3432052399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:32.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:32 smithi165 ceph-mon[34540]: Deploying daemon mds.h on smithi165 2024-05-09T08:23:32.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:32 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] up:boot 2024-05-09T08:23:32.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:32 smithi165 ceph-mon[34540]: fsmap 7 up:standby 2024-05-09T08:23:32.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:32 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:23:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:32 smithi184 ceph-mon[34509]: Deploying daemon mds.h on smithi165 2024-05-09T08:23:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:32 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] up:boot 2024-05-09T08:23:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:32 smithi184 ceph-mon[34509]: fsmap 7 up:standby 2024-05-09T08:23:32.422 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:32 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:23:32.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:32 smithi028 ceph-mon[30119]: Deploying daemon mds.h on smithi165 2024-05-09T08:23:32.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:32 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] up:boot 2024-05-09T08:23:32.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:32 smithi028 ceph-mon[30119]: fsmap 7 up:standby 2024-05-09T08:23:32.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:32 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:23:33.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.261+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2692826080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 msgr2=0x7ffb7c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.261+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2692826080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 0x7ffb7c1099c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009ca0 tx=0x7ffb7402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.261+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2692826080 shutdown_connections 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.261+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2692826080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 0x7ffb7c1099c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.261+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2692826080 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb7c104c40 0x7ffb7c107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.261+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2692826080 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb7c102310 0x7ffb7c104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.261+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2692826080 >> 172.21.15.28:0/2692826080 conn(0x7ffb7c0fbd20 msgr2=0x7ffb7c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.261+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2692826080 shutdown_connections 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.261+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2692826080 wait complete. 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb83b4f640 1 Processor -- start 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb83b4f640 1 -- start start 2024-05-09T08:23:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb83b4f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb7c102310 0x7ffb7c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb83b4f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb7c104c40 0x7ffb7c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb83b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 0x7ffb7c19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb83b4f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c10d560 con 0x7ffb7c107570 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb83b4f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c10d3e0 con 0x7ffb7c102310 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb83b4f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c10d6e0 con 0x7ffb7c104c40 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb820ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 0x7ffb7c19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb820ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 0x7ffb7c19f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60390/0 (socket says 172.21.15.28:60390) 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb818eb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb7c102310 0x7ffb7c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb820ec640 1 -- 172.21.15.28:0/2137847791 learned_addr learned my addr 172.21.15.28:0/2137847791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.262+0000 7ffb810ea640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb7c104c40 0x7ffb7c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb820ec640 1 -- 172.21.15.28:0/2137847791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb7c104c40 msgr2=0x7ffb7c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb820ec640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb7c104c40 0x7ffb7c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb820ec640 1 -- 172.21.15.28:0/2137847791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb7c102310 msgr2=0x7ffb7c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb820ec640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb7c102310 0x7ffb7c197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb820ec640 1 -- 172.21.15.28:0/2137847791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7c19f770 con 0x7ffb7c107570 2024-05-09T08:23:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb818eb640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb7c102310 0x7ffb7c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:23:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb810ea640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb7c104c40 0x7ffb7c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:23:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb820ec640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 0x7ffb7c19f100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb7402f540 tx=0x7ffb74031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb72ffd640 1 -- 172.21.15.28:0/2137847791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74003b70 con 0x7ffb7c107570 2024-05-09T08:23:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb72ffd640 1 -- 172.21.15.28:0/2137847791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb74003d10 con 0x7ffb7c107570 2024-05-09T08:23:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7c19fa00 con 0x7ffb7c107570 2024-05-09T08:23:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb72ffd640 1 -- 172.21.15.28:0/2137847791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7402fb50 con 0x7ffb7c107570 2024-05-09T08:23:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.263+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb7c19ffa0 con 0x7ffb7c107570 2024-05-09T08:23:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.264+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb7c0fd810 con 0x7ffb7c107570 2024-05-09T08:23:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.265+0000 7ffb72ffd640 1 -- 172.21.15.28:0/2137847791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ffb7402fcf0 con 0x7ffb7c107570 2024-05-09T08:23:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.266+0000 7ffb72ffd640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb680a2c10 0x7ffb680a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.266+0000 7ffb818eb640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb680a2c10 0x7ffb680a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.266+0000 7ffb72ffd640 1 -- 172.21.15.28:0/2137847791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffb740eff70 con 0x7ffb7c107570 2024-05-09T08:23:33.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.267+0000 7ffb818eb640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb680a2c10 0x7ffb680a50d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009fc0 tx=0x7ffb78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.271+0000 7ffb72ffd640 1 -- 172.21.15.28:0/2137847791 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffb740bd2d0 con 0x7ffb7c107570 2024-05-09T08:23:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:33 smithi165 ceph-mon[34540]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:33.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:33 smithi028 ceph-mon[30119]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.495+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb7c062430 con 0x7ffb7c107570 2024-05-09T08:23:33.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.496+0000 7ffb72ffd640 1 -- 172.21.15.28:0/2137847791 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7ffb740bcbd0 con 0x7ffb7c107570 2024-05-09T08:23:33.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-09T08:23:33.496 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-05-09T08:23:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb680a2c10 msgr2=0x7ffb680a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb680a2c10 0x7ffb680a50d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009fc0 tx=0x7ffb78005f90 comp rx=0 tx=0).stop 2024-05-09T08:23:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 msgr2=0x7ffb7c19f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 0x7ffb7c19f100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb7402f540 tx=0x7ffb74031750 comp rx=0 tx=0).stop 2024-05-09T08:23:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 shutdown_connections 2024-05-09T08:23:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb680a2c10 0x7ffb680a50d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7c107570 0x7ffb7c19f100 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb7c104c40 0x7ffb7c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 --2- 172.21.15.28:0/2137847791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb7c102310 0x7ffb7c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 >> 172.21.15.28:0/2137847791 conn(0x7ffb7c0fbd20 msgr2=0x7ffb7c102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 shutdown_connections 2024-05-09T08:23:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:33.498+0000 7ffb83b4f640 1 -- 172.21.15.28:0/2137847791 wait complete. 2024-05-09T08:23:33.602 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:23:33 smithi165 podman[64286]: 2024-05-09 08:23:33.532676141 +0000 UTC m=+0.127585060 container create 65dfbf08fba2f7180771c8a821bf5f4e6bc3fffdbb9c40833c1d1d8cf6b0a38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-09T08:23:33.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:33 smithi184 ceph-mon[34509]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:34.102 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:23:33 smithi165 podman[64286]: 2024-05-09 08:23:33.794242728 +0000 UTC m=+0.389151651 container init 65dfbf08fba2f7180771c8a821bf5f4e6bc3fffdbb9c40833c1d1d8cf6b0a38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-09T08:23:34.102 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:23:33 smithi165 podman[64286]: 2024-05-09 08:23:33.796826512 +0000 UTC m=+0.391735434 container start 65dfbf08fba2f7180771c8a821bf5f4e6bc3fffdbb9c40833c1d1d8cf6b0a38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-09T08:23:34.102 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:23:33 smithi165 ceph-mds[64304]: -- 172.21.15.165:0/248410887 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c597233a40 con 0x55c597207000 2024-05-09T08:23:34.102 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:23:33 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: starting mds.h at 2024-05-09T08:23:34.102 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:23:33 smithi165 bash[64286]: 65dfbf08fba2f7180771c8a821bf5f4e6bc3fffdbb9c40833c1d1d8cf6b0a38a 2024-05-09T08:23:34.103 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:23:33 smithi165 systemd[1]: Started Ceph mds.h for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:34.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:34 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2137847791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:34.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:34 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:34.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:34 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:34.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:34 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:34.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:34 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:34.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:34 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:34.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:34 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:34.598 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:34 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2137847791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:34.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:34 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:34.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:34 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:34.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:34 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:34.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:34 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:34.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:34 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:34.599 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:34 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:34.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:34 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2137847791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:34.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:34 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:34.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:34 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:34.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:34 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:34.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:34 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:34.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:34 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:34.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:34 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:35.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:35 smithi028 ceph-mon[30119]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:35.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:35 smithi028 ceph-mon[30119]: Deploying daemon mds.i on smithi184 2024-05-09T08:23:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:35 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] up:boot 2024-05-09T08:23:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:35 smithi028 ceph-mon[30119]: fsmap 8 up:standby 2024-05-09T08:23:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:35 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:23:35.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:35 smithi165 ceph-mon[34540]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:35 smithi165 ceph-mon[34540]: Deploying daemon mds.i on smithi184 2024-05-09T08:23:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:35 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] up:boot 2024-05-09T08:23:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:35 smithi165 ceph-mon[34540]: fsmap 8 up:standby 2024-05-09T08:23:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:35 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:23:35.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:35 smithi184 ceph-mon[34509]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:35.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:35 smithi184 ceph-mon[34509]: Deploying daemon mds.i on smithi184 2024-05-09T08:23:35.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:35 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] up:boot 2024-05-09T08:23:35.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:35 smithi184 ceph-mon[34509]: fsmap 8 up:standby 2024-05-09T08:23:35.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:35 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:23:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.627+0000 7f91014bc640 1 -- 172.21.15.28:0/3290502779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc107570 msgr2=0x7f90fc1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.627+0000 7f91014bc640 1 --2- 172.21.15.28:0/3290502779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc107570 0x7f90fc1099c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009ca0 tx=0x7f90f402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.627+0000 7f91014bc640 1 -- 172.21.15.28:0/3290502779 shutdown_connections 2024-05-09T08:23:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.627+0000 7f91014bc640 1 --2- 172.21.15.28:0/3290502779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc107570 0x7f90fc1099c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.627+0000 7f91014bc640 1 --2- 172.21.15.28:0/3290502779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc104c40 0x7f90fc107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.627+0000 7f91014bc640 1 --2- 172.21.15.28:0/3290502779 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90fc102310 0x7f90fc104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.627+0000 7f91014bc640 1 -- 172.21.15.28:0/3290502779 >> 172.21.15.28:0/3290502779 conn(0x7f90fc0fbd20 msgr2=0x7f90fc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.627+0000 7f91014bc640 1 -- 172.21.15.28:0/3290502779 shutdown_connections 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.627+0000 7f91014bc640 1 -- 172.21.15.28:0/3290502779 wait complete. 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f91014bc640 1 Processor -- start 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f91014bc640 1 -- start start 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f91014bc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90fc102310 0x7f90fc101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f91014bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc104c40 0x7f90fc1002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f91014bc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc107570 0x7f90fc100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f91014bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90fc10d5b0 con 0x7f90fc104c40 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f91014bc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f90fc10d430 con 0x7f90fc102310 2024-05-09T08:23:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f91014bc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90fc10d730 con 0x7f90fc107570 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f90f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc104c40 0x7f90fc1002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f90f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc104c40 0x7f90fc1002c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35006/0 (socket says 172.21.15.28:35006) 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f90f27fc640 1 -- 172.21.15.28:0/3194437614 learned_addr learned my addr 172.21.15.28:0/3194437614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f90fb7fe640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc107570 0x7f90fc100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.628+0000 7f90faffd640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90fc102310 0x7f90fc101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90f27fc640 1 -- 172.21.15.28:0/3194437614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc107570 msgr2=0x7f90fc100800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90f27fc640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc107570 0x7f90fc100800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90f27fc640 1 -- 172.21.15.28:0/3194437614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90fc102310 msgr2=0x7f90fc101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90f27fc640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90fc102310 0x7f90fc101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90f27fc640 1 -- 172.21.15.28:0/3194437614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90fc101090 con 0x7f90fc104c40 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90faffd640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90fc102310 0x7f90fc101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90f27fc640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc104c40 0x7f90fc1002c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f90e400c970 tx=0x7f90e400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90f8ff9640 1 -- 172.21.15.28:0/3194437614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4007bb0 con 0x7f90fc104c40 2024-05-09T08:23:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90fc1a1cf0 con 0x7f90fc104c40 2024-05-09T08:23:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90f8ff9640 1 -- 172.21.15.28:0/3194437614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90e4007d50 con 0x7f90fc104c40 2024-05-09T08:23:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f90f8ff9640 1 -- 172.21.15.28:0/3194437614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e40187a0 con 0x7f90fc104c40 2024-05-09T08:23:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.629+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90fc1a2210 con 0x7f90fc104c40 2024-05-09T08:23:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.630+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90fc0fd810 con 0x7f90fc104c40 2024-05-09T08:23:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.631+0000 7f90f8ff9640 1 -- 172.21.15.28:0/3194437614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f90e400f450 con 0x7f90fc104c40 2024-05-09T08:23:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.632+0000 7f90f8ff9640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f90e80a2b90 0x7f90e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.632+0000 7f90faffd640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f90e80a2b90 0x7f90e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.632+0000 7f90f8ff9640 1 -- 172.21.15.28:0/3194437614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f90e40cec40 con 0x7f90fc104c40 2024-05-09T08:23:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.633+0000 7f90faffd640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f90e80a2b90 0x7f90e80a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009930 tx=0x7f90ec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.637+0000 7f90f8ff9640 1 -- 172.21.15.28:0/3194437614 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f90e4014030 con 0x7f90fc104c40 2024-05-09T08:23:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.861+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90fc02d370 con 0x7f90fc104c40 2024-05-09T08:23:35.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.862+0000 7f90f8ff9640 1 -- 172.21.15.28:0/3194437614 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7f90e409c040 con 0x7f90fc104c40 2024-05-09T08:23:35.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:35.863 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-09T08:23:35.863 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-05-09T08:23:35.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f90e80a2b90 msgr2=0x7f90e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:35.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f90e80a2b90 0x7f90e80a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009930 tx=0x7f90ec008040 comp rx=0 tx=0).stop 2024-05-09T08:23:35.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc104c40 msgr2=0x7f90fc1002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc104c40 0x7f90fc1002c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f90e400c970 tx=0x7f90e400ce30 comp rx=0 tx=0).stop 2024-05-09T08:23:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 shutdown_connections 2024-05-09T08:23:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f90e80a2b90 0x7f90e80a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90fc107570 0x7f90fc100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90fc104c40 0x7f90fc1002c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 --2- 172.21.15.28:0/3194437614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90fc102310 0x7f90fc101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 >> 172.21.15.28:0/3194437614 conn(0x7f90fc0fbd20 msgr2=0x7f90fc102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 shutdown_connections 2024-05-09T08:23:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:35.864+0000 7f91014bc640 1 -- 172.21.15.28:0/3194437614 wait complete. 2024-05-09T08:23:36.169 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:23:36 smithi184 systemd[1]: Starting Ceph mds.i for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:23:36.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:36 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3194437614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:36.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:36 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3194437614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:36.463 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:23:36 smithi184 podman[65377]: 2024-05-09 08:23:36.21357879 +0000 UTC m=+0.017818094 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:23:36.464 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:23:36 smithi184 podman[65377]: 2024-05-09 08:23:36.334749817 +0000 UTC m=+0.138989111 container create 5514586d1cef6e2b9a420749fe873b47fdafda8b1c96cbb3e906a3d3f55ae89b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0) 2024-05-09T08:23:36.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:36 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3194437614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:36.858 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:23:36 smithi184 podman[65377]: 2024-05-09 08:23:36.596051936 +0000 UTC m=+0.400291229 container init 5514586d1cef6e2b9a420749fe873b47fdafda8b1c96cbb3e906a3d3f55ae89b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc) 2024-05-09T08:23:36.858 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:23:36 smithi184 podman[65377]: 2024-05-09 08:23:36.599452867 +0000 UTC m=+0.403692157 container start 5514586d1cef6e2b9a420749fe873b47fdafda8b1c96cbb3e906a3d3f55ae89b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-09T08:23:36.858 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:23:36 smithi184 ceph-mds[65397]: -- 172.21.15.184:0/4128672825 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c79c561a40 con 0x55c79c534c00 2024-05-09T08:23:36.858 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:23:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: starting mds.i at 2024-05-09T08:23:36.858 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:23:36 smithi184 bash[65377]: 5514586d1cef6e2b9a420749fe873b47fdafda8b1c96cbb3e906a3d3f55ae89b 2024-05-09T08:23:36.858 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:23:36 smithi184 systemd[1]: Started Ceph mds.i for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:37.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:37 smithi028 ceph-mon[30119]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:37.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:37.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:37.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:37.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:37.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:37.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:37 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:37 smithi165 ceph-mon[34540]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:37.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:37.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:37 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:37.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:37 smithi184 ceph-mon[34509]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:37.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:37.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:37.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:37.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:37.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:37.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:37 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:37.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.987+0000 7f94feb09640 1 -- 172.21.15.28:0/2777580731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8104c40 msgr2=0x7f94f8107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.987+0000 7f94feb09640 1 --2- 172.21.15.28:0/2777580731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8104c40 0x7f94f8107030 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009900 tx=0x7f94ec02a410 comp rx=0 tx=0).stop 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.987+0000 7f94feb09640 1 -- 172.21.15.28:0/2777580731 shutdown_connections 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.987+0000 7f94feb09640 1 --2- 172.21.15.28:0/2777580731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8107570 0x7f94f81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.987+0000 7f94feb09640 1 --2- 172.21.15.28:0/2777580731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8104c40 0x7f94f8107030 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.987+0000 7f94feb09640 1 --2- 172.21.15.28:0/2777580731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f8102310 0x7f94f8104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.987+0000 7f94feb09640 1 -- 172.21.15.28:0/2777580731 >> 172.21.15.28:0/2777580731 conn(0x7f94f80fbd20 msgr2=0x7f94f80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.987+0000 7f94feb09640 1 -- 172.21.15.28:0/2777580731 shutdown_connections 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.987+0000 7f94feb09640 1 -- 172.21.15.28:0/2777580731 wait complete. 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94feb09640 1 Processor -- start 2024-05-09T08:23:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94feb09640 1 -- start start 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94feb09640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8102310 0x7f94f8101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94feb09640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f8104c40 0x7f94f81002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94feb09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8107570 0x7f94f8100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94feb09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94f810d510 con 0x7f94f8107570 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94feb09640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f94f810d390 con 0x7f94f8104c40 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94feb09640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f94f810d690 con 0x7f94f8102310 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94fd0a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8107570 0x7f94f8100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94fd0a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8107570 0x7f94f8100800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35024/0 (socket says 172.21.15.28:35024) 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94fd0a6640 1 -- 172.21.15.28:0/1270228858 learned_addr learned my addr 172.21.15.28:0/1270228858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.988+0000 7f94fc8a5640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8102310 0x7f94f8101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94f7fff640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f8104c40 0x7f94f81002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94fd0a6640 1 -- 172.21.15.28:0/1270228858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8102310 msgr2=0x7f94f8101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94fd0a6640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8102310 0x7f94f8101c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94fd0a6640 1 -- 172.21.15.28:0/1270228858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f8104c40 msgr2=0x7f94f81002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94fd0a6640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f8104c40 0x7f94f81002c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94fd0a6640 1 -- 172.21.15.28:0/1270228858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f81010c0 con 0x7f94f8107570 2024-05-09T08:23:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94fd0a6640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8107570 0x7f94f8100800 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f94f000cad0 tx=0x7f94f000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94f5ffb640 1 -- 172.21.15.28:0/1270228858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f000ae20 con 0x7f94f8107570 2024-05-09T08:23:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94f5ffb640 1 -- 172.21.15.28:0/1270228858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f0005020 con 0x7f94f8107570 2024-05-09T08:23:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f81a1d00 con 0x7f94f8107570 2024-05-09T08:23:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94f5ffb640 1 -- 172.21.15.28:0/1270228858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f000f6f0 con 0x7f94f8107570 2024-05-09T08:23:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.989+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f81a2210 con 0x7f94f8107570 2024-05-09T08:23:37.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.990+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94f80fd810 con 0x7f94f8107570 2024-05-09T08:23:37.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.991+0000 7f94f5ffb640 1 -- 172.21.15.28:0/1270228858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f94f00051d0 con 0x7f94f8107570 2024-05-09T08:23:37.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.992+0000 7f94f5ffb640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f94e00a2c10 0x7f94e00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:37.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.992+0000 7f94fc8a5640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f94e00a2c10 0x7f94e00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:37.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.992+0000 7f94f5ffb640 1 -- 172.21.15.28:0/1270228858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f94f00cee70 con 0x7f94f8107570 2024-05-09T08:23:37.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.992+0000 7f94fc8a5640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f94e00a2c10 0x7f94e00a50d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f94e80058c0 tx=0x7f94e800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:37.996+0000 7f94f5ffb640 1 -- 172.21.15.28:0/1270228858 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f94f0014030 con 0x7f94f8107570 2024-05-09T08:23:38.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.223+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94f8062430 con 0x7f94f8107570 2024-05-09T08:23:38.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.224+0000 7f94f5ffb640 1 -- 172.21.15.28:0/1270228858 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7f94f009c270 con 0x7f94f8107570 2024-05-09T08:23:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:38.224 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T08:23:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f94e00a2c10 msgr2=0x7f94e00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f94e00a2c10 0x7f94e00a50d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f94e80058c0 tx=0x7f94e800a2d0 comp rx=0 tx=0).stop 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8107570 msgr2=0x7f94f8100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8107570 0x7f94f8100800 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f94f000cad0 tx=0x7f94f000cf90 comp rx=0 tx=0).stop 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 shutdown_connections 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f94e00a2c10 0x7f94e00a50d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94f8107570 0x7f94f8100800 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f8104c40 0x7f94f81002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 --2- 172.21.15.28:0/1270228858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f94f8102310 0x7f94f8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 >> 172.21.15.28:0/1270228858 conn(0x7f94f80fbd20 msgr2=0x7f94f8102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 shutdown_connections 2024-05-09T08:23:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:38.226+0000 7f94feb09640 1 -- 172.21.15.28:0/1270228858 wait complete. 2024-05-09T08:23:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:38 smithi028 ceph-mon[30119]: Deploying daemon mds.g on smithi028 2024-05-09T08:23:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:38 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] up:boot 2024-05-09T08:23:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:38 smithi028 ceph-mon[30119]: fsmap 9 up:standby 2024-05-09T08:23:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:38 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:23:38.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:38 smithi165 ceph-mon[34540]: Deploying daemon mds.g on smithi028 2024-05-09T08:23:38.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:38 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] up:boot 2024-05-09T08:23:38.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:38 smithi165 ceph-mon[34540]: fsmap 9 up:standby 2024-05-09T08:23:38.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:38 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:23:38.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:38 smithi184 ceph-mon[34509]: Deploying daemon mds.g on smithi028 2024-05-09T08:23:38.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:38 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] up:boot 2024-05-09T08:23:38.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:38 smithi184 ceph-mon[34509]: fsmap 9 up:standby 2024-05-09T08:23:38.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:38 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:23:39.183 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:23:39 smithi028 systemd[1]: Starting Ceph mds.g for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:23:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:39 smithi028 ceph-mon[30119]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:39.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:39 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1270228858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:39.433 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:23:39 smithi028 podman[68678]: 2024-05-09 08:23:39.217494572 +0000 UTC m=+0.018524385 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:23:39.433 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:23:39 smithi028 podman[68678]: 2024-05-09 08:23:39.331918895 +0000 UTC m=+0.132948696 container create a6350a7cbf4a1d273a40f97d2ea3fee247bddf142e24d49b59e9adc8e10127d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-09T08:23:39.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:39 smithi165 ceph-mon[34540]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:39.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:39 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1270228858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:39.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:39 smithi184 ceph-mon[34509]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:39.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:39 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1270228858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:39.933 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:23:39 smithi028 podman[68678]: 2024-05-09 08:23:39.568311902 +0000 UTC m=+0.369341708 container init a6350a7cbf4a1d273a40f97d2ea3fee247bddf142e24d49b59e9adc8e10127d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-09T08:23:39.933 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:23:39 smithi028 podman[68678]: 2024-05-09 08:23:39.571243216 +0000 UTC m=+0.372273026 container start a6350a7cbf4a1d273a40f97d2ea3fee247bddf142e24d49b59e9adc8e10127d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-09T08:23:39.934 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:23:39 smithi028 ceph-mds[68697]: -- 172.21.15.28:0/3145002777 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a7d31b1a40 con 0x55a7d3185000 2024-05-09T08:23:39.934 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:23:39 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g[68693]: starting mds.g at 2024-05-09T08:23:39.934 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:23:39 smithi028 bash[68678]: a6350a7cbf4a1d273a40f97d2ea3fee247bddf142e24d49b59e9adc8e10127d0 2024-05-09T08:23:39.934 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:23:39 smithi028 systemd[1]: Started Ceph mds.g for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:40.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.357+0000 7fe633d42640 1 -- 172.21.15.28:0/1269851416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c103430 msgr2=0x7fe62c109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.357+0000 7fe633d42640 1 --2- 172.21.15.28:0/1269851416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c103430 0x7fe62c109cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe624009ca0 tx=0x7fe62402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.357+0000 7fe633d42640 1 -- 172.21.15.28:0/1269851416 shutdown_connections 2024-05-09T08:23:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.357+0000 7fe633d42640 1 --2- 172.21.15.28:0/1269851416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c103430 0x7fe62c109cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.357+0000 7fe633d42640 1 --2- 172.21.15.28:0/1269851416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe62c102a70 0x7fe62c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.357+0000 7fe633d42640 1 --2- 172.21.15.28:0/1269851416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62c101880 0x7fe62c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.357+0000 7fe633d42640 1 -- 172.21.15.28:0/1269851416 >> 172.21.15.28:0/1269851416 conn(0x7fe62c0fce10 msgr2=0x7fe62c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.357+0000 7fe633d42640 1 -- 172.21.15.28:0/1269851416 shutdown_connections 2024-05-09T08:23:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.357+0000 7fe633d42640 1 -- 172.21.15.28:0/1269851416 wait complete. 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe633d42640 1 Processor -- start 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe633d42640 1 -- start start 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe633d42640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62c101880 0x7fe62c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe633d42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c102a70 0x7fe62c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe633d42640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe62c103430 0x7fe62c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe633d42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c10d4e0 con 0x7fe62c102a70 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe633d42640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c10d360 con 0x7fe62c103430 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe633d42640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c10d660 con 0x7fe62c101880 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe6312dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c102a70 0x7fe62c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe6312dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c102a70 0x7fe62c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35044/0 (socket says 172.21.15.28:35044) 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.358+0000 7fe6312dd640 1 -- 172.21.15.28:0/1129686506 learned_addr learned my addr 172.21.15.28:0/1129686506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe631ade640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62c101880 0x7fe62c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe6322df640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe62c103430 0x7fe62c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe6312dd640 1 -- 172.21.15.28:0/1129686506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62c101880 msgr2=0x7fe62c197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe6312dd640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62c101880 0x7fe62c197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe6312dd640 1 -- 172.21.15.28:0/1129686506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe62c103430 msgr2=0x7fe62c19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe6312dd640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe62c103430 0x7fe62c19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe6312dd640 1 -- 172.21.15.28:0/1129686506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c19f690 con 0x7fe62c102a70 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe631ade640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62c101880 0x7fe62c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe6322df640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe62c103430 0x7fe62c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe6312dd640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c102a70 0x7fe62c197fb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe62000d950 tx=0x7fe62000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe61effd640 1 -- 172.21.15.28:0/1129686506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620017030 con 0x7fe62c102a70 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe61effd640 1 -- 172.21.15.28:0/1129686506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe620004020 con 0x7fe62c102a70 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62c19f980 con 0x7fe62c102a70 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe61effd640 1 -- 172.21.15.28:0/1129686506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6200047a0 con 0x7fe62c102a70 2024-05-09T08:23:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.359+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62c19fed0 con 0x7fe62c102a70 2024-05-09T08:23:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.360+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62c108ad0 con 0x7fe62c102a70 2024-05-09T08:23:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.361+0000 7fe61effd640 1 -- 172.21.15.28:0/1129686506 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe6200041d0 con 0x7fe62c102a70 2024-05-09T08:23:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.362+0000 7fe61effd640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6180a2b90 0x7fe6180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.362+0000 7fe631ade640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6180a2b90 0x7fe6180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.362+0000 7fe61effd640 1 -- 172.21.15.28:0/1129686506 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe6200ce6e0 con 0x7fe62c102a70 2024-05-09T08:23:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.363+0000 7fe631ade640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6180a2b90 0x7fe6180a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe62c1028d0 tx=0x7fe628005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.367+0000 7fe61effd640 1 -- 172.21.15.28:0/1129686506 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe62009ba40 con 0x7fe62c102a70 2024-05-09T08:23:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.594+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe62c02d1c0 con 0x7fe62c102a70 2024-05-09T08:23:40.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.595+0000 7fe61effd640 1 -- 172.21.15.28:0/1129686506 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7fe62009b340 con 0x7fe62c102a70 2024-05-09T08:23:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:40.596 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T08:23:40.596 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 11 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6180a2b90 msgr2=0x7fe6180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6180a2b90 0x7fe6180a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe62c1028d0 tx=0x7fe628005db0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c102a70 msgr2=0x7fe62c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c102a70 0x7fe62c197fb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe62000d950 tx=0x7fe62000de10 comp rx=0 tx=0).stop 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 shutdown_connections 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe6180a2b90 0x7fe6180a5050 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe62c103430 0x7fe62c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62c102a70 0x7fe62c197fb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 --2- 172.21.15.28:0/1129686506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62c101880 0x7fe62c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.597+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 >> 172.21.15.28:0/1129686506 conn(0x7fe62c0fce10 msgr2=0x7fe62c105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:40.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.598+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 shutdown_connections 2024-05-09T08:23:40.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:40.598+0000 7fe633d42640 1 -- 172.21.15.28:0/1129686506 wait complete. 2024-05-09T08:23:40.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:40.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:40.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:40.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: Deploying daemon mds.a on smithi028 2024-05-09T08:23:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.28:6838/612632862,v1:172.21.15.28:6839/612632862] up:boot 2024-05-09T08:23:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: fsmap 10 up:standby 2024-05-09T08:23:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:23:40.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:40 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1129686506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:41.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: Deploying daemon mds.a on smithi028 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.28:6838/612632862,v1:172.21.15.28:6839/612632862] up:boot 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: fsmap 10 up:standby 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:23:41.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:40 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1129686506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: Deploying daemon mds.a on smithi028 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.28:6838/612632862,v1:172.21.15.28:6839/612632862] up:boot 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: fsmap 10 up:standby 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:23:41.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:40 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1129686506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:42.013 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:23:41 smithi028 systemd[1]: Starting Ceph mds.a for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:23:42.305 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:23:42 smithi028 podman[69030]: 2024-05-09 08:23:42.090622834 +0000 UTC m=+0.018786971 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:23:42.305 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:23:42 smithi028 podman[69030]: 2024-05-09 08:23:42.193717736 +0000 UTC m=+0.121881846 container create 9c62fc1c03aded0f1d12e49ff45a5b30126d7d9361b104789c09117eff835439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-96805dc) 2024-05-09T08:23:42.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:42.638 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:23:42 smithi028 podman[69030]: 2024-05-09 08:23:42.421393529 +0000 UTC m=+0.349557642 container init 9c62fc1c03aded0f1d12e49ff45a5b30126d7d9361b104789c09117eff835439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T08:23:42.638 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:23:42 smithi028 podman[69030]: 2024-05-09 08:23:42.424966113 +0000 UTC m=+0.353130226 container start 9c62fc1c03aded0f1d12e49ff45a5b30126d7d9361b104789c09117eff835439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-09T08:23:42.638 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:23:42 smithi028 ceph-mds[69057]: -- 172.21.15.28:0/146967561 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557d67fa1a40 con 0x557d67f74c00 2024-05-09T08:23:42.638 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:23:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a[69053]: starting mds.a at 2024-05-09T08:23:42.638 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:23:42 smithi028 bash[69030]: 9c62fc1c03aded0f1d12e49ff45a5b30126d7d9361b104789c09117eff835439 2024-05-09T08:23:42.638 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:23:42 smithi028 systemd[1]: Started Ceph mds.a for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/1279019309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68103430 msgr2=0x7f2e68109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/1279019309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68103430 0x7f2e68109cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009ca0 tx=0x7f2e6002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/1279019309 shutdown_connections 2024-05-09T08:23:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/1279019309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68103430 0x7f2e68109cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/1279019309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e68102a70 0x7f2e68102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/1279019309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e68101880 0x7f2e68101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/1279019309 >> 172.21.15.28:0/1279019309 conn(0x7f2e680fce10 msgr2=0x7f2e680ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/1279019309 shutdown_connections 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/1279019309 wait complete. 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.731+0000 7f2e6f6bd640 1 Processor -- start 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6f6bd640 1 -- start start 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6f6bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e68101880 0x7f2e68197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6f6bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68102a70 0x7f2e681980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6f6bd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e68103430 0x7f2e6819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6f6bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6810d550 con 0x7f2e68102a70 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6f6bd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6810d3d0 con 0x7f2e68103430 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6f6bd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6810d6d0 con 0x7f2e68101880 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6cc58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68102a70 0x7f2e681980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6cc58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68102a70 0x7f2e681980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35094/0 (socket says 172.21.15.28:35094) 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6cc58640 1 -- 172.21.15.28:0/3204344648 learned_addr learned my addr 172.21.15.28:0/3204344648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6d459640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e68101880 0x7f2e68197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6dc5a640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e68103430 0x7f2e6819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6cc58640 1 -- 172.21.15.28:0/3204344648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e68101880 msgr2=0x7f2e68197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6cc58640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e68101880 0x7f2e68197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6cc58640 1 -- 172.21.15.28:0/3204344648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e68103430 msgr2=0x7f2e6819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6cc58640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e68103430 0x7f2e6819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6cc58640 1 -- 172.21.15.28:0/3204344648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6819f780 con 0x7f2e68102a70 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.732+0000 7f2e6cc58640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68102a70 0x7f2e681980a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800e9f0 tx=0x7f2e5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.733+0000 7f2e5e7fc640 1 -- 172.21.15.28:0/3204344648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5800cd80 con 0x7f2e68102a70 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.733+0000 7f2e5e7fc640 1 -- 172.21.15.28:0/3204344648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e58005500 con 0x7f2e68102a70 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.733+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6819fa70 con 0x7f2e68102a70 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.733+0000 7f2e5e7fc640 1 -- 172.21.15.28:0/3204344648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e58010690 con 0x7f2e68102a70 2024-05-09T08:23:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.733+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6819ffc0 con 0x7f2e68102a70 2024-05-09T08:23:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.734+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e68108b40 con 0x7f2e68102a70 2024-05-09T08:23:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.735+0000 7f2e5e7fc640 1 -- 172.21.15.28:0/3204344648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2e580046c0 con 0x7f2e68102a70 2024-05-09T08:23:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.736+0000 7f2e5e7fc640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e540a2c10 0x7f2e540a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.736+0000 7f2e6d459640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e540a2c10 0x7f2e540a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.736+0000 7f2e5e7fc640 1 -- 172.21.15.28:0/3204344648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2e580cf0e0 con 0x7f2e68102a70 2024-05-09T08:23:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.737+0000 7f2e6d459640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e540a2c10 0x7f2e540a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2e681028d0 tx=0x7f2e64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:42.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.739+0000 7f2e5e7fc640 1 -- 172.21.15.28:0/3204344648 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2e58053740 con 0x7f2e68102a70 2024-05-09T08:23:42.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:42 smithi165 ceph-mon[34540]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:42.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:42.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:42.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:42.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:42.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:42.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:42 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:42.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:42 smithi028 ceph-mon[30119]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:42.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:42.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:42.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:42.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:42 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.966+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e6802d1c0 con 0x7f2e68102a70 2024-05-09T08:23:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.967+0000 7f2e5e7fc640 1 -- 172.21.15.28:0/3204344648 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7f2e5809c4e0 con 0x7f2e68102a70 2024-05-09T08:23:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:42.968 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T08:23:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-05-09T08:23:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e540a2c10 msgr2=0x7f2e540a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e540a2c10 0x7f2e540a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2e681028d0 tx=0x7f2e64005f90 comp rx=0 tx=0).stop 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68102a70 msgr2=0x7f2e681980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68102a70 0x7f2e681980a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800e9f0 tx=0x7f2e5800edf0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 shutdown_connections 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2e540a2c10 0x7f2e540a50d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e68103430 0x7f2e6819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68102a70 0x7f2e681980a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 --2- 172.21.15.28:0/3204344648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e68101880 0x7f2e68197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 >> 172.21.15.28:0/3204344648 conn(0x7f2e680fce10 msgr2=0x7f2e68105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 shutdown_connections 2024-05-09T08:23:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:42.969+0000 7f2e6f6bd640 1 -- 172.21.15.28:0/3204344648 wait complete. 2024-05-09T08:23:43.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:42 smithi184 ceph-mon[34509]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:43.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:43.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:43.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:43.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:23:43.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:23:43.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:42 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:44.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:43 smithi165 ceph-mon[34540]: Deploying daemon mds.b on smithi165 2024-05-09T08:23:44.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:43 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.28:6840/9520094,v1:172.21.15.28:6841/9520094] up:boot 2024-05-09T08:23:44.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:43 smithi165 ceph-mon[34540]: fsmap 11 up:standby 2024-05-09T08:23:44.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:43 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:23:44.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:43 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3204344648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:44.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:43 smithi184 ceph-mon[34509]: Deploying daemon mds.b on smithi165 2024-05-09T08:23:44.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:43 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.28:6840/9520094,v1:172.21.15.28:6841/9520094] up:boot 2024-05-09T08:23:44.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:43 smithi184 ceph-mon[34509]: fsmap 11 up:standby 2024-05-09T08:23:44.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:43 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:23:44.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:43 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3204344648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:44.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:43 smithi028 ceph-mon[30119]: Deploying daemon mds.b on smithi165 2024-05-09T08:23:44.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:43 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.28:6840/9520094,v1:172.21.15.28:6841/9520094] up:boot 2024-05-09T08:23:44.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:43 smithi028 ceph-mon[30119]: fsmap 11 up:standby 2024-05-09T08:23:44.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:43 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:23:44.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:43 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3204344648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:44.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:45.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:44 smithi028 ceph-mon[30119]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:45.089 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:44 smithi165 ceph-mon[34540]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:45.090 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:44 smithi165 systemd[1]: Starting Ceph mds.b for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:23:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.105+0000 7ff2cc815640 1 -- 172.21.15.28:0/3622819550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 msgr2=0x7ff2c8101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.105+0000 7ff2cc815640 1 --2- 172.21.15.28:0/3622819550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 0x7ff2c8101de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009900 tx=0x7ff2b002a410 comp rx=0 tx=0).stop 2024-05-09T08:23:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.105+0000 7ff2cc815640 1 -- 172.21.15.28:0/3622819550 shutdown_connections 2024-05-09T08:23:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.105+0000 7ff2cc815640 1 --2- 172.21.15.28:0/3622819550 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2c8102320 0x7ff2c8108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.105+0000 7ff2cc815640 1 --2- 172.21.15.28:0/3622819550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 0x7ff2c8101de0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.105+0000 7ff2cc815640 1 --2- 172.21.15.28:0/3622819550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c8100770 0x7ff2c8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.105+0000 7ff2cc815640 1 -- 172.21.15.28:0/3622819550 >> 172.21.15.28:0/3622819550 conn(0x7ff2c80fbce0 msgr2=0x7ff2c80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.106+0000 7ff2cc815640 1 -- 172.21.15.28:0/3622819550 shutdown_connections 2024-05-09T08:23:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.106+0000 7ff2cc815640 1 -- 172.21.15.28:0/3622819550 wait complete. 2024-05-09T08:23:45.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:44 smithi184 ceph-mon[34509]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:45.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.106+0000 7ff2cc815640 1 Processor -- start 2024-05-09T08:23:45.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.106+0000 7ff2cc815640 1 -- start start 2024-05-09T08:23:45.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.106+0000 7ff2cc815640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c8100770 0x7ff2c810b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.106+0000 7ff2cc815640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 0x7ff2c8109730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2cc815640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2c8102320 0x7ff2c8109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2cc815640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c810d570 con 0x7ff2c8101960 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2cc815640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c810d3f0 con 0x7ff2c8102320 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2cc815640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c810d6f0 con 0x7ff2c8100770 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 0x7ff2c8109730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 0x7ff2c8109730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55868/0 (socket says 172.21.15.28:55868) 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bffff640 1 -- 172.21.15.28:0/1657905196 learned_addr learned my addr 172.21.15.28:0/1657905196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2c659c640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c8100770 0x7ff2c810b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2c6d9d640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2c8102320 0x7ff2c8109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bffff640 1 -- 172.21.15.28:0/1657905196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c8100770 msgr2=0x7ff2c810b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bffff640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c8100770 0x7ff2c810b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bffff640 1 -- 172.21.15.28:0/1657905196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2c8102320 msgr2=0x7ff2c8109c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bffff640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2c8102320 0x7ff2c8109c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bffff640 1 -- 172.21.15.28:0/1657905196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c810a500 con 0x7ff2c8101960 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2c6d9d640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2c8102320 0x7ff2c8109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bffff640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 0x7ff2c8109730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0031440 tx=0x7ff2b0032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bf7fe640 1 -- 172.21.15.28:0/1657905196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b003f030 con 0x7ff2c8101960 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c81a1ca0 con 0x7ff2c8101960 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bf7fe640 1 -- 172.21.15.28:0/1657905196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2b0003ce0 con 0x7ff2c8101960 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.107+0000 7ff2bf7fe640 1 -- 172.21.15.28:0/1657905196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0035070 con 0x7ff2c8101960 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.108+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c81a2190 con 0x7ff2c8101960 2024-05-09T08:23:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.109+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2c8107a30 con 0x7ff2c8101960 2024-05-09T08:23:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.110+0000 7ff2bf7fe640 1 -- 172.21.15.28:0/1657905196 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff2b0042030 con 0x7ff2c8101960 2024-05-09T08:23:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.110+0000 7ff2bf7fe640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff2b40a2b90 0x7ff2b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.111+0000 7ff2c659c640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff2b40a2b90 0x7ff2b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.111+0000 7ff2bf7fe640 1 -- 172.21.15.28:0/1657905196 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff2b00eec90 con 0x7ff2c8101960 2024-05-09T08:23:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.111+0000 7ff2c659c640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff2b40a2b90 0x7ff2b40a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff2c81017c0 tx=0x7ff2b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.115+0000 7ff2bf7fe640 1 -- 172.21.15.28:0/1657905196 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff2b00bc090 con 0x7ff2c8101960 2024-05-09T08:23:45.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.342+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2c802d390 con 0x7ff2c8101960 2024-05-09T08:23:45.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.343+0000 7ff2bf7fe640 1 -- 172.21.15.28:0/1657905196 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7ff2b00bb990 con 0x7ff2c8101960 2024-05-09T08:23:45.345 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:45.345 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T08:23:45.345 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff2b40a2b90 msgr2=0x7ff2b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff2b40a2b90 0x7ff2b40a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff2c81017c0 tx=0x7ff2b8005f90 comp rx=0 tx=0).stop 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 msgr2=0x7ff2c8109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 0x7ff2c8109730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0031440 tx=0x7ff2b0032ce0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 shutdown_connections 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff2b40a2b90 0x7ff2b40a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2c8102320 0x7ff2c8109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c8101960 0x7ff2c8109730 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 --2- 172.21.15.28:0/1657905196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c8100770 0x7ff2c810b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 >> 172.21.15.28:0/1657905196 conn(0x7ff2c80fbce0 msgr2=0x7ff2c8104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 shutdown_connections 2024-05-09T08:23:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:45.345+0000 7ff2cc815640 1 -- 172.21.15.28:0/1657905196 wait complete. 2024-05-09T08:23:45.352 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:45 smithi165 podman[64600]: 2024-05-09 08:23:44.991970339 +0000 UTC m=+0.018134365 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:23:45.352 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:45 smithi165 podman[64600]: 2024-05-09 08:23:45.113130686 +0000 UTC m=+0.139294699 container create 55ee3e387e7a6563ba17ff278aac220fc807a1560a8ad123e3c1910f612cbe48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:23:45.739 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:45 smithi165 podman[64600]: 2024-05-09 08:23:45.370556175 +0000 UTC m=+0.396720192 container init 55ee3e387e7a6563ba17ff278aac220fc807a1560a8ad123e3c1910f612cbe48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:23:45.739 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:45 smithi165 podman[64600]: 2024-05-09 08:23:45.373298132 +0000 UTC m=+0.399462146 container start 55ee3e387e7a6563ba17ff278aac220fc807a1560a8ad123e3c1910f612cbe48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-09T08:23:45.740 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:45 smithi165 ceph-mds[64619]: -- 172.21.15.165:0/725148928 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f3dd60ba40 con 0x55f3dd5df000 2024-05-09T08:23:45.740 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:45 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b[64615]: starting mds.b at 2024-05-09T08:23:45.740 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:45 smithi165 bash[64600]: 55ee3e387e7a6563ba17ff278aac220fc807a1560a8ad123e3c1910f612cbe48 2024-05-09T08:23:45.740 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:45 smithi165 systemd[1]: Started Ceph mds.b for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:23:46.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:45 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1657905196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:46.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:45 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:45 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:45 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:45 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:45 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:45 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:46.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:45 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1657905196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:46.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:45 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:45 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:45 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:45 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:45 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:45 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:46.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:45 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1657905196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:46.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:45 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:45 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:45 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:45 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:45 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:46.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:45 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:46.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:46 smithi165 ceph-mon[34540]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:46.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:46 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] up:boot 2024-05-09T08:23:46.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:46 smithi165 ceph-mon[34540]: fsmap 12 up:standby 2024-05-09T08:23:46.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:46 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:23:46.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:46 smithi184 ceph-mon[34509]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:46.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:46 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] up:boot 2024-05-09T08:23:46.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:46 smithi184 ceph-mon[34509]: fsmap 12 up:standby 2024-05-09T08:23:46.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:46 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:23:46.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:46 smithi028 ceph-mon[30119]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:46.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:46 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] up:boot 2024-05-09T08:23:46.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:46 smithi028 ceph-mon[30119]: fsmap 12 up:standby 2024-05-09T08:23:46.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:46 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:23:47.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.476+0000 7faacec22640 1 -- 172.21.15.28:0/2813144282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac80ff8e0 msgr2=0x7faac80ffd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.476+0000 7faacec22640 1 --2- 172.21.15.28:0/2813144282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac80ff8e0 0x7faac80ffd60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faab8009ca0 tx=0x7faab802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.476+0000 7faacec22640 1 -- 172.21.15.28:0/2813144282 shutdown_connections 2024-05-09T08:23:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.476+0000 7faacec22640 1 --2- 172.21.15.28:0/2813144282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faac81002a0 0x7faac8109e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.477+0000 7faacec22640 1 --2- 172.21.15.28:0/2813144282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac80ff8e0 0x7faac80ffd60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.477+0000 7faacec22640 1 --2- 172.21.15.28:0/2813144282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faac8102e70 0x7faac8103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.477+0000 7faacec22640 1 -- 172.21.15.28:0/2813144282 >> 172.21.15.28:0/2813144282 conn(0x7faac80fb4a0 msgr2=0x7faac80fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.477+0000 7faacec22640 1 -- 172.21.15.28:0/2813144282 shutdown_connections 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.477+0000 7faacec22640 1 -- 172.21.15.28:0/2813144282 wait complete. 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.477+0000 7faacec22640 1 Processor -- start 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.477+0000 7faacec22640 1 -- start start 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacec22640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faac80ff8e0 0x7faac8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacec22640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faac81002a0 0x7faac81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacec22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac8102e70 0x7faac819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacec22640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faac810d570 con 0x7faac8102e70 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacec22640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faac810d3f0 con 0x7faac81002a0 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacec22640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faac810d6f0 con 0x7faac80ff8e0 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacd1bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac8102e70 0x7faac819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacd1bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac8102e70 0x7faac819f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55896/0 (socket says 172.21.15.28:55896) 2024-05-09T08:23:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacd1bf640 1 -- 172.21.15.28:0/2202614265 learned_addr learned my addr 172.21.15.28:0/2202614265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacc9be640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faac80ff8e0 0x7faac8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faac7fff640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faac81002a0 0x7faac81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacd1bf640 1 -- 172.21.15.28:0/2202614265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faac80ff8e0 msgr2=0x7faac8197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacd1bf640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faac80ff8e0 0x7faac8197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacd1bf640 1 -- 172.21.15.28:0/2202614265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faac81002a0 msgr2=0x7faac81980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacd1bf640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faac81002a0 0x7faac81980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacd1bf640 1 -- 172.21.15.28:0/2202614265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac819f7d0 con 0x7faac8102e70 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacd1bf640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac8102e70 0x7faac819f160 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faac000d950 tx=0x7faac000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faac5ffb640 1 -- 172.21.15.28:0/2202614265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0017030 con 0x7faac8102e70 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faac5ffb640 1 -- 172.21.15.28:0/2202614265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faac0004020 con 0x7faac8102e70 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac819fac0 con 0x7faac8102e70 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.478+0000 7faac5ffb640 1 -- 172.21.15.28:0/2202614265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0003c00 con 0x7faac8102e70 2024-05-09T08:23:47.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.479+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac819ffe0 con 0x7faac8102e70 2024-05-09T08:23:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.480+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac8108d30 con 0x7faac8102e70 2024-05-09T08:23:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.481+0000 7faac5ffb640 1 -- 172.21.15.28:0/2202614265 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7faac001a070 con 0x7faac8102e70 2024-05-09T08:23:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.481+0000 7faac5ffb640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faab40a2c10 0x7faab40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.482+0000 7faacc9be640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faab40a2c10 0x7faab40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.482+0000 7faac5ffb640 1 -- 172.21.15.28:0/2202614265 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faac00ce380 con 0x7faac8102e70 2024-05-09T08:23:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.482+0000 7faacc9be640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faab40a2c10 0x7faab40a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faab00058c0 tx=0x7faab000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.486+0000 7faac5ffb640 1 -- 172.21.15.28:0/2202614265 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faac0021070 con 0x7faac8102e70 2024-05-09T08:23:47.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.717+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faac802d1c0 con 0x7faac8102e70 2024-05-09T08:23:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.718+0000 7faac5ffb640 1 -- 172.21.15.28:0/2202614265 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7faac009b780 con 0x7faac8102e70 2024-05-09T08:23:47.719 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:47.719 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24512,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-09T08:23:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-05-09T08:23:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faab40a2c10 msgr2=0x7faab40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faab40a2c10 0x7faab40a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faab00058c0 tx=0x7faab000a2d0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac8102e70 msgr2=0x7faac819f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac8102e70 0x7faac819f160 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faac000d950 tx=0x7faac000de10 comp rx=0 tx=0).stop 2024-05-09T08:23:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 shutdown_connections 2024-05-09T08:23:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faab40a2c10 0x7faab40a50d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac8102e70 0x7faac819f160 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faac81002a0 0x7faac81980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 --2- 172.21.15.28:0/2202614265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faac80ff8e0 0x7faac8197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 >> 172.21.15.28:0/2202614265 conn(0x7faac80fb4a0 msgr2=0x7faac8105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 shutdown_connections 2024-05-09T08:23:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.720+0000 7faacec22640 1 -- 172.21.15.28:0/2202614265 wait complete. 2024-05-09T08:23:47.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:23:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.838+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/4287495276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38102320 msgr2=0x7f8a38108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.838+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/4287495276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38102320 0x7f8a38108bb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009ca0 tx=0x7f8a3002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.838+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/4287495276 shutdown_connections 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.838+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/4287495276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38102320 0x7f8a38108bb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.838+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/4287495276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a38101960 0x7f8a38101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.838+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/4287495276 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a38100770 0x7f8a38100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.838+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/4287495276 >> 172.21.15.28:0/4287495276 conn(0x7f8a380fbce0 msgr2=0x7f8a380fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.838+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/4287495276 shutdown_connections 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.838+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/4287495276 wait complete. 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3c8ee640 1 Processor -- start 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3c8ee640 1 -- start start 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3c8ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38100770 0x7f8a38197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3c8ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a38101960 0x7f8a381980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3c8ee640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a38102320 0x7f8a3819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3c8ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810c3f0 con 0x7f8a38100770 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3c8ee640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810c270 con 0x7f8a38102320 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3c8ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810c570 con 0x7f8a38101960 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38100770 0x7f8a38197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38100770 0x7f8a38197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55904/0 (socket says 172.21.15.28:55904) 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3659c640 1 -- 172.21.15.28:0/354006126 learned_addr learned my addr 172.21.15.28:0/354006126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a36d9d640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a38102320 0x7f8a3819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a35d9b640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a38101960 0x7f8a381980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3659c640 1 -- 172.21.15.28:0/354006126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a38101960 msgr2=0x7f8a381980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3659c640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a38101960 0x7f8a381980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3659c640 1 -- 172.21.15.28:0/354006126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a38102320 msgr2=0x7f8a3819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3659c640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a38102320 0x7f8a3819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a3659c640 1 -- 172.21.15.28:0/354006126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3819f780 con 0x7f8a38100770 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.839+0000 7f8a36d9d640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a38102320 0x7f8a3819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.840+0000 7f8a3659c640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38100770 0x7f8a38197b60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800e9c0 tx=0x7f8a2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.840+0000 7f8a277fe640 1 -- 172.21.15.28:0/354006126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2800cd80 con 0x7f8a38100770 2024-05-09T08:23:47.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.840+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3819fa70 con 0x7f8a38100770 2024-05-09T08:23:47.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.840+0000 7f8a277fe640 1 -- 172.21.15.28:0/354006126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a280035f0 con 0x7f8a38100770 2024-05-09T08:23:47.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.840+0000 7f8a277fe640 1 -- 172.21.15.28:0/354006126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28010690 con 0x7f8a38100770 2024-05-09T08:23:47.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.840+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a3819ff70 con 0x7f8a38100770 2024-05-09T08:23:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.841+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a381a0200 con 0x7f8a38100770 2024-05-09T08:23:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.842+0000 7f8a277fe640 1 -- 172.21.15.28:0/354006126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8a280037a0 con 0x7f8a38100770 2024-05-09T08:23:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.843+0000 7f8a277fe640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a200a2b90 0x7f8a200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.843+0000 7f8a277fe640 1 -- 172.21.15.28:0/354006126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8a280cefe0 con 0x7f8a38100770 2024-05-09T08:23:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.843+0000 7f8a35d9b640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a200a2b90 0x7f8a200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.843+0000 7f8a35d9b640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a200a2b90 0x7f8a200a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a38199070 tx=0x7f8a2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:47.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:47.850+0000 7f8a277fe640 1 -- 172.21.15.28:0/354006126 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8a2809c3e0 con 0x7f8a38100770 2024-05-09T08:23:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.040+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a381a08d0 con 0x7f8a38100770 2024-05-09T08:23:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.041+0000 7f8a277fe640 1 -- 172.21.15.28:0/354006126 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18468 (secure 0 0 0) 0x7f8a2809bce0 con 0x7f8a38100770 2024-05-09T08:23:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:48.042 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:01.846859+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:23:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a200a2b90 msgr2=0x7f8a200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a200a2b90 0x7f8a200a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a38199070 tx=0x7f8a2c005f90 comp rx=0 tx=0).stop 2024-05-09T08:23:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38100770 msgr2=0x7f8a38197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38100770 0x7f8a38197b60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800e9c0 tx=0x7f8a2800ee80 comp rx=0 tx=0).stop 2024-05-09T08:23:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 shutdown_connections 2024-05-09T08:23:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a200a2b90 0x7f8a200a5050 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a38102320 0x7f8a3819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a38101960 0x7f8a381980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 --2- 172.21.15.28:0/354006126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a38100770 0x7f8a38197b60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 >> 172.21.15.28:0/354006126 conn(0x7f8a380fbce0 msgr2=0x7f8a380689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 shutdown_connections 2024-05-09T08:23:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.043+0000 7f8a3c8ee640 1 -- 172.21.15.28:0/354006126 wait complete. 2024-05-09T08:23:48.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T08:23:48.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:23:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.165+0000 7f9bea2b3640 1 -- 172.21.15.28:0/2367140638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 msgr2=0x7f9be4102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.165+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/2367140638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 0x7f9be4102f00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009900 tx=0x7f9bd802a410 comp rx=0 tx=0).stop 2024-05-09T08:23:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.166+0000 7f9bea2b3640 1 -- 172.21.15.28:0/2367140638 shutdown_connections 2024-05-09T08:23:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.166+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/2367140638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9be4103440 0x7f9be4109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.166+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/2367140638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 0x7f9be4102f00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.166+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/2367140638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be4101890 0x7f9be4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.166+0000 7f9bea2b3640 1 -- 172.21.15.28:0/2367140638 >> 172.21.15.28:0/2367140638 conn(0x7f9be40fce00 msgr2=0x7f9be40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.166+0000 7f9bea2b3640 1 -- 172.21.15.28:0/2367140638 shutdown_connections 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.166+0000 7f9bea2b3640 1 -- 172.21.15.28:0/2367140638 wait complete. 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.166+0000 7f9bea2b3640 1 Processor -- start 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.166+0000 7f9bea2b3640 1 -- start start 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9bea2b3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9be4101890 0x7f9be4197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9bea2b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 0x7f9be41980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9bea2b3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be4103440 0x7f9be419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9bea2b3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9be410d510 con 0x7f9be4102a80 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9bea2b3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9be410d390 con 0x7f9be4101890 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9bea2b3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9be410d690 con 0x7f9be4103440 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 0x7f9be41980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 0x7f9be41980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55918/0 (socket says 172.21.15.28:55918) 2024-05-09T08:23:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be37fe640 1 -- 172.21.15.28:0/312003553 learned_addr learned my addr 172.21.15.28:0/312003553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be3fff640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9be4101890 0x7f9be4197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be8850640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be4103440 0x7f9be419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be37fe640 1 -- 172.21.15.28:0/312003553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be4103440 msgr2=0x7f9be419f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be37fe640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be4103440 0x7f9be419f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be37fe640 1 -- 172.21.15.28:0/312003553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9be4101890 msgr2=0x7f9be4197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be37fe640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9be4101890 0x7f9be4197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be37fe640 1 -- 172.21.15.28:0/312003553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be419f7d0 con 0x7f9be4102a80 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be8850640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be4103440 0x7f9be419f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be37fe640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 0x7f9be41980f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009230 tx=0x7f9bd8031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be17fa640 1 -- 172.21.15.28:0/312003553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd803f030 con 0x7f9be4102a80 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.167+0000 7f9be17fa640 1 -- 172.21.15.28:0/312003553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bd8003ad0 con 0x7f9be4102a80 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.168+0000 7f9be17fa640 1 -- 172.21.15.28:0/312003553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8035070 con 0x7f9be4102a80 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.168+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be419fa60 con 0x7f9be4102a80 2024-05-09T08:23:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.168+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be419ffa0 con 0x7f9be4102a80 2024-05-09T08:23:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.169+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9be4108bb0 con 0x7f9be4102a80 2024-05-09T08:23:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.170+0000 7f9be17fa640 1 -- 172.21.15.28:0/312003553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9bd8042030 con 0x7f9be4102a80 2024-05-09T08:23:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.171+0000 7f9be17fa640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9bcc0a2c10 0x7f9bcc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.171+0000 7f9be3fff640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9bcc0a2c10 0x7f9bcc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.171+0000 7f9be17fa640 1 -- 172.21.15.28:0/312003553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9bd80ef4b0 con 0x7f9be4102a80 2024-05-09T08:23:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.171+0000 7f9be3fff640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9bcc0a2c10 0x7f9bcc0a50d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9be41028e0 tx=0x7f9bd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.178+0000 7f9be17fa640 1 -- 172.21.15.28:0/312003553 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9bd80bc8b0 con 0x7f9be4102a80 2024-05-09T08:23:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.395+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9be402d350 con 0x7f9be4102a80 2024-05-09T08:23:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.396+0000 7f9be17fa640 1 -- 172.21.15.28:0/312003553 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f9bd80bc1b0 con 0x7f9be4102a80 2024-05-09T08:23:48.397 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:23:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9bcc0a2c10 msgr2=0x7f9bcc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9bcc0a2c10 0x7f9bcc0a50d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9be41028e0 tx=0x7f9bd4005f90 comp rx=0 tx=0).stop 2024-05-09T08:23:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 msgr2=0x7f9be41980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 0x7f9be41980f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009230 tx=0x7f9bd8031b60 comp rx=0 tx=0).stop 2024-05-09T08:23:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 shutdown_connections 2024-05-09T08:23:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f9bcc0a2c10 0x7f9bcc0a50d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9be4103440 0x7f9be419f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be4102a80 0x7f9be41980f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 --2- 172.21.15.28:0/312003553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9be4101890 0x7f9be4197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 >> 172.21.15.28:0/312003553 conn(0x7f9be40fce00 msgr2=0x7f9be4105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 shutdown_connections 2024-05-09T08:23:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.398+0000 7f9bea2b3640 1 -- 172.21.15.28:0/312003553 wait complete. 2024-05-09T08:23:48.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T08:23:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.514+0000 7fbbde534640 1 -- 172.21.15.28:0/3324969547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 msgr2=0x7fbbd8100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.514+0000 7fbbde534640 1 --2- 172.21.15.28:0/3324969547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 0x7fbbd8100bb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009900 tx=0x7fbbc802a410 comp rx=0 tx=0).stop 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.514+0000 7fbbde534640 1 -- 172.21.15.28:0/3324969547 shutdown_connections 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.514+0000 7fbbde534640 1 --2- 172.21.15.28:0/3324969547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbd8102340 0x7fbbd8108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.514+0000 7fbbde534640 1 --2- 172.21.15.28:0/3324969547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbd8101980 0x7fbbd8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.514+0000 7fbbde534640 1 --2- 172.21.15.28:0/3324969547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 0x7fbbd8100bb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.514+0000 7fbbde534640 1 -- 172.21.15.28:0/3324969547 >> 172.21.15.28:0/3324969547 conn(0x7fbbd80fbd20 msgr2=0x7fbbd80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.514+0000 7fbbde534640 1 -- 172.21.15.28:0/3324969547 shutdown_connections 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.514+0000 7fbbde534640 1 -- 172.21.15.28:0/3324969547 wait complete. 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbde534640 1 Processor -- start 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbde534640 1 -- start start 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbde534640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 0x7fbbd806aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbde534640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbd8101980 0x7fbbd819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbde534640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbd8102340 0x7fbbd819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbde534640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd810c400 con 0x7fbbd8100790 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbde534640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd810c280 con 0x7fbbd8102340 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbde534640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd810c580 con 0x7fbbd8101980 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 0x7fbbd806aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 0x7fbbd806aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55930/0 (socket says 172.21.15.28:55930) 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbd7fff640 1 -- 172.21.15.28:0/3923855043 learned_addr learned my addr 172.21.15.28:0/3923855043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbdcad1640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbd8102340 0x7fbbd819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbcffff640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbd8101980 0x7fbbd819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbd7fff640 1 -- 172.21.15.28:0/3923855043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbd8101980 msgr2=0x7fbbd819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.515+0000 7fbbd7fff640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbd8101980 0x7fbbd819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.516+0000 7fbbd7fff640 1 -- 172.21.15.28:0/3923855043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbd8102340 msgr2=0x7fbbd819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.516+0000 7fbbd7fff640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbd8102340 0x7fbbd819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.516+0000 7fbbd7fff640 1 -- 172.21.15.28:0/3923855043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd819f880 con 0x7fbbd8100790 2024-05-09T08:23:48.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.516+0000 7fbbd7fff640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 0x7fbbd806aa00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8031610 tx=0x7fbbc80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:48.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.516+0000 7fbbd5ffb640 1 -- 172.21.15.28:0/3923855043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc803f030 con 0x7fbbd8100790 2024-05-09T08:23:48.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.516+0000 7fbbd5ffb640 1 -- 172.21.15.28:0/3923855043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbc8003ce0 con 0x7fbbd8100790 2024-05-09T08:23:48.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.516+0000 7fbbd5ffb640 1 -- 172.21.15.28:0/3923855043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8035070 con 0x7fbbd8100790 2024-05-09T08:23:48.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.516+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd819fab0 con 0x7fbbd8100790 2024-05-09T08:23:48.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.516+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbd819ffa0 con 0x7fbbd8100790 2024-05-09T08:23:48.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.517+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbd8107b30 con 0x7fbbd8100790 2024-05-09T08:23:48.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.518+0000 7fbbd5ffb640 1 -- 172.21.15.28:0/3923855043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbbc8042030 con 0x7fbbd8100790 2024-05-09T08:23:48.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.519+0000 7fbbd5ffb640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbbc40a2b90 0x7fbbc40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.519+0000 7fbbcffff640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbbc40a2b90 0x7fbbc40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.519+0000 7fbbd5ffb640 1 -- 172.21.15.28:0/3923855043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbbc80ef130 con 0x7fbbd8100790 2024-05-09T08:23:48.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.519+0000 7fbbcffff640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbbc40a2b90 0x7fbbc40a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbbd806a770 tx=0x7fbbc000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.523+0000 7fbbd5ffb640 1 -- 172.21.15.28:0/3923855043 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbbc80bbe40 con 0x7fbbd8100790 2024-05-09T08:23:48.653 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2202614265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:48.654 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:48 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/354006126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:48.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:48.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:48.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:48.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2202614265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:48.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:48 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/354006126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:48.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:48.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:48.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:48.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:48.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2202614265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:48.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:48 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/354006126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.712+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbbd8062460 con 0x7fbbd8100790 2024-05-09T08:23:48.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.713+0000 7fbbd5ffb640 1 -- 172.21.15.28:0/3923855043 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fbbc80c0140 con 0x7fbbd8100790 2024-05-09T08:23:48.714 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:48.714 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-05-09T08:23:48.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:23:48.714 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-05-09T08:23:48.714 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-05-09T08:23:48.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:23:48.715 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-05-09T08:23:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbbc40a2b90 msgr2=0x7fbbc40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbbc40a2b90 0x7fbbc40a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbbd806a770 tx=0x7fbbc000b390 comp rx=0 tx=0).stop 2024-05-09T08:23:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 msgr2=0x7fbbd806aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 0x7fbbd806aa00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8031610 tx=0x7fbbc80043b0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 shutdown_connections 2024-05-09T08:23:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbbc40a2b90 0x7fbbc40a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbd8102340 0x7fbbd819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbbd8101980 0x7fbbd819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 --2- 172.21.15.28:0/3923855043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd8100790 0x7fbbd806aa00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 >> 172.21.15.28:0/3923855043 conn(0x7fbbd80fbd20 msgr2=0x7fbbd8104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 shutdown_connections 2024-05-09T08:23:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.715+0000 7fbbde534640 1 -- 172.21.15.28:0/3923855043 wait complete. 2024-05-09T08:23:48.727 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T08:23:48.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T08:23:48.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 -- 172.21.15.28:0/2171716928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4102a80 msgr2=0x7fb7e4102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2171716928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4102a80 0x7fb7e4102f00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009900 tx=0x7fb7d402a410 comp rx=0 tx=0).stop 2024-05-09T08:23:48.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 -- 172.21.15.28:0/2171716928 shutdown_connections 2024-05-09T08:23:48.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2171716928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4103440 0x7fb7e4109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2171716928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4102a80 0x7fb7e4102f00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2171716928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7e4101890 0x7fb7e4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 -- 172.21.15.28:0/2171716928 >> 172.21.15.28:0/2171716928 conn(0x7fb7e40fce00 msgr2=0x7fb7e40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 -- 172.21.15.28:0/2171716928 shutdown_connections 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 -- 172.21.15.28:0/2171716928 wait complete. 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 Processor -- start 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.837+0000 7fb7ec212640 1 -- start start 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ec212640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4101890 0x7fb7e4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ec212640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7e4102a80 0x7fb7e4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ec212640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4103440 0x7fb7e419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ec212640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410d540 con 0x7fb7e4101890 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ec212640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410d3c0 con 0x7fb7e4103440 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ec212640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410d6c0 con 0x7fb7e4102a80 2024-05-09T08:23:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e9fae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4101890 0x7fb7e4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e9fae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4101890 0x7fb7e4197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55948/0 (socket says 172.21.15.28:55948) 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e9fae640 1 -- 172.21.15.28:0/2534662611 learned_addr learned my addr 172.21.15.28:0/2534662611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e97ad640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7e4102a80 0x7fb7e4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ea7af640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4103440 0x7fb7e419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e9fae640 1 -- 172.21.15.28:0/2534662611 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7e4102a80 msgr2=0x7fb7e4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e9fae640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7e4102a80 0x7fb7e4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e9fae640 1 -- 172.21.15.28:0/2534662611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4103440 msgr2=0x7fb7e419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e9fae640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4103440 0x7fb7e419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e9fae640 1 -- 172.21.15.28:0/2534662611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e419f740 con 0x7fb7e4101890 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ea7af640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4103440 0x7fb7e419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7e9fae640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4101890 0x7fb7e4197b20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000d770 tx=0x7fb7e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7daffd640 1 -- 172.21.15.28:0/2534662611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0017070 con 0x7fb7e4101890 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e419fa30 con 0x7fb7e4101890 2024-05-09T08:23:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7daffd640 1 -- 172.21.15.28:0/2534662611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7e0009c30 con 0x7fb7e4101890 2024-05-09T08:23:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7daffd640 1 -- 172.21.15.28:0/2534662611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0003dd0 con 0x7fb7e4101890 2024-05-09T08:23:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.838+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e419ff50 con 0x7fb7e4101890 2024-05-09T08:23:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.839+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e4108b50 con 0x7fb7e4101890 2024-05-09T08:23:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.841+0000 7fb7daffd640 1 -- 172.21.15.28:0/2534662611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb7e0003fb0 con 0x7fb7e4101890 2024-05-09T08:23:48.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.841+0000 7fb7daffd640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb7d00a2b90 0x7fb7d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:48.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.841+0000 7fb7daffd640 1 -- 172.21.15.28:0/2534662611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb7e00ce5e0 con 0x7fb7e4101890 2024-05-09T08:23:48.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.841+0000 7fb7e97ad640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb7d00a2b90 0x7fb7d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:48.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.842+0000 7fb7e97ad640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb7d00a2b90 0x7fb7d00a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4006010 tx=0x7fb7d4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:48.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:48.846+0000 7fb7daffd640 1 -- 172.21.15.28:0/2534662611 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb7e009b9e0 con 0x7fb7e4101890 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.031+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb7e402d3d0 con 0x7fb7e4101890 2024-05-09T08:23:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.547+0000 7fb7daffd640 1 -- 172.21.15.28:0/2534662611 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fb7e0009de0 con 0x7fb7e4101890 2024-05-09T08:23:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-05-09T08:23:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb7d00a2b90 msgr2=0x7fb7d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb7d00a2b90 0x7fb7d00a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4006010 tx=0x7fb7d4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4101890 msgr2=0x7fb7e4197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4101890 0x7fb7e4197b20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000d770 tx=0x7fb7e000dc30 comp rx=0 tx=0).stop 2024-05-09T08:23:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 shutdown_connections 2024-05-09T08:23:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb7d00a2b90 0x7fb7d00a5050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4103440 0x7fb7e419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7e4102a80 0x7fb7e4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 --2- 172.21.15.28:0/2534662611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4101890 0x7fb7e4197b20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 >> 172.21.15.28:0/2534662611 conn(0x7fb7e40fce00 msgr2=0x7fb7e4105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 shutdown_connections 2024-05-09T08:23:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.550+0000 7fb7ec212640 1 -- 172.21.15.28:0/2534662611 wait complete. 2024-05-09T08:23:49.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-09T08:23:49.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.679+0000 7f61dc06d640 1 -- 172.21.15.28:0/3636254523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4102370 msgr2=0x7f61d4104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.679+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3636254523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4102370 0x7f61d4104760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f61c4009ca0 tx=0x7f61c402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.679+0000 7f61dc06d640 1 -- 172.21.15.28:0/3636254523 shutdown_connections 2024-05-09T08:23:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.679+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3636254523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d41075d0 0x7f61d41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.679+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3636254523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d4104ca0 0x7f61d4107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.679+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3636254523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4102370 0x7f61d4104760 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.679+0000 7f61dc06d640 1 -- 172.21.15.28:0/3636254523 >> 172.21.15.28:0/3636254523 conn(0x7f61d40fbd20 msgr2=0x7f61d40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.679+0000 7f61dc06d640 1 -- 172.21.15.28:0/3636254523 shutdown_connections 2024-05-09T08:23:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.679+0000 7f61dc06d640 1 -- 172.21.15.28:0/3636254523 wait complete. 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61dc06d640 1 Processor -- start 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61dc06d640 1 -- start start 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61dc06d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d4102370 0x7f61d4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61dc06d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4104ca0 0x7f61d4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61dc06d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d41075d0 0x7f61d419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61dc06d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61d410d500 con 0x7f61d4104ca0 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61dc06d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f61d410d380 con 0x7f61d41075d0 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61dc06d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f61d410d680 con 0x7f61d4102370 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9608640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4104ca0 0x7f61d4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9608640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4104ca0 0x7f61d4198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55958/0 (socket says 172.21.15.28:55958) 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9608640 1 -- 172.21.15.28:0/3985917239 learned_addr learned my addr 172.21.15.28:0/3985917239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61da60a640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d41075d0 0x7f61d419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9e09640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d4102370 0x7f61d4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9608640 1 -- 172.21.15.28:0/3985917239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d4102370 msgr2=0x7f61d4197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9608640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d4102370 0x7f61d4197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9608640 1 -- 172.21.15.28:0/3985917239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d41075d0 msgr2=0x7f61d419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9608640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d41075d0 0x7f61d419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9608640 1 -- 172.21.15.28:0/3985917239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d419f770 con 0x7f61d4104ca0 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d9608640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4104ca0 0x7f61d4198090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f61c800d770 tx=0x7f61c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d2ffd640 1 -- 172.21.15.28:0/3985917239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8017070 con 0x7f61d4104ca0 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d2ffd640 1 -- 172.21.15.28:0/3985917239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61c8009c30 con 0x7f61d4104ca0 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61d419fa60 con 0x7f61d4104ca0 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.680+0000 7f61d2ffd640 1 -- 172.21.15.28:0/3985917239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c800f620 con 0x7f61d4104ca0 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.681+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61d419ffb0 con 0x7f61d4104ca0 2024-05-09T08:23:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.681+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61d40fd810 con 0x7f61d4104ca0 2024-05-09T08:23:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.683+0000 7f61d2ffd640 1 -- 172.21.15.28:0/3985917239 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f61c801a070 con 0x7f61d4104ca0 2024-05-09T08:23:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.684+0000 7f61d2ffd640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f61bc0a2b90 0x7f61bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.684+0000 7f61d9e09640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f61bc0a2b90 0x7f61bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.684+0000 7f61d2ffd640 1 -- 172.21.15.28:0/3985917239 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f61c80ce720 con 0x7f61d4104ca0 2024-05-09T08:23:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.684+0000 7f61d9e09640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f61bc0a2b90 0x7f61bc0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f61c4006010 tx=0x7f61c4000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.688+0000 7f61d2ffd640 1 -- 172.21.15.28:0/3985917239 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f61c8009de0 con 0x7f61d4104ca0 2024-05-09T08:23:49.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:49 smithi165 ceph-mon[34540]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:49.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:49 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/312003553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:23:49.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:49 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3923855043' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T08:23:49.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:49 smithi184 ceph-mon[34509]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:49.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:49 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/312003553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:23:49.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:49 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3923855043' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T08:23:49.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:49.872+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f61d402d1c0 con 0x7f61d4104ca0 2024-05-09T08:23:49.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:49 smithi028 ceph-mon[30119]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:49.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:49 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/312003553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:23:49.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:49 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3923855043' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T08:23:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.549+0000 7f61d2ffd640 1 -- 172.21.15.28:0/3985917239 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f61c809b360 con 0x7f61d4104ca0 2024-05-09T08:23:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-05-09T08:23:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f61bc0a2b90 msgr2=0x7f61bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f61bc0a2b90 0x7f61bc0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f61c4006010 tx=0x7f61c4000f40 comp rx=0 tx=0).stop 2024-05-09T08:23:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4104ca0 msgr2=0x7f61d4198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4104ca0 0x7f61d4198090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f61c800d770 tx=0x7f61c800dc30 comp rx=0 tx=0).stop 2024-05-09T08:23:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 shutdown_connections 2024-05-09T08:23:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f61bc0a2b90 0x7f61bc0a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d41075d0 0x7f61d419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61d4104ca0 0x7f61d4198090 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 --2- 172.21.15.28:0/3985917239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d4102370 0x7f61d4197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 >> 172.21.15.28:0/3985917239 conn(0x7f61d40fbd20 msgr2=0x7f61d4102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.552+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 shutdown_connections 2024-05-09T08:23:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.553+0000 7f61dc06d640 1 -- 172.21.15.28:0/3985917239 wait complete. 2024-05-09T08:23:50.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T08:23:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.681+0000 7fb0ee590640 1 -- 172.21.15.28:0/2575420420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e8109730 msgr2=0x7fb0e810bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.681+0000 7fb0ee590640 1 --2- 172.21.15.28:0/2575420420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e8109730 0x7fb0e810bbc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009ca0 tx=0x7fb0e002a950 comp rx=0 tx=0).stop 2024-05-09T08:23:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.681+0000 7fb0ee590640 1 -- 172.21.15.28:0/2575420420 shutdown_connections 2024-05-09T08:23:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.681+0000 7fb0ee590640 1 --2- 172.21.15.28:0/2575420420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e8109730 0x7fb0e810bbc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.681+0000 7fb0ee590640 1 --2- 172.21.15.28:0/2575420420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0e8074210 0x7fb0e8074670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.681+0000 7fb0ee590640 1 --2- 172.21.15.28:0/2575420420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0e80757f0 0x7fb0e8073c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.681+0000 7fb0ee590640 1 -- 172.21.15.28:0/2575420420 >> 172.21.15.28:0/2575420420 conn(0x7fb0e80fce10 msgr2=0x7fb0e80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 -- 172.21.15.28:0/2575420420 shutdown_connections 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 -- 172.21.15.28:0/2575420420 wait complete. 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 Processor -- start 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 -- start start 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0e8074210 0x7fb0e8199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e80757f0 0x7fb0e819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0e8109730 0x7fb0e81a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810f760 con 0x7fb0e80757f0 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810f5e0 con 0x7fb0e8074210 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.682+0000 7fb0ee590640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810f8e0 con 0x7fb0e8109730 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e80757f0 0x7fb0e819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e80757f0 0x7fb0e819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55980/0 (socket says 172.21.15.28:55980) 2024-05-09T08:23:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e77fe640 1 -- 172.21.15.28:0/4241176768 learned_addr learned my addr 172.21.15.28:0/4241176768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0ecb2d640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0e8109730 0x7fb0e81a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e7fff640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0e8074210 0x7fb0e8199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e77fe640 1 -- 172.21.15.28:0/4241176768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0e8109730 msgr2=0x7fb0e81a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e77fe640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0e8109730 0x7fb0e81a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e77fe640 1 -- 172.21.15.28:0/4241176768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0e8074210 msgr2=0x7fb0e8199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e77fe640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0e8074210 0x7fb0e8199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e77fe640 1 -- 172.21.15.28:0/4241176768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e81a1980 con 0x7fb0e80757f0 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0ecb2d640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0e8109730 0x7fb0e81a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e77fe640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e80757f0 0x7fb0e819a2a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00d770 tx=0x7fb0dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e57fa640 1 -- 172.21.15.28:0/4241176768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc017070 con 0x7fb0e80757f0 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e57fa640 1 -- 172.21.15.28:0/4241176768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0dc005500 con 0x7fb0e80757f0 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e81a1c70 con 0x7fb0e80757f0 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0e57fa640 1 -- 172.21.15.28:0/4241176768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc004e40 con 0x7fb0e80757f0 2024-05-09T08:23:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.683+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e81a2170 con 0x7fb0e80757f0 2024-05-09T08:23:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.684+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e81084f0 con 0x7fb0e80757f0 2024-05-09T08:23:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.686+0000 7fb0e57fa640 1 -- 172.21.15.28:0/4241176768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb0dc009c30 con 0x7fb0e80757f0 2024-05-09T08:23:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.687+0000 7fb0e57fa640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0d00a2b90 0x7fb0d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.687+0000 7fb0e7fff640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0d00a2b90 0x7fb0d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.687+0000 7fb0e57fa640 1 -- 172.21.15.28:0/4241176768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7fb0dc0ceda0 con 0x7fb0e80757f0 2024-05-09T08:23:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.688+0000 7fb0e7fff640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0d00a2b90 0x7fb0d00a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009890 tx=0x7fb0d8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.691+0000 7fb0e57fa640 1 -- 172.21.15.28:0/4241176768 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb0dc09beb0 con 0x7fb0e80757f0 2024-05-09T08:23:50.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2534662611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T08:23:50.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2534662611' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: pgmap v282: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:50.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:50 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3985917239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T08:23:50.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2534662611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2534662611' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: pgmap v282: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:50.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:50 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3985917239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T08:23:50.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:50.916+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb0e802d1c0 con 0x7fb0e80757f0 2024-05-09T08:23:50.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2534662611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T08:23:50.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2534662611' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: pgmap v282: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:50.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:50 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3985917239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T08:23:51.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.629+0000 7fb0e57fa640 1 -- 172.21.15.28:0/4241176768 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fb0dc09b7b0 con 0x7fb0e80757f0 2024-05-09T08:23:51.631 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-05-09T08:23:51.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0d00a2b90 msgr2=0x7fb0d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:51.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0d00a2b90 0x7fb0d00a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009890 tx=0x7fb0d8008040 comp rx=0 tx=0).stop 2024-05-09T08:23:51.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e80757f0 msgr2=0x7fb0e819a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:51.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e80757f0 0x7fb0e819a2a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00d770 tx=0x7fb0dc00dc30 comp rx=0 tx=0).stop 2024-05-09T08:23:51.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 shutdown_connections 2024-05-09T08:23:51.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0d00a2b90 0x7fb0d00a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:51.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0e8109730 0x7fb0e81a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:51.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e80757f0 0x7fb0e819a2a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:51.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 --2- 172.21.15.28:0/4241176768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0e8074210 0x7fb0e8199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:51.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 >> 172.21.15.28:0/4241176768 conn(0x7fb0e80fce10 msgr2=0x7fb0e81056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:51.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 shutdown_connections 2024-05-09T08:23:51.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.632+0000 7fb0ee590640 1 -- 172.21.15.28:0/4241176768 wait complete. 2024-05-09T08:23:51.649 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-09T08:23:51.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.762+0000 7faa51f21640 1 -- 172.21.15.28:0/1604024273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c102a70 msgr2=0x7faa4c102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.762+0000 7faa51f21640 1 --2- 172.21.15.28:0/1604024273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c102a70 0x7faa4c102ef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faa40009900 tx=0x7faa4002a410 comp rx=0 tx=0).stop 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.762+0000 7faa51f21640 1 -- 172.21.15.28:0/1604024273 shutdown_connections 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.762+0000 7faa51f21640 1 --2- 172.21.15.28:0/1604024273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa4c103430 0x7faa4c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.762+0000 7faa51f21640 1 --2- 172.21.15.28:0/1604024273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c102a70 0x7faa4c102ef0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.762+0000 7faa51f21640 1 --2- 172.21.15.28:0/1604024273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa4c101880 0x7faa4c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.762+0000 7faa51f21640 1 -- 172.21.15.28:0/1604024273 >> 172.21.15.28:0/1604024273 conn(0x7faa4c0fce10 msgr2=0x7faa4c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.762+0000 7faa51f21640 1 -- 172.21.15.28:0/1604024273 shutdown_connections 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.762+0000 7faa51f21640 1 -- 172.21.15.28:0/1604024273 wait complete. 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa51f21640 1 Processor -- start 2024-05-09T08:23:51.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa51f21640 1 -- start start 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa51f21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c101880 0x7faa4c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa51f21640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa4c102a70 0x7faa4c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa51f21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa4c103430 0x7faa4c19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa51f21640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c10d4e0 con 0x7faa4c101880 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa51f21640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c10d360 con 0x7faa4c102a70 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa51f21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c10d660 con 0x7faa4c103430 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c101880 0x7faa4c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c101880 0x7faa4c197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55996/0 (socket says 172.21.15.28:55996) 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.763+0000 7faa4b7fe640 1 -- 172.21.15.28:0/65164164 learned_addr learned my addr 172.21.15.28:0/65164164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa4bfff640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa4c103430 0x7faa4c19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa4affd640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa4c102a70 0x7faa4c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa4b7fe640 1 -- 172.21.15.28:0/65164164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa4c103430 msgr2=0x7faa4c19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa4b7fe640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa4c103430 0x7faa4c19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa4b7fe640 1 -- 172.21.15.28:0/65164164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa4c102a70 msgr2=0x7faa4c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa4b7fe640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa4c102a70 0x7faa4c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:51.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa4b7fe640 1 -- 172.21.15.28:0/65164164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c19f7a0 con 0x7faa4c101880 2024-05-09T08:23:51.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa4affd640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa4c102a70 0x7faa4c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:51.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa4b7fe640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c101880 0x7faa4c197b80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00c970 tx=0x7faa3c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:51.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa48ff9640 1 -- 172.21.15.28:0/65164164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c007c10 con 0x7faa4c101880 2024-05-09T08:23:51.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa48ff9640 1 -- 172.21.15.28:0/65164164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa3c007db0 con 0x7faa4c101880 2024-05-09T08:23:51.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c19fa90 con 0x7faa4c101880 2024-05-09T08:23:51.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa48ff9640 1 -- 172.21.15.28:0/65164164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c00f940 con 0x7faa4c101880 2024-05-09T08:23:51.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.764+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c1a0030 con 0x7faa4c101880 2024-05-09T08:23:51.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.765+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c108ba0 con 0x7faa4c101880 2024-05-09T08:23:51.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.766+0000 7faa48ff9640 1 -- 172.21.15.28:0/65164164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7faa3c00fae0 con 0x7faa4c101880 2024-05-09T08:23:51.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.767+0000 7faa48ff9640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa340a2c10 0x7faa340a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:51.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.767+0000 7faa4affd640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa340a2c10 0x7faa340a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:51.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.767+0000 7faa48ff9640 1 -- 172.21.15.28:0/65164164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7faa3c0cf370 con 0x7faa4c101880 2024-05-09T08:23:51.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.768+0000 7faa4affd640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa340a2c10 0x7faa340a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faa40000c00 tx=0x7faa40031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:51.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.772+0000 7faa48ff9640 1 -- 172.21.15.28:0/65164164 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faa3c09c410 con 0x7faa4c101880 2024-05-09T08:23:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:51 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3985917239' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T08:23:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:51 smithi165 ceph-mon[34540]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T08:23:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:51 smithi165 ceph-mon[34540]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:23:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:51 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4241176768' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T08:23:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:51 smithi165 ceph-mon[34540]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-09T08:23:51.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:51 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3985917239' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T08:23:51.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:51 smithi184 ceph-mon[34509]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T08:23:51.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:51 smithi184 ceph-mon[34509]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:23:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:51 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4241176768' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T08:23:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:51 smithi184 ceph-mon[34509]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-09T08:23:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:51 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3985917239' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T08:23:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:51 smithi028 ceph-mon[30119]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T08:23:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:51 smithi028 ceph-mon[30119]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:23:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:51 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4241176768' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T08:23:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:51 smithi028 ceph-mon[30119]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-09T08:23:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:51.970+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7faa4c02d1c0 con 0x7faa4c101880 2024-05-09T08:23:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.556+0000 7faa48ff9640 1 -- 172.21.15.28:0/65164164 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7faa3c09bd10 con 0x7faa4c101880 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa340a2c10 msgr2=0x7faa340a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa340a2c10 0x7faa340a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faa40000c00 tx=0x7faa40031400 comp rx=0 tx=0).stop 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c101880 msgr2=0x7faa4c197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c101880 0x7faa4c197b80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00c970 tx=0x7faa3c00ce30 comp rx=0 tx=0).stop 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 shutdown_connections 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7faa340a2c10 0x7faa340a50d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa4c103430 0x7faa4c19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa4c102a70 0x7faa4c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 --2- 172.21.15.28:0/65164164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4c101880 0x7faa4c197b80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.559+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 >> 172.21.15.28:0/65164164 conn(0x7faa4c0fce10 msgr2=0x7faa4c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.560+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 shutdown_connections 2024-05-09T08:23:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.560+0000 7faa51f21640 1 -- 172.21.15.28:0/65164164 wait complete. 2024-05-09T08:23:52.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T08:23:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 -- 172.21.15.28:0/1347771282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 msgr2=0x7f1830100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 --2- 172.21.15.28:0/1347771282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 0x7f1830100bb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1820009900 tx=0x7f182002a410 comp rx=0 tx=0).stop 2024-05-09T08:23:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 -- 172.21.15.28:0/1347771282 shutdown_connections 2024-05-09T08:23:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 --2- 172.21.15.28:0/1347771282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1830102340 0x7f1830108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 --2- 172.21.15.28:0/1347771282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1830101980 0x7f1830101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 --2- 172.21.15.28:0/1347771282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 0x7f1830100bb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 -- 172.21.15.28:0/1347771282 >> 172.21.15.28:0/1347771282 conn(0x7f18300fbd20 msgr2=0x7f18300fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 -- 172.21.15.28:0/1347771282 shutdown_connections 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 -- 172.21.15.28:0/1347771282 wait complete. 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 Processor -- start 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.691+0000 7f1835942640 1 -- start start 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1835942640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 0x7f1830197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1835942640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1830101980 0x7f1830198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1835942640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1830102340 0x7f183019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1835942640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f183010c460 con 0x7f1830100790 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1835942640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f183010c2e0 con 0x7f1830102340 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1835942640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f183010c5e0 con 0x7f1830101980 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f182effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 0x7f1830197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f182effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 0x7f1830197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56028/0 (socket says 172.21.15.28:56028) 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f182effd640 1 -- 172.21.15.28:0/6005181 learned_addr learned my addr 172.21.15.28:0/6005181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f182effd640 1 -- 172.21.15.28:0/6005181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1830101980 msgr2=0x7f1830198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f182effd640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1830101980 0x7f1830198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f182effd640 1 -- 172.21.15.28:0/6005181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1830102340 msgr2=0x7f183019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f182effd640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1830102340 0x7f183019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f182effd640 1 -- 172.21.15.28:0/6005181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183019f810 con 0x7f1830100790 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f182effd640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 0x7f1830197bf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1820031610 tx=0x7f18200043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1834940640 1 -- 172.21.15.28:0/6005181 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182003f030 con 0x7f1830100790 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1834940640 1 -- 172.21.15.28:0/6005181 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1820003ce0 con 0x7f1830100790 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183019faa0 con 0x7f1830100790 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1834940640 1 -- 172.21.15.28:0/6005181 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1820035070 con 0x7f1830100790 2024-05-09T08:23:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.692+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18301a0060 con 0x7f1830100790 2024-05-09T08:23:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.693+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18301a0290 con 0x7f1830100790 2024-05-09T08:23:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.695+0000 7f1834940640 1 -- 172.21.15.28:0/6005181 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1820042030 con 0x7f1830100790 2024-05-09T08:23:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.695+0000 7f1834940640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f181c0a2c10 0x7f181c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.695+0000 7f1827fff640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f181c0a2c10 0x7f181c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.695+0000 7f1834940640 1 -- 172.21.15.28:0/6005181 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f18200ef900 con 0x7f1830100790 2024-05-09T08:23:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.696+0000 7f1827fff640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f181c0a2c10 0x7f181c0a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1830199100 tx=0x7f1818006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.699+0000 7f1834940640 1 -- 172.21.15.28:0/6005181 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f18200bc830 con 0x7f1830100790 2024-05-09T08:23:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:52 smithi165 ceph-mon[34540]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T08:23:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:52 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4241176768' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T08:23:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:52 smithi165 ceph-mon[34540]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-09T08:23:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:52 smithi165 ceph-mon[34540]: pgmap v285: 97 pgs: 26 creating+peering, 12 active+clean, 59 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:52 smithi165 ceph-mon[34540]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-09T08:23:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:52 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/65164164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T08:23:52.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:52 smithi184 ceph-mon[34509]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T08:23:52.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:52 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4241176768' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T08:23:52.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:52 smithi184 ceph-mon[34509]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-09T08:23:52.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:52 smithi184 ceph-mon[34509]: pgmap v285: 97 pgs: 26 creating+peering, 12 active+clean, 59 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:52.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:52 smithi184 ceph-mon[34509]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-09T08:23:52.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:52 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/65164164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T08:23:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:52.885+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f18301a0510 con 0x7f1830100790 2024-05-09T08:23:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:52 smithi028 ceph-mon[30119]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T08:23:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:52 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4241176768' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T08:23:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:52 smithi028 ceph-mon[30119]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-09T08:23:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:52 smithi028 ceph-mon[30119]: pgmap v285: 97 pgs: 26 creating+peering, 12 active+clean, 59 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:23:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:52 smithi028 ceph-mon[30119]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-09T08:23:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:52 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/65164164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T08:23:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:53 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/65164164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T08:23:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:53 smithi165 ceph-mon[34540]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T08:23:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:53 smithi165 ceph-mon[34540]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:23:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:53 smithi165 ceph-mon[34540]: Cluster is now healthy 2024-05-09T08:23:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:53 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] up:active 2024-05-09T08:23:53.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:53 smithi165 ceph-mon[34540]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:23:53.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:53 smithi165 ceph-mon[34540]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:23:53.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:53 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/6005181' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T08:23:53.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:53 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/65164164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T08:23:53.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:53 smithi184 ceph-mon[34509]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T08:23:53.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:53 smithi184 ceph-mon[34509]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:23:53.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:53 smithi184 ceph-mon[34509]: Cluster is now healthy 2024-05-09T08:23:53.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:53 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] up:active 2024-05-09T08:23:53.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:53 smithi184 ceph-mon[34509]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:23:53.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:53 smithi184 ceph-mon[34509]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:23:53.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:53 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/6005181' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T08:23:53.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:53 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/65164164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T08:23:53.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:53 smithi028 ceph-mon[30119]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T08:23:53.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:53 smithi028 ceph-mon[30119]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:23:53.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:53 smithi028 ceph-mon[30119]: Cluster is now healthy 2024-05-09T08:23:53.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:53 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] up:active 2024-05-09T08:23:53.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:53 smithi028 ceph-mon[30119]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:23:53.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:53 smithi028 ceph-mon[30119]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:23:53.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:53 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/6005181' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T08:23:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.643+0000 7f1834940640 1 -- 172.21.15.28:0/6005181 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f18200bc130 con 0x7f1830100790 2024-05-09T08:23:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' created 2024-05-09T08:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.647+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f181c0a2c10 msgr2=0x7f181c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.647+0000 7f1835942640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f181c0a2c10 0x7f181c0a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1830199100 tx=0x7f1818006c90 comp rx=0 tx=0).stop 2024-05-09T08:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.647+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 msgr2=0x7f1830197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.647+0000 7f1835942640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 0x7f1830197bf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1820031610 tx=0x7f18200043b0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.648+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 shutdown_connections 2024-05-09T08:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.648+0000 7f1835942640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f181c0a2c10 0x7f181c0a50d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.648+0000 7f1835942640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1830102340 0x7f183019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.648+0000 7f1835942640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1830101980 0x7f1830198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.648+0000 7f1835942640 1 --2- 172.21.15.28:0/6005181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1830100790 0x7f1830197bf0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.648+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 >> 172.21.15.28:0/6005181 conn(0x7f18300fbd20 msgr2=0x7f18300fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.648+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 shutdown_connections 2024-05-09T08:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.648+0000 7f1835942640 1 -- 172.21.15.28:0/6005181 wait complete. 2024-05-09T08:23:54.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-09T08:23:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.781+0000 7f6dfab42640 1 -- 172.21.15.28:0/2390015570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4101890 msgr2=0x7f6df4101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.781+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2390015570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4101890 0x7f6df4101cb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009900 tx=0x7f6de802a410 comp rx=0 tx=0).stop 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.781+0000 7f6dfab42640 1 -- 172.21.15.28:0/2390015570 shutdown_connections 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.781+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2390015570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df4103440 0x7f6df4109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.781+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2390015570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6df4102a80 0x7f6df4102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.781+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2390015570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4101890 0x7f6df4101cb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.781+0000 7f6dfab42640 1 -- 172.21.15.28:0/2390015570 >> 172.21.15.28:0/2390015570 conn(0x7f6df40fce00 msgr2=0x7f6df40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.781+0000 7f6dfab42640 1 -- 172.21.15.28:0/2390015570 shutdown_connections 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.781+0000 7f6dfab42640 1 -- 172.21.15.28:0/2390015570 wait complete. 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6dfab42640 1 Processor -- start 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6dfab42640 1 -- start start 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6dfab42640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6df4101890 0x7f6df4197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6dfab42640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df4102a80 0x7f6df4198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6dfab42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4103440 0x7f6df419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6dfab42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6df410d510 con 0x7f6df4103440 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6dfab42640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6df410d390 con 0x7f6df4101890 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6dfab42640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6df410d690 con 0x7f6df4102a80 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df90df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4103440 0x7f6df419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df90df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4103440 0x7f6df419f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42432/0 (socket says 172.21.15.28:42432) 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df90df640 1 -- 172.21.15.28:0/2812742616 learned_addr learned my addr 172.21.15.28:0/2812742616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df88de640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6df4101890 0x7f6df4197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df3fff640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df4102a80 0x7f6df4198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df90df640 1 -- 172.21.15.28:0/2812742616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df4102a80 msgr2=0x7f6df4198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df90df640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df4102a80 0x7f6df4198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df90df640 1 -- 172.21.15.28:0/2812742616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6df4101890 msgr2=0x7f6df4197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df90df640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6df4101890 0x7f6df4197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df90df640 1 -- 172.21.15.28:0/2812742616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df419f7f0 con 0x7f6df4103440 2024-05-09T08:23:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.782+0000 7f6df88de640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6df4101890 0x7f6df4197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.783+0000 7f6df90df640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4103440 0x7f6df419f180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00cad0 tx=0x7f6dec00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.783+0000 7f6df1ffb640 1 -- 172.21.15.28:0/2812742616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec00ae20 con 0x7f6df4103440 2024-05-09T08:23:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.783+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df419fae0 con 0x7f6df4103440 2024-05-09T08:23:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.783+0000 7f6df1ffb640 1 -- 172.21.15.28:0/2812742616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dec005020 con 0x7f6df4103440 2024-05-09T08:23:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.783+0000 7f6df1ffb640 1 -- 172.21.15.28:0/2812742616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec00f6b0 con 0x7f6df4103440 2024-05-09T08:23:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.783+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df41a0030 con 0x7f6df4103440 2024-05-09T08:23:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.784+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df4108c80 con 0x7f6df4103440 2024-05-09T08:23:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.785+0000 7f6df1ffb640 1 -- 172.21.15.28:0/2812742616 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6dec0051d0 con 0x7f6df4103440 2024-05-09T08:23:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.786+0000 7f6df1ffb640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6de40a2b90 0x7f6de40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.786+0000 7f6df1ffb640 1 -- 172.21.15.28:0/2812742616 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f6dec0cf640 con 0x7f6df4103440 2024-05-09T08:23:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.786+0000 7f6df88de640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6de40a2b90 0x7f6de40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.786+0000 7f6df88de640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6de40a2b90 0x7f6de40a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6de8006010 tx=0x7f6de8032dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.790+0000 7f6df1ffb640 1 -- 172.21.15.28:0/2812742616 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6dec014030 con 0x7f6df4103440 2024-05-09T08:23:54.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:54 smithi028 ceph-mon[30119]: pgmap v287: 97 pgs: 26 creating+peering, 64 active+clean, 7 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-05-09T08:23:54.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:54 smithi028 ceph-mon[30119]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T08:23:54.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:54.976+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f6df402d1c0 con 0x7f6df4103440 2024-05-09T08:23:55.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:54 smithi165 ceph-mon[34540]: pgmap v287: 97 pgs: 26 creating+peering, 64 active+clean, 7 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-05-09T08:23:55.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:54 smithi165 ceph-mon[34540]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T08:23:55.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:54 smithi184 ceph-mon[34509]: pgmap v287: 97 pgs: 26 creating+peering, 64 active+clean, 7 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-05-09T08:23:55.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:54 smithi184 ceph-mon[34509]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T08:23:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.648+0000 7f6df1ffb640 1 -- 172.21.15.28:0/2812742616 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f6dec09c380 con 0x7f6df4103440 2024-05-09T08:23:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 4 allow_ec_overwrites to true 2024-05-09T08:23:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6de40a2b90 msgr2=0x7f6de40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6de40a2b90 0x7f6de40a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6de8006010 tx=0x7f6de8032dd0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4103440 msgr2=0x7f6df419f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4103440 0x7f6df419f180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00cad0 tx=0x7f6dec00cf90 comp rx=0 tx=0).stop 2024-05-09T08:23:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 shutdown_connections 2024-05-09T08:23:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6de40a2b90 0x7f6de40a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6df4103440 0x7f6df419f180 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6df4102a80 0x7f6df4198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 --2- 172.21.15.28:0/2812742616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6df4101890 0x7f6df4197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 >> 172.21.15.28:0/2812742616 conn(0x7f6df40fce00 msgr2=0x7f6df4105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.652+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 shutdown_connections 2024-05-09T08:23:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.653+0000 7f6dfab42640 1 -- 172.21.15.28:0/2812742616 wait complete. 2024-05-09T08:23:55.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-09T08:23:55.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 -- 172.21.15.28:0/1959066208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078102320 msgr2=0x7f3078108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:55.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 --2- 172.21.15.28:0/1959066208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078102320 0x7f3078108bb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3070009ca0 tx=0x7f307002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 -- 172.21.15.28:0/1959066208 shutdown_connections 2024-05-09T08:23:55.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 --2- 172.21.15.28:0/1959066208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078102320 0x7f3078108bb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 --2- 172.21.15.28:0/1959066208 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078101960 0x7f3078101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 --2- 172.21.15.28:0/1959066208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3078100770 0x7f3078100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 -- 172.21.15.28:0/1959066208 >> 172.21.15.28:0/1959066208 conn(0x7f30780fbce0 msgr2=0x7f30780fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 -- 172.21.15.28:0/1959066208 shutdown_connections 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 -- 172.21.15.28:0/1959066208 wait complete. 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 Processor -- start 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.788+0000 7f307cd70640 1 -- start start 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307cd70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078100770 0x7f3078196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307cd70640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3078101960 0x7f3078197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307cd70640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102320 0x7f307819e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307cd70640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f307810c4e0 con 0x7f3078100770 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307cd70640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f307810c360 con 0x7f3078101960 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307cd70640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f307810c660 con 0x7f3078102320 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078100770 0x7f3078196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078100770 0x7f3078196be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42448/0 (socket says 172.21.15.28:42448) 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307659c640 1 -- 172.21.15.28:0/3691025050 learned_addr learned my addr 172.21.15.28:0/3691025050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f3076d9d640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102320 0x7f307819e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f3075d9b640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3078101960 0x7f3078197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307659c640 1 -- 172.21.15.28:0/3691025050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102320 msgr2=0x7f307819e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307659c640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102320 0x7f307819e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307659c640 1 -- 172.21.15.28:0/3691025050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3078101960 msgr2=0x7f3078197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307659c640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3078101960 0x7f3078197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307659c640 1 -- 172.21.15.28:0/3691025050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307819e800 con 0x7f3078100770 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f3076d9d640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102320 0x7f307819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307659c640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078100770 0x7f3078196be0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f306800b450 tx=0x7f306800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f30677fe640 1 -- 172.21.15.28:0/3691025050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068015020 con 0x7f3078100770 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f30677fe640 1 -- 172.21.15.28:0/3691025050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3068011d70 con 0x7f3078100770 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307819eaf0 con 0x7f3078100770 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f30677fe640 1 -- 172.21.15.28:0/3691025050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306800f340 con 0x7f3078100770 2024-05-09T08:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.789+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f307819f040 con 0x7f3078100770 2024-05-09T08:23:55.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.791+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3078107b60 con 0x7f3078100770 2024-05-09T08:23:55.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.792+0000 7f30677fe640 1 -- 172.21.15.28:0/3691025050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3068018020 con 0x7f3078100770 2024-05-09T08:23:55.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.793+0000 7f30677fe640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f30600a2b90 0x7f30600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.793+0000 7f3075d9b640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f30600a2b90 0x7f30600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.793+0000 7f30677fe640 1 -- 172.21.15.28:0/3691025050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f30680cf2f0 con 0x7f3078100770 2024-05-09T08:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.793+0000 7f3075d9b640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f30600a2b90 0x7f30600a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f30781980f0 tx=0x7f306c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:55.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:55.797+0000 7f30677fe640 1 -- 172.21.15.28:0/3691025050 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f306801f020 con 0x7f3078100770 2024-05-09T08:23:55.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:55 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/6005181' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T08:23:55.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:55 smithi028 ceph-mon[30119]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T08:23:55.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:55 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2812742616' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T08:23:56.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.025+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f307802d1c0 con 0x7f3078100770 2024-05-09T08:23:56.102 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:23:55 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b[64615]: 2024-05-09T08:23:55.742+0000 7fd5675ca700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T08:23:56.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:55 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/6005181' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T08:23:56.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:55 smithi165 ceph-mon[34540]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T08:23:56.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:55 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2812742616' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T08:23:56.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:55 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/6005181' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T08:23:56.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:55 smithi184 ceph-mon[34509]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T08:23:56.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:55 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2812742616' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T08:23:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.650+0000 7f30677fe640 1 -- 172.21.15.28:0/3691025050 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f306800d040 con 0x7f3078100770 2024-05-09T08:23:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:added data pool 4 to fsmap 2024-05-09T08:23:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.653+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f30600a2b90 msgr2=0x7f30600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.653+0000 7f307cd70640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f30600a2b90 0x7f30600a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f30781980f0 tx=0x7f306c008040 comp rx=0 tx=0).stop 2024-05-09T08:23:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.653+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078100770 msgr2=0x7f3078196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.653+0000 7f307cd70640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078100770 0x7f3078196be0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f306800b450 tx=0x7f306800b910 comp rx=0 tx=0).stop 2024-05-09T08:23:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.653+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 shutdown_connections 2024-05-09T08:23:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.653+0000 7f307cd70640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f30600a2b90 0x7f30600a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.653+0000 7f307cd70640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3078102320 0x7f307819e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.653+0000 7f307cd70640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3078101960 0x7f3078197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.653+0000 7f307cd70640 1 --2- 172.21.15.28:0/3691025050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3078100770 0x7f3078196be0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.654+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 >> 172.21.15.28:0/3691025050 conn(0x7f30780fbce0 msgr2=0x7f3078104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.654+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 shutdown_connections 2024-05-09T08:23:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.654+0000 7f307cd70640 1 -- 172.21.15.28:0/3691025050 wait complete. 2024-05-09T08:23:56.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 -- 172.21.15.28:0/4267288707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600102340 msgr2=0x7fd600108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 --2- 172.21.15.28:0/4267288707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600102340 0x7fd600108bd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009ca0 tx=0x7fd5f802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 -- 172.21.15.28:0/4267288707 shutdown_connections 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 --2- 172.21.15.28:0/4267288707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600102340 0x7fd600108bd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 --2- 172.21.15.28:0/4267288707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd600101980 0x7fd600101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 --2- 172.21.15.28:0/4267288707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd600100790 0x7fd600100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 -- 172.21.15.28:0/4267288707 >> 172.21.15.28:0/4267288707 conn(0x7fd6000fbd20 msgr2=0x7fd6000fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 -- 172.21.15.28:0/4267288707 shutdown_connections 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 -- 172.21.15.28:0/4267288707 wait complete. 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 Processor -- start 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.781+0000 7fd605810640 1 -- start start 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd605810640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd600100790 0x7fd60006aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd605810640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600101980 0x7fd60019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd605810640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd600102340 0x7fd60019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd605810640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd60010c410 con 0x7fd600101980 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd605810640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd60010c290 con 0x7fd600102340 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd605810640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd60010c590 con 0x7fd600100790 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600101980 0x7fd60019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600101980 0x7fd60019c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42468/0 (socket says 172.21.15.28:42468) 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5f7fff640 1 -- 172.21.15.28:0/1154205181 learned_addr learned my addr 172.21.15.28:0/1154205181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5ff7fe640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd600102340 0x7fd60019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5feffd640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd600100790 0x7fd60006aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5f7fff640 1 -- 172.21.15.28:0/1154205181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd600100790 msgr2=0x7fd60006aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5f7fff640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd600100790 0x7fd60006aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5f7fff640 1 -- 172.21.15.28:0/1154205181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd600102340 msgr2=0x7fd60019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5f7fff640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd600102340 0x7fd60019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5f7fff640 1 -- 172.21.15.28:0/1154205181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60019f6e0 con 0x7fd600101980 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd5f7fff640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600101980 0x7fd60019c7b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800d770 tx=0x7fd5e800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd60480e640 1 -- 172.21.15.28:0/1154205181 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8017070 con 0x7fd600101980 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60019f970 con 0x7fd600101980 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd60480e640 1 -- 172.21.15.28:0/1154205181 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5e8005500 con 0x7fd600101980 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.782+0000 7fd60480e640 1 -- 172.21.15.28:0/1154205181 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8004e00 con 0x7fd600101980 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.783+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60019fe70 con 0x7fd600101980 2024-05-09T08:23:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.783+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6001079e0 con 0x7fd600101980 2024-05-09T08:23:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.785+0000 7fd60480e640 1 -- 172.21.15.28:0/1154205181 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd5e8004fa0 con 0x7fd600101980 2024-05-09T08:23:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.785+0000 7fd60480e640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5ec0a2b90 0x7fd5ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.786+0000 7fd5feffd640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5ec0a2b90 0x7fd5ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.786+0000 7fd60480e640 1 -- 172.21.15.28:0/1154205181 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd5e80cecb0 con 0x7fd600101980 2024-05-09T08:23:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.786+0000 7fd5feffd640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5ec0a2b90 0x7fd5ec0a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd5f00058c0 tx=0x7fd5f0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:56.790+0000 7fd60480e640 1 -- 172.21.15.28:0/1154205181 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd5e809b9d0 con 0x7fd600101980 2024-05-09T08:23:56.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:56 smithi028 ceph-mon[30119]: pgmap v290: 129 pgs: 30 unknown, 20 creating+peering, 79 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-05-09T08:23:56.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:56 smithi028 ceph-mon[30119]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:23:56.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:56 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2812742616' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T08:23:56.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:56 smithi028 ceph-mon[30119]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T08:23:56.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:56 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3691025050' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T08:23:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.013+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd600062430 con 0x7fd600101980 2024-05-09T08:23:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.014+0000 7fd60480e640 1 -- 172.21.15.28:0/1154205181 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11797 (secure 0 0 0) 0x7fd5e809b2d0 con 0x7fd600101980 2024-05-09T08:23:57.016 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:57.016 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:23:56.027187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24512,"qdb_cluster":[24512]},"id":1}]} 2024-05-09T08:23:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-05-09T08:23:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5ec0a2b90 msgr2=0x7fd5ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5ec0a2b90 0x7fd5ec0a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd5f00058c0 tx=0x7fd5f0005850 comp rx=0 tx=0).stop 2024-05-09T08:23:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600101980 msgr2=0x7fd60019c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600101980 0x7fd60019c7b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800d770 tx=0x7fd5e800dc30 comp rx=0 tx=0).stop 2024-05-09T08:23:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 shutdown_connections 2024-05-09T08:23:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5ec0a2b90 0x7fd5ec0a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd600102340 0x7fd60019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd600101980 0x7fd60019c7b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 --2- 172.21.15.28:0/1154205181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd600100790 0x7fd60006aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 >> 172.21.15.28:0/1154205181 conn(0x7fd6000fbd20 msgr2=0x7fd600104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 shutdown_connections 2024-05-09T08:23:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.016+0000 7fd605810640 1 -- 172.21.15.28:0/1154205181 wait complete. 2024-05-09T08:23:57.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:23:57.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:56 smithi165 ceph-mon[34540]: pgmap v290: 129 pgs: 30 unknown, 20 creating+peering, 79 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-05-09T08:23:57.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:56 smithi165 ceph-mon[34540]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:23:57.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:56 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2812742616' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T08:23:57.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:56 smithi165 ceph-mon[34540]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T08:23:57.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:56 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3691025050' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T08:23:57.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:56 smithi184 ceph-mon[34509]: pgmap v290: 129 pgs: 30 unknown, 20 creating+peering, 79 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-05-09T08:23:57.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:56 smithi184 ceph-mon[34509]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:23:57.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:56 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2812742616' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T08:23:57.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:56 smithi184 ceph-mon[34509]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T08:23:57.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:56 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3691025050' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T08:23:57.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 -- 172.21.15.28:0/4203980923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f0102a90 msgr2=0x7f00f0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/4203980923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f0102a90 0x7f00f0102f10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009ca0 tx=0x7f00e002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 -- 172.21.15.28:0/4203980923 shutdown_connections 2024-05-09T08:23:57.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/4203980923 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00f0103450 0x7f00f0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/4203980923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f0102a90 0x7f00f0102f10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/4203980923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00f01018a0 0x7f00f0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 -- 172.21.15.28:0/4203980923 >> 172.21.15.28:0/4203980923 conn(0x7f00f00fce30 msgr2=0x7f00f00ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 -- 172.21.15.28:0/4203980923 shutdown_connections 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 -- 172.21.15.28:0/4203980923 wait complete. 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 Processor -- start 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 -- start start 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01018a0 0x7f00f0197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.133+0000 7f00f6ce1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00f0102a90 0x7f00f0198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f6ce1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00f0103450 0x7f00f019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f6ce1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00f010d520 con 0x7f00f01018a0 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f6ce1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f00f010d3a0 con 0x7f00f0103450 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f6ce1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f00f010d6a0 con 0x7f00f0102a90 2024-05-09T08:23:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f4a7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01018a0 0x7f00f0197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f4a7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01018a0 0x7f00f0197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42496/0 (socket says 172.21.15.28:42496) 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f4a7d640 1 -- 172.21.15.28:0/1544468579 learned_addr learned my addr 172.21.15.28:0/1544468579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00effff640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00f0102a90 0x7f00f0198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f527e640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00f0103450 0x7f00f019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f4a7d640 1 -- 172.21.15.28:0/1544468579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00f0102a90 msgr2=0x7f00f0198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f4a7d640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00f0102a90 0x7f00f0198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f4a7d640 1 -- 172.21.15.28:0/1544468579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00f0103450 msgr2=0x7f00f019f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f4a7d640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00f0103450 0x7f00f019f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f4a7d640 1 -- 172.21.15.28:0/1544468579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f019f7e0 con 0x7f00f01018a0 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f4a7d640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01018a0 0x7f00f0197bc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f00d800e9c0 tx=0x7f00d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00edffb640 1 -- 172.21.15.28:0/1544468579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d800cd80 con 0x7f00f01018a0 2024-05-09T08:23:57.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00edffb640 1 -- 172.21.15.28:0/1544468579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00d80035f0 con 0x7f00f01018a0 2024-05-09T08:23:57.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f019fad0 con 0x7f00f01018a0 2024-05-09T08:23:57.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.134+0000 7f00edffb640 1 -- 172.21.15.28:0/1544468579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8010690 con 0x7f00f01018a0 2024-05-09T08:23:57.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.135+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f019ffd0 con 0x7f00f01018a0 2024-05-09T08:23:57.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.136+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f0108bc0 con 0x7f00f01018a0 2024-05-09T08:23:57.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.137+0000 7f00edffb640 1 -- 172.21.15.28:0/1544468579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f00d80037a0 con 0x7f00f01018a0 2024-05-09T08:23:57.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.137+0000 7f00edffb640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f00dc0a2b90 0x7f00dc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.138+0000 7f00edffb640 1 -- 172.21.15.28:0/1544468579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f00d80cf5d0 con 0x7f00f01018a0 2024-05-09T08:23:57.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.138+0000 7f00effff640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f00dc0a2b90 0x7f00dc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.138+0000 7f00effff640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f00dc0a2b90 0x7f00dc0a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f00f01990d0 tx=0x7f00e0000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:57.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.142+0000 7f00edffb640 1 -- 172.21.15.28:0/1544468579 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f00d809c2f0 con 0x7f00f01018a0 2024-05-09T08:23:57.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.327+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00f0062460 con 0x7f00f01018a0 2024-05-09T08:23:57.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.329+0000 7f00edffb640 1 -- 172.21.15.28:0/1544468579 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7f00d809bbf0 con 0x7f00f01018a0 2024-05-09T08:23:57.330 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:57.331 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:23:57.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f00dc0a2b90 msgr2=0x7f00dc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f00dc0a2b90 0x7f00dc0a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f00f01990d0 tx=0x7f00e0000f40 comp rx=0 tx=0).stop 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01018a0 msgr2=0x7f00f0197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01018a0 0x7f00f0197bc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f00d800e9c0 tx=0x7f00d800ee80 comp rx=0 tx=0).stop 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 shutdown_connections 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f00dc0a2b90 0x7f00dc0a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00f0103450 0x7f00f019f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00f0102a90 0x7f00f0198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 --2- 172.21.15.28:0/1544468579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00f01018a0 0x7f00f0197bc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 >> 172.21.15.28:0/1544468579 conn(0x7f00f00fce30 msgr2=0x7f00f0105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 shutdown_connections 2024-05-09T08:23:57.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.331+0000 7f00f6ce1640 1 -- 172.21.15.28:0/1544468579 wait complete. 2024-05-09T08:23:57.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:23:57.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.448+0000 7f56fdad4640 1 -- 172.21.15.28:0/502416366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 msgr2=0x7f56f8107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.448+0000 7f56fdad4640 1 --2- 172.21.15.28:0/502416366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 0x7f56f8107030 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009900 tx=0x7f56ec02a410 comp rx=0 tx=0).stop 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.448+0000 7f56fdad4640 1 -- 172.21.15.28:0/502416366 shutdown_connections 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.448+0000 7f56fdad4640 1 --2- 172.21.15.28:0/502416366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56f8107570 0x7f56f81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.448+0000 7f56fdad4640 1 --2- 172.21.15.28:0/502416366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 0x7f56f8107030 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.448+0000 7f56fdad4640 1 --2- 172.21.15.28:0/502416366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56f8102310 0x7f56f8104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.448+0000 7f56fdad4640 1 -- 172.21.15.28:0/502416366 >> 172.21.15.28:0/502416366 conn(0x7f56f80fbd20 msgr2=0x7f56f80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.448+0000 7f56fdad4640 1 -- 172.21.15.28:0/502416366 shutdown_connections 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.448+0000 7f56fdad4640 1 -- 172.21.15.28:0/502416366 wait complete. 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56fdad4640 1 Processor -- start 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56fdad4640 1 -- start start 2024-05-09T08:23:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56fdad4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56f8102310 0x7f56f8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56fdad4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 0x7f56f81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56fdad4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56f8107570 0x7f56f819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56fdad4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f56f810d500 con 0x7f56f8104c40 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56fdad4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f56f810d380 con 0x7f56f8107570 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56fdad4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f56f810d680 con 0x7f56f8102310 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 0x7f56f81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 0x7f56f81980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42524/0 (socket says 172.21.15.28:42524) 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.449+0000 7f56f6ffd640 1 -- 172.21.15.28:0/2903145387 learned_addr learned my addr 172.21.15.28:0/2903145387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f77fe640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56f8102310 0x7f56f8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f6ffd640 1 -- 172.21.15.28:0/2903145387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56f8102310 msgr2=0x7f56f8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f6ffd640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56f8102310 0x7f56f8197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f6ffd640 1 -- 172.21.15.28:0/2903145387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56f8107570 msgr2=0x7f56f819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f6ffd640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56f8107570 0x7f56f819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f6ffd640 1 -- 172.21.15.28:0/2903145387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f819f780 con 0x7f56f8104c40 2024-05-09T08:23:57.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f6ffd640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 0x7f56f81980a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f56ec031610 tx=0x7f56ec0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:57.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f4ff9640 1 -- 172.21.15.28:0/2903145387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec03f030 con 0x7f56f8104c40 2024-05-09T08:23:57.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f4ff9640 1 -- 172.21.15.28:0/2903145387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56ec003ce0 con 0x7f56f8104c40 2024-05-09T08:23:57.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f819fa10 con 0x7f56f8104c40 2024-05-09T08:23:57.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56f4ff9640 1 -- 172.21.15.28:0/2903145387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec035070 con 0x7f56f8104c40 2024-05-09T08:23:57.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.450+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f819ff00 con 0x7f56f8104c40 2024-05-09T08:23:57.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.451+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f80fd810 con 0x7f56f8104c40 2024-05-09T08:23:57.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.452+0000 7f56f4ff9640 1 -- 172.21.15.28:0/2903145387 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f56ec042030 con 0x7f56f8104c40 2024-05-09T08:23:57.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.453+0000 7f56f4ff9640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56e00a2b90 0x7f56e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.453+0000 7f56f77fe640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56e00a2b90 0x7f56e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.453+0000 7f56f4ff9640 1 -- 172.21.15.28:0/2903145387 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f56ec0ef950 con 0x7f56f8104c40 2024-05-09T08:23:57.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.453+0000 7f56f77fe640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56e00a2b90 0x7f56e00a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f56e800ba40 tx=0x7f56e8005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:57.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.457+0000 7f56f4ff9640 1 -- 172.21.15.28:0/2903145387 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f56ec0bc610 con 0x7f56f8104c40 2024-05-09T08:23:57.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.645+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56f80624d0 con 0x7f56f8104c40 2024-05-09T08:23:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.646+0000 7f56f4ff9640 1 -- 172.21.15.28:0/2903145387 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7f56ec0bbf10 con 0x7f56f8104c40 2024-05-09T08:23:57.647 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:57.648 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:23:57.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56e00a2b90 msgr2=0x7f56e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56e00a2b90 0x7f56e00a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f56e800ba40 tx=0x7f56e8005cc0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 msgr2=0x7f56f81980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 0x7f56f81980a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f56ec031610 tx=0x7f56ec0043b0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 shutdown_connections 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56e00a2b90 0x7f56e00a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56f8107570 0x7f56f819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8104c40 0x7f56f81980a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 --2- 172.21.15.28:0/2903145387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56f8102310 0x7f56f8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 >> 172.21.15.28:0/2903145387 conn(0x7f56f80fbd20 msgr2=0x7f56f8102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 shutdown_connections 2024-05-09T08:23:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.648+0000 7f56fdad4640 1 -- 172.21.15.28:0/2903145387 wait complete. 2024-05-09T08:23:57.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:23:57.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 -- 172.21.15.28:0/4235856844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c100790 msgr2=0x7f5d9c100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 --2- 172.21.15.28:0/4235856844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c100790 0x7f5d9c100bb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5d98009900 tx=0x7f5d9802a410 comp rx=0 tx=0).stop 2024-05-09T08:23:57.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 -- 172.21.15.28:0/4235856844 shutdown_connections 2024-05-09T08:23:57.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 --2- 172.21.15.28:0/4235856844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c102340 0x7f5d9c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 --2- 172.21.15.28:0/4235856844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d9c101980 0x7f5d9c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 --2- 172.21.15.28:0/4235856844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c100790 0x7f5d9c100bb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 -- 172.21.15.28:0/4235856844 >> 172.21.15.28:0/4235856844 conn(0x7f5d9c0fbd20 msgr2=0x7f5d9c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 -- 172.21.15.28:0/4235856844 shutdown_connections 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 -- 172.21.15.28:0/4235856844 wait complete. 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 Processor -- start 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.781+0000 7f5da3aba640 1 -- start start 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da3aba640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d9c100790 0x7f5d9c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da3aba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c101980 0x7f5d9c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da3aba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c102340 0x7f5d9c19f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da3aba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c10d4f0 con 0x7f5d9c102340 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da3aba640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c10d370 con 0x7f5d9c100790 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da3aba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c10d670 con 0x7f5d9c101980 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da2057640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c102340 0x7f5d9c19f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da2057640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c102340 0x7f5d9c19f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42544/0 (socket says 172.21.15.28:42544) 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da2057640 1 -- 172.21.15.28:0/2849783921 learned_addr learned my addr 172.21.15.28:0/2849783921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da1856640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d9c100790 0x7f5d9c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da1055640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c101980 0x7f5d9c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da2057640 1 -- 172.21.15.28:0/2849783921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c101980 msgr2=0x7f5d9c198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da2057640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c101980 0x7f5d9c198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da2057640 1 -- 172.21.15.28:0/2849783921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d9c100790 msgr2=0x7f5d9c197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da2057640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d9c100790 0x7f5d9c197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da2057640 1 -- 172.21.15.28:0/2849783921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d9c19f7e0 con 0x7f5d9c102340 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da1856640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d9c100790 0x7f5d9c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da2057640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c102340 0x7f5d9c19f170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009fd0 tx=0x7f5d9400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5d8effd640 1 -- 172.21.15.28:0/2849783921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d94017070 con 0x7f5d9c102340 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5d8effd640 1 -- 172.21.15.28:0/2849783921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d94003e50 con 0x7f5d9c102340 2024-05-09T08:23:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d9c19fad0 con 0x7f5d9c102340 2024-05-09T08:23:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5d8effd640 1 -- 172.21.15.28:0/2849783921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d940106c0 con 0x7f5d9c102340 2024-05-09T08:23:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.782+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d9c1a0020 con 0x7f5d9c102340 2024-05-09T08:23:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.783+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d9c107bd0 con 0x7f5d9c102340 2024-05-09T08:23:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.784+0000 7f5d8effd640 1 -- 172.21.15.28:0/2849783921 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5d94003970 con 0x7f5d9c102340 2024-05-09T08:23:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.785+0000 7f5d8effd640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5d880a2c10 0x7f5d880a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.785+0000 7f5da1856640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5d880a2c10 0x7f5d880a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.785+0000 7f5d8effd640 1 -- 172.21.15.28:0/2849783921 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5d940cf040 con 0x7f5d9c102340 2024-05-09T08:23:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.786+0000 7f5da1856640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5d880a2c10 0x7f5d880a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5d98006010 tx=0x7f5d98032cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:57.789+0000 7f5d8effd640 1 -- 172.21.15.28:0/2849783921 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5d9409bd60 con 0x7f5d9c102340 2024-05-09T08:23:57.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:57 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3691025050' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T08:23:57.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:57 smithi028 ceph-mon[30119]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T08:23:57.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:57 smithi028 ceph-mon[30119]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:23:57.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:57 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1154205181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:57.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:57 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1544468579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:57.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:57 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2903145387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.012+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d9c0624d0 con 0x7f5d9c102340 2024-05-09T08:23:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.013+0000 7f5d8effd640 1 -- 172.21.15.28:0/2849783921 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11797 (secure 0 0 0) 0x7f5d9409b660 con 0x7f5d9c102340 2024-05-09T08:23:58.015 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:58.015 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:23:56.027187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24512,"qdb_cluster":[24512]},"id":1}]} 2024-05-09T08:23:58.015 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5d880a2c10 msgr2=0x7f5d880a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5d880a2c10 0x7f5d880a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5d98006010 tx=0x7f5d98032cf0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c102340 msgr2=0x7f5d9c19f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c102340 0x7f5d9c19f170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009fd0 tx=0x7f5d9400b260 comp rx=0 tx=0).stop 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 shutdown_connections 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5d880a2c10 0x7f5d880a50d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d9c102340 0x7f5d9c19f170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c101980 0x7f5d9c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 --2- 172.21.15.28:0/2849783921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d9c100790 0x7f5d9c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 >> 172.21.15.28:0/2849783921 conn(0x7f5d9c0fbd20 msgr2=0x7f5d9c104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 shutdown_connections 2024-05-09T08:23:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.015+0000 7f5da3aba640 1 -- 172.21.15.28:0/2849783921 wait complete. 2024-05-09T08:23:58.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:23:58.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:57 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3691025050' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T08:23:58.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:57 smithi165 ceph-mon[34540]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T08:23:58.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:57 smithi165 ceph-mon[34540]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:23:58.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:57 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1154205181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:58.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:57 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1544468579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:58.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:57 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2903145387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:58.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:57 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3691025050' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T08:23:58.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:57 smithi184 ceph-mon[34509]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T08:23:58.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:57 smithi184 ceph-mon[34509]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:23:58.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:57 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1154205181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:58.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:57 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1544468579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:58.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:57 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2903145387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:58.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 -- 172.21.15.28:0/402836547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 msgr2=0x7fafc01099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 --2- 172.21.15.28:0/402836547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 0x7fafc01099c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009ca0 tx=0x7fafb802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 -- 172.21.15.28:0/402836547 shutdown_connections 2024-05-09T08:23:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 --2- 172.21.15.28:0/402836547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 0x7fafc01099c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 --2- 172.21.15.28:0/402836547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafc0104c40 0x7fafc0107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 --2- 172.21.15.28:0/402836547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafc0102310 0x7fafc0104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 -- 172.21.15.28:0/402836547 >> 172.21.15.28:0/402836547 conn(0x7fafc00fbd20 msgr2=0x7fafc00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 -- 172.21.15.28:0/402836547 shutdown_connections 2024-05-09T08:23:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 -- 172.21.15.28:0/402836547 wait complete. 2024-05-09T08:23:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.133+0000 7fafc6110640 1 Processor -- start 2024-05-09T08:23:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafc6110640 1 -- start start 2024-05-09T08:23:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafc6110640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafc0102310 0x7fafc0101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafc6110640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafc0104c40 0x7fafc01002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafc6110640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 0x7fafc0100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafc6110640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafc010d530 con 0x7fafc0107570 2024-05-09T08:23:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafc6110640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fafc010d3b0 con 0x7fafc0102310 2024-05-09T08:23:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafc6110640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fafc010d6b0 con 0x7fafc0104c40 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 0x7fafc0100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 0x7fafc0100800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42568/0 (socket says 172.21.15.28:42568) 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbffff640 1 -- 172.21.15.28:0/2908203356 learned_addr learned my addr 172.21.15.28:0/2908203356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbf7fe640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafc0102310 0x7fafc0101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbeffd640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafc0104c40 0x7fafc01002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbffff640 1 -- 172.21.15.28:0/2908203356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafc0104c40 msgr2=0x7fafc01002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbffff640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafc0104c40 0x7fafc01002c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbffff640 1 -- 172.21.15.28:0/2908203356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafc0102310 msgr2=0x7fafc0101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbffff640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafc0102310 0x7fafc0101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbffff640 1 -- 172.21.15.28:0/2908203356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc0101090 con 0x7fafc0107570 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbeffd640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafc0104c40 0x7fafc01002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.134+0000 7fafbffff640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 0x7fafc0100800 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fafb8031ae0 tx=0x7fafb8031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.135+0000 7fafbcff9640 1 -- 172.21.15.28:0/2908203356 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8003c30 con 0x7fafc0107570 2024-05-09T08:23:58.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.135+0000 7fafbcff9640 1 -- 172.21.15.28:0/2908203356 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb8003dd0 con 0x7fafc0107570 2024-05-09T08:23:58.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.135+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc01a1c80 con 0x7fafc0107570 2024-05-09T08:23:58.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.135+0000 7fafbcff9640 1 -- 172.21.15.28:0/2908203356 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb802fcb0 con 0x7fafc0107570 2024-05-09T08:23:58.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.135+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc01a21d0 con 0x7fafc0107570 2024-05-09T08:23:58.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.136+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc00fd810 con 0x7fafc0107570 2024-05-09T08:23:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.137+0000 7fafbcff9640 1 -- 172.21.15.28:0/2908203356 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fafb8004050 con 0x7fafc0107570 2024-05-09T08:23:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.138+0000 7fafbcff9640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafa80a2c10 0x7fafa80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.138+0000 7fafbf7fe640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafa80a2c10 0x7fafa80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.138+0000 7fafbcff9640 1 -- 172.21.15.28:0/2908203356 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fafb80f07c0 con 0x7fafc0107570 2024-05-09T08:23:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.138+0000 7fafbf7fe640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafa80a2c10 0x7fafa80a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009b60 tx=0x7fafb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.142+0000 7fafbcff9640 1 -- 172.21.15.28:0/2908203356 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fafb80bd4e0 con 0x7fafc0107570 2024-05-09T08:23:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.333+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fafc002d340 con 0x7fafc0107570 2024-05-09T08:23:58.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.334+0000 7fafbcff9640 1 -- 172.21.15.28:0/2908203356 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7fafb80bcde0 con 0x7fafc0107570 2024-05-09T08:23:58.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:23:58.336 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:23:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafa80a2c10 msgr2=0x7fafa80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafa80a2c10 0x7fafa80a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009b60 tx=0x7fafb0005f90 comp rx=0 tx=0).stop 2024-05-09T08:23:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 msgr2=0x7fafc0100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 0x7fafc0100800 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fafb8031ae0 tx=0x7fafb8031ee0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 shutdown_connections 2024-05-09T08:23:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fafa80a2c10 0x7fafa80a50d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc0107570 0x7fafc0100800 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafc0104c40 0x7fafc01002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 --2- 172.21.15.28:0/2908203356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafc0102310 0x7fafc0101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.336+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 >> 172.21.15.28:0/2908203356 conn(0x7fafc00fbd20 msgr2=0x7fafc0102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.337+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 shutdown_connections 2024-05-09T08:23:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.337+0000 7fafc6110640 1 -- 172.21.15.28:0/2908203356 wait complete. 2024-05-09T08:23:58.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T08:23:50.918821+0000', 'modified': '2024-05-09T08:23:56.027187+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24512}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24512': {'gid': 24512, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.165:6839/1723030158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6838', 'nonce': 1723030158}, {'type': 'v1', 'addr': '172.21.15.165:6839', 'nonce': 1723030158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24512, 'qdb_cluster': [24512]} 2024-05-09T08:23:58.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T08:23:58.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T08:23:58.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.459+0000 7f95329c8640 1 -- 172.21.15.28:0/508745465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c102320 msgr2=0x7f952c108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.459+0000 7f95329c8640 1 --2- 172.21.15.28:0/508745465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c102320 0x7f952c108bb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9524009ca0 tx=0x7f952402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.459+0000 7f95329c8640 1 -- 172.21.15.28:0/508745465 shutdown_connections 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.459+0000 7f95329c8640 1 --2- 172.21.15.28:0/508745465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c102320 0x7f952c108bb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.459+0000 7f95329c8640 1 --2- 172.21.15.28:0/508745465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f952c101960 0x7f952c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.459+0000 7f95329c8640 1 --2- 172.21.15.28:0/508745465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f952c100770 0x7f952c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.459+0000 7f95329c8640 1 -- 172.21.15.28:0/508745465 >> 172.21.15.28:0/508745465 conn(0x7f952c0fbce0 msgr2=0x7f952c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 -- 172.21.15.28:0/508745465 shutdown_connections 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 -- 172.21.15.28:0/508745465 wait complete. 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 Processor -- start 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 -- start start 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c100770 0x7f952c196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f952c101960 0x7f952c196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f952c102320 0x7f952c19e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f952c10c3f0 con 0x7f952c100770 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f952c10c270 con 0x7f952c101960 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f95329c8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f952c10c570 con 0x7f952c102320 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f952bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c100770 0x7f952c196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f952bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c100770 0x7f952c196a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42582/0 (socket says 172.21.15.28:42582) 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.460+0000 7f952bfff640 1 -- 172.21.15.28:0/3093522738 learned_addr learned my addr 172.21.15.28:0/3093522738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f9530f65640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f952c102320 0x7f952c19e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f952bfff640 1 -- 172.21.15.28:0/3093522738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f952c102320 msgr2=0x7f952c19e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f952bfff640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f952c102320 0x7f952c19e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f952bfff640 1 -- 172.21.15.28:0/3093522738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f952c101960 msgr2=0x7f952c196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f9523fff640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f952c101960 0x7f952c196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f952bfff640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f952c101960 0x7f952c196fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f952bfff640 1 -- 172.21.15.28:0/3093522738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952c19e680 con 0x7f952c100770 2024-05-09T08:23:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f952bfff640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c100770 0x7f952c196a60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f951c00e9c0 tx=0x7f951c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f9529ffb640 1 -- 172.21.15.28:0/3093522738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c00cd80 con 0x7f952c100770 2024-05-09T08:23:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f9529ffb640 1 -- 172.21.15.28:0/3093522738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f951c0035f0 con 0x7f952c100770 2024-05-09T08:23:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f95329c8640 1 -- 172.21.15.28:0/3093522738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f952c19e970 con 0x7f952c100770 2024-05-09T08:23:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f9529ffb640 1 -- 172.21.15.28:0/3093522738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c010690 con 0x7f952c100770 2024-05-09T08:23:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.461+0000 7f95329c8640 1 -- 172.21.15.28:0/3093522738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f952c19eec0 con 0x7f952c100770 2024-05-09T08:23:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f9529ffb640 1 -- 172.21.15.28:0/3093522738 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f951c0037a0 con 0x7f952c100770 2024-05-09T08:23:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f95357b2740 1 Processor -- start 2024-05-09T08:23:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f95357b2740 1 -- start start 2024-05-09T08:23:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f95357b2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 0x55770f3b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f95357b2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3b94c0 0x55770f3bdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f95357b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3be630 0x55770f3c0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f95357b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55770f31a4d0 con 0x55770f3be630 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f95357b2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55770f2c23e0 con 0x55770f3b94c0 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f95357b2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55770f316490 con 0x55770f3b8ad0 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f9523fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3be630 0x55770f3c0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f9523fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3be630 0x55770f3c0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42592/0 (socket says 172.21.15.28:42592) 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f9523fff640 1 -- 172.21.15.28:0/1374763703 learned_addr learned my addr 172.21.15.28:0/1374763703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f9529ffb640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95180a2c10 0x7f95180a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f9530f65640 1 --2- 172.21.15.28:0/1374763703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 0x55770f3b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.464+0000 7f952bfff640 1 --2- 172.21.15.28:0/1374763703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3b94c0 0x55770f3bdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f9530f65640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95180a2c10 0x7f95180a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f9523fff640 1 -- 172.21.15.28:0/1374763703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 msgr2=0x55770f3b8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f9523fff640 1 --2- 172.21.15.28:0/1374763703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 0x55770f3b8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f9523fff640 1 -- 172.21.15.28:0/1374763703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3b94c0 msgr2=0x55770f3bdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f9523fff640 1 --2- 172.21.15.28:0/1374763703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3b94c0 0x55770f3bdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f9523fff640 1 -- 172.21.15.28:0/1374763703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55770f02e2d0 con 0x55770f3be630 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f9529ffb640 1 -- 172.21.15.28:0/3093522738 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f951c0cf6b0 con 0x7f952c100770 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f9523fff640 1 --2- 172.21.15.28:0/1374763703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3be630 0x55770f3c0ac0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9514009b60 tx=0x7f951400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a0aaf9b7f11b2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f9530f65640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95180a2c10 0x7f95180a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9524005cc0 tx=0x7f95240095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f95227fc640 1 -- 172.21.15.28:0/1374763703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514016070 con 0x55770f3be630 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f95227fc640 1 -- 172.21.15.28:0/1374763703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9514009e50 con 0x55770f3be630 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.465+0000 7f95227fc640 1 -- 172.21.15.28:0/1374763703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514011470 con 0x55770f3be630 2024-05-09T08:23:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 -- 172.21.15.28:0/1374763703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3be630 msgr2=0x55770f3c0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 --2- 172.21.15.28:0/1374763703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3be630 0x55770f3c0ac0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9514009b60 tx=0x7f951400e6c0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 -- 172.21.15.28:0/1374763703 shutdown_connections 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 --2- 172.21.15.28:0/1374763703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3be630 0x55770f3c0ac0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 --2- 172.21.15.28:0/1374763703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3b94c0 0x55770f3bdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 --2- 172.21.15.28:0/1374763703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 0x55770f3b8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 -- 172.21.15.28:0/1374763703 >> 172.21.15.28:0/1374763703 conn(0x55770f3b1f30 msgr2=0x55770f3b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 -- 172.21.15.28:0/1374763703 shutdown_connections 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 -- 172.21.15.28:0/1374763703 wait complete. 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.466+0000 7f95357b2740 1 -- start start 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f95357b2740 1 Processor -- start 2024-05-09T08:23:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f95357b2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 0x55770f3c2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f95357b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b94c0 0x55770f3c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f95357b2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f95357b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55770f3c15e0 con 0x55770f3b94c0 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f95357b2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55770f2d58e0 con 0x55770f3be630 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f95357b2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55770f3c1760 con 0x55770f3b8ad0 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f95357b2740 10 client.0 _ll_register_callbacks cb 0x55770f2a29c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9523fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b94c0 0x55770f3c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9523fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b94c0 0x55770f3c33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42606/0 (socket says 172.21.15.28:42606) 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9523fff640 1 -- 172.21.15.28:0/362661583 learned_addr learned my addr 172.21.15.28:0/362661583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f952bfff640 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 0x55770f3c2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9530f65640 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9523fff640 1 -- 172.21.15.28:0/362661583 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 msgr2=0x55770f3c2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9523fff640 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 0x55770f3c2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9523fff640 1 -- 172.21.15.28:0/362661583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 msgr2=0x55770f3ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9523fff640 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9523fff640 1 -- 172.21.15.28:0/362661583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55770f0f86b0 con 0x55770f3b94c0 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9530f65640 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f9523fff640 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b94c0 0x55770f3c33a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f951400ebc0 tx=0x7f9514005530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0cd0c3b888abe3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.467+0000 7f950b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f950b7fe640 1 -- 172.21.15.28:0/362661583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514028060 con 0x55770f3b94c0 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f95357b2740 1 -- 172.21.15.28:0/362661583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9514009e50 con 0x55770f3b94c0 2024-05-09T08:23:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f950b7fe640 1 -- 172.21.15.28:0/362661583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9514009c50 con 0x55770f3b94c0 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f950b7fe640 1 -- 172.21.15.28:0/362661583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f951402f050 con 0x55770f3b94c0 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f95357b2740 10 client.14748 fetch_fsmap learned FSMap version 17 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f95357b2740 1 -- 172.21.15.28:0/362661583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55770f3cd3c0 con 0x55770f3b94c0 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f950b7fe640 1 -- 172.21.15.28:0/362661583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9514009e50 con 0x55770f3b94c0 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f950b7fe640 1 -- 172.21.15.28:0/362661583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f951400f040 con 0x55770f3b94c0 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f95357b2740 10 client.14748 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f95357b2740 10 client.14748 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f95357b2740 20 client.14748 populate_metadata read hostname 'smithi028' 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f95357b2740 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55770f3cf6d0 0x55770f3d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f95357b2740 4 client.14748 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:23:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.468+0000 7f95357b2740 1 -- 172.21.15.28:0/362661583 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55770f327c00 con 0x55770f3cf6d0 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.469+0000 7f952bfff640 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55770f3cf6d0 0x55770f3d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.469+0000 7f950b7fe640 10 client.14748 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.469+0000 7f952bfff640 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55770f3cf6d0 0x55770f3d0b40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df470e98e705f8be server_cookie=63fdb0dab8d69d78 in_seq=0 out_seq=0 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f950b7fe640 1 -- 172.21.15.28:0/362661583 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55770f327c00 con 0x55770f3cf6d0 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f950b7fe640 10 client.14748 handle_command_reply: tid=0 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 client.14748 shutdown 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 -- 172.21.15.28:0/362661583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b94c0 msgr2=0x55770f3c33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b94c0 0x55770f3c33a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f951400ebc0 tx=0x7f9514005530 comp rx=0 tx=0).stop 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 -- 172.21.15.28:0/362661583 shutdown_connections 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55770f3cf6d0 0x55770f3d0b40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b94c0 0x55770f3c33a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f950b7fe640 0 client.14748 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:23:58.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 --2- 172.21.15.28:0/362661583 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b8ad0 0x55770f3c2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 -- 172.21.15.28:0/362661583 >> 172.21.15.28:0/362661583 conn(0x55770f3b1f30 msgr2=0x55770f3b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 -- 172.21.15.28:0/362661583 shutdown_connections 2024-05-09T08:23:58.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 1 -- 172.21.15.28:0/362661583 wait complete. 2024-05-09T08:23:58.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.470+0000 7f95357b2740 20 client.14748 trim_cache size 0 max 16384 2024-05-09T08:23:58.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.488+0000 7f95357b2740 1 Processor -- start 2024-05-09T08:23:58.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f95357b2740 1 -- start start 2024-05-09T08:23:58.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f95357b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3b9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f95357b2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 0x55770f3ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f95357b2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f95357b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55770f3cee80 con 0x55770f3b8ad0 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f95357b2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55770f3cf550 con 0x55770f3be630 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f95357b2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55770f3cfc20 con 0x55770f3b94c0 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f952bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3b9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f952bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3b9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42614/0 (socket says 172.21.15.28:42614) 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f952bfff640 1 -- 172.21.15.28:0/3729843780 learned_addr learned my addr 172.21.15.28:0/3729843780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f9530f65640 1 --2- 172.21.15.28:0/3729843780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f9523fff640 1 --2- 172.21.15.28:0/3729843780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 0x55770f3ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f952bfff640 1 -- 172.21.15.28:0/3729843780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 msgr2=0x55770f3ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f952bfff640 1 --2- 172.21.15.28:0/3729843780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 0x55770f3ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f952bfff640 1 -- 172.21.15.28:0/3729843780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 msgr2=0x55770f3b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:23:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f952bfff640 1 --2- 172.21.15.28:0/3729843780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f952bfff640 1 -- 172.21.15.28:0/3729843780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55770f3c3f90 con 0x55770f3b8ad0 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f952bfff640 1 --2- 172.21.15.28:0/3729843780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3b9cd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f951c0ccf70 tx=0x7f951c06ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28d77932118218b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f9521ffb640 1 -- 172.21.15.28:0/3729843780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c0a0080 con 0x55770f3b8ad0 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f9521ffb640 1 -- 172.21.15.28:0/3729843780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f951c0a4940 con 0x55770f3b8ad0 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.489+0000 7f9521ffb640 1 -- 172.21.15.28:0/3729843780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c098340 con 0x55770f3b8ad0 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.490+0000 7f95357b2740 1 -- 172.21.15.28:0/3729843780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 msgr2=0x55770f3b9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.490+0000 7f95357b2740 1 --2- 172.21.15.28:0/3729843780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3b9cd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f951c0ccf70 tx=0x7f951c06ac50 comp rx=0 tx=0).stop 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.490+0000 7f95357b2740 1 -- 172.21.15.28:0/3729843780 shutdown_connections 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.490+0000 7f95357b2740 1 --2- 172.21.15.28:0/3729843780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.490+0000 7f95357b2740 1 --2- 172.21.15.28:0/3729843780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 0x55770f3ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.490+0000 7f95357b2740 1 --2- 172.21.15.28:0/3729843780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3b9cd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.490+0000 7f95357b2740 1 -- 172.21.15.28:0/3729843780 >> 172.21.15.28:0/3729843780 conn(0x55770f3b1f30 msgr2=0x55770f3b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.490+0000 7f95357b2740 1 -- 172.21.15.28:0/3729843780 shutdown_connections 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.490+0000 7f95357b2740 1 -- 172.21.15.28:0/3729843780 wait complete. 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f95357b2740 1 -- start start 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f95357b2740 1 Processor -- start 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f95357b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f95357b2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 0x55770f3c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f95357b2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f95357b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55770f3d09c0 con 0x55770f3b8ad0 2024-05-09T08:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f95357b2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55770f3d02f0 con 0x55770f3be630 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f95357b2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55770f3d1090 con 0x55770f3b94c0 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f95357b2740 10 client.0 _ll_register_callbacks cb 0x7f95140099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f952bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f952bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42620/0 (socket says 172.21.15.28:42620) 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f952bfff640 1 -- 172.21.15.28:0/1694628864 learned_addr learned my addr 172.21.15.28:0/1694628864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f9530f65640 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f9523fff640 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 0x55770f3c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f952bfff640 1 -- 172.21.15.28:0/1694628864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 msgr2=0x55770f3c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f952bfff640 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 0x55770f3c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f952bfff640 1 -- 172.21.15.28:0/1694628864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 msgr2=0x55770f3c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:23:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f952bfff640 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f952bfff640 1 -- 172.21.15.28:0/1694628864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55770f3c9910 con 0x55770f3b8ad0 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f9530f65640 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.491+0000 7f9523fff640 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 0x55770f3c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f952bfff640 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3c85e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f951c0058c0 tx=0x7f951c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e863c11638ba8b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f950b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f950b7fe640 1 -- 172.21.15.28:0/1694628864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c099d70 con 0x55770f3b8ad0 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f950b7fe640 1 -- 172.21.15.28:0/1694628864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f951c098340 con 0x55770f3b8ad0 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f95357b2740 1 -- 172.21.15.28:0/1694628864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f951c0a4940 con 0x55770f3b8ad0 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f950b7fe640 1 -- 172.21.15.28:0/1694628864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f951c0a0080 con 0x55770f3b8ad0 2024-05-09T08:23:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f95357b2740 10 client.14754 fetch_fsmap learned FSMap version 17 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f95357b2740 1 -- 172.21.15.28:0/1694628864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55770f435970 con 0x55770f3b8ad0 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f950b7fe640 1 -- 172.21.15.28:0/1694628864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f951c0a4940 con 0x55770f3b8ad0 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f950b7fe640 1 -- 172.21.15.28:0/1694628864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f951c0cfc80 con 0x55770f3b8ad0 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f95357b2740 10 client.14754 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f95357b2740 10 client.14754 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f95357b2740 20 client.14754 populate_metadata read hostname 'smithi028' 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f95357b2740 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55770f3cf6d0 0x55770f438d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f95357b2740 4 client.14754 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.492+0000 7f95357b2740 1 -- 172.21.15.28:0/1694628864 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "status"}) -- 0x55770f321f50 con 0x55770f3cf6d0 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.493+0000 7f9523fff640 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55770f3cf6d0 0x55770f438d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.493+0000 7f950b7fe640 10 client.14754 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.493+0000 7f9523fff640 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55770f3cf6d0 0x55770f438d10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f90608e3924ecdd5 server_cookie=4e6827b1f66ce67a in_seq=0 out_seq=0 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.493+0000 7f950b7fe640 1 -- 172.21.15.28:0/1694628864 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55770f321f50 con 0x55770f3cf6d0 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.493+0000 7f950b7fe640 10 client.14754 handle_command_reply: tid=0 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.493+0000 7f95357b2740 1 client.14754 shutdown 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.493+0000 7f95357b2740 1 -- 172.21.15.28:0/1694628864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 msgr2=0x55770f3c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.493+0000 7f95357b2740 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3c85e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f951c0058c0 tx=0x7f951c003f10 comp rx=0 tx=0).stop 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95357b2740 1 -- 172.21.15.28:0/1694628864 shutdown_connections 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95357b2740 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55770f3cf6d0 0x55770f438d10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95357b2740 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55770f3be630 0x55770f3c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95357b2740 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55770f3b94c0 0x55770f3c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f950b7fe640 0 client.14754 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:23:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95357b2740 1 --2- 172.21.15.28:0/1694628864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55770f3b8ad0 0x55770f3c85e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95357b2740 1 -- 172.21.15.28:0/1694628864 >> 172.21.15.28:0/1694628864 conn(0x55770f3b1f30 msgr2=0x55770f3b9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95357b2740 1 -- 172.21.15.28:0/1694628864 shutdown_connections 2024-05-09T08:23:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95357b2740 1 -- 172.21.15.28:0/1694628864 wait complete. 2024-05-09T08:23:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95357b2740 20 client.14754 trim_cache size 0 max 16384 2024-05-09T08:23:58.504 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:23:58.504 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24512, 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.8624178259999997, 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 89, 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 13.095468760999999 2024-05-09T08:23:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:23:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95329c8640 1 -- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95180a2c10 msgr2=0x7f95180a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95329c8640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95180a2c10 0x7f95180a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9524005cc0 tx=0x7f95240095c0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95329c8640 1 -- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c100770 msgr2=0x7f952c196a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.494+0000 7f95329c8640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c100770 0x7f952c196a60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f951c00e9c0 tx=0x7f951c00ee80 comp rx=0 tx=0).stop 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.495+0000 7f95329c8640 1 -- 172.21.15.28:0/3093522738 shutdown_connections 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.495+0000 7f95329c8640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95180a2c10 0x7f95180a50d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.495+0000 7f95329c8640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f952c102320 0x7f952c19e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.495+0000 7f95329c8640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f952c101960 0x7f952c196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.495+0000 7f95329c8640 1 --2- 172.21.15.28:0/3093522738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f952c100770 0x7f952c196a60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.495+0000 7f95329c8640 1 -- 172.21.15.28:0/3093522738 >> 172.21.15.28:0/3093522738 conn(0x7f952c0fbce0 msgr2=0x7f952c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.495+0000 7f95329c8640 1 -- 172.21.15.28:0/3093522738 shutdown_connections 2024-05-09T08:23:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:23:58.495+0000 7f95329c8640 1 -- 172.21.15.28:0/3093522738 wait complete. 2024-05-09T08:23:58.511 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T08:23:58.511 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:23:58.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-09T08:23:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:23:58.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:58 smithi028 ceph-mon[30119]: pgmap v293: 129 pgs: 13 unknown, 1 creating+activating, 2 creating+peering, 113 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 3 op/s 2024-05-09T08:23:58.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:58 smithi028 ceph-mon[30119]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:23:58.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:58 smithi028 ceph-mon[30119]: Cluster is now healthy 2024-05-09T08:23:58.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:58 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2849783921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:58.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:23:58 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2908203356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:59.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:58 smithi165 ceph-mon[34540]: pgmap v293: 129 pgs: 13 unknown, 1 creating+activating, 2 creating+peering, 113 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 3 op/s 2024-05-09T08:23:59.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:58 smithi165 ceph-mon[34540]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:23:59.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:58 smithi165 ceph-mon[34540]: Cluster is now healthy 2024-05-09T08:23:59.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:58 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2849783921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:59.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:23:58 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2908203356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:23:59.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:58 smithi184 ceph-mon[34509]: pgmap v293: 129 pgs: 13 unknown, 1 creating+activating, 2 creating+peering, 113 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 3 op/s 2024-05-09T08:23:59.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:58 smithi184 ceph-mon[34509]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:23:59.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:58 smithi184 ceph-mon[34509]: Cluster is now healthy 2024-05-09T08:23:59.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:58 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2849783921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:23:59.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:23:58 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2908203356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.506+0000 7fe44e117700 1 -- 172.21.15.28:0/4113506689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 msgr2=0x7fe448105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.506+0000 7fe44e117700 1 --2- 172.21.15.28:0/4113506689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 0x7fe448105ea0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0099c0 tx=0x7fe43c009cd0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.506+0000 7fe44e117700 1 -- 172.21.15.28:0/4113506689 shutdown_connections 2024-05-09T08:24:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.506+0000 7fe44e117700 1 --2- 172.21.15.28:0/4113506689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe448106580 0x7fe44810e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.506+0000 7fe44e117700 1 --2- 172.21.15.28:0/4113506689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4481014a0 0x7fe448101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.506+0000 7fe44e117700 1 --2- 172.21.15.28:0/4113506689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 0x7fe448105ea0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.506+0000 7fe44e117700 1 -- 172.21.15.28:0/4113506689 >> 172.21.15.28:0/4113506689 conn(0x7fe4480782c0 msgr2=0x7fe4480786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.506+0000 7fe44e117700 1 -- 172.21.15.28:0/4113506689 shutdown_connections 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.506+0000 7fe44e117700 1 -- 172.21.15.28:0/4113506689 wait complete. 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.507+0000 7fe44e117700 1 Processor -- start 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.507+0000 7fe44e117700 1 -- start start 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe44e117700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4481014a0 0x7fe4481998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe44e117700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 0x7fe448199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe44e117700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe448106580 0x7fe4481a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe44e117700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe448111540 con 0x7fe448101e50 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe44e117700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe4481113c0 con 0x7fe4481014a0 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe44e117700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4481116c0 con 0x7fe448106580 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe446ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 0x7fe448199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe446ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 0x7fe448199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42642/0 (socket says 172.21.15.28:42642) 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe446ffd700 1 -- 172.21.15.28:0/1201812198 learned_addr learned my addr 172.21.15.28:0/1201812198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe4477fe700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4481014a0 0x7fe4481998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe447fff700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe448106580 0x7fe4481a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe446ffd700 1 -- 172.21.15.28:0/1201812198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe448106580 msgr2=0x7fe4481a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe446ffd700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe448106580 0x7fe4481a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe446ffd700 1 -- 172.21.15.28:0/1201812198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4481014a0 msgr2=0x7fe4481998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe446ffd700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4481014a0 0x7fe4481998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe446ffd700 1 -- 172.21.15.28:0/1201812198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4481a4ea0 con 0x7fe448101e50 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe447fff700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe448106580 0x7fe4481a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe446ffd700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 0x7fe448199e20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00c010 tx=0x7fe43c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe444ff9700 1 -- 172.21.15.28:0/1201812198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c027070 con 0x7fe448101e50 2024-05-09T08:24:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4481a5070 con 0x7fe448101e50 2024-05-09T08:24:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe444ff9700 1 -- 172.21.15.28:0/1201812198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43c003730 con 0x7fe448101e50 2024-05-09T08:24:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.508+0000 7fe444ff9700 1 -- 172.21.15.28:0/1201812198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c022d40 con 0x7fe448101e50 2024-05-09T08:24:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.509+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4481a54d0 con 0x7fe448101e50 2024-05-09T08:24:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.509+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe448069010 con 0x7fe448101e50 2024-05-09T08:24:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.511+0000 7fe444ff9700 1 -- 172.21.15.28:0/1201812198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe43c02a070 con 0x7fe448101e50 2024-05-09T08:24:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.511+0000 7fe444ff9700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe4300a2c20 0x7fe4300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.511+0000 7fe4477fe700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe4300a2c20 0x7fe4300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.512+0000 7fe444ff9700 1 -- 172.21.15.28:0/1201812198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe43c0d0760 con 0x7fe448101e50 2024-05-09T08:24:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.512+0000 7fe4477fe700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe4300a2c20 0x7fe4300a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe438005950 tx=0x7fe43800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.515+0000 7fe444ff9700 1 -- 172.21.15.28:0/1201812198 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe43c031070 con 0x7fe448101e50 2024-05-09T08:24:00.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.772+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe44802db80 con 0x7fe448101e50 2024-05-09T08:24:00.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:00 smithi028 ceph-mon[30119]: pgmap v294: 129 pgs: 1 creating+activating, 2 creating+peering, 126 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1 KiB/s wr, 2 op/s 2024-05-09T08:24:00.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.773+0000 7fe444ff9700 1 -- 172.21.15.28:0/1201812198 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fe43c013070 con 0x7fe448101e50 2024-05-09T08:24:00.774 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T08:24:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe4300a2c20 msgr2=0x7fe4300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe4300a2c20 0x7fe4300a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe438005950 tx=0x7fe43800b410 comp rx=0 tx=0).stop 2024-05-09T08:24:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 msgr2=0x7fe448199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 0x7fe448199e20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00c010 tx=0x7fe43c0047b0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 shutdown_connections 2024-05-09T08:24:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe4300a2c20 0x7fe4300a50e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe448106580 0x7fe4481a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4481014a0 0x7fe4481998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 --2- 172.21.15.28:0/1201812198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe448101e50 0x7fe448199e20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 >> 172.21.15.28:0/1201812198 conn(0x7fe4480782c0 msgr2=0x7fe4480ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 shutdown_connections 2024-05-09T08:24:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:00.775+0000 7fe44e117700 1 -- 172.21.15.28:0/1201812198 wait complete. 2024-05-09T08:24:01.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:00 smithi165 ceph-mon[34540]: pgmap v294: 129 pgs: 1 creating+activating, 2 creating+peering, 126 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1 KiB/s wr, 2 op/s 2024-05-09T08:24:01.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:00 smithi184 ceph-mon[34509]: pgmap v294: 129 pgs: 1 creating+activating, 2 creating+peering, 126 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1 KiB/s wr, 2 op/s 2024-05-09T08:24:01.413 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T08:24:01.413 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T08:24:01.413 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-05-09T08:24:01.413 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-09T08:24:01.413 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:24:01.413 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T08:24:01.413 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T08:24:01.414 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T08:24:01.414 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T08:24:01.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:01.414 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-05-09T08:24:01.434 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T08:24:01.434 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-05-09T08:24:01.435 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-05-09T08:24:01.436 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T08:24:01.436 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42235sec preferred_lft 42235sec 2024-05-09T08:24:01.436 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-05-09T08:24:01.436 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:24:01.436 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:24:01.436 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-05-09T08:24:01.436 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-05-09T08:24:01.436 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T08:24:01.437 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:24:01.437 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:24:01.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-05-09T08:24:01.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-05-09T08:24:01.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-05-09T08:24:01.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T08:24:01.437 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:24:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:24:01.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:24:01.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:01.634 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T08:24:01.660 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-05-09T08:24:01.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:01.661 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-05-09T08:24:01.735 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-05-09T08:24:01.735 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T08:24:01.735 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T08:24:01.735 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T08:24:01.735 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T08:24:01.735 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:24:01.736 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:24:01.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T08:24:01.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T08:24:01.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T08:24:01.736 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:24:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:24:01.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:01 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1201812198' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:24:02.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:01 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1201812198' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:24:02.009 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:01 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1201812198' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:24:02.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:24:02.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:02.046 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-05-09T08:24:02.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:02.064 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-05-09T08:24:02.119 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:24:02.119 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:24:02.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:24:02.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-09T08:24:02.120 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:24:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:24:02.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:24:02.242 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-09T08:24:02.242 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:24:02.242 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:24:02.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-09T08:24:02.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T08:24:02.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-09T08:24:02.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-09T08:24:02.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-09T08:24:02.242 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:24:02.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:24:02.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:24:02.371 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:24:02.371 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:24:02.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-05-09T08:24:02.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T08:24:02.371 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:24:02.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:24:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:24:02.452 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-09T08:24:02.452 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:24:02.452 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:24:02.468 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T08:24:02.468 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:24:02.468 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:24:02.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-05-09T08:24:02.589 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T08:24:02.646 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:24:02.647 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/05b14529aa821067819b7b858702ed1cabda4f960539bbf8aef9bb31e506f93a/merged 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/9e29e043cb0f99851477bef0e8243fff06c0780f471adffdda849438320684ae/merged 2024-05-09T08:24:02.648 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/e85dfc3bf834ebb66efc0c712a04e99ce596d4bf22a90a4bfec01ec6d384689a/merged 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/b686dfedf8ba946ae6ee1dbd06370cb9c7234ae18e9f56e6daec8e4663c40556/merged 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/b90ab13cc012be3841d18882691d4c956c3cae5a3d826c9feb12aaad89bb74bb/merged 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/e0fa84bf13c97723fed68a6002b28740f58f75b17ede8c6fbe775d0466cc7f6a/merged 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/13122165aec782bb6f91f2bcc16b2288f927757583d37dd6a187c3948b3539fc/merged 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/d13b1d5a2e77a6a1c47a439b151ab4e682ba3cd951b63891f22e0c29c227fd03/merged 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/7b9cd4b08b919ec11955417b98a460d3256d0701b85b85e7a712210d7046a9a0/merged 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/6242d42840ec4f81eb664998050edd1802ab005e4edcf00509ee37b7d3f09a4e/merged 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:24:02.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:02.650 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-05-09T08:24:02.703 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T08:24:02.703 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-09T08:24:02.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-05-09T08:24:02.772 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-05-09T08:24:02.830 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T08:24:02.831 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/05b14529aa821067819b7b858702ed1cabda4f960539bbf8aef9bb31e506f93a/merged 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/9e29e043cb0f99851477bef0e8243fff06c0780f471adffdda849438320684ae/merged 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/e85dfc3bf834ebb66efc0c712a04e99ce596d4bf22a90a4bfec01ec6d384689a/merged 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/b686dfedf8ba946ae6ee1dbd06370cb9c7234ae18e9f56e6daec8e4663c40556/merged 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/b90ab13cc012be3841d18882691d4c956c3cae5a3d826c9feb12aaad89bb74bb/merged 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/e0fa84bf13c97723fed68a6002b28740f58f75b17ede8c6fbe775d0466cc7f6a/merged 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/13122165aec782bb6f91f2bcc16b2288f927757583d37dd6a187c3948b3539fc/merged 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/d13b1d5a2e77a6a1c47a439b151ab4e682ba3cd951b63891f22e0c29c227fd03/merged 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/7b9cd4b08b919ec11955417b98a460d3256d0701b85b85e7a712210d7046a9a0/merged 2024-05-09T08:24:02.832 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/6242d42840ec4f81eb664998050edd1802ab005e4edcf00509ee37b7d3f09a4e/merged 2024-05-09T08:24:02.833 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-05-09T08:24:02.833 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:24:02.833 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:24:02.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:02.833 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-05-09T08:24:02.858 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:2024-05-09T08:24:02.857+0000 7fa983359480 -1 init, newargv = 0x5643a0112350 newargc=15 2024-05-09T08:24:02.858 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[71236]: starting ceph client 2024-05-09T08:24:02.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[71236]: starting fuse 2024-05-09T08:24:02.874 INFO:teuthology.orchestra.run.smithi028.stdout:114 2024-05-09T08:24:02.875 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-05-09T08:24:02.875 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> 2024-05-09T08:24:02.876 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.admin")) 2024-05-09T08:24:02.877 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-05-09T08:24:02.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:02 smithi028 ceph-mon[30119]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 878 B/s wr, 2 op/s 2024-05-09T08:24:02.997 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.admin.71236.asok 2024-05-09T08:24:03.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:24:03.007 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.71236.asok 2024-05-09T08:24:03.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:03.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.71236.asok status 2024-05-09T08:24:03.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:02 smithi165 ceph-mon[34540]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 878 B/s wr, 2 op/s 2024-05-09T08:24:03.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:02 smithi184 ceph-mon[34509]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 878 B/s wr, 2 op/s 2024-05-09T08:24:03.147 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "96805dcf909541c4ca9a1e90258de8ae5b6fa1bc", 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "admin", 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "71236", 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:24:03.148 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 14766, 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 14766 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3869519441 2024-05-09T08:24:03.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3869519441 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.14766 192.168.0.1:0/3869519441", 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/3869519441", 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 17, 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 89, 2024-05-09T08:24:03.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-05-09T08:24:03.151 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-05-09T08:24:03.151 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-05-09T08:24:03.151 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:24:03.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:03.156 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-05-09T08:24:03.156 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:24:03.173 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-05-09T08:24:03.173 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:24:03.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:03.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:24:03.241 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-09T08:24:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T08:24:03.349 INFO:teuthology.orchestra.run.smithi028.stdout:# file: . 2024-05-09T08:24:03.350 INFO:teuthology.orchestra.run.smithi028.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-09T08:24:03.350 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:24:03.356 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:24:03.356 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:24:03.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:24:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 -- 172.21.15.28:0/2804224190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 msgr2=0x7f51e4100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 --2- 172.21.15.28:0/2804224190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 0x7f51e4100ba0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009900 tx=0x7f51d802a410 comp rx=0 tx=0).stop 2024-05-09T08:24:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 -- 172.21.15.28:0/2804224190 shutdown_connections 2024-05-09T08:24:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 --2- 172.21.15.28:0/2804224190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e4102330 0x7f51e4108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 --2- 172.21.15.28:0/2804224190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e4101970 0x7f51e4101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 --2- 172.21.15.28:0/2804224190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 0x7f51e4100ba0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 -- 172.21.15.28:0/2804224190 >> 172.21.15.28:0/2804224190 conn(0x7f51e40fbd10 msgr2=0x7f51e40fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 -- 172.21.15.28:0/2804224190 shutdown_connections 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 -- 172.21.15.28:0/2804224190 wait complete. 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.518+0000 7f51eaeee640 1 Processor -- start 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51eaeee640 1 -- start start 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51eaeee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 0x7f51e4197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51eaeee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e4101970 0x7f51e4198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51eaeee640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e4102330 0x7f51e419f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51eaeee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51e410c3f0 con 0x7f51e4100780 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51eaeee640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f51e410c270 con 0x7f51e4102330 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51eaeee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f51e410c570 con 0x7f51e4101970 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e8c8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 0x7f51e4197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e8c8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 0x7f51e4197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42658/0 (socket says 172.21.15.28:42658) 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e8c8a640 1 -- 172.21.15.28:0/3370765039 learned_addr learned my addr 172.21.15.28:0/3370765039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e3fff640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e4101970 0x7f51e4198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e948b640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e4102330 0x7f51e419f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e8c8a640 1 -- 172.21.15.28:0/3370765039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e4101970 msgr2=0x7f51e4198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e8c8a640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e4101970 0x7f51e4198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e8c8a640 1 -- 172.21.15.28:0/3370765039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e4102330 msgr2=0x7f51e419f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e8c8a640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e4102330 0x7f51e419f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e8c8a640 1 -- 172.21.15.28:0/3370765039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e419f810 con 0x7f51e4100780 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.519+0000 7f51e8c8a640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 0x7f51e4197bf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f51d8031610 tx=0x7f51d8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.520+0000 7f51e1ffb640 1 -- 172.21.15.28:0/3370765039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d803f030 con 0x7f51e4100780 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.520+0000 7f51e1ffb640 1 -- 172.21.15.28:0/3370765039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d8003ad0 con 0x7f51e4100780 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.520+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e419faa0 con 0x7f51e4100780 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.520+0000 7f51e1ffb640 1 -- 172.21.15.28:0/3370765039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8035070 con 0x7f51e4100780 2024-05-09T08:24:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.520+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e419ff90 con 0x7f51e4100780 2024-05-09T08:24:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.521+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e41a01c0 con 0x7f51e4100780 2024-05-09T08:24:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.522+0000 7f51e1ffb640 1 -- 172.21.15.28:0/3370765039 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f51d8042030 con 0x7f51e4100780 2024-05-09T08:24:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.523+0000 7f51e1ffb640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f51d40a2b90 0x7f51d40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.523+0000 7f51e1ffb640 1 -- 172.21.15.28:0/3370765039 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f51d80efbd0 con 0x7f51e4100780 2024-05-09T08:24:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.523+0000 7f51e3fff640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f51d40a2b90 0x7f51d40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.523+0000 7f51e3fff640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f51d40a2b90 0x7f51d40a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51e4199100 tx=0x7f51d000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.526+0000 7f51e1ffb640 1 -- 172.21.15.28:0/3370765039 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f51d80bc240 con 0x7f51e4100780 2024-05-09T08:24:03.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.715+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51e41a0390 con 0x7f51e4100780 2024-05-09T08:24:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.716+0000 7f51e1ffb640 1 -- 172.21.15.28:0/3370765039 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7f51d80c0540 con 0x7f51e4100780 2024-05-09T08:24:03.718 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:03.718 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:24:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f51d40a2b90 msgr2=0x7f51d40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f51d40a2b90 0x7f51d40a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51e4199100 tx=0x7f51d000b390 comp rx=0 tx=0).stop 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 msgr2=0x7f51e4197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 0x7f51e4197bf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f51d8031610 tx=0x7f51d8004270 comp rx=0 tx=0).stop 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 shutdown_connections 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f51d40a2b90 0x7f51d40a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e4102330 0x7f51e419f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f51e4101970 0x7f51e4198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 --2- 172.21.15.28:0/3370765039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51e4100780 0x7f51e4197bf0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 >> 172.21.15.28:0/3370765039 conn(0x7f51e40fbd10 msgr2=0x7f51e4069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 shutdown_connections 2024-05-09T08:24:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.718+0000 7f51eaeee640 1 -- 172.21.15.28:0/3370765039 wait complete. 2024-05-09T08:24:03.731 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:24:03.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:24:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.838+0000 7f1b94df4640 1 -- 172.21.15.28:0/2252781520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90100790 msgr2=0x7f1b90100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.838+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2252781520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90100790 0x7f1b90100bb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1b80009900 tx=0x7f1b8002a410 comp rx=0 tx=0).stop 2024-05-09T08:24:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.838+0000 7f1b94df4640 1 -- 172.21.15.28:0/2252781520 shutdown_connections 2024-05-09T08:24:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.838+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2252781520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b90102340 0x7f1b90108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.838+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2252781520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b90101980 0x7f1b90101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.838+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2252781520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90100790 0x7f1b90100bb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.838+0000 7f1b94df4640 1 -- 172.21.15.28:0/2252781520 >> 172.21.15.28:0/2252781520 conn(0x7f1b900fbd20 msgr2=0x7f1b900fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.838+0000 7f1b94df4640 1 -- 172.21.15.28:0/2252781520 shutdown_connections 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.838+0000 7f1b94df4640 1 -- 172.21.15.28:0/2252781520 wait complete. 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b94df4640 1 Processor -- start 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b94df4640 1 -- start start 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b94df4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b90100790 0x7f1b90197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b94df4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90101980 0x7f1b90198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b94df4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b90102340 0x7f1b9019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b94df4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9010c410 con 0x7f1b90101980 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b94df4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9010c290 con 0x7f1b90100790 2024-05-09T08:24:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b94df4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9010c590 con 0x7f1b90102340 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90101980 0x7f1b90198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90101980 0x7f1b90198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42676/0 (socket says 172.21.15.28:42676) 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8dd9b640 1 -- 172.21.15.28:0/2024987200 learned_addr learned my addr 172.21.15.28:0/2024987200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8e59c640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b90100790 0x7f1b90197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8ed9d640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b90102340 0x7f1b9019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8dd9b640 1 -- 172.21.15.28:0/2024987200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b90102340 msgr2=0x7f1b9019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8dd9b640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b90102340 0x7f1b9019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8dd9b640 1 -- 172.21.15.28:0/2024987200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b90100790 msgr2=0x7f1b90197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8dd9b640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b90100790 0x7f1b90197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8dd9b640 1 -- 172.21.15.28:0/2024987200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9019f810 con 0x7f1b90101980 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.839+0000 7f1b8dd9b640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90101980 0x7f1b90198130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400e9f0 tx=0x7f1b8400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.840+0000 7f1b7f7fe640 1 -- 172.21.15.28:0/2024987200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400cd80 con 0x7f1b90101980 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.840+0000 7f1b7f7fe640 1 -- 172.21.15.28:0/2024987200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b84010910 con 0x7f1b90101980 2024-05-09T08:24:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.840+0000 7f1b7f7fe640 1 -- 172.21.15.28:0/2024987200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b84018990 con 0x7f1b90101980 2024-05-09T08:24:03.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.840+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b9019fb00 con 0x7f1b90101980 2024-05-09T08:24:03.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.840+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b901a00a0 con 0x7f1b90101980 2024-05-09T08:24:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.841+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b901a0270 con 0x7f1b90101980 2024-05-09T08:24:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.842+0000 7f1b7f7fe640 1 -- 172.21.15.28:0/2024987200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1b84010ac0 con 0x7f1b90101980 2024-05-09T08:24:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.843+0000 7f1b7f7fe640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1b780a2c10 0x7f1b780a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.843+0000 7f1b8e59c640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1b780a2c10 0x7f1b780a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.843+0000 7f1b7f7fe640 1 -- 172.21.15.28:0/2024987200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1b840cf4f0 con 0x7f1b90101980 2024-05-09T08:24:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.843+0000 7f1b8e59c640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1b780a2c10 0x7f1b780a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1b80005ab0 tx=0x7f1b80000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:03.847+0000 7f1b7f7fe640 1 -- 172.21.15.28:0/2024987200 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b8409bbc0 con 0x7f1b90101980 2024-05-09T08:24:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.032+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b901a0440 con 0x7f1b90101980 2024-05-09T08:24:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.032+0000 7f1b7f7fe640 1 -- 172.21.15.28:0/2024987200 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f1b8409fec0 con 0x7f1b90101980 2024-05-09T08:24:04.033 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:24:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1b780a2c10 msgr2=0x7f1b780a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1b780a2c10 0x7f1b780a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1b80005ab0 tx=0x7f1b80000bc0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90101980 msgr2=0x7f1b90198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90101980 0x7f1b90198130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400e9f0 tx=0x7f1b8400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 shutdown_connections 2024-05-09T08:24:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1b780a2c10 0x7f1b780a50d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b90102340 0x7f1b9019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b90101980 0x7f1b90198130 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 --2- 172.21.15.28:0/2024987200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b90100790 0x7f1b90197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 >> 172.21.15.28:0/2024987200 conn(0x7f1b900fbd20 msgr2=0x7f1b900fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 shutdown_connections 2024-05-09T08:24:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.034+0000 7f1b94df4640 1 -- 172.21.15.28:0/2024987200 wait complete. 2024-05-09T08:24:04.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:24:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.153+0000 7f0f39dc1640 1 -- 172.21.15.28:0/527046125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f34100790 msgr2=0x7f0f34100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.153+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/527046125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f34100790 0x7f0f34100bb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009860 tx=0x7f0f2402a340 comp rx=0 tx=0).stop 2024-05-09T08:24:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.153+0000 7f0f39dc1640 1 -- 172.21.15.28:0/527046125 shutdown_connections 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.153+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/527046125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f34102340 0x7f0f34108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.153+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/527046125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34101980 0x7f0f34101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.153+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/527046125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f34100790 0x7f0f34100bb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.153+0000 7f0f39dc1640 1 -- 172.21.15.28:0/527046125 >> 172.21.15.28:0/527046125 conn(0x7f0f340fbd20 msgr2=0x7f0f340fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.153+0000 7f0f39dc1640 1 -- 172.21.15.28:0/527046125 shutdown_connections 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.153+0000 7f0f39dc1640 1 -- 172.21.15.28:0/527046125 wait complete. 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f39dc1640 1 Processor -- start 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f39dc1640 1 -- start start 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f39dc1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f34100790 0x7f0f34104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f39dc1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f34101980 0x7f0f34102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f39dc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34102340 0x7f0f341032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f39dc1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3410c410 con 0x7f0f34102340 2024-05-09T08:24:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f39dc1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3410c290 con 0x7f0f34100790 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f39dc1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3410c590 con 0x7f0f34101980 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f33fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34102340 0x7f0f341032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f33fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34102340 0x7f0f341032a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42696/0 (socket says 172.21.15.28:42696) 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f33fff640 1 -- 172.21.15.28:0/2822313527 learned_addr learned my addr 172.21.15.28:0/2822313527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.154+0000 7f0f337fe640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f34100790 0x7f0f34104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f33fff640 1 -- 172.21.15.28:0/2822313527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f34101980 msgr2=0x7f0f34102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f33fff640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f34101980 0x7f0f34102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f33fff640 1 -- 172.21.15.28:0/2822313527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f34100790 msgr2=0x7f0f34104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f33fff640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f34100790 0x7f0f34104680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f33fff640 1 -- 172.21.15.28:0/2822313527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f34103aa0 con 0x7f0f34102340 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f33fff640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34102340 0x7f0f341032a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00eb10 tx=0x7f0f2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f30ff9640 1 -- 172.21.15.28:0/2822313527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c004b20 con 0x7f0f34102340 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f341a1d50 con 0x7f0f34102340 2024-05-09T08:24:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f30ff9640 1 -- 172.21.15.28:0/2822313527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f2c004cc0 con 0x7f0f34102340 2024-05-09T08:24:04.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f30ff9640 1 -- 172.21.15.28:0/2822313527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c010760 con 0x7f0f34102340 2024-05-09T08:24:04.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.155+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f341a2270 con 0x7f0f34102340 2024-05-09T08:24:04.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.156+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f341a2500 con 0x7f0f34102340 2024-05-09T08:24:04.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.157+0000 7f0f30ff9640 1 -- 172.21.15.28:0/2822313527 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0f2c010900 con 0x7f0f34102340 2024-05-09T08:24:04.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.158+0000 7f0f30ff9640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0f1c0a2b90 0x7f0f1c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.158+0000 7f0f30ff9640 1 -- 172.21.15.28:0/2822313527 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0f2c0cfd20 con 0x7f0f34102340 2024-05-09T08:24:04.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.158+0000 7f0f337fe640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0f1c0a2b90 0x7f0f1c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.159+0000 7f0f337fe640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0f1c0a2b90 0x7f0f1c0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009830 tx=0x7f0f24030570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.161+0000 7f0f30ff9640 1 -- 172.21.15.28:0/2822313527 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0f2c09ca40 con 0x7f0f34102340 2024-05-09T08:24:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.345+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0f341a27f0 con 0x7f0f34102340 2024-05-09T08:24:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.346+0000 7f0f30ff9640 1 -- 172.21.15.28:0/2822313527 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f0f2c09c340 con 0x7f0f34102340 2024-05-09T08:24:04.347 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:24:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0f1c0a2b90 msgr2=0x7f0f1c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0f1c0a2b90 0x7f0f1c0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009830 tx=0x7f0f24030570 comp rx=0 tx=0).stop 2024-05-09T08:24:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34102340 msgr2=0x7f0f341032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34102340 0x7f0f341032a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00eb10 tx=0x7f0f2c00c2d0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 shutdown_connections 2024-05-09T08:24:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0f1c0a2b90 0x7f0f1c0a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34102340 0x7f0f341032a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f34101980 0x7f0f34102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 --2- 172.21.15.28:0/2822313527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f34100790 0x7f0f34104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 >> 172.21.15.28:0/2822313527 conn(0x7f0f340fbd20 msgr2=0x7f0f340fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 shutdown_connections 2024-05-09T08:24:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.348+0000 7f0f39dc1640 1 -- 172.21.15.28:0/2822313527 wait complete. 2024-05-09T08:24:04.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:24:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.467+0000 7f85e319e640 1 -- 172.21.15.28:0/793579419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc102a80 msgr2=0x7f85dc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.467+0000 7f85e319e640 1 --2- 172.21.15.28:0/793579419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc102a80 0x7f85dc102f00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009900 tx=0x7f85cc02a410 comp rx=0 tx=0).stop 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.467+0000 7f85e319e640 1 -- 172.21.15.28:0/793579419 shutdown_connections 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.467+0000 7f85e319e640 1 --2- 172.21.15.28:0/793579419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85dc103440 0x7f85dc109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.467+0000 7f85e319e640 1 --2- 172.21.15.28:0/793579419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc102a80 0x7f85dc102f00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.467+0000 7f85e319e640 1 --2- 172.21.15.28:0/793579419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc101890 0x7f85dc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.467+0000 7f85e319e640 1 -- 172.21.15.28:0/793579419 >> 172.21.15.28:0/793579419 conn(0x7f85dc0fce00 msgr2=0x7f85dc0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.467+0000 7f85e319e640 1 -- 172.21.15.28:0/793579419 shutdown_connections 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.467+0000 7f85e319e640 1 -- 172.21.15.28:0/793579419 wait complete. 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e319e640 1 Processor -- start 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e319e640 1 -- start start 2024-05-09T08:24:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e319e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85dc101890 0x7f85dc197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e319e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc102a80 0x7f85dc1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e319e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc103440 0x7f85dc19f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e319e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc10d510 con 0x7f85dc103440 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e319e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc10d390 con 0x7f85dc102a80 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e319e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc10d690 con 0x7f85dc101890 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e173b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc103440 0x7f85dc19f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e173b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc103440 0x7f85dc19f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42714/0 (socket says 172.21.15.28:42714) 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e173b640 1 -- 172.21.15.28:0/5084732 learned_addr learned my addr 172.21.15.28:0/5084732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85d3fff640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc102a80 0x7f85dc1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.468+0000 7f85e0f3a640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85dc101890 0x7f85dc197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85e173b640 1 -- 172.21.15.28:0/5084732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85dc101890 msgr2=0x7f85dc197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85e173b640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85dc101890 0x7f85dc197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85e173b640 1 -- 172.21.15.28:0/5084732 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc102a80 msgr2=0x7f85dc1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85e173b640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc102a80 0x7f85dc1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85e173b640 1 -- 172.21.15.28:0/5084732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85dc19f7d0 con 0x7f85dc103440 2024-05-09T08:24:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85e173b640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc103440 0x7f85dc19f160 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f85d400d750 tx=0x7f85d400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85d1ffb640 1 -- 172.21.15.28:0/5084732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d4017070 con 0x7f85dc103440 2024-05-09T08:24:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85dc19fac0 con 0x7f85dc103440 2024-05-09T08:24:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85d1ffb640 1 -- 172.21.15.28:0/5084732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85d4005500 con 0x7f85dc103440 2024-05-09T08:24:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85d1ffb640 1 -- 172.21.15.28:0/5084732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d4004d90 con 0x7f85dc103440 2024-05-09T08:24:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.469+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85dc1a0030 con 0x7f85dc103440 2024-05-09T08:24:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.470+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85dc108c00 con 0x7f85dc103440 2024-05-09T08:24:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.471+0000 7f85d1ffb640 1 -- 172.21.15.28:0/5084732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f85d4004f30 con 0x7f85dc103440 2024-05-09T08:24:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.472+0000 7f85d1ffb640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f85c40a2b90 0x7f85c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.472+0000 7f85e0f3a640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f85c40a2b90 0x7f85c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.472+0000 7f85d1ffb640 1 -- 172.21.15.28:0/5084732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f85d40ceb30 con 0x7f85dc103440 2024-05-09T08:24:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.472+0000 7f85e0f3a640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f85c40a2b90 0x7f85c40a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85dc1028e0 tx=0x7f85d8009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.476+0000 7f85d1ffb640 1 -- 172.21.15.28:0/5084732 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f85d409b1a0 con 0x7f85dc103440 2024-05-09T08:24:04.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.659+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f85dc062430 con 0x7f85dc103440 2024-05-09T08:24:04.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.659+0000 7f85d1ffb640 1 -- 172.21.15.28:0/5084732 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f85d409f4a0 con 0x7f85dc103440 2024-05-09T08:24:04.660 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f85c40a2b90 msgr2=0x7f85c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f85c40a2b90 0x7f85c40a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85dc1028e0 tx=0x7f85d8009720 comp rx=0 tx=0).stop 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc103440 msgr2=0x7f85dc19f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc103440 0x7f85dc19f160 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f85d400d750 tx=0x7f85d400dc10 comp rx=0 tx=0).stop 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 shutdown_connections 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f85c40a2b90 0x7f85c40a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc103440 0x7f85dc19f160 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc102a80 0x7f85dc1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 --2- 172.21.15.28:0/5084732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85dc101890 0x7f85dc197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 >> 172.21.15.28:0/5084732 conn(0x7f85dc0fce00 msgr2=0x7f85dc105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 shutdown_connections 2024-05-09T08:24:04.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.661+0000 7f85e319e640 1 -- 172.21.15.28:0/5084732 wait complete. 2024-05-09T08:24:04.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.781+0000 7fd0418e1640 1 -- 172.21.15.28:0/2750720027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c100a10 msgr2=0x7fd03c100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.781+0000 7fd0418e1640 1 --2- 172.21.15.28:0/2750720027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c100a10 0x7fd03c100e90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd024009930 tx=0x7fd02402a420 comp rx=0 tx=0).stop 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.781+0000 7fd0418e1640 1 -- 172.21.15.28:0/2750720027 shutdown_connections 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.781+0000 7fd0418e1640 1 --2- 172.21.15.28:0/2750720027 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd03c1013d0 0x7fd03c108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.781+0000 7fd0418e1640 1 --2- 172.21.15.28:0/2750720027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c100a10 0x7fd03c100e90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.781+0000 7fd0418e1640 1 --2- 172.21.15.28:0/2750720027 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c101d70 0x7fd03c102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.781+0000 7fd0418e1640 1 -- 172.21.15.28:0/2750720027 >> 172.21.15.28:0/2750720027 conn(0x7fd03c0fb460 msgr2=0x7fd03c0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.781+0000 7fd0418e1640 1 -- 172.21.15.28:0/2750720027 shutdown_connections 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.781+0000 7fd0418e1640 1 -- 172.21.15.28:0/2750720027 wait complete. 2024-05-09T08:24:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd0418e1640 1 Processor -- start 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd0418e1640 1 -- start start 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd0418e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c100a10 0x7fd03c1047c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd0418e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c1013d0 0x7fd03c102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd0418e1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd03c101d70 0x7fd03c103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd0418e1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c10c430 con 0x7fd03c1013d0 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd0418e1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c10c2b0 con 0x7fd03c101d70 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd0418e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c10c5b0 con 0x7fd03c100a10 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd033fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c1013d0 0x7fd03c102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd033fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c1013d0 0x7fd03c102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42426/0 (socket says 172.21.15.28:42426) 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd033fff640 1 -- 172.21.15.28:0/3631804992 learned_addr learned my addr 172.21.15.28:0/3631804992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.782+0000 7fd03affd640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c100a10 0x7fd03c1047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd03b7fe640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd03c101d70 0x7fd03c103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd033fff640 1 -- 172.21.15.28:0/3631804992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c100a10 msgr2=0x7fd03c1047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd033fff640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c100a10 0x7fd03c1047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd033fff640 1 -- 172.21.15.28:0/3631804992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd03c101d70 msgr2=0x7fd03c103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd033fff640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd03c101d70 0x7fd03c103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd033fff640 1 -- 172.21.15.28:0/3631804992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c103bd0 con 0x7fd03c1013d0 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd03b7fe640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd03c101d70 0x7fd03c103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd03affd640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c100a10 0x7fd03c1047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd033fff640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c1013d0 0x7fd03c102e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd024005ab0 tx=0x7fd0240042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd0408df640 1 -- 172.21.15.28:0/3631804992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02403f030 con 0x7fd03c1013d0 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03c1a1cf0 con 0x7fd03c1013d0 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd0408df640 1 -- 172.21.15.28:0/3631804992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0240044d0 con 0x7fd03c1013d0 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd0408df640 1 -- 172.21.15.28:0/3631804992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024035070 con 0x7fd03c1013d0 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.783+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd03c1a2190 con 0x7fd03c1013d0 2024-05-09T08:24:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.784+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd03c1a2480 con 0x7fd03c1013d0 2024-05-09T08:24:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.785+0000 7fd0408df640 1 -- 172.21.15.28:0/3631804992 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd024042030 con 0x7fd03c1013d0 2024-05-09T08:24:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.786+0000 7fd0408df640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0280a2b90 0x7fd0280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.786+0000 7fd03affd640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0280a2b90 0x7fd0280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.786+0000 7fd0408df640 1 -- 172.21.15.28:0/3631804992 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd0240ef800 con 0x7fd03c1013d0 2024-05-09T08:24:04.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.786+0000 7fd03affd640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0280a2b90 0x7fd0280a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0058c0 tx=0x7fd02c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:04.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.790+0000 7fd0408df640 1 -- 172.21.15.28:0/3631804992 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd0240bc520 con 0x7fd03c1013d0 2024-05-09T08:24:04.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:04 smithi028 ceph-mon[30119]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-05-09T08:24:04.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3370765039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:04.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2024987200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:24:04.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2822313527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:24:04.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/5084732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:24:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.975+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd03c1a2650 con 0x7fd03c1013d0 2024-05-09T08:24:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.976+0000 7fd0408df640 1 -- 172.21.15.28:0/3631804992 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fd0240bbe20 con 0x7fd03c1013d0 2024-05-09T08:24:04.977 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:24:04.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0280a2b90 msgr2=0x7fd0280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0280a2b90 0x7fd0280a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0058c0 tx=0x7fd02c005850 comp rx=0 tx=0).stop 2024-05-09T08:24:04.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c1013d0 msgr2=0x7fd03c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c1013d0 0x7fd03c102e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd024005ab0 tx=0x7fd0240042e0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 shutdown_connections 2024-05-09T08:24:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd0280a2b90 0x7fd0280a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd03c101d70 0x7fd03c103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03c1013d0 0x7fd03c102e00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 --2- 172.21.15.28:0/3631804992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c100a10 0x7fd03c1047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 >> 172.21.15.28:0/3631804992 conn(0x7fd03c0fb460 msgr2=0x7fd03c0fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 shutdown_connections 2024-05-09T08:24:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:04.978+0000 7fd0418e1640 1 -- 172.21.15.28:0/3631804992 wait complete. 2024-05-09T08:24:04.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T08:24:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.095+0000 7f0d4a3af640 1 -- 172.21.15.28:0/2889531030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d44107610 msgr2=0x7f0d44109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.095+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/2889531030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d44107610 0x7f0d44109a00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c009ca0 tx=0x7f0d3c02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.095+0000 7f0d4a3af640 1 -- 172.21.15.28:0/2889531030 shutdown_connections 2024-05-09T08:24:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.095+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/2889531030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d44107610 0x7f0d44109a00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.095+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/2889531030 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d44104ce0 0x7f0d441070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.095+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/2889531030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d441023b0 0x7f0d441047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.095+0000 7f0d4a3af640 1 -- 172.21.15.28:0/2889531030 >> 172.21.15.28:0/2889531030 conn(0x7f0d440fbd00 msgr2=0x7f0d440fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.095+0000 7f0d4a3af640 1 -- 172.21.15.28:0/2889531030 shutdown_connections 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.095+0000 7f0d4a3af640 1 -- 172.21.15.28:0/2889531030 wait complete. 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d4a3af640 1 Processor -- start 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d4a3af640 1 -- start start 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d4a3af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d441023b0 0x7f0d44197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d4a3af640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d44104ce0 0x7f0d44198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d4a3af640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d44107610 0x7f0d4419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d4a3af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4410d530 con 0x7f0d441023b0 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d4a3af640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4410d3b0 con 0x7f0d44104ce0 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d4a3af640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4410d6b0 con 0x7f0d44107610 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d441023b0 0x7f0d44197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d441023b0 0x7f0d44197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42454/0 (socket says 172.21.15.28:42454) 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d43fff640 1 -- 172.21.15.28:0/1345310401 learned_addr learned my addr 172.21.15.28:0/1345310401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d4894c640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d44107610 0x7f0d4419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.096+0000 7f0d437fe640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d44104ce0 0x7f0d44198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d43fff640 1 -- 172.21.15.28:0/1345310401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d44107610 msgr2=0x7f0d4419f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d43fff640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d44107610 0x7f0d4419f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d43fff640 1 -- 172.21.15.28:0/1345310401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d44104ce0 msgr2=0x7f0d44198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d43fff640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d44104ce0 0x7f0d44198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d43fff640 1 -- 172.21.15.28:0/1345310401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4419f7f0 con 0x7f0d441023b0 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d437fe640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d44104ce0 0x7f0d44198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d43fff640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d441023b0 0x7f0d44197bd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400d770 tx=0x7f0d3400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d417fa640 1 -- 172.21.15.28:0/1345310401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d34017070 con 0x7f0d441023b0 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d4419fae0 con 0x7f0d441023b0 2024-05-09T08:24:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d417fa640 1 -- 172.21.15.28:0/1345310401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d34005500 con 0x7f0d441023b0 2024-05-09T08:24:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d417fa640 1 -- 172.21.15.28:0/1345310401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d34004e00 con 0x7f0d441023b0 2024-05-09T08:24:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.097+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d4419ffe0 con 0x7f0d441023b0 2024-05-09T08:24:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.098+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d440fd7f0 con 0x7f0d441023b0 2024-05-09T08:24:05.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:04 smithi165 ceph-mon[34540]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-05-09T08:24:05.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3370765039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:05.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2024987200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:24:05.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2822313527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:24:05.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/5084732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:24:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.099+0000 7f0d417fa640 1 -- 172.21.15.28:0/1345310401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0d34004fa0 con 0x7f0d441023b0 2024-05-09T08:24:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.100+0000 7f0d417fa640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0d2c0a2b90 0x7f0d2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.100+0000 7f0d417fa640 1 -- 172.21.15.28:0/1345310401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0d340cf190 con 0x7f0d441023b0 2024-05-09T08:24:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.100+0000 7f0d437fe640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0d2c0a2b90 0x7f0d2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.100+0000 7f0d437fe640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0d2c0a2b90 0x7f0d2c0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0d380058c0 tx=0x7f0d38005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.103+0000 7f0d417fa640 1 -- 172.21.15.28:0/1345310401 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0d3409be50 con 0x7f0d441023b0 2024-05-09T08:24:05.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:04 smithi184 ceph-mon[34509]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-05-09T08:24:05.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3370765039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:05.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2024987200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:24:05.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2822313527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:24:05.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/5084732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:24:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.299+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0d4402d1c0 con 0x7f0d441023b0 2024-05-09T08:24:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.300+0000 7f0d417fa640 1 -- 172.21.15.28:0/1345310401 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7f0d3409b750 con 0x7f0d441023b0 2024-05-09T08:24:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2931568341 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3510522803 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/934935086 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1194745694 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3920156841 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/1179097296 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:05.302 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/1179097296 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:05.302 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2931568341 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:05.302 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/466716681 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:05.302 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2416478420 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:05.302 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-05-09T08:24:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0d2c0a2b90 msgr2=0x7f0d2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0d2c0a2b90 0x7f0d2c0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0d380058c0 tx=0x7f0d38005850 comp rx=0 tx=0).stop 2024-05-09T08:24:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d441023b0 msgr2=0x7f0d44197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d441023b0 0x7f0d44197bd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400d770 tx=0x7f0d3400dc30 comp rx=0 tx=0).stop 2024-05-09T08:24:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 shutdown_connections 2024-05-09T08:24:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0d2c0a2b90 0x7f0d2c0a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d44107610 0x7f0d4419f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d44104ce0 0x7f0d44198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 --2- 172.21.15.28:0/1345310401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d441023b0 0x7f0d44197bd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 >> 172.21.15.28:0/1345310401 conn(0x7f0d440fbd00 msgr2=0x7f0d44102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:05.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 shutdown_connections 2024-05-09T08:24:05.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.303+0000 7f0d4a3af640 1 -- 172.21.15.28:0/1345310401 wait complete. 2024-05-09T08:24:05.314 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:24:05.314 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:24:05.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:24:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.475+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/4006268894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c94104ca0 msgr2=0x7f0c94107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.475+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/4006268894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c94104ca0 0x7f0c94107090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009900 tx=0x7f0c8402a410 comp rx=0 tx=0).stop 2024-05-09T08:24:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.475+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/4006268894 shutdown_connections 2024-05-09T08:24:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.475+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/4006268894 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c941075d0 0x7f0c941099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.475+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/4006268894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c94104ca0 0x7f0c94107090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.475+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/4006268894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c94102370 0x7f0c94104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.475+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/4006268894 >> 172.21.15.28:0/4006268894 conn(0x7f0c940fbd20 msgr2=0x7f0c940fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.475+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/4006268894 shutdown_connections 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.475+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/4006268894 wait complete. 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c9b7b0640 1 Processor -- start 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c9b7b0640 1 -- start start 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c9b7b0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c94102370 0x7f0c94197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c9b7b0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c94104ca0 0x7f0c94198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c9b7b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c941075d0 0x7f0c9419f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c9b7b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9410d510 con 0x7f0c941075d0 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c9b7b0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9410d390 con 0x7f0c94104ca0 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c9b7b0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9410d690 con 0x7f0c94102370 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c99d4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c941075d0 0x7f0c9419f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c99d4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c941075d0 0x7f0c9419f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42474/0 (socket says 172.21.15.28:42474) 2024-05-09T08:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c9954c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c94102370 0x7f0c94197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c99d4d640 1 -- 172.21.15.28:0/3067440167 learned_addr learned my addr 172.21.15.28:0/3067440167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c98d4b640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c94104ca0 0x7f0c94198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c99d4d640 1 -- 172.21.15.28:0/3067440167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c94102370 msgr2=0x7f0c94197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c99d4d640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c94102370 0x7f0c94197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c99d4d640 1 -- 172.21.15.28:0/3067440167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c94104ca0 msgr2=0x7f0c94198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c99d4d640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c94104ca0 0x7f0c94198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.476+0000 7f0c99d4d640 1 -- 172.21.15.28:0/3067440167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9419f810 con 0x7f0c941075d0 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.477+0000 7f0c99d4d640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c941075d0 0x7f0c9419f1a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00cad0 tx=0x7f0c8c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.477+0000 7f0c8a7fc640 1 -- 172.21.15.28:0/3067440167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c00ae20 con 0x7f0c941075d0 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.477+0000 7f0c8a7fc640 1 -- 172.21.15.28:0/3067440167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c8c005020 con 0x7f0c941075d0 2024-05-09T08:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.477+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c9419fb00 con 0x7f0c941075d0 2024-05-09T08:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.477+0000 7f0c8a7fc640 1 -- 172.21.15.28:0/3067440167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c00f6f0 con 0x7f0c941075d0 2024-05-09T08:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.477+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c941a0070 con 0x7f0c941075d0 2024-05-09T08:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.478+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c940fd810 con 0x7f0c941075d0 2024-05-09T08:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.479+0000 7f0c8a7fc640 1 -- 172.21.15.28:0/3067440167 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0c8c0051d0 con 0x7f0c941075d0 2024-05-09T08:24:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.480+0000 7f0c8a7fc640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c800a2c10 0x7f0c800a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.480+0000 7f0c8a7fc640 1 -- 172.21.15.28:0/3067440167 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0c8c0cf550 con 0x7f0c941075d0 2024-05-09T08:24:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.480+0000 7f0c9954c640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c800a2c10 0x7f0c800a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.480+0000 7f0c9954c640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c800a2c10 0x7f0c800a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009bd0 tx=0x7f0c90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.485+0000 7f0c8a7fc640 1 -- 172.21.15.28:0/3067440167 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0c8c014030 con 0x7f0c941075d0 2024-05-09T08:24:05.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.670+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c940624d0 con 0x7f0c941075d0 2024-05-09T08:24:05.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.672+0000 7f0c8a7fc640 1 -- 172.21.15.28:0/3067440167 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7f0c8c09c270 con 0x7f0c941075d0 2024-05-09T08:24:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:05.673 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c800a2c10 msgr2=0x7f0c800a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c800a2c10 0x7f0c800a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009bd0 tx=0x7f0c90005f90 comp rx=0 tx=0).stop 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c941075d0 msgr2=0x7f0c9419f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c941075d0 0x7f0c9419f1a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00cad0 tx=0x7f0c8c00cf90 comp rx=0 tx=0).stop 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 shutdown_connections 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c800a2c10 0x7f0c800a50d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c941075d0 0x7f0c9419f1a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c94104ca0 0x7f0c94198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 --2- 172.21.15.28:0/3067440167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c94102370 0x7f0c94197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 >> 172.21.15.28:0/3067440167 conn(0x7f0c940fbd20 msgr2=0x7f0c94102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 shutdown_connections 2024-05-09T08:24:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.674+0000 7f0c9b7b0640 1 -- 172.21.15.28:0/3067440167 wait complete. 2024-05-09T08:24:05.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:24:05.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:24:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3916261867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4104c40 msgr2=0x7fdce4107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3916261867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4104c40 0x7fdce4107030 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009900 tx=0x7fdcd402a410 comp rx=0 tx=0).stop 2024-05-09T08:24:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3916261867 shutdown_connections 2024-05-09T08:24:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3916261867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdce4107570 0x7fdce41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3916261867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4104c40 0x7fdce4107030 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3916261867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdce4102310 0x7fdce4104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3916261867 >> 172.21.15.28:0/3916261867 conn(0x7fdce40fbd20 msgr2=0x7fdce40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3916261867 shutdown_connections 2024-05-09T08:24:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3916261867 wait complete. 2024-05-09T08:24:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.797+0000 7fdcebeb3640 1 Processor -- start 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcebeb3640 1 -- start start 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcebeb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4102310 0x7fdce4197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcebeb3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdce4104c40 0x7fdce4198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcebeb3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdce4107570 0x7fdce419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcebeb3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdce410d510 con 0x7fdce4102310 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcebeb3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdce410d390 con 0x7fdce4107570 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcebeb3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdce410d690 con 0x7fdce4104c40 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce9c4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4102310 0x7fdce4197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce9c4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4102310 0x7fdce4197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42484/0 (socket says 172.21.15.28:42484) 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce9c4f640 1 -- 172.21.15.28:0/3275814746 learned_addr learned my addr 172.21.15.28:0/3275814746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce944e640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdce4104c40 0x7fdce4198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcea450640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdce4107570 0x7fdce419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce9c4f640 1 -- 172.21.15.28:0/3275814746 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdce4104c40 msgr2=0x7fdce4198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce9c4f640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdce4104c40 0x7fdce4198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce9c4f640 1 -- 172.21.15.28:0/3275814746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdce4107570 msgr2=0x7fdce419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce9c4f640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdce4107570 0x7fdce419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce9c4f640 1 -- 172.21.15.28:0/3275814746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce419f7e0 con 0x7fdce4102310 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcea450640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdce4107570 0x7fdce419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce944e640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdce4104c40 0x7fdce4198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdce9c4f640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4102310 0x7fdce4197bc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdce000e9f0 tx=0x7fdce000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcdaffd640 1 -- 172.21.15.28:0/3275814746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce000cd80 con 0x7fdce4102310 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce419fad0 con 0x7fdce4102310 2024-05-09T08:24:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcdaffd640 1 -- 172.21.15.28:0/3275814746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdce0010910 con 0x7fdce4102310 2024-05-09T08:24:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.798+0000 7fdcdaffd640 1 -- 172.21.15.28:0/3275814746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce00188f0 con 0x7fdce4102310 2024-05-09T08:24:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.799+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce41a0040 con 0x7fdce4102310 2024-05-09T08:24:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.800+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdce40fd810 con 0x7fdce4102310 2024-05-09T08:24:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.801+0000 7fdcdaffd640 1 -- 172.21.15.28:0/3275814746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdce0010ac0 con 0x7fdce4102310 2024-05-09T08:24:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.802+0000 7fdcdaffd640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdcd00a2b90 0x7fdcd00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.802+0000 7fdce944e640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdcd00a2b90 0x7fdcd00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.802+0000 7fdcdaffd640 1 -- 172.21.15.28:0/3275814746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdce00cf790 con 0x7fdce4102310 2024-05-09T08:24:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.802+0000 7fdce944e640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdcd00a2b90 0x7fdcd00a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4006010 tx=0x7fdcd4032cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.806+0000 7fdcdaffd640 1 -- 172.21.15.28:0/3275814746 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdce009c4b0 con 0x7fdce4102310 2024-05-09T08:24:05.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:05 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3631804992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:24:05.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:05 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1345310401' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:24:05.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:05 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3067440167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:05.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.994+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdce4062460 con 0x7fdce4102310 2024-05-09T08:24:05.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.994+0000 7fdcdaffd640 1 -- 172.21.15.28:0/3275814746 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fdce009bdb0 con 0x7fdce4102310 2024-05-09T08:24:05.994 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:24:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdcd00a2b90 msgr2=0x7fdcd00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdcd00a2b90 0x7fdcd00a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4006010 tx=0x7fdcd4032cf0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4102310 msgr2=0x7fdce4197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4102310 0x7fdce4197bc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdce000e9f0 tx=0x7fdce000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 shutdown_connections 2024-05-09T08:24:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdcd00a2b90 0x7fdcd00a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdce4107570 0x7fdce419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdce4104c40 0x7fdce4198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 --2- 172.21.15.28:0/3275814746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4102310 0x7fdce4197bc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 >> 172.21.15.28:0/3275814746 conn(0x7fdce40fbd20 msgr2=0x7fdce4102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 shutdown_connections 2024-05-09T08:24:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:05.996+0000 7fdcebeb3640 1 -- 172.21.15.28:0/3275814746 wait complete. 2024-05-09T08:24:06.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:24:06.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:05 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3631804992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:24:06.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:05 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1345310401' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:24:06.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:05 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3067440167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:06.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:05 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3631804992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:24:06.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:05 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1345310401' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:24:06.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:05 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3067440167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.116+0000 7f6f40309640 1 -- 172.21.15.28:0/2482161680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38102350 msgr2=0x7f6f38104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.116+0000 7f6f40309640 1 --2- 172.21.15.28:0/2482161680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38102350 0x7f6f38104740 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6f34009ca0 tx=0x7f6f3402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.116+0000 7f6f40309640 1 -- 172.21.15.28:0/2482161680 shutdown_connections 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.116+0000 7f6f40309640 1 --2- 172.21.15.28:0/2482161680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f381075b0 0x7f6f381099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.116+0000 7f6f40309640 1 --2- 172.21.15.28:0/2482161680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f38104c80 0x7f6f38107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.116+0000 7f6f40309640 1 --2- 172.21.15.28:0/2482161680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38102350 0x7f6f38104740 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.116+0000 7f6f40309640 1 -- 172.21.15.28:0/2482161680 >> 172.21.15.28:0/2482161680 conn(0x7f6f380fbce0 msgr2=0x7f6f380fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.116+0000 7f6f40309640 1 -- 172.21.15.28:0/2482161680 shutdown_connections 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.116+0000 7f6f40309640 1 -- 172.21.15.28:0/2482161680 wait complete. 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f40309640 1 Processor -- start 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f40309640 1 -- start start 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f40309640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f38102350 0x7f6f38197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f40309640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38104c80 0x7f6f38198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f40309640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f381075b0 0x7f6f3819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f40309640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3810d4e0 con 0x7f6f38104c80 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f40309640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3810d360 con 0x7f6f381075b0 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f40309640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3810d660 con 0x7f6f38102350 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f3d8a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38104c80 0x7f6f38198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f3d8a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38104c80 0x7f6f38198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42500/0 (socket says 172.21.15.28:42500) 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f3d8a4640 1 -- 172.21.15.28:0/3580559002 learned_addr learned my addr 172.21.15.28:0/3580559002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f3e0a5640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f38102350 0x7f6f38197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f3d8a4640 1 -- 172.21.15.28:0/3580559002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f38102350 msgr2=0x7f6f38197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f3d8a4640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f38102350 0x7f6f38197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f3d8a4640 1 -- 172.21.15.28:0/3580559002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f381075b0 msgr2=0x7f6f3819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f3d8a4640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f381075b0 0x7f6f3819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.117+0000 7f6f3d8a4640 1 -- 172.21.15.28:0/3580559002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3819f800 con 0x7f6f38104c80 2024-05-09T08:24:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.118+0000 7f6f3d8a4640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38104c80 0x7f6f38198120 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6f2800e9c0 tx=0x7f6f2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.118+0000 7f6f2f7fe640 1 -- 172.21.15.28:0/3580559002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2800cd80 con 0x7f6f38104c80 2024-05-09T08:24:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.118+0000 7f6f2f7fe640 1 -- 172.21.15.28:0/3580559002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f28005500 con 0x7f6f38104c80 2024-05-09T08:24:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.118+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f3819faf0 con 0x7f6f38104c80 2024-05-09T08:24:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.118+0000 7f6f2f7fe640 1 -- 172.21.15.28:0/3580559002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f28010690 con 0x7f6f38104c80 2024-05-09T08:24:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.118+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f381a0090 con 0x7f6f38104c80 2024-05-09T08:24:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.119+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f380fd7d0 con 0x7f6f38104c80 2024-05-09T08:24:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.120+0000 7f6f2f7fe640 1 -- 172.21.15.28:0/3580559002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6f280046c0 con 0x7f6f38104c80 2024-05-09T08:24:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.121+0000 7f6f2f7fe640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6f240a2c10 0x7f6f240a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.121+0000 7f6f3e0a5640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6f240a2c10 0x7f6f240a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.121+0000 7f6f2f7fe640 1 -- 172.21.15.28:0/3580559002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6f280cf690 con 0x7f6f38104c80 2024-05-09T08:24:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.121+0000 7f6f3e0a5640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6f240a2c10 0x7f6f240a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6f34006010 tx=0x7f6f34000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:06.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.126+0000 7f6f2f7fe640 1 -- 172.21.15.28:0/3580559002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6f2809c350 con 0x7f6f38104c80 2024-05-09T08:24:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.313+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6f380624d0 con 0x7f6f38104c80 2024-05-09T08:24:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.313+0000 7f6f2f7fe640 1 -- 172.21.15.28:0/3580559002 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f6f2809bc50 con 0x7f6f38104c80 2024-05-09T08:24:06.314 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:24:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.315+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6f240a2c10 msgr2=0x7f6f240a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.315+0000 7f6f40309640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6f240a2c10 0x7f6f240a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6f34006010 tx=0x7f6f34000f40 comp rx=0 tx=0).stop 2024-05-09T08:24:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.315+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38104c80 msgr2=0x7f6f38198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.315+0000 7f6f40309640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38104c80 0x7f6f38198120 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6f2800e9c0 tx=0x7f6f2800ee80 comp rx=0 tx=0).stop 2024-05-09T08:24:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.316+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 shutdown_connections 2024-05-09T08:24:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.316+0000 7f6f40309640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6f240a2c10 0x7f6f240a50d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.316+0000 7f6f40309640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f381075b0 0x7f6f3819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.316+0000 7f6f40309640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f38104c80 0x7f6f38198120 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.316+0000 7f6f40309640 1 --2- 172.21.15.28:0/3580559002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f38102350 0x7f6f38197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.316+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 >> 172.21.15.28:0/3580559002 conn(0x7f6f380fbce0 msgr2=0x7f6f38102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.316+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 shutdown_connections 2024-05-09T08:24:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.316+0000 7f6f40309640 1 -- 172.21.15.28:0/3580559002 wait complete. 2024-05-09T08:24:06.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:24:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.437+0000 7f70712da640 1 -- 172.21.15.28:0/1602368067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c108830 msgr2=0x7f706c108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.437+0000 7f70712da640 1 --2- 172.21.15.28:0/1602368067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c108830 0x7f706c108c50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f705c009930 tx=0x7f705c02a420 comp rx=0 tx=0).stop 2024-05-09T08:24:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 -- 172.21.15.28:0/1602368067 shutdown_connections 2024-05-09T08:24:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 --2- 172.21.15.28:0/1602368067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f706c1100b0 0x7f706c1124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 --2- 172.21.15.28:0/1602368067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f706c109190 0x7f706c10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 --2- 172.21.15.28:0/1602368067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c108830 0x7f706c108c50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 -- 172.21.15.28:0/1602368067 >> 172.21.15.28:0/1602368067 conn(0x7f706c06e6c0 msgr2=0x7f706c070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 -- 172.21.15.28:0/1602368067 shutdown_connections 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 -- 172.21.15.28:0/1602368067 wait complete. 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 Processor -- start 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 -- start start 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f706c108830 0x7f706c1a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c109190 0x7f706c1a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f706c1100b0 0x7f706c1a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f706c115d20 con 0x7f706c109190 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f706c115ba0 con 0x7f706c1100b0 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f70712da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f706c115ea0 con 0x7f706c108830 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f7063fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c109190 0x7f706c1a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f7063fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c109190 0x7f706c1a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42514/0 (socket says 172.21.15.28:42514) 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.438+0000 7f7063fff640 1 -- 172.21.15.28:0/2066079164 learned_addr learned my addr 172.21.15.28:0/2066079164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f706affd640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f706c108830 0x7f706c1a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f706b7fe640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f706c1100b0 0x7f706c1a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f7063fff640 1 -- 172.21.15.28:0/2066079164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f706c108830 msgr2=0x7f706c1a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f7063fff640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f706c108830 0x7f706c1a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f7063fff640 1 -- 172.21.15.28:0/2066079164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f706c1100b0 msgr2=0x7f706c1a79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f7063fff640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f706c1100b0 0x7f706c1a79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f7063fff640 1 -- 172.21.15.28:0/2066079164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706c1a8050 con 0x7f706c109190 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f706b7fe640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f706c1100b0 0x7f706c1a79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f7063fff640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c109190 0x7f706c1a0970 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f705400e9c0 tx=0x7f705400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f7068ff9640 1 -- 172.21.15.28:0/2066079164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705400cd80 con 0x7f706c109190 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f7068ff9640 1 -- 172.21.15.28:0/2066079164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70540035f0 con 0x7f706c109190 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706c1a8340 con 0x7f706c109190 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f7068ff9640 1 -- 172.21.15.28:0/2066079164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054010690 con 0x7f706c109190 2024-05-09T08:24:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.439+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706c1a8890 con 0x7f706c109190 2024-05-09T08:24:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.440+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f706c10f270 con 0x7f706c109190 2024-05-09T08:24:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.442+0000 7f7068ff9640 1 -- 172.21.15.28:0/2066079164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f70540037a0 con 0x7f706c109190 2024-05-09T08:24:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.443+0000 7f7068ff9640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f70580a2b90 0x7f70580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.443+0000 7f706affd640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f70580a2b90 0x7f70580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.443+0000 7f7068ff9640 1 -- 172.21.15.28:0/2066079164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f70540cf710 con 0x7f706c109190 2024-05-09T08:24:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.443+0000 7f706affd640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f70580a2b90 0x7f70580a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f705c005ab0 tx=0x7f705c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.447+0000 7f7068ff9640 1 -- 172.21.15.28:0/2066079164 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f705409c430 con 0x7f706c109190 2024-05-09T08:24:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.632+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f706c062460 con 0x7f706c109190 2024-05-09T08:24:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.632+0000 7f7068ff9640 1 -- 172.21.15.28:0/2066079164 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f705409bd30 con 0x7f706c109190 2024-05-09T08:24:06.633 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:24:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f70580a2b90 msgr2=0x7f70580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f70580a2b90 0x7f70580a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f705c005ab0 tx=0x7f705c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c109190 msgr2=0x7f706c1a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c109190 0x7f706c1a0970 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f705400e9c0 tx=0x7f705400ee80 comp rx=0 tx=0).stop 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 shutdown_connections 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f70580a2b90 0x7f70580a5050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f706c1100b0 0x7f706c1a79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f706c109190 0x7f706c1a0970 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 --2- 172.21.15.28:0/2066079164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f706c108830 0x7f706c1a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 >> 172.21.15.28:0/2066079164 conn(0x7f706c06e6c0 msgr2=0x7f706c070410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 shutdown_connections 2024-05-09T08:24:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.634+0000 7f70712da640 1 -- 172.21.15.28:0/2066079164 wait complete. 2024-05-09T08:24:06.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:24:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.754+0000 7fe8b0d63640 1 -- 172.21.15.28:0/3823127844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac101980 msgr2=0x7fe8ac101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.754+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/3823127844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac101980 0x7fe8ac101e00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe894009900 tx=0x7fe89402a410 comp rx=0 tx=0).stop 2024-05-09T08:24:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.754+0000 7fe8b0d63640 1 -- 172.21.15.28:0/3823127844 shutdown_connections 2024-05-09T08:24:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.754+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/3823127844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8ac102340 0x7fe8ac108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.754+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/3823127844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac101980 0x7fe8ac101e00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.754+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/3823127844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8ac100790 0x7fe8ac100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.754+0000 7fe8b0d63640 1 -- 172.21.15.28:0/3823127844 >> 172.21.15.28:0/3823127844 conn(0x7fe8ac0fbd20 msgr2=0x7fe8ac0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.754+0000 7fe8b0d63640 1 -- 172.21.15.28:0/3823127844 shutdown_connections 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.754+0000 7fe8b0d63640 1 -- 172.21.15.28:0/3823127844 wait complete. 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8b0d63640 1 Processor -- start 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8b0d63640 1 -- start start 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8b0d63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8ac100790 0x7fe8ac06a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8b0d63640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8ac101980 0x7fe8ac19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8b0d63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac102340 0x7fe8ac19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8b0d63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac10c490 con 0x7fe8ac102340 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8b0d63640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac10c310 con 0x7fe8ac101980 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8b0d63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac10c610 con 0x7fe8ac100790 2024-05-09T08:24:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac102340 0x7fe8ac19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac102340 0x7fe8ac19efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42540/0 (socket says 172.21.15.28:42540) 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8aad9d640 1 -- 172.21.15.28:0/2820891898 learned_addr learned my addr 172.21.15.28:0/2820891898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8aa59c640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8ac100790 0x7fe8ac06a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8a3fff640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8ac101980 0x7fe8ac19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8aad9d640 1 -- 172.21.15.28:0/2820891898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8ac100790 msgr2=0x7fe8ac06a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8aad9d640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8ac100790 0x7fe8ac06a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8aad9d640 1 -- 172.21.15.28:0/2820891898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8ac101980 msgr2=0x7fe8ac19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8aad9d640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8ac101980 0x7fe8ac19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8aad9d640 1 -- 172.21.15.28:0/2820891898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8ac19f8b0 con 0x7fe8ac102340 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.755+0000 7fe8a3fff640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8ac101980 0x7fe8ac19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.756+0000 7fe8aad9d640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac102340 0x7fe8ac19efa0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe8a400b970 tx=0x7fe8a400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.756+0000 7fe8a37fe640 1 -- 172.21.15.28:0/2820891898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4019070 con 0x7fe8ac102340 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.756+0000 7fe8a37fe640 1 -- 172.21.15.28:0/2820891898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8a400cb40 con 0x7fe8ac102340 2024-05-09T08:24:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.756+0000 7fe8a37fe640 1 -- 172.21.15.28:0/2820891898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4007650 con 0x7fe8ac102340 2024-05-09T08:24:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.756+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8ac19fb40 con 0x7fe8ac102340 2024-05-09T08:24:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.756+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8ac1a0090 con 0x7fe8ac102340 2024-05-09T08:24:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.757+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8ac107ab0 con 0x7fe8ac102340 2024-05-09T08:24:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.758+0000 7fe8a37fe640 1 -- 172.21.15.28:0/2820891898 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe8a401c070 con 0x7fe8ac102340 2024-05-09T08:24:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.759+0000 7fe8a37fe640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8980a2c10 0x7fe8980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.759+0000 7fe8aa59c640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8980a2c10 0x7fe8980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.759+0000 7fe8a37fe640 1 -- 172.21.15.28:0/2820891898 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe8a40ce830 con 0x7fe8ac102340 2024-05-09T08:24:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.759+0000 7fe8aa59c640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8980a2c10 0x7fe8980a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe89c005fd0 tx=0x7fe89c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.763+0000 7fe8a37fe640 1 -- 172.21.15.28:0/2820891898 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe8a409aea0 con 0x7fe8ac102340 2024-05-09T08:24:06.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:06 smithi028 ceph-mon[30119]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 615 B/s wr, 1 op/s 2024-05-09T08:24:06.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3275814746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:24:06.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3580559002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:24:06.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2066079164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:24:06.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.947+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe8ac0624d0 con 0x7fe8ac102340 2024-05-09T08:24:06.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.947+0000 7fe8a37fe640 1 -- 172.21.15.28:0/2820891898 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fe8a409f1a0 con 0x7fe8ac102340 2024-05-09T08:24:06.947 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8980a2c10 msgr2=0x7fe8980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8980a2c10 0x7fe8980a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe89c005fd0 tx=0x7fe89c00b390 comp rx=0 tx=0).stop 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac102340 msgr2=0x7fe8ac19efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac102340 0x7fe8ac19efa0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe8a400b970 tx=0x7fe8a400be30 comp rx=0 tx=0).stop 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 shutdown_connections 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8980a2c10 0x7fe8980a50d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8ac102340 0x7fe8ac19efa0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8ac101980 0x7fe8ac19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 --2- 172.21.15.28:0/2820891898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe8ac100790 0x7fe8ac06a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 >> 172.21.15.28:0/2820891898 conn(0x7fe8ac0fbd20 msgr2=0x7fe8ac104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:06.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 shutdown_connections 2024-05-09T08:24:06.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:06.949+0000 7fe8b0d63640 1 -- 172.21.15.28:0/2820891898 wait complete. 2024-05-09T08:24:06.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T08:24:07.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.067+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/301956554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8104c40 msgr2=0x7fcdf8107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:07.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.067+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/301956554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8104c40 0x7fcdf8107030 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009900 tx=0x7fcdec02a410 comp rx=0 tx=0).stop 2024-05-09T08:24:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.067+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/301956554 shutdown_connections 2024-05-09T08:24:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.067+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/301956554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8107570 0x7fcdf81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.067+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/301956554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8104c40 0x7fcdf8107030 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.067+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/301956554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdf8102310 0x7fcdf8104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.067+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/301956554 >> 172.21.15.28:0/301956554 conn(0x7fcdf80fbd20 msgr2=0x7fcdf80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.067+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/301956554 shutdown_connections 2024-05-09T08:24:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.067+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/301956554 wait complete. 2024-05-09T08:24:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdfe1ce640 1 Processor -- start 2024-05-09T08:24:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdfe1ce640 1 -- start start 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdfe1ce640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8102310 0x7fcdf8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdfe1ce640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdf8104c40 0x7fcdf81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdfe1ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8107570 0x7fcdf819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdfe1ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf810d510 con 0x7fcdf8107570 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdfe1ce640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf810d390 con 0x7fcdf8104c40 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdfe1ce640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf810d690 con 0x7fcdf8102310 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8107570 0x7fcdf819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8107570 0x7fcdf819f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42554/0 (socket says 172.21.15.28:42554) 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf7fff640 1 -- 172.21.15.28:0/3212732905 learned_addr learned my addr 172.21.15.28:0/3212732905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf77fe640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8102310 0x7fcdf8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf6ffd640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdf8104c40 0x7fcdf81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf7fff640 1 -- 172.21.15.28:0/3212732905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8102310 msgr2=0x7fcdf8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf7fff640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8102310 0x7fcdf8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf7fff640 1 -- 172.21.15.28:0/3212732905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdf8104c40 msgr2=0x7fcdf81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf7fff640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdf8104c40 0x7fcdf81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf7fff640 1 -- 172.21.15.28:0/3212732905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf819f780 con 0x7fcdf8107570 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.068+0000 7fcdf77fe640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8102310 0x7fcdf8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.069+0000 7fcdf7fff640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8107570 0x7fcdf819f110 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000eb10 tx=0x7fcdf000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.069+0000 7fcdf4ff9640 1 -- 172.21.15.28:0/3212732905 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0004b20 con 0x7fcdf8107570 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.069+0000 7fcdf4ff9640 1 -- 172.21.15.28:0/3212732905 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdf0004cc0 con 0x7fcdf8107570 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.069+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf819fa70 con 0x7fcdf8107570 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.069+0000 7fcdf4ff9640 1 -- 172.21.15.28:0/3212732905 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0010760 con 0x7fcdf8107570 2024-05-09T08:24:07.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.069+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf819ff90 con 0x7fcdf8107570 2024-05-09T08:24:07.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.070+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf80fd810 con 0x7fcdf8107570 2024-05-09T08:24:07.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.071+0000 7fcdf4ff9640 1 -- 172.21.15.28:0/3212732905 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcdf0005000 con 0x7fcdf8107570 2024-05-09T08:24:07.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.072+0000 7fcdf4ff9640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcde00a2b90 0x7fcde00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.072+0000 7fcdf77fe640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcde00a2b90 0x7fcde00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.072+0000 7fcdf4ff9640 1 -- 172.21.15.28:0/3212732905 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcdf00cfe50 con 0x7fcdf8107570 2024-05-09T08:24:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.072+0000 7fcdf77fe640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcde00a2b90 0x7fcde00a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcde800abe0 tx=0x7fcde800a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:07.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.075+0000 7fcdf4ff9640 1 -- 172.21.15.28:0/3212732905 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcdf009cb10 con 0x7fcdf8107570 2024-05-09T08:24:07.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:06 smithi165 ceph-mon[34540]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 615 B/s wr, 1 op/s 2024-05-09T08:24:07.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:06 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3275814746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:24:07.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:06 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3580559002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:24:07.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:06 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2066079164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:24:07.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:06 smithi184 ceph-mon[34509]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 615 B/s wr, 1 op/s 2024-05-09T08:24:07.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:06 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3275814746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:24:07.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:06 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3580559002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:24:07.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:06 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2066079164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:24:07.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.274+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcdf802d1c0 con 0x7fcdf8107570 2024-05-09T08:24:07.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.274+0000 7fcdf4ff9640 1 -- 172.21.15.28:0/3212732905 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7fcdf802d1c0 con 0x7fcdf8107570 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2931568341 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3510522803 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/934935086 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1194745694 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3920156841 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/1179097296 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/1179097296 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2931568341 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/466716681 2024-05-10T08:16:49.552676+0000 2024-05-09T08:24:07.275 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2416478420 2024-05-10T08:16:22.859488+0000 2024-05-09T08:24:07.276 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-05-09T08:24:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcde00a2b90 msgr2=0x7fcde00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcde00a2b90 0x7fcde00a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcde800abe0 tx=0x7fcde800a5c0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8107570 msgr2=0x7fcdf819f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8107570 0x7fcdf819f110 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000eb10 tx=0x7fcdf000c2d0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 shutdown_connections 2024-05-09T08:24:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcde00a2b90 0x7fcde00a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8107570 0x7fcdf819f110 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdf8104c40 0x7fcdf81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 --2- 172.21.15.28:0/3212732905 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdf8102310 0x7fcdf8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 >> 172.21.15.28:0/3212732905 conn(0x7fcdf80fbd20 msgr2=0x7fcdf8102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:07.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 shutdown_connections 2024-05-09T08:24:07.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:07.277+0000 7fcdfe1ce640 1 -- 172.21.15.28:0/3212732905 wait complete. 2024-05-09T08:24:07.288 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-05-09T08:24:07.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:07.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:24:07.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[71236]: fuse finished with error 0 and tester_r 0 2024-05-09T08:24:07.314 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T08:24:08.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2820891898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:24:08.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3212732905' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:24:08.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2820891898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:24:08.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3212732905' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:24:08.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2820891898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:24:08.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3212732905' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:24:09.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:08 smithi165 ceph-mon[34540]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-05-09T08:24:09.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:08 smithi184 ceph-mon[34509]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-05-09T08:24:09.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:08 smithi028 ceph-mon[30119]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-05-09T08:24:11.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:10 smithi165 ceph-mon[34540]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-09T08:24:11.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:10 smithi184 ceph-mon[34509]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-09T08:24:11.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:10 smithi028 ceph-mon[30119]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-09T08:24:13.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:12 smithi165 ceph-mon[34540]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:24:13.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:12 smithi184 ceph-mon[34509]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:24:13.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:12 smithi028 ceph-mon[30119]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:24:13.415 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:24:13.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:24:13.416 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-09T08:24:13.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:24:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.603+0000 7f3299970640 1 -- 172.21.15.28:0/4001362269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294101980 msgr2=0x7f3294101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.603+0000 7f3299970640 1 --2- 172.21.15.28:0/4001362269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294101980 0x7f3294101e00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3288009900 tx=0x7f328802a410 comp rx=0 tx=0).stop 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.604+0000 7f3299970640 1 -- 172.21.15.28:0/4001362269 shutdown_connections 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.604+0000 7f3299970640 1 --2- 172.21.15.28:0/4001362269 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3294102340 0x7f3294108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.604+0000 7f3299970640 1 --2- 172.21.15.28:0/4001362269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294101980 0x7f3294101e00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.604+0000 7f3299970640 1 --2- 172.21.15.28:0/4001362269 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3294100790 0x7f3294100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.604+0000 7f3299970640 1 -- 172.21.15.28:0/4001362269 >> 172.21.15.28:0/4001362269 conn(0x7f32940fbd20 msgr2=0x7f32940fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.604+0000 7f3299970640 1 -- 172.21.15.28:0/4001362269 shutdown_connections 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.604+0000 7f3299970640 1 -- 172.21.15.28:0/4001362269 wait complete. 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.604+0000 7f3299970640 1 Processor -- start 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.604+0000 7f3299970640 1 -- start start 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f3299970640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3294100790 0x7f3294197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f3299970640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3294101980 0x7f3294198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f3299970640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294102340 0x7f329419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f3299970640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f329410c410 con 0x7f3294102340 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f3299970640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f329410c290 con 0x7f3294100790 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f3299970640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f329410c590 con 0x7f3294101980 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294102340 0x7f329419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294102340 0x7f329419f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42570/0 (socket says 172.21.15.28:42570) 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32937fe640 1 -- 172.21.15.28:0/3144716918 learned_addr learned my addr 172.21.15.28:0/3144716918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f3292ffd640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3294100790 0x7f3294197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32927fc640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3294101980 0x7f3294198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32937fe640 1 -- 172.21.15.28:0/3144716918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3294101980 msgr2=0x7f3294198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32937fe640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3294101980 0x7f3294198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32937fe640 1 -- 172.21.15.28:0/3144716918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3294100790 msgr2=0x7f3294197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32937fe640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3294100790 0x7f3294197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32937fe640 1 -- 172.21.15.28:0/3144716918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329419f7f0 con 0x7f3294102340 2024-05-09T08:24:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.605+0000 7f32937fe640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294102340 0x7f329419f180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f328c00cad0 tx=0x7f328c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.606+0000 7f329896e640 1 -- 172.21.15.28:0/3144716918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c00ae20 con 0x7f3294102340 2024-05-09T08:24:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.606+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f329419fae0 con 0x7f3294102340 2024-05-09T08:24:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.606+0000 7f329896e640 1 -- 172.21.15.28:0/3144716918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f328c005020 con 0x7f3294102340 2024-05-09T08:24:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.606+0000 7f329896e640 1 -- 172.21.15.28:0/3144716918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c00f6f0 con 0x7f3294102340 2024-05-09T08:24:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.606+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32941a0000 con 0x7f3294102340 2024-05-09T08:24:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.607+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32941a0230 con 0x7f3294102340 2024-05-09T08:24:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.608+0000 7f329896e640 1 -- 172.21.15.28:0/3144716918 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f328c0051d0 con 0x7f3294102340 2024-05-09T08:24:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.608+0000 7f329896e640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f327c0a2b90 0x7f327c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.609+0000 7f329896e640 1 -- 172.21.15.28:0/3144716918 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f328c0cf5b0 con 0x7f3294102340 2024-05-09T08:24:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.609+0000 7f3292ffd640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f327c0a2b90 0x7f327c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.609+0000 7f3292ffd640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f327c0a2b90 0x7f327c0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3284005fd0 tx=0x7f328400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:13.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.612+0000 7f329896e640 1 -- 172.21.15.28:0/3144716918 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f328c014030 con 0x7f3294102340 2024-05-09T08:24:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.798+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32941a04a0 con 0x7f3294102340 2024-05-09T08:24:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.800+0000 7f329896e640 1 -- 172.21.15.28:0/3144716918 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7f328c09bc80 con 0x7f3294102340 2024-05-09T08:24:13.800 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:13.800 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:24:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f327c0a2b90 msgr2=0x7f327c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f327c0a2b90 0x7f327c0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3284005fd0 tx=0x7f328400b390 comp rx=0 tx=0).stop 2024-05-09T08:24:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294102340 msgr2=0x7f329419f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294102340 0x7f329419f180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f328c00cad0 tx=0x7f328c00cf90 comp rx=0 tx=0).stop 2024-05-09T08:24:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 shutdown_connections 2024-05-09T08:24:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f327c0a2b90 0x7f327c0a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3294102340 0x7f329419f180 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3294101980 0x7f3294198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 --2- 172.21.15.28:0/3144716918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3294100790 0x7f3294197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 >> 172.21.15.28:0/3144716918 conn(0x7f32940fbd20 msgr2=0x7f3294069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 shutdown_connections 2024-05-09T08:24:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.802+0000 7f3299970640 1 -- 172.21.15.28:0/3144716918 wait complete. 2024-05-09T08:24:13.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.919+0000 7f4029a6b640 1 -- 172.21.15.28:0/1062528938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 msgr2=0x7f4024100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.919+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1062528938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 0x7f4024100ba0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4014009900 tx=0x7f401402a410 comp rx=0 tx=0).stop 2024-05-09T08:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.919+0000 7f4029a6b640 1 -- 172.21.15.28:0/1062528938 shutdown_connections 2024-05-09T08:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.919+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1062528938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4024102330 0x7f4024108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.919+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1062528938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4024101970 0x7f4024101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.919+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1062528938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 0x7f4024100ba0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.919+0000 7f4029a6b640 1 -- 172.21.15.28:0/1062528938 >> 172.21.15.28:0/1062528938 conn(0x7f40240fbd10 msgr2=0x7f40240fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.919+0000 7f4029a6b640 1 -- 172.21.15.28:0/1062528938 shutdown_connections 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.919+0000 7f4029a6b640 1 -- 172.21.15.28:0/1062528938 wait complete. 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f4029a6b640 1 Processor -- start 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f4029a6b640 1 -- start start 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f4029a6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 0x7f4024196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f4029a6b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4024101970 0x7f4024196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f4029a6b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4024102330 0x7f402419e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f4029a6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f402410c400 con 0x7f4024100780 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f4029a6b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f402410c280 con 0x7f4024102330 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f4029a6b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f402410c580 con 0x7f4024101970 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f40237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 0x7f4024196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f40237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 0x7f4024196aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42586/0 (socket says 172.21.15.28:42586) 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f40237fe640 1 -- 172.21.15.28:0/1906597139 learned_addr learned my addr 172.21.15.28:0/1906597139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.920+0000 7f40237fe640 1 -- 172.21.15.28:0/1906597139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4024101970 msgr2=0x7f4024196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f40237fe640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4024101970 0x7f4024196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f40237fe640 1 -- 172.21.15.28:0/1906597139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4024102330 msgr2=0x7f402419e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f40237fe640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4024102330 0x7f402419e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f40237fe640 1 -- 172.21.15.28:0/1906597139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402419e6c0 con 0x7f4024100780 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f40237fe640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 0x7f4024196aa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4014031610 tx=0x7f40140043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f4020ff9640 1 -- 172.21.15.28:0/1906597139 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401403f030 con 0x7f4024100780 2024-05-09T08:24:13.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f402419e950 con 0x7f4024100780 2024-05-09T08:24:13.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f4020ff9640 1 -- 172.21.15.28:0/1906597139 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4014003ce0 con 0x7f4024100780 2024-05-09T08:24:13.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f4020ff9640 1 -- 172.21.15.28:0/1906597139 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4014035070 con 0x7f4024100780 2024-05-09T08:24:13.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.921+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f402419ee90 con 0x7f4024100780 2024-05-09T08:24:13.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.922+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4024107aa0 con 0x7f4024100780 2024-05-09T08:24:13.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.923+0000 7f4020ff9640 1 -- 172.21.15.28:0/1906597139 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4014042030 con 0x7f4024100780 2024-05-09T08:24:13.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.924+0000 7f4020ff9640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f400c0a2c10 0x7f400c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.924+0000 7f4022ffd640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f400c0a2c10 0x7f400c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:13.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.924+0000 7f4020ff9640 1 -- 172.21.15.28:0/1906597139 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f40140ef570 con 0x7f4024100780 2024-05-09T08:24:13.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.925+0000 7f4022ffd640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f400c0a2c10 0x7f400c0a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4024197fb0 tx=0x7f4018005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:13.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:13.929+0000 7f4020ff9640 1 -- 172.21.15.28:0/1906597139 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f40140bc230 con 0x7f4024100780 2024-05-09T08:24:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.114+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4024062430 con 0x7f4024100780 2024-05-09T08:24:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.116+0000 7f4020ff9640 1 -- 172.21.15.28:0/1906597139 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7f40140bbb30 con 0x7f4024100780 2024-05-09T08:24:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:14.116 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:24:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f400c0a2c10 msgr2=0x7f400c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f400c0a2c10 0x7f400c0a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4024197fb0 tx=0x7f4018005ed0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 msgr2=0x7f4024196aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 0x7f4024196aa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4014031610 tx=0x7f40140043b0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 shutdown_connections 2024-05-09T08:24:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f400c0a2c10 0x7f400c0a50d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4024102330 0x7f402419e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4024101970 0x7f4024196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 --2- 172.21.15.28:0/1906597139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4024100780 0x7f4024196aa0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 >> 172.21.15.28:0/1906597139 conn(0x7f40240fbd10 msgr2=0x7f4024104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 shutdown_connections 2024-05-09T08:24:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.118+0000 7f4029a6b640 1 -- 172.21.15.28:0/1906597139 wait complete. 2024-05-09T08:24:14.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T08:24:14.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.238+0000 7f151f0b3640 1 -- 172.21.15.28:0/2277224518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518101980 msgr2=0x7f1518101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.238+0000 7f151f0b3640 1 --2- 172.21.15.28:0/2277224518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518101980 0x7f1518101e00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1504009900 tx=0x7f150402a410 comp rx=0 tx=0).stop 2024-05-09T08:24:14.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.238+0000 7f151f0b3640 1 -- 172.21.15.28:0/2277224518 shutdown_connections 2024-05-09T08:24:14.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.238+0000 7f151f0b3640 1 --2- 172.21.15.28:0/2277224518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1518102340 0x7f1518108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.238+0000 7f151f0b3640 1 --2- 172.21.15.28:0/2277224518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518101980 0x7f1518101e00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.238+0000 7f151f0b3640 1 --2- 172.21.15.28:0/2277224518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1518100790 0x7f1518100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.238+0000 7f151f0b3640 1 -- 172.21.15.28:0/2277224518 >> 172.21.15.28:0/2277224518 conn(0x7f15180fbd20 msgr2=0x7f15180fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.238+0000 7f151f0b3640 1 -- 172.21.15.28:0/2277224518 shutdown_connections 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.238+0000 7f151f0b3640 1 -- 172.21.15.28:0/2277224518 wait complete. 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151f0b3640 1 Processor -- start 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151f0b3640 1 -- start start 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151f0b3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1518100790 0x7f1518197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151f0b3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1518101980 0x7f1518198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151f0b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518102340 0x7f151819f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151f0b3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f151810c3f0 con 0x7f1518102340 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151f0b3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f151810c270 con 0x7f1518101980 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151f0b3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f151810c570 con 0x7f1518100790 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151d650640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518102340 0x7f151819f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151d650640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518102340 0x7f151819f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42608/0 (socket says 172.21.15.28:42608) 2024-05-09T08:24:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151d650640 1 -- 172.21.15.28:0/215150616 learned_addr learned my addr 172.21.15.28:0/215150616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f1517fff640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1518101980 0x7f1518198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.239+0000 7f151ce4f640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1518100790 0x7f1518197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f151d650640 1 -- 172.21.15.28:0/215150616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1518100790 msgr2=0x7f1518197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f151d650640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1518100790 0x7f1518197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f151d650640 1 -- 172.21.15.28:0/215150616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1518101980 msgr2=0x7f1518198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f151d650640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1518101980 0x7f1518198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f151d650640 1 -- 172.21.15.28:0/215150616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151819f7e0 con 0x7f1518102340 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f151ce4f640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1518100790 0x7f1518197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f1517fff640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1518101980 0x7f1518198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f151d650640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518102340 0x7f151819f170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f151000eaa0 tx=0x7f151000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f1515ffb640 1 -- 172.21.15.28:0/215150616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151000cdb0 con 0x7f1518102340 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f1515ffb640 1 -- 172.21.15.28:0/215150616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1510005020 con 0x7f1518102340 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151819fad0 con 0x7f1518102340 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f1515ffb640 1 -- 172.21.15.28:0/215150616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15100116b0 con 0x7f1518102340 2024-05-09T08:24:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.240+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151819ffa0 con 0x7f1518102340 2024-05-09T08:24:14.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.241+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15181a0290 con 0x7f1518102340 2024-05-09T08:24:14.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.242+0000 7f1515ffb640 1 -- 172.21.15.28:0/215150616 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1510005560 con 0x7f1518102340 2024-05-09T08:24:14.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.243+0000 7f1515ffb640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f15080a2b90 0x7f15080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:14.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.243+0000 7f1515ffb640 1 -- 172.21.15.28:0/215150616 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f15100cf520 con 0x7f1518102340 2024-05-09T08:24:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.243+0000 7f151ce4f640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f15080a2b90 0x7f15080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.244+0000 7f151ce4f640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f15080a2b90 0x7f15080a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f150c005d00 tx=0x7f150c005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.247+0000 7f1515ffb640 1 -- 172.21.15.28:0/215150616 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1510016030 con 0x7f1518102340 2024-05-09T08:24:14.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.474+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f15181a04c0 con 0x7f1518102340 2024-05-09T08:24:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.696+0000 7f1515ffb640 1 -- 172.21.15.28:0/215150616 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f151009c240 con 0x7f1518102340 2024-05-09T08:24:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f15080a2b90 msgr2=0x7f15080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f15080a2b90 0x7f15080a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f150c005d00 tx=0x7f150c005c90 comp rx=0 tx=0).stop 2024-05-09T08:24:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518102340 msgr2=0x7f151819f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518102340 0x7f151819f170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f151000eaa0 tx=0x7f151000ef60 comp rx=0 tx=0).stop 2024-05-09T08:24:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 shutdown_connections 2024-05-09T08:24:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f15080a2b90 0x7f15080a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1518102340 0x7f151819f170 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1518101980 0x7f1518198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 --2- 172.21.15.28:0/215150616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1518100790 0x7f1518197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 >> 172.21.15.28:0/215150616 conn(0x7f15180fbd20 msgr2=0x7f15180fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 shutdown_connections 2024-05-09T08:24:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.699+0000 7f151f0b3640 1 -- 172.21.15.28:0/215150616 wait complete. 2024-05-09T08:24:14.711 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-05-09T08:24:14.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-05-09T08:24:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.820+0000 7f899fa31640 1 -- 172.21.15.28:0/1271737101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998100790 msgr2=0x7f8998100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.820+0000 7f899fa31640 1 --2- 172.21.15.28:0/1271737101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998100790 0x7f8998100bb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8990009900 tx=0x7f899002a410 comp rx=0 tx=0).stop 2024-05-09T08:24:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 -- 172.21.15.28:0/1271737101 shutdown_connections 2024-05-09T08:24:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 --2- 172.21.15.28:0/1271737101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8998102340 0x7f8998108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 --2- 172.21.15.28:0/1271737101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998101980 0x7f8998101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 --2- 172.21.15.28:0/1271737101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998100790 0x7f8998100bb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 -- 172.21.15.28:0/1271737101 >> 172.21.15.28:0/1271737101 conn(0x7f89980fbd20 msgr2=0x7f89980fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 -- 172.21.15.28:0/1271737101 shutdown_connections 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 -- 172.21.15.28:0/1271737101 wait complete. 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 Processor -- start 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 -- start start 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.821+0000 7f899fa31640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8998100790 0x7f899806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899fa31640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998101980 0x7f899819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899fa31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998102340 0x7f899819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899fa31640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f899810c460 con 0x7f8998102340 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899fa31640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f899810c2e0 con 0x7f8998101980 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899fa31640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f899810c5e0 con 0x7f8998100790 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899dfce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998102340 0x7f899819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899dfce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998102340 0x7f899819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35540/0 (socket says 172.21.15.28:35540) 2024-05-09T08:24:14.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899dfce640 1 -- 172.21.15.28:0/3040899379 learned_addr learned my addr 172.21.15.28:0/3040899379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899cfcc640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998101980 0x7f899819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899d7cd640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8998100790 0x7f899806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899dfce640 1 -- 172.21.15.28:0/3040899379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8998100790 msgr2=0x7f899806a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899dfce640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8998100790 0x7f899806a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899dfce640 1 -- 172.21.15.28:0/3040899379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998101980 msgr2=0x7f899819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899dfce640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998101980 0x7f899819c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899dfce640 1 -- 172.21.15.28:0/3040899379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f899819f830 con 0x7f8998102340 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899d7cd640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8998100790 0x7f899806a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899cfcc640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998101980 0x7f899819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899dfce640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998102340 0x7f899819ef70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f898800c970 tx=0x7f898800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f89967fc640 1 -- 172.21.15.28:0/3040899379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8988007bb0 con 0x7f8998102340 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f89967fc640 1 -- 172.21.15.28:0/3040899379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8988007d50 con 0x7f8998102340 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f899819fb20 con 0x7f8998102340 2024-05-09T08:24:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f89967fc640 1 -- 172.21.15.28:0/3040899379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898800f450 con 0x7f8998102340 2024-05-09T08:24:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.822+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89981a00c0 con 0x7f8998102340 2024-05-09T08:24:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.823+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8998107ab0 con 0x7f8998102340 2024-05-09T08:24:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.824+0000 7f89967fc640 1 -- 172.21.15.28:0/3040899379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8988004f20 con 0x7f8998102340 2024-05-09T08:24:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.825+0000 7f89967fc640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89800a2c10 0x7f89800a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.825+0000 7f899d7cd640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89800a2c10 0x7f89800a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.825+0000 7f89967fc640 1 -- 172.21.15.28:0/3040899379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f89880cefb0 con 0x7f8998102340 2024-05-09T08:24:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.826+0000 7f899d7cd640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89800a2c10 0x7f89800a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f899002a930 tx=0x7f8990000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:14.829+0000 7f89967fc640 1 -- 172.21.15.28:0/3040899379 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8988014030 con 0x7f8998102340 2024-05-09T08:24:15.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.055+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f8998062430 con 0x7f8998102340 2024-05-09T08:24:15.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:14 smithi165 ceph-mon[34540]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:24:15.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:14 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3144716918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:15.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:14 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1906597139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:15.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:14 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/215150616' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T08:24:15.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:14 smithi184 ceph-mon[34509]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:24:15.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:14 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3144716918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:15.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:14 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1906597139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:15.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:14 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/215150616' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T08:24:15.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:14 smithi028 ceph-mon[30119]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:24:15.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:14 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3144716918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:15.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:14 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1906597139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:15.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:14 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/215150616' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T08:24:15.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.367+0000 7f89967fc640 1 -- 172.21.15.28:0/3040899379 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f89880bb070 con 0x7f8998102340 2024-05-09T08:24:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89800a2c10 msgr2=0x7f89800a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89800a2c10 0x7f89800a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f899002a930 tx=0x7f8990000bc0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998102340 msgr2=0x7f899819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998102340 0x7f899819ef70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f898800c970 tx=0x7f898800ce30 comp rx=0 tx=0).stop 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 shutdown_connections 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89800a2c10 0x7f89800a50d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8998102340 0x7f899819ef70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998101980 0x7f899819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 --2- 172.21.15.28:0/3040899379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8998100790 0x7f899806a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 >> 172.21.15.28:0/3040899379 conn(0x7f89980fbd20 msgr2=0x7f8998104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 shutdown_connections 2024-05-09T08:24:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.369+0000 7f899fa31640 1 -- 172.21.15.28:0/3040899379 wait complete. 2024-05-09T08:24:15.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-05-09T08:24:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.494+0000 7fb0c4aca640 1 -- 172.21.15.28:0/2176560950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 msgr2=0x7fb0c0108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.494+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/2176560950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 0x7fb0c0108bb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009900 tx=0x7fb0b802a410 comp rx=0 tx=0).stop 2024-05-09T08:24:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.494+0000 7fb0c4aca640 1 -- 172.21.15.28:0/2176560950 shutdown_connections 2024-05-09T08:24:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.494+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/2176560950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 0x7fb0c0108bb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.494+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/2176560950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0101960 0x7fb0c0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.494+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/2176560950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0c0100770 0x7fb0c0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.494+0000 7fb0c4aca640 1 -- 172.21.15.28:0/2176560950 >> 172.21.15.28:0/2176560950 conn(0x7fb0c00fbce0 msgr2=0x7fb0c00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 -- 172.21.15.28:0/2176560950 shutdown_connections 2024-05-09T08:24:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 -- 172.21.15.28:0/2176560950 wait complete. 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 Processor -- start 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 -- start start 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0100770 0x7fb0c0069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0c0101960 0x7fb0c019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 0x7fb0c019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010c460 con 0x7fb0c0102320 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010c2e0 con 0x7fb0c0100770 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.495+0000 7fb0c4aca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010c5e0 con 0x7fb0c0101960 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 0x7fb0c019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 0x7fb0c019eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35560/0 (socket says 172.21.15.28:35560) 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bed9d640 1 -- 172.21.15.28:0/4148863803 learned_addr learned my addr 172.21.15.28:0/4148863803 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0be59c640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0100770 0x7fb0c0069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bdd9b640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0c0101960 0x7fb0c019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bed9d640 1 -- 172.21.15.28:0/4148863803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0c0101960 msgr2=0x7fb0c019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bed9d640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0c0101960 0x7fb0c019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bed9d640 1 -- 172.21.15.28:0/4148863803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0100770 msgr2=0x7fb0c0069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bed9d640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0100770 0x7fb0c0069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bed9d640 1 -- 172.21.15.28:0/4148863803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c019f7c0 con 0x7fb0c0102320 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0bed9d640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 0x7fb0c019eeb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80314e0 tx=0x7fb0b80043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0b77fe640 1 -- 172.21.15.28:0/4148863803 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b803f030 con 0x7fb0c0102320 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c019f9f0 con 0x7fb0c0102320 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0b77fe640 1 -- 172.21.15.28:0/4148863803 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b8003ce0 con 0x7fb0c0102320 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0b77fe640 1 -- 172.21.15.28:0/4148863803 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8035070 con 0x7fb0c0102320 2024-05-09T08:24:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.496+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c019ffb0 con 0x7fb0c0102320 2024-05-09T08:24:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.497+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c0107a90 con 0x7fb0c0102320 2024-05-09T08:24:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.498+0000 7fb0b77fe640 1 -- 172.21.15.28:0/4148863803 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb0b8042030 con 0x7fb0c0102320 2024-05-09T08:24:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.499+0000 7fb0b77fe640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0ac0a2b90 0x7fb0ac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.499+0000 7fb0be59c640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0ac0a2b90 0x7fb0ac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.499+0000 7fb0b77fe640 1 -- 172.21.15.28:0/4148863803 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb0b80efb00 con 0x7fb0c0102320 2024-05-09T08:24:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.500+0000 7fb0be59c640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0ac0a2b90 0x7fb0ac0a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb0c01017c0 tx=0x7fb0b000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.504+0000 7fb0b77fe640 1 -- 172.21.15.28:0/4148863803 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb0b80bc8e0 con 0x7fb0c0102320 2024-05-09T08:24:15.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:15.734+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fb0c002d1c0 con 0x7fb0c0102320 2024-05-09T08:24:16.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:15 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/215150616' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T08:24:16.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:15 smithi165 ceph-mon[34540]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:24:16.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:15 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3040899379' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-09T08:24:16.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:15 smithi165 ceph-mon[34540]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T08:24:16.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:15 smithi165 ceph-mon[34540]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T08:24:16.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:15 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3040899379' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-09T08:24:16.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:15 smithi165 ceph-mon[34540]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-05-09T08:24:16.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:15 smithi165 ceph-mon[34540]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-09T08:24:16.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:15 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/215150616' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T08:24:16.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:15 smithi184 ceph-mon[34509]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:24:16.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:15 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3040899379' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-09T08:24:16.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:15 smithi184 ceph-mon[34509]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T08:24:16.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:15 smithi184 ceph-mon[34509]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T08:24:16.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:15 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3040899379' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-09T08:24:16.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:15 smithi184 ceph-mon[34509]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-05-09T08:24:16.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:15 smithi184 ceph-mon[34509]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-09T08:24:16.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:15 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/215150616' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T08:24:16.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:15 smithi028 ceph-mon[30119]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T08:24:16.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:15 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3040899379' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-09T08:24:16.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:15 smithi028 ceph-mon[30119]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T08:24:16.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:15 smithi028 ceph-mon[30119]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T08:24:16.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:15 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3040899379' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-09T08:24:16.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:15 smithi028 ceph-mon[30119]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-05-09T08:24:16.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:15 smithi028 ceph-mon[30119]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-09T08:24:16.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.371+0000 7fb0b77fe640 1 -- 172.21.15.28:0/4148863803 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7fb0b80bc1e0 con 0x7fb0c0102320 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0ac0a2b90 msgr2=0x7fb0ac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0ac0a2b90 0x7fb0ac0a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb0c01017c0 tx=0x7fb0b000d040 comp rx=0 tx=0).stop 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 msgr2=0x7fb0c019eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 0x7fb0c019eeb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80314e0 tx=0x7fb0b80043c0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 shutdown_connections 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb0ac0a2b90 0x7fb0ac0a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c0102320 0x7fb0c019eeb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0c0101960 0x7fb0c019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 --2- 172.21.15.28:0/4148863803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0100770 0x7fb0c0069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 >> 172.21.15.28:0/4148863803 conn(0x7fb0c00fbce0 msgr2=0x7fb0c0104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:16.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 shutdown_connections 2024-05-09T08:24:16.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.374+0000 7fb0c4aca640 1 -- 172.21.15.28:0/4148863803 wait complete. 2024-05-09T08:24:16.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-09T08:24:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.503+0000 7f482e7a9640 1 -- 172.21.15.28:0/669521547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828101980 msgr2=0x7f4828101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.503+0000 7f482e7a9640 1 --2- 172.21.15.28:0/669521547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828101980 0x7f4828101e00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f481c009900 tx=0x7f481c02a410 comp rx=0 tx=0).stop 2024-05-09T08:24:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.503+0000 7f482e7a9640 1 -- 172.21.15.28:0/669521547 shutdown_connections 2024-05-09T08:24:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.503+0000 7f482e7a9640 1 --2- 172.21.15.28:0/669521547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4828102340 0x7f4828108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.503+0000 7f482e7a9640 1 --2- 172.21.15.28:0/669521547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828101980 0x7f4828101e00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.503+0000 7f482e7a9640 1 --2- 172.21.15.28:0/669521547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4828100790 0x7f4828100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.503+0000 7f482e7a9640 1 -- 172.21.15.28:0/669521547 >> 172.21.15.28:0/669521547 conn(0x7f48280fbd20 msgr2=0x7f48280fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.503+0000 7f482e7a9640 1 -- 172.21.15.28:0/669521547 shutdown_connections 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.503+0000 7f482e7a9640 1 -- 172.21.15.28:0/669521547 wait complete. 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482e7a9640 1 Processor -- start 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482e7a9640 1 -- start start 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482e7a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828100790 0x7f4828104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482e7a9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4828101980 0x7f4828102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482e7a9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4828102340 0x7f48281032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482e7a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f482810c460 con 0x7f4828100790 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482e7a9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f482810c2e0 con 0x7f4828102340 2024-05-09T08:24:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482e7a9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f482810c5e0 con 0x7f4828101980 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f4827fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828100790 0x7f4828104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f4827fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828100790 0x7f4828104720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35576/0 (socket says 172.21.15.28:35576) 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f4827fff640 1 -- 172.21.15.28:0/1939977135 learned_addr learned my addr 172.21.15.28:0/1939977135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482cd46640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4828102340 0x7f48281032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f48277fe640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4828101980 0x7f4828102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f4827fff640 1 -- 172.21.15.28:0/1939977135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4828101980 msgr2=0x7f4828102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f4827fff640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4828101980 0x7f4828102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f4827fff640 1 -- 172.21.15.28:0/1939977135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4828102340 msgr2=0x7f48281032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f4827fff640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4828102340 0x7f48281032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f4827fff640 1 -- 172.21.15.28:0/1939977135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4828103b30 con 0x7f4828100790 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.504+0000 7f482cd46640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4828102340 0x7f48281032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.505+0000 7f4827fff640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828100790 0x7f4828104720 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f481800d950 tx=0x7f481800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.505+0000 7f48257fa640 1 -- 172.21.15.28:0/1939977135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4818017070 con 0x7f4828100790 2024-05-09T08:24:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.505+0000 7f48257fa640 1 -- 172.21.15.28:0/1939977135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4818009c30 con 0x7f4828100790 2024-05-09T08:24:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.505+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48281a1cd0 con 0x7f4828100790 2024-05-09T08:24:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.505+0000 7f48257fa640 1 -- 172.21.15.28:0/1939977135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481800f600 con 0x7f4828100790 2024-05-09T08:24:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.505+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48281a21a0 con 0x7f4828100790 2024-05-09T08:24:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.506+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48281a2370 con 0x7f4828100790 2024-05-09T08:24:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.507+0000 7f48257fa640 1 -- 172.21.15.28:0/1939977135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f481801a070 con 0x7f4828100790 2024-05-09T08:24:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.508+0000 7f48257fa640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f48100a2c10 0x7f48100a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.508+0000 7f48277fe640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f48100a2c10 0x7f48100a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.508+0000 7f48257fa640 1 -- 172.21.15.28:0/1939977135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f48180cec10 con 0x7f4828100790 2024-05-09T08:24:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.508+0000 7f48277fe640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f48100a2c10 0x7f48100a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f481c031440 tx=0x7f481c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:16.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.512+0000 7f48257fa640 1 -- 172.21.15.28:0/1939977135 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4818009de0 con 0x7f4828100790 2024-05-09T08:24:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:16.740+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f48281a2640 con 0x7f4828100790 2024-05-09T08:24:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:16 smithi165 ceph-mon[34540]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:24:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:16 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4148863803' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T08:24:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:16 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4148863803' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T08:24:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:16 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] up:active 2024-05-09T08:24:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:16 smithi165 ceph-mon[34540]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T08:24:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:16 smithi165 ceph-mon[34540]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-05-09T08:24:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:16 smithi165 ceph-mon[34540]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-05-09T08:24:17.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:16 smithi165 ceph-mon[34540]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-09T08:24:17.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:16 smithi184 ceph-mon[34509]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:24:17.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:16 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4148863803' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T08:24:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:16 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4148863803' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T08:24:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:16 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] up:active 2024-05-09T08:24:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:16 smithi184 ceph-mon[34509]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T08:24:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:16 smithi184 ceph-mon[34509]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-05-09T08:24:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:16 smithi184 ceph-mon[34509]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-05-09T08:24:17.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:16 smithi184 ceph-mon[34509]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-09T08:24:17.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:16 smithi028 ceph-mon[30119]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:24:17.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:16 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4148863803' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T08:24:17.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:16 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4148863803' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T08:24:17.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:16 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] up:active 2024-05-09T08:24:17.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:16 smithi028 ceph-mon[30119]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T08:24:17.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:16 smithi028 ceph-mon[30119]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-05-09T08:24:17.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:16 smithi028 ceph-mon[30119]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-05-09T08:24:17.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:16 smithi028 ceph-mon[30119]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-09T08:24:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.378+0000 7f48257fa640 1 -- 172.21.15.28:0/1939977135 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f481809b280 con 0x7f4828100790 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.381+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f48100a2c10 msgr2=0x7f48100a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.381+0000 7f482e7a9640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f48100a2c10 0x7f48100a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f481c031440 tx=0x7f481c0315b0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.381+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828100790 msgr2=0x7f4828104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.381+0000 7f482e7a9640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828100790 0x7f4828104720 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f481800d950 tx=0x7f481800de10 comp rx=0 tx=0).stop 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.381+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 shutdown_connections 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.382+0000 7f482e7a9640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f48100a2c10 0x7f48100a50d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.382+0000 7f482e7a9640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4828102340 0x7f48281032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.382+0000 7f482e7a9640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4828101980 0x7f4828102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.382+0000 7f482e7a9640 1 --2- 172.21.15.28:0/1939977135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828100790 0x7f4828104720 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.382+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 >> 172.21.15.28:0/1939977135 conn(0x7f48280fbd20 msgr2=0x7f48280fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.382+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 shutdown_connections 2024-05-09T08:24:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.382+0000 7f482e7a9640 1 -- 172.21.15.28:0/1939977135 wait complete. 2024-05-09T08:24:17.400 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-05-09T08:24:17.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-05-09T08:24:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.514+0000 7fb59ffd5640 1 -- 172.21.15.28:0/1570099517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598100790 msgr2=0x7fb598100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.514+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/1570099517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598100790 0x7fb598100bb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb594009900 tx=0x7fb59402a3f0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.514+0000 7fb59ffd5640 1 -- 172.21.15.28:0/1570099517 shutdown_connections 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.514+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/1570099517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb598102340 0x7fb598108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.514+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/1570099517 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb598101980 0x7fb598101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.514+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/1570099517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598100790 0x7fb598100bb0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.514+0000 7fb59ffd5640 1 -- 172.21.15.28:0/1570099517 >> 172.21.15.28:0/1570099517 conn(0x7fb5980fbd20 msgr2=0x7fb5980fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.514+0000 7fb59ffd5640 1 -- 172.21.15.28:0/1570099517 shutdown_connections 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.514+0000 7fb59ffd5640 1 -- 172.21.15.28:0/1570099517 wait complete. 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59ffd5640 1 Processor -- start 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59ffd5640 1 -- start start 2024-05-09T08:24:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59ffd5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb598100790 0x7fb598197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59ffd5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb598101980 0x7fb5981981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59ffd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598102340 0x7fb59819f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59ffd5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb59810c460 con 0x7fb598102340 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59ffd5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb59810c2e0 con 0x7fb598100790 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59ffd5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb59810c5e0 con 0x7fb598101980 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59dd71640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb598100790 0x7fb598197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59e572640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598102340 0x7fb59819f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59dd71640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb598100790 0x7fb598197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:38270/0 (socket says 172.21.15.28:38270) 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59dd71640 1 -- 172.21.15.28:0/4053201700 learned_addr learned my addr 172.21.15.28:0/4053201700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59d570640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb598101980 0x7fb5981981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59e572640 1 -- 172.21.15.28:0/4053201700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb598101980 msgr2=0x7fb5981981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59e572640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb598101980 0x7fb5981981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59e572640 1 -- 172.21.15.28:0/4053201700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb598100790 msgr2=0x7fb598197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59e572640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb598100790 0x7fb598197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59e572640 1 -- 172.21.15.28:0/4053201700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59819f8d0 con 0x7fb598102340 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59d570640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb598101980 0x7fb5981981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.515+0000 7fb59dd71640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb598100790 0x7fb598197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.516+0000 7fb59e572640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598102340 0x7fb59819f230 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb59000be20 tx=0x7fb59000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.516+0000 7fb58effd640 1 -- 172.21.15.28:0/4053201700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb590019070 con 0x7fb598102340 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.516+0000 7fb58effd640 1 -- 172.21.15.28:0/4053201700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb590014920 con 0x7fb598102340 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.516+0000 7fb58effd640 1 -- 172.21.15.28:0/4053201700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb590012920 con 0x7fb598102340 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.516+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb59819fbc0 con 0x7fb598102340 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.516+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5981a0160 con 0x7fb598102340 2024-05-09T08:24:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.517+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5981a0390 con 0x7fb598102340 2024-05-09T08:24:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.518+0000 7fb58effd640 1 -- 172.21.15.28:0/4053201700 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb59001c070 con 0x7fb598102340 2024-05-09T08:24:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.519+0000 7fb58effd640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb5840a2c10 0x7fb5840a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.519+0000 7fb59dd71640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb5840a2c10 0x7fb5840a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.519+0000 7fb58effd640 1 -- 172.21.15.28:0/4053201700 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb5900cea40 con 0x7fb598102340 2024-05-09T08:24:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.519+0000 7fb59dd71640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb5840a2c10 0x7fb5840a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb5940314e0 tx=0x7fb594000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.523+0000 7fb58effd640 1 -- 172.21.15.28:0/4053201700 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb59009b0b0 con 0x7fb598102340 2024-05-09T08:24:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.735+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb5981a0560 con 0x7fb5840a2c10 2024-05-09T08:24:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.759+0000 7fb58effd640 1 -- 172.21.15.28:0/4053201700 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb5981a0560 con 0x7fb5840a2c10 2024-05-09T08:24:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb5840a2c10 msgr2=0x7fb5840a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb5840a2c10 0x7fb5840a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb5940314e0 tx=0x7fb594000bc0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598102340 msgr2=0x7fb59819f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598102340 0x7fb59819f230 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb59000be20 tx=0x7fb59000bf10 comp rx=0 tx=0).stop 2024-05-09T08:24:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 shutdown_connections 2024-05-09T08:24:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb5840a2c10 0x7fb5840a50d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598102340 0x7fb59819f230 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb598101980 0x7fb5981981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 --2- 172.21.15.28:0/4053201700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb598100790 0x7fb598197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 >> 172.21.15.28:0/4053201700 conn(0x7fb5980fbd20 msgr2=0x7fb5980fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 shutdown_connections 2024-05-09T08:24:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.761+0000 7fb59ffd5640 1 -- 172.21.15.28:0/4053201700 wait complete. 2024-05-09T08:24:17.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-09T08:24:17.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.883+0000 7f8a53722640 1 -- 172.21.15.28:0/3700003362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 msgr2=0x7f8a4c101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.883+0000 7f8a53722640 1 --2- 172.21.15.28:0/3700003362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 0x7f8a4c101cb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009900 tx=0x7f8a4802a410 comp rx=0 tx=0).stop 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.883+0000 7f8a53722640 1 -- 172.21.15.28:0/3700003362 shutdown_connections 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.883+0000 7f8a53722640 1 --2- 172.21.15.28:0/3700003362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a4c103440 0x7f8a4c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.883+0000 7f8a53722640 1 --2- 172.21.15.28:0/3700003362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c102a80 0x7f8a4c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.883+0000 7f8a53722640 1 --2- 172.21.15.28:0/3700003362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 0x7f8a4c101cb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.883+0000 7f8a53722640 1 -- 172.21.15.28:0/3700003362 >> 172.21.15.28:0/3700003362 conn(0x7f8a4c0fce00 msgr2=0x7f8a4c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.883+0000 7f8a53722640 1 -- 172.21.15.28:0/3700003362 shutdown_connections 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.883+0000 7f8a53722640 1 -- 172.21.15.28:0/3700003362 wait complete. 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a53722640 1 Processor -- start 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a53722640 1 -- start start 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a53722640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 0x7f8a4c0755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a53722640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a4c102a80 0x7f8a4c073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:17.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a53722640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c103440 0x7f8a4c074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a53722640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c10d500 con 0x7f8a4c101890 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a53722640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c10d380 con 0x7f8a4c102a80 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a53722640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c10d680 con 0x7f8a4c103440 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a514be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 0x7f8a4c0755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a514be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 0x7f8a4c0755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35610/0 (socket says 172.21.15.28:35610) 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a514be640 1 -- 172.21.15.28:0/332983704 learned_addr learned my addr 172.21.15.28:0/332983704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a50cbd640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a4c102a80 0x7f8a4c073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a51cbf640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c103440 0x7f8a4c074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a514be640 1 -- 172.21.15.28:0/332983704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c103440 msgr2=0x7f8a4c074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a514be640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c103440 0x7f8a4c074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a514be640 1 -- 172.21.15.28:0/332983704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a4c102a80 msgr2=0x7f8a4c073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a514be640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a4c102a80 0x7f8a4c073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a514be640 1 -- 172.21.15.28:0/332983704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c0749f0 con 0x7f8a4c101890 2024-05-09T08:24:17.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.884+0000 7f8a51cbf640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c103440 0x7f8a4c074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:17.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.885+0000 7f8a514be640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 0x7f8a4c0755e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8a48031610 tx=0x7f8a48004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:17.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.885+0000 7f8a427fc640 1 -- 172.21.15.28:0/332983704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4803f030 con 0x7f8a4c101890 2024-05-09T08:24:17.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.885+0000 7f8a427fc640 1 -- 172.21.15.28:0/332983704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a48003ad0 con 0x7f8a4c101890 2024-05-09T08:24:17.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.885+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c1a1c70 con 0x7f8a4c101890 2024-05-09T08:24:17.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.885+0000 7f8a427fc640 1 -- 172.21.15.28:0/332983704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48035070 con 0x7f8a4c101890 2024-05-09T08:24:17.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.885+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c1a21c0 con 0x7f8a4c101890 2024-05-09T08:24:17.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.886+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4c108c30 con 0x7f8a4c101890 2024-05-09T08:24:17.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.887+0000 7f8a427fc640 1 -- 172.21.15.28:0/332983704 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8a48042030 con 0x7f8a4c101890 2024-05-09T08:24:17.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.888+0000 7f8a427fc640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a380a2b90 0x7f8a380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:17.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.888+0000 7f8a50cbd640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a380a2b90 0x7f8a380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:17.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.888+0000 7f8a427fc640 1 -- 172.21.15.28:0/332983704 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8a480ef6d0 con 0x7f8a4c101890 2024-05-09T08:24:17.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.888+0000 7f8a50cbd640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a380a2b90 0x7f8a380a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c075350 tx=0x7f8a3c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:17.892+0000 7f8a427fc640 1 -- 172.21.15.28:0/332983704 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8a480bc390 con 0x7f8a4c101890 2024-05-09T08:24:18.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:17 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1939977135' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T08:24:18.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:17 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1939977135' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T08:24:18.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:17 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] up:active 2024-05-09T08:24:18.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:17 smithi165 ceph-mon[34540]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-09T08:24:18.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:17 smithi165 ceph-mon[34540]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-05-09T08:24:18.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:17 smithi165 ceph-mon[34540]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-05-09T08:24:18.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:17 smithi165 ceph-mon[34540]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-05-09T08:24:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.104+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f8a4c061d00 con 0x7f8a380a2b90 2024-05-09T08:24:18.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:17 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1939977135' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T08:24:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:17 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1939977135' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T08:24:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:17 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] up:active 2024-05-09T08:24:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:17 smithi184 ceph-mon[34509]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-09T08:24:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:17 smithi184 ceph-mon[34509]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-05-09T08:24:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:17 smithi184 ceph-mon[34509]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-05-09T08:24:18.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:17 smithi184 ceph-mon[34509]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-05-09T08:24:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.128+0000 7f8a427fc640 1 -- 172.21.15.28:0/332983704 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8a4c061d00 con 0x7f8a380a2b90 2024-05-09T08:24:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a380a2b90 msgr2=0x7f8a380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a380a2b90 0x7f8a380a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c075350 tx=0x7f8a3c00b390 comp rx=0 tx=0).stop 2024-05-09T08:24:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 msgr2=0x7f8a4c0755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 0x7f8a4c0755e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8a48031610 tx=0x7f8a48004270 comp rx=0 tx=0).stop 2024-05-09T08:24:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 shutdown_connections 2024-05-09T08:24:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8a380a2b90 0x7f8a380a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c103440 0x7f8a4c074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a4c102a80 0x7f8a4c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 --2- 172.21.15.28:0/332983704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a4c101890 0x7f8a4c0755e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 >> 172.21.15.28:0/332983704 conn(0x7f8a4c0fce00 msgr2=0x7f8a4c105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 shutdown_connections 2024-05-09T08:24:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.130+0000 7f8a53722640 1 -- 172.21.15.28:0/332983704 wait complete. 2024-05-09T08:24:18.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1939977135' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T08:24:18.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1939977135' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T08:24:18.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:17 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] up:active 2024-05-09T08:24:18.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:17 smithi028 ceph-mon[30119]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-09T08:24:18.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:17 smithi028 ceph-mon[30119]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-05-09T08:24:18.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:17 smithi028 ceph-mon[30119]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-05-09T08:24:18.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:17 smithi028 ceph-mon[30119]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-05-09T08:24:18.142 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:24:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:24:17.740+0000 7fa5cdc3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:24:18.142 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:24:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:24:17.740+0000 7fa5cdc3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:24:18.142 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:24:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:24:17.740+0000 7fa5cdc3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:24:18.142 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:24:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:24:17.740+0000 7fa5cdc3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:24:18.142 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:24:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:24:17.740+0000 7fa5cdc3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:24:18.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-09T08:24:18.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.253+0000 7fb37f4da640 1 -- 172.21.15.28:0/2077291660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378103440 msgr2=0x7fb378109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.253+0000 7fb37f4da640 1 --2- 172.21.15.28:0/2077291660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378103440 0x7fb378109cd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb370009ca0 tx=0x7fb37002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.253+0000 7fb37f4da640 1 -- 172.21.15.28:0/2077291660 shutdown_connections 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.253+0000 7fb37f4da640 1 --2- 172.21.15.28:0/2077291660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378103440 0x7fb378109cd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.253+0000 7fb37f4da640 1 --2- 172.21.15.28:0/2077291660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb378102a80 0x7fb378102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.253+0000 7fb37f4da640 1 --2- 172.21.15.28:0/2077291660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb378101890 0x7fb378101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.253+0000 7fb37f4da640 1 -- 172.21.15.28:0/2077291660 >> 172.21.15.28:0/2077291660 conn(0x7fb3780fce00 msgr2=0x7fb3780ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.253+0000 7fb37f4da640 1 -- 172.21.15.28:0/2077291660 shutdown_connections 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.253+0000 7fb37f4da640 1 -- 172.21.15.28:0/2077291660 wait complete. 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37f4da640 1 Processor -- start 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37f4da640 1 -- start start 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37f4da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378101890 0x7fb378197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37f4da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb378102a80 0x7fb378198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37f4da640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb378103440 0x7fb37819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37f4da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb37810d4f0 con 0x7fb378101890 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37f4da640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb37810d370 con 0x7fb378103440 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37f4da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb37810d670 con 0x7fb378102a80 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37d276640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378101890 0x7fb378197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37d276640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378101890 0x7fb378197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35628/0 (socket says 172.21.15.28:35628) 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37d276640 1 -- 172.21.15.28:0/3679231301 learned_addr learned my addr 172.21.15.28:0/3679231301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37da77640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb378103440 0x7fb37819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37ca75640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb378102a80 0x7fb378198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37d276640 1 -- 172.21.15.28:0/3679231301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb378102a80 msgr2=0x7fb378198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37d276640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb378102a80 0x7fb378198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37d276640 1 -- 172.21.15.28:0/3679231301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb378103440 msgr2=0x7fb37819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37d276640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb378103440 0x7fb37819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37d276640 1 -- 172.21.15.28:0/3679231301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37819f730 con 0x7fb378101890 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37ca75640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb378102a80 0x7fb378198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.254+0000 7fb37da77640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb378103440 0x7fb37819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.255+0000 7fb37d276640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378101890 0x7fb378197b10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb37400c970 tx=0x7fb37400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.255+0000 7fb36e7fc640 1 -- 172.21.15.28:0/3679231301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb374007c10 con 0x7fb378101890 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.255+0000 7fb36e7fc640 1 -- 172.21.15.28:0/3679231301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb374007db0 con 0x7fb378101890 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.255+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb37819fa20 con 0x7fb378101890 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.255+0000 7fb36e7fc640 1 -- 172.21.15.28:0/3679231301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37400f940 con 0x7fb378101890 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.255+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb37819ff70 con 0x7fb378101890 2024-05-09T08:24:18.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.256+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb378108b50 con 0x7fb378101890 2024-05-09T08:24:18.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.257+0000 7fb36e7fc640 1 -- 172.21.15.28:0/3679231301 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb37400fae0 con 0x7fb378101890 2024-05-09T08:24:18.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.257+0000 7fb36e7fc640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb3640a2b90 0x7fb3640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.258+0000 7fb37ca75640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb3640a2b90 0x7fb3640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.258+0000 7fb36e7fc640 1 -- 172.21.15.28:0/3679231301 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb3740cf770 con 0x7fb378101890 2024-05-09T08:24:18.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.258+0000 7fb37ca75640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb3640a2b90 0x7fb3640a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb378199020 tx=0x7fb368005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.262+0000 7fb36e7fc640 1 -- 172.21.15.28:0/3679231301 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb37409c4c0 con 0x7fb378101890 2024-05-09T08:24:18.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.487+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fb378061c70 con 0x7fb3640a2b90 2024-05-09T08:24:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.492+0000 7fb36e7fc640 1 -- 172.21.15.28:0/3679231301 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fb378061c70 con 0x7fb3640a2b90 2024-05-09T08:24:18.493 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/18e460ab-40a2-4925-9dfa-b84dd50731a5 2024-05-09T08:24:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.494+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb3640a2b90 msgr2=0x7fb3640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.494+0000 7fb37f4da640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb3640a2b90 0x7fb3640a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb378199020 tx=0x7fb368005f90 comp rx=0 tx=0).stop 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.494+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378101890 msgr2=0x7fb378197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.494+0000 7fb37f4da640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378101890 0x7fb378197b10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb37400c970 tx=0x7fb37400ce30 comp rx=0 tx=0).stop 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.495+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 shutdown_connections 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.495+0000 7fb37f4da640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb3640a2b90 0x7fb3640a5050 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.495+0000 7fb37f4da640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb378103440 0x7fb37819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.495+0000 7fb37f4da640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb378102a80 0x7fb378198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.495+0000 7fb37f4da640 1 --2- 172.21.15.28:0/3679231301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb378101890 0x7fb378197b10 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.495+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 >> 172.21.15.28:0/3679231301 conn(0x7fb3780fce00 msgr2=0x7fb378105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.495+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 shutdown_connections 2024-05-09T08:24:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.495+0000 7fb37f4da640 1 -- 172.21.15.28:0/3679231301 wait complete. 2024-05-09T08:24:18.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-09T08:24:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 -- 172.21.15.28:0/1089653623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 msgr2=0x7fdd9c0fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 --2- 172.21.15.28:0/1089653623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 0x7fdd9c0fff60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c009900 tx=0x7fdd8c02a400 comp rx=0 tx=0).stop 2024-05-09T08:24:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 -- 172.21.15.28:0/1089653623 shutdown_connections 2024-05-09T08:24:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 --2- 172.21.15.28:0/1089653623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd9c103570 0x7fdd9c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 --2- 172.21.15.28:0/1089653623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd9c102bb0 0x7fdd9c103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 --2- 172.21.15.28:0/1089653623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 0x7fdd9c0fff60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 -- 172.21.15.28:0/1089653623 >> 172.21.15.28:0/1089653623 conn(0x7fdd9c0fb470 msgr2=0x7fdd9c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 -- 172.21.15.28:0/1089653623 shutdown_connections 2024-05-09T08:24:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 -- 172.21.15.28:0/1089653623 wait complete. 2024-05-09T08:24:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.616+0000 7fdda21c8640 1 Processor -- start 2024-05-09T08:24:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdda21c8640 1 -- start start 2024-05-09T08:24:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdda21c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 0x7fdd9c197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdda21c8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd9c102bb0 0x7fdd9c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdda21c8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd9c103570 0x7fdd9c19f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdda21c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c10d570 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdda21c8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c10d3f0 con 0x7fdd9c103570 2024-05-09T08:24:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdda21c8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c10d6f0 con 0x7fdd9c102bb0 2024-05-09T08:24:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 0x7fdd9c197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 0x7fdd9c197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35636/0 (socket says 172.21.15.28:35636) 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9b7fe640 1 -- 172.21.15.28:0/2431584891 learned_addr learned my addr 172.21.15.28:0/2431584891 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9bfff640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd9c103570 0x7fdd9c19f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9affd640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd9c102bb0 0x7fdd9c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9b7fe640 1 -- 172.21.15.28:0/2431584891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd9c102bb0 msgr2=0x7fdd9c1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9b7fe640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd9c102bb0 0x7fdd9c1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9b7fe640 1 -- 172.21.15.28:0/2431584891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd9c103570 msgr2=0x7fdd9c19f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9b7fe640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd9c103570 0x7fdd9c19f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9b7fe640 1 -- 172.21.15.28:0/2431584891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9c19f8b0 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.617+0000 7fdd9b7fe640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 0x7fdd9c197c70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c031610 tx=0x7fdd8c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.618+0000 7fdd98ff9640 1 -- 172.21.15.28:0/2431584891 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c03f030 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.618+0000 7fdd98ff9640 1 -- 172.21.15.28:0/2431584891 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd8c003ce0 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.618+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9c19fb40 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.618+0000 7fdd98ff9640 1 -- 172.21.15.28:0/2431584891 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c035070 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.618+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd9c1a0080 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.619+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd9c108db0 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.620+0000 7fdd98ff9640 1 -- 172.21.15.28:0/2431584891 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdd8c042030 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.620+0000 7fdd98ff9640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdd840a2c10 0x7fdd840a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.621+0000 7fdd9affd640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdd840a2c10 0x7fdd840a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.621+0000 7fdd98ff9640 1 -- 172.21.15.28:0/2431584891 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdd8c0ef700 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.621+0000 7fdd9affd640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdd840a2c10 0x7fdd840a50d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c199170 tx=0x7fdd90009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.625+0000 7fdd98ff9640 1 -- 172.21.15.28:0/2431584891 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdd8c0bc420 con 0x7fdd9c0ffb40 2024-05-09T08:24:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.839+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdd9c061d00 con 0x7fdd840a2c10 2024-05-09T08:24:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.868+0000 7fdd98ff9640 1 -- 172.21.15.28:0/2431584891 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdd9c061d00 con 0x7fdd840a2c10 2024-05-09T08:24:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdd840a2c10 msgr2=0x7fdd840a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdd840a2c10 0x7fdd840a50d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c199170 tx=0x7fdd90009110 comp rx=0 tx=0).stop 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 msgr2=0x7fdd9c197c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 0x7fdd9c197c70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c031610 tx=0x7fdd8c0043b0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 shutdown_connections 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fdd840a2c10 0x7fdd840a50d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd9c103570 0x7fdd9c19f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd9c102bb0 0x7fdd9c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 --2- 172.21.15.28:0/2431584891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd9c0ffb40 0x7fdd9c197c70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 >> 172.21.15.28:0/2431584891 conn(0x7fdd9c0fb470 msgr2=0x7fdd9c105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 shutdown_connections 2024-05-09T08:24:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.870+0000 7fdda21c8640 1 -- 172.21.15.28:0/2431584891 wait complete. 2024-05-09T08:24:18.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-09T08:24:18.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 -- 172.21.15.28:0/1561693043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 msgr2=0x7fd700100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 --2- 172.21.15.28:0/1561693043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 0x7fd700100b90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009900 tx=0x7fd6f402a410 comp rx=0 tx=0).stop 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 -- 172.21.15.28:0/1561693043 shutdown_connections 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 --2- 172.21.15.28:0/1561693043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd700102320 0x7fd700108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 --2- 172.21.15.28:0/1561693043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd700101960 0x7fd700101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 --2- 172.21.15.28:0/1561693043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 0x7fd700100b90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 -- 172.21.15.28:0/1561693043 >> 172.21.15.28:0/1561693043 conn(0x7fd7000fbce0 msgr2=0x7fd7000fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 -- 172.21.15.28:0/1561693043 shutdown_connections 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 -- 172.21.15.28:0/1561693043 wait complete. 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.993+0000 7fd707b14640 1 Processor -- start 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd707b14640 1 -- start start 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd707b14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 0x7fd700197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd707b14640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd700101960 0x7fd700198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd707b14640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd700102320 0x7fd70019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd707b14640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd70010d570 con 0x7fd700100770 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd707b14640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd70010d3f0 con 0x7fd700102320 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd707b14640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd70010d6f0 con 0x7fd700101960 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7058b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 0x7fd700197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7058b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 0x7fd700197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35662/0 (socket says 172.21.15.28:35662) 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7058b0640 1 -- 172.21.15.28:0/745783222 learned_addr learned my addr 172.21.15.28:0/745783222 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7060b1640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd700102320 0x7fd70019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7050af640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd700101960 0x7fd700198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7058b0640 1 -- 172.21.15.28:0/745783222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd700101960 msgr2=0x7fd700198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7058b0640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd700101960 0x7fd700198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7058b0640 1 -- 172.21.15.28:0/745783222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd700102320 msgr2=0x7fd70019f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7058b0640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd700102320 0x7fd70019f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7058b0640 1 -- 172.21.15.28:0/745783222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70019f800 con 0x7fd700100770 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd7058b0640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 0x7fd700197be0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4031440 tx=0x7fd6f4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd6f2ffd640 1 -- 172.21.15.28:0/745783222 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f403f030 con 0x7fd700100770 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd6f2ffd640 1 -- 172.21.15.28:0/745783222 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6f4003ce0 con 0x7fd700100770 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70019fa90 con 0x7fd700100770 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.994+0000 7fd6f2ffd640 1 -- 172.21.15.28:0/745783222 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4035070 con 0x7fd700100770 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.995+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd70019ffd0 con 0x7fd700100770 2024-05-09T08:24:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.995+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd700107ae0 con 0x7fd700100770 2024-05-09T08:24:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.997+0000 7fd6f2ffd640 1 -- 172.21.15.28:0/745783222 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6f4042030 con 0x7fd700100770 2024-05-09T08:24:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.997+0000 7fd6f2ffd640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.997+0000 7fd7050af640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.997+0000 7fd6f2ffd640 1 -- 172.21.15.28:0/745783222 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6f40ef6f0 con 0x7fd700100770 2024-05-09T08:24:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:18.998+0000 7fd7050af640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd7001990f0 tx=0x7fd6fc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.002+0000 7fd6f2ffd640 1 -- 172.21.15.28:0/745783222 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd6f40bbdc0 con 0x7fd700100770 2024-05-09T08:24:19.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.216+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fd700061d00 con 0x7fd6ec0a2c10 2024-05-09T08:24:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.221+0000 7fd6f2ffd640 1 -- 172.21.15.28:0/745783222 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd700061d00 con 0x7fd6ec0a2c10 2024-05-09T08:24:19.222 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00 2024-05-09T08:24:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6ec0a2c10 msgr2=0x7fd6ec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd7001990f0 tx=0x7fd6fc00b390 comp rx=0 tx=0).stop 2024-05-09T08:24:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 msgr2=0x7fd700197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 0x7fd700197be0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4031440 tx=0x7fd6f4032ce0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 shutdown_connections 2024-05-09T08:24:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd700102320 0x7fd70019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd700101960 0x7fd700198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 --2- 172.21.15.28:0/745783222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd700100770 0x7fd700197be0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 >> 172.21.15.28:0/745783222 conn(0x7fd7000fbce0 msgr2=0x7fd700104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 shutdown_connections 2024-05-09T08:24:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.223+0000 7fd707b14640 1 -- 172.21.15.28:0/745783222 wait complete. 2024-05-09T08:24:19.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:24:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.339+0000 7fa144a8c640 1 -- 172.21.15.28:0/1461796118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa140103450 msgr2=0x7fa140109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.339+0000 7fa144a8c640 1 --2- 172.21.15.28:0/1461796118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa140103450 0x7fa140109ce0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa138009ca0 tx=0x7fa13802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.339+0000 7fa144a8c640 1 -- 172.21.15.28:0/1461796118 shutdown_connections 2024-05-09T08:24:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.339+0000 7fa144a8c640 1 --2- 172.21.15.28:0/1461796118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa140103450 0x7fa140109ce0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.339+0000 7fa144a8c640 1 --2- 172.21.15.28:0/1461796118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa140102a90 0x7fa140102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.339+0000 7fa144a8c640 1 --2- 172.21.15.28:0/1461796118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1401018a0 0x7fa140101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.339+0000 7fa144a8c640 1 -- 172.21.15.28:0/1461796118 >> 172.21.15.28:0/1461796118 conn(0x7fa1400fce30 msgr2=0x7fa1400ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.339+0000 7fa144a8c640 1 -- 172.21.15.28:0/1461796118 shutdown_connections 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.339+0000 7fa144a8c640 1 -- 172.21.15.28:0/1461796118 wait complete. 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa144a8c640 1 Processor -- start 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa144a8c640 1 -- start start 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa144a8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1401018a0 0x7fa140197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa144a8c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa140102a90 0x7fa140198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa144a8c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa140103450 0x7fa14019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa144a8c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa14010d500 con 0x7fa1401018a0 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa144a8c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa14010d380 con 0x7fa140102a90 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa144a8c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa14010d680 con 0x7fa140103450 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa13e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1401018a0 0x7fa140197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa13e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1401018a0 0x7fa140197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35684/0 (socket says 172.21.15.28:35684) 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa13e59c640 1 -- 172.21.15.28:0/2509947952 learned_addr learned my addr 172.21.15.28:0/2509947952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.340+0000 7fa13dd9b640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa140102a90 0x7fa140198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa13e59c640 1 -- 172.21.15.28:0/2509947952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa140103450 msgr2=0x7fa14019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa13e59c640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa140103450 0x7fa14019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa13e59c640 1 -- 172.21.15.28:0/2509947952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa140102a90 msgr2=0x7fa140198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa13e59c640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa140102a90 0x7fa140198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa13e59c640 1 -- 172.21.15.28:0/2509947952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14019f730 con 0x7fa1401018a0 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa13e59c640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1401018a0 0x7fa140197b10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa13000e9c0 tx=0x7fa13000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa12f7fe640 1 -- 172.21.15.28:0/2509947952 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13000cd80 con 0x7fa1401018a0 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa12f7fe640 1 -- 172.21.15.28:0/2509947952 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1300035f0 con 0x7fa1401018a0 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa14019fa20 con 0x7fa1401018a0 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa12f7fe640 1 -- 172.21.15.28:0/2509947952 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa130010690 con 0x7fa1401018a0 2024-05-09T08:24:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.341+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa14019ff70 con 0x7fa1401018a0 2024-05-09T08:24:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.342+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa140108af0 con 0x7fa1401018a0 2024-05-09T08:24:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.343+0000 7fa12f7fe640 1 -- 172.21.15.28:0/2509947952 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa1300037a0 con 0x7fa1401018a0 2024-05-09T08:24:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.344+0000 7fa12f7fe640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1280a2b90 0x7fa1280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.344+0000 7fa13dd9b640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1280a2b90 0x7fa1280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.344+0000 7fa12f7fe640 1 -- 172.21.15.28:0/2509947952 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa1300cf770 con 0x7fa1401018a0 2024-05-09T08:24:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.344+0000 7fa13dd9b640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1280a2b90 0x7fa1280a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa140199020 tx=0x7fa13400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.348+0000 7fa12f7fe640 1 -- 172.21.15.28:0/2509947952 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa13009c430 con 0x7fa1401018a0 2024-05-09T08:24:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.577+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa14002d1c0 con 0x7fa1401018a0 2024-05-09T08:24:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.578+0000 7fa12f7fe640 1 -- 172.21.15.28:0/2509947952 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7fa14002d1c0 con 0x7fa1401018a0 2024-05-09T08:24:19.579 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:19.580 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:24:19.404649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24481,24499,24476,24512,24463]},"id":1}]} 2024-05-09T08:24:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-05-09T08:24:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1280a2b90 msgr2=0x7fa1280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1280a2b90 0x7fa1280a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa140199020 tx=0x7fa13400a2d0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1401018a0 msgr2=0x7fa140197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1401018a0 0x7fa140197b10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa13000e9c0 tx=0x7fa13000ee80 comp rx=0 tx=0).stop 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 shutdown_connections 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa1280a2b90 0x7fa1280a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa140103450 0x7fa14019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa140102a90 0x7fa140198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 --2- 172.21.15.28:0/2509947952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1401018a0 0x7fa140197b10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 >> 172.21.15.28:0/2509947952 conn(0x7fa1400fce30 msgr2=0x7fa140105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 shutdown_connections 2024-05-09T08:24:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.580+0000 7fa144a8c640 1 -- 172.21.15.28:0/2509947952 wait complete. 2024-05-09T08:24:19.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: pgmap v303: 129 pgs: 129 active+clean; 584 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-05-09T08:24:19.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:19.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:19.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] up:active 2024-05-09T08:24:19.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-09T08:24:19.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T08:24:19.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: Cluster is now healthy 2024-05-09T08:24:19.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-05-09T08:24:19.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-09T08:24:19.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:19 smithi028 ceph-mon[30119]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-05-09T08:24:19.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:24:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 -- 172.21.15.28:0/1274841861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc102320 msgr2=0x7f8adc108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 --2- 172.21.15.28:0/1274841861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc102320 0x7f8adc108bb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009ca0 tx=0x7f8ad402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 -- 172.21.15.28:0/1274841861 shutdown_connections 2024-05-09T08:24:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 --2- 172.21.15.28:0/1274841861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc102320 0x7f8adc108bb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 --2- 172.21.15.28:0/1274841861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8adc101960 0x7f8adc101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 --2- 172.21.15.28:0/1274841861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8adc100770 0x7f8adc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 -- 172.21.15.28:0/1274841861 >> 172.21.15.28:0/1274841861 conn(0x7f8adc0fbce0 msgr2=0x7f8adc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 -- 172.21.15.28:0/1274841861 shutdown_connections 2024-05-09T08:24:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 -- 172.21.15.28:0/1274841861 wait complete. 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 Processor -- start 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 -- start start 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8adc100770 0x7f8adc069490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc101960 0x7f8adc19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8adc102320 0x7f8adc19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8adc10c440 con 0x7f8adc101960 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8adc10c2c0 con 0x7f8adc102320 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.701+0000 7f8ae3231640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8adc10c5c0 con 0x7f8adc100770 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8adbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc101960 0x7f8adc19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8adbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc101960 0x7f8adc19c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35702/0 (socket says 172.21.15.28:35702) 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8adbfff640 1 -- 172.21.15.28:0/2852229510 learned_addr learned my addr 172.21.15.28:0/2852229510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8ae17ce640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8adc102320 0x7f8adc19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8ae0fcd640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8adc100770 0x7f8adc069490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8adbfff640 1 -- 172.21.15.28:0/2852229510 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8adc100770 msgr2=0x7f8adc069490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8adbfff640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8adc100770 0x7f8adc069490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8adbfff640 1 -- 172.21.15.28:0/2852229510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8adc102320 msgr2=0x7f8adc19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8adbfff640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8adc102320 0x7f8adc19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8adbfff640 1 -- 172.21.15.28:0/2852229510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8adc19f7d0 con 0x7f8adc101960 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8adbfff640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc101960 0x7f8adc19c9a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400e9f0 tx=0x7f8ac400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8ad9ffb640 1 -- 172.21.15.28:0/2852229510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac400cd80 con 0x7f8adc101960 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8adc19fac0 con 0x7f8adc101960 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8ad9ffb640 1 -- 172.21.15.28:0/2852229510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ac4010910 con 0x7f8adc101960 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.702+0000 7f8ad9ffb640 1 -- 172.21.15.28:0/2852229510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40188f0 con 0x7f8adc101960 2024-05-09T08:24:19.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.703+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8adc1a0010 con 0x7f8adc101960 2024-05-09T08:24:19.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.703+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8adc107a30 con 0x7f8adc101960 2024-05-09T08:24:19.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.705+0000 7f8ad9ffb640 1 -- 172.21.15.28:0/2852229510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8ac4010ac0 con 0x7f8adc101960 2024-05-09T08:24:19.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.705+0000 7f8ad9ffb640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8ac80a2b90 0x7f8ac80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.705+0000 7f8ae0fcd640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8ac80a2b90 0x7f8ac80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.705+0000 7f8ad9ffb640 1 -- 172.21.15.28:0/2852229510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8ac40cf9f0 con 0x7f8adc101960 2024-05-09T08:24:19.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.706+0000 7f8ae0fcd640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8ac80a2b90 0x7f8ac80a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0058c0 tx=0x7f8acc00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:19.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.709+0000 7f8ad9ffb640 1 -- 172.21.15.28:0/2852229510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8ac409c060 con 0x7f8adc101960 2024-05-09T08:24:19.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: pgmap v303: 129 pgs: 129 active+clean; 584 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-05-09T08:24:19.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:19.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:19.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] up:active 2024-05-09T08:24:19.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-09T08:24:19.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T08:24:19.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: Cluster is now healthy 2024-05-09T08:24:19.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-05-09T08:24:19.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-09T08:24:19.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:19 smithi165 ceph-mon[34540]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-05-09T08:24:19.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: pgmap v303: 129 pgs: 129 active+clean; 584 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-05-09T08:24:19.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:19.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:19.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] up:active 2024-05-09T08:24:19.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-09T08:24:19.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T08:24:19.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: Cluster is now healthy 2024-05-09T08:24:19.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-05-09T08:24:19.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-09T08:24:19.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:19 smithi184 ceph-mon[34509]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-05-09T08:24:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.897+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8adc0624d0 con 0x7f8adc101960 2024-05-09T08:24:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.899+0000 7f8ad9ffb640 1 -- 172.21.15.28:0/2852229510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7f8ac40a0360 con 0x7f8adc101960 2024-05-09T08:24:19.900 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:19.900 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:24:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8ac80a2b90 msgr2=0x7f8ac80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8ac80a2b90 0x7f8ac80a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0058c0 tx=0x7f8acc00a2d0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc101960 msgr2=0x7f8adc19c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc101960 0x7f8adc19c9a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400e9f0 tx=0x7f8ac400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 shutdown_connections 2024-05-09T08:24:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8ac80a2b90 0x7f8ac80a5050 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8adc102320 0x7f8adc19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8adc101960 0x7f8adc19c9a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 --2- 172.21.15.28:0/2852229510 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8adc100770 0x7f8adc069490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 >> 172.21.15.28:0/2852229510 conn(0x7f8adc0fbce0 msgr2=0x7f8adc104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 shutdown_connections 2024-05-09T08:24:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:19.901+0000 7f8ae3231640 1 -- 172.21.15.28:0/2852229510 wait complete. 2024-05-09T08:24:19.914 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-09T08:24:19.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:24:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.018+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/755800944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 msgr2=0x7f4bac101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.018+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/755800944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 0x7f4bac101de0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0009900 tx=0x7f4ba002a410 comp rx=0 tx=0).stop 2024-05-09T08:24:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.019+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/755800944 shutdown_connections 2024-05-09T08:24:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.019+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/755800944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bac102320 0x7f4bac108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.019+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/755800944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 0x7f4bac101de0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.019+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/755800944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac100770 0x7f4bac100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.019+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/755800944 >> 172.21.15.28:0/755800944 conn(0x7f4bac0fbce0 msgr2=0x7f4bac0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.019+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/755800944 shutdown_connections 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.019+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/755800944 wait complete. 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.019+0000 7f4bb1fa9640 1 Processor -- start 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.019+0000 7f4bb1fa9640 1 -- start start 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4bb1fa9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac100770 0x7f4bac069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4bb1fa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 0x7f4bac19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4bb1fa9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bac102320 0x7f4bac19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4bb1fa9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac10c3d0 con 0x7f4bac101960 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4bb1fa9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac10c250 con 0x7f4bac100770 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4bb1fa9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac10c550 con 0x7f4bac102320 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4baaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 0x7f4bac19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4baaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 0x7f4bac19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35722/0 (socket says 172.21.15.28:35722) 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4baaffd640 1 -- 172.21.15.28:0/109145735 learned_addr learned my addr 172.21.15.28:0/109145735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4bab7fe640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac100770 0x7f4bac069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4babfff640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bac102320 0x7f4bac19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4baaffd640 1 -- 172.21.15.28:0/109145735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bac102320 msgr2=0x7f4bac19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4baaffd640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bac102320 0x7f4bac19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4baaffd640 1 -- 172.21.15.28:0/109145735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac100770 msgr2=0x7f4bac069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4baaffd640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac100770 0x7f4bac069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4baaffd640 1 -- 172.21.15.28:0/109145735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac19f7f0 con 0x7f4bac101960 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4bab7fe640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac100770 0x7f4bac069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4baaffd640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 0x7f4bac19c8c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0009230 tx=0x7f4ba0031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4ba8ff9640 1 -- 172.21.15.28:0/109145735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba003f030 con 0x7f4bac101960 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4ba8ff9640 1 -- 172.21.15.28:0/109145735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba0003ad0 con 0x7f4bac101960 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bac19fa20 con 0x7f4bac101960 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.020+0000 7f4ba8ff9640 1 -- 172.21.15.28:0/109145735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0035070 con 0x7f4bac101960 2024-05-09T08:24:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.021+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bac19ff10 con 0x7f4bac101960 2024-05-09T08:24:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.022+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bac107a90 con 0x7f4bac101960 2024-05-09T08:24:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.023+0000 7f4ba8ff9640 1 -- 172.21.15.28:0/109145735 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4ba0042030 con 0x7f4bac101960 2024-05-09T08:24:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.023+0000 7f4ba8ff9640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b940a2b90 0x7f4b940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.023+0000 7f4bab7fe640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b940a2b90 0x7f4b940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.023+0000 7f4ba8ff9640 1 -- 172.21.15.28:0/109145735 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4ba00ef4c0 con 0x7f4bac101960 2024-05-09T08:24:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.024+0000 7f4bab7fe640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b940a2b90 0x7f4b940a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4bac1017c0 tx=0x7f4b9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:20.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.028+0000 7f4ba8ff9640 1 -- 172.21.15.28:0/109145735 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4ba00bc1e0 con 0x7f4bac101960 2024-05-09T08:24:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.225+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4bac061c70 con 0x7f4b940a2b90 2024-05-09T08:24:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.235+0000 7f4ba8ff9640 1 -- 172.21.15.28:0/109145735 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340623 (secure 0 0 0) 0x7f4bac061c70 con 0x7f4b940a2b90 2024-05-09T08:24:20.241 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:20.243 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b940a2b90 msgr2=0x7f4b940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b940a2b90 0x7f4b940a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4bac1017c0 tx=0x7f4b9c005f90 comp rx=0 tx=0).stop 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 msgr2=0x7f4bac19c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 0x7f4bac19c8c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0009230 tx=0x7f4ba0031b60 comp rx=0 tx=0).stop 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 shutdown_connections 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b940a2b90 0x7f4b940a5050 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bac102320 0x7f4bac19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac101960 0x7f4bac19c8c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 --2- 172.21.15.28:0/109145735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac100770 0x7f4bac069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 >> 172.21.15.28:0/109145735 conn(0x7f4bac0fbce0 msgr2=0x7f4bac104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 shutdown_connections 2024-05-09T08:24:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.239+0000 7f4bb1fa9640 1 -- 172.21.15.28:0/109145735 wait complete. 2024-05-09T08:24:20.252 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-05-09T08:24:19.643870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598855,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":315,"num_write_kb":2985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":258,"ondisk_log_size":258,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373268,"kb_used_data":11612,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568556,"statfs":{"total":1151940427776,"available":1151558201344,"internally_reserved":0,"allocated":11890688,"data_stored":7523513,"data_compressed":284457,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3988,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003355"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.330072+0000","last_change":"2024-05-09T08:23:51.579564+0000","last_active":"2024-05-09T08:23:57.330072+0000","last_peered":"2024-05-09T08:23:57.330072+0000","last_clean":"2024-05-09T08:23:57.330072+0000","last_became_active":"2024-05-09T08:23:51.578884+0000","last_became_peered":"2024-05-09T08:23:51.578884+0000","last_unstale":"2024-05-09T08:23:57.330072+0000","last_undegraded":"2024-05-09T08:23:57.330072+0000","last_fullsized":"2024-05-09T08:23:57.330072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:40.473483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.301084+0000","last_change":"2024-05-09T08:23:51.574187+0000","last_active":"2024-05-09T08:23:58.301084+0000","last_peered":"2024-05-09T08:23:58.301084+0000","last_clean":"2024-05-09T08:23:58.301084+0000","last_became_active":"2024-05-09T08:23:51.573837+0000","last_became_peered":"2024-05-09T08:23:51.573837+0000","last_unstale":"2024-05-09T08:23:58.301084+0000","last_undegraded":"2024-05-09T08:23:58.301084+0000","last_fullsized":"2024-05-09T08:23:58.301084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:31.448038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300979+0000","last_change":"2024-05-09T08:23:51.577224+0000","last_active":"2024-05-09T08:23:58.300979+0000","last_peered":"2024-05-09T08:23:58.300979+0000","last_clean":"2024-05-09T08:23:58.300979+0000","last_became_active":"2024-05-09T08:23:51.576822+0000","last_became_peered":"2024-05-09T08:23:51.576822+0000","last_unstale":"2024-05-09T08:23:58.300979+0000","last_undegraded":"2024-05-09T08:23:58.300979+0000","last_fullsized":"2024-05-09T08:23:58.300979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:04.902530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921941+0000","last_change":"2024-05-09T08:23:51.577172+0000","last_active":"2024-05-09T08:23:56.921941+0000","last_peered":"2024-05-09T08:23:56.921941+0000","last_clean":"2024-05-09T08:23:56.921941+0000","last_became_active":"2024-05-09T08:23:51.576861+0000","last_became_peered":"2024-05-09T08:23:51.576861+0000","last_unstale":"2024-05-09T08:23:56.921941+0000","last_undegraded":"2024-05-09T08:23:56.921941+0000","last_fullsized":"2024-05-09T08:23:56.921941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:10.925739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469607+0000","last_change":"2024-05-09T08:23:51.579408+0000","last_active":"2024-05-09T08:23:57.469607+0000","last_peered":"2024-05-09T08:23:57.469607+0000","last_clean":"2024-05-09T08:23:57.469607+0000","last_became_active":"2024-05-09T08:23:51.578229+0000","last_became_peered":"2024-05-09T08:23:51.578229+0000","last_unstale":"2024-05-09T08:23:57.469607+0000","last_undegraded":"2024-05-09T08:23:57.469607+0000","last_fullsized":"2024-05-09T08:23:57.469607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:47:27.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654282+0000","last_change":"2024-05-09T08:23:51.576924+0000","last_active":"2024-05-09T08:23:56.654282+0000","last_peered":"2024-05-09T08:23:56.654282+0000","last_clean":"2024-05-09T08:23:56.654282+0000","last_became_active":"2024-05-09T08:23:51.576601+0000","last_became_peered":"2024-05-09T08:23:51.576601+0000","last_unstale":"2024-05-09T08:23:56.654282+0000","last_undegraded":"2024-05-09T08:23:56.654282+0000","last_fullsized":"2024-05-09T08:23:56.654282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:07:16.910271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022073+0000","last_change":"2024-05-09T08:23:51.574843+0000","last_active":"2024-05-09T08:23:58.022073+0000","last_peered":"2024-05-09T08:23:58.022073+0000","last_clean":"2024-05-09T08:23:58.022073+0000","last_became_active":"2024-05-09T08:23:51.574498+0000","last_became_peered":"2024-05-09T08:23:51.574498+0000","last_unstale":"2024-05-09T08:23:58.022073+0000","last_undegraded":"2024-05-09T08:23:58.022073+0000","last_fullsized":"2024-05-09T08:23:58.022073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:59:50.974491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022179+0000","last_change":"2024-05-09T08:23:51.574001+0000","last_active":"2024-05-09T08:23:58.022179+0000","last_peered":"2024-05-09T08:23:58.022179+0000","last_clean":"2024-05-09T08:23:58.022179+0000","last_became_active":"2024-05-09T08:23:51.573632+0000","last_became_peered":"2024-05-09T08:23:51.573632+0000","last_unstale":"2024-05-09T08:23:58.022179+0000","last_undegraded":"2024-05-09T08:23:58.022179+0000","last_fullsized":"2024-05-09T08:23:58.022179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:55:16.418951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488452+0000","last_change":"2024-05-09T08:23:51.567635+0000","last_active":"2024-05-09T08:23:57.488452+0000","last_peered":"2024-05-09T08:23:57.488452+0000","last_clean":"2024-05-09T08:23:57.488452+0000","last_became_active":"2024-05-09T08:23:51.567283+0000","last_became_peered":"2024-05-09T08:23:51.567283+0000","last_unstale":"2024-05-09T08:23:57.488452+0000","last_undegraded":"2024-05-09T08:23:57.488452+0000","last_fullsized":"2024-05-09T08:23:57.488452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:42.346856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654839+0000","last_change":"2024-05-09T08:23:51.576090+0000","last_active":"2024-05-09T08:23:56.654839+0000","last_peered":"2024-05-09T08:23:56.654839+0000","last_clean":"2024-05-09T08:23:56.654839+0000","last_became_active":"2024-05-09T08:23:51.575768+0000","last_became_peered":"2024-05-09T08:23:51.575768+0000","last_unstale":"2024-05-09T08:23:56.654839+0000","last_undegraded":"2024-05-09T08:23:56.654839+0000","last_fullsized":"2024-05-09T08:23:56.654839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:00:50.016924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022633+0000","last_change":"2024-05-09T08:23:51.578957+0000","last_active":"2024-05-09T08:23:58.022633+0000","last_peered":"2024-05-09T08:23:58.022633+0000","last_clean":"2024-05-09T08:23:58.022633+0000","last_became_active":"2024-05-09T08:23:51.578210+0000","last_became_peered":"2024-05-09T08:23:51.578210+0000","last_unstale":"2024-05-09T08:23:58.022633+0000","last_undegraded":"2024-05-09T08:23:58.022633+0000","last_fullsized":"2024-05-09T08:23:58.022633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:28:28.953434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921864+0000","last_change":"2024-05-09T08:23:51.575389+0000","last_active":"2024-05-09T08:23:56.921864+0000","last_peered":"2024-05-09T08:23:56.921864+0000","last_clean":"2024-05-09T08:23:56.921864+0000","last_became_active":"2024-05-09T08:23:51.575071+0000","last_became_peered":"2024-05-09T08:23:51.575071+0000","last_unstale":"2024-05-09T08:23:56.921864+0000","last_undegraded":"2024-05-09T08:23:56.921864+0000","last_fullsized":"2024-05-09T08:23:56.921864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:34.844611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022544+0000","last_change":"2024-05-09T08:23:51.572574+0000","last_active":"2024-05-09T08:23:58.022544+0000","last_peered":"2024-05-09T08:23:58.022544+0000","last_clean":"2024-05-09T08:23:58.022544+0000","last_became_active":"2024-05-09T08:23:51.572243+0000","last_became_peered":"2024-05-09T08:23:51.572243+0000","last_unstale":"2024-05-09T08:23:58.022544+0000","last_undegraded":"2024-05-09T08:23:58.022544+0000","last_fullsized":"2024-05-09T08:23:58.022544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:52.534365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300681+0000","last_change":"2024-05-09T08:23:51.574963+0000","last_active":"2024-05-09T08:23:58.300681+0000","last_peered":"2024-05-09T08:23:58.300681+0000","last_clean":"2024-05-09T08:23:58.300681+0000","last_became_active":"2024-05-09T08:23:51.574597+0000","last_became_peered":"2024-05-09T08:23:51.574597+0000","last_unstale":"2024-05-09T08:23:58.300681+0000","last_undegraded":"2024-05-09T08:23:58.300681+0000","last_fullsized":"2024-05-09T08:23:58.300681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:33.348828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300607+0000","last_change":"2024-05-09T08:23:51.577777+0000","last_active":"2024-05-09T08:23:58.300607+0000","last_peered":"2024-05-09T08:23:58.300607+0000","last_clean":"2024-05-09T08:23:58.300607+0000","last_became_active":"2024-05-09T08:23:51.577073+0000","last_became_peered":"2024-05-09T08:23:51.577073+0000","last_unstale":"2024-05-09T08:23:58.300607+0000","last_undegraded":"2024-05-09T08:23:58.300607+0000","last_fullsized":"2024-05-09T08:23:58.300607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:26.249488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.487831+0000","last_change":"2024-05-09T08:23:51.574476+0000","last_active":"2024-05-09T08:23:57.487831+0000","last_peered":"2024-05-09T08:23:57.487831+0000","last_clean":"2024-05-09T08:23:57.487831+0000","last_became_active":"2024-05-09T08:23:51.572748+0000","last_became_peered":"2024-05-09T08:23:51.572748+0000","last_unstale":"2024-05-09T08:23:57.487831+0000","last_undegraded":"2024-05-09T08:23:57.487831+0000","last_fullsized":"2024-05-09T08:23:57.487831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:43.515419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329835+0000","last_change":"2024-05-09T08:23:51.580568+0000","last_active":"2024-05-09T08:23:57.329835+0000","last_peered":"2024-05-09T08:23:57.329835+0000","last_clean":"2024-05-09T08:23:57.329835+0000","last_became_active":"2024-05-09T08:23:51.579626+0000","last_became_peered":"2024-05-09T08:23:51.579626+0000","last_unstale":"2024-05-09T08:23:57.329835+0000","last_undegraded":"2024-05-09T08:23:57.329835+0000","last_fullsized":"2024-05-09T08:23:57.329835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:28.756079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654855+0000","last_change":"2024-05-09T08:23:51.574363+0000","last_active":"2024-05-09T08:23:56.654855+0000","last_peered":"2024-05-09T08:23:56.654855+0000","last_clean":"2024-05-09T08:23:56.654855+0000","last_became_active":"2024-05-09T08:23:51.573858+0000","last_became_peered":"2024-05-09T08:23:51.573858+0000","last_unstale":"2024-05-09T08:23:56.654855+0000","last_undegraded":"2024-05-09T08:23:56.654855+0000","last_fullsized":"2024-05-09T08:23:56.654855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:30.752939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654980+0000","last_change":"2024-05-09T08:23:51.579361+0000","last_active":"2024-05-09T08:23:56.654980+0000","last_peered":"2024-05-09T08:23:56.654980+0000","last_clean":"2024-05-09T08:23:56.654980+0000","last_became_active":"2024-05-09T08:23:51.579025+0000","last_became_peered":"2024-05-09T08:23:51.579025+0000","last_unstale":"2024-05-09T08:23:56.654980+0000","last_undegraded":"2024-05-09T08:23:56.654980+0000","last_fullsized":"2024-05-09T08:23:56.654980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:02.273999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654896+0000","last_change":"2024-05-09T08:23:55.670107+0000","last_active":"2024-05-09T08:23:56.654896+0000","last_peered":"2024-05-09T08:23:56.654896+0000","last_clean":"2024-05-09T08:23:56.654896+0000","last_became_active":"2024-05-09T08:23:55.669621+0000","last_became_peered":"2024-05-09T08:23:55.669621+0000","last_unstale":"2024-05-09T08:23:56.654896+0000","last_undegraded":"2024-05-09T08:23:56.654896+0000","last_fullsized":"2024-05-09T08:23:56.654896+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:00.547137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654971+0000","last_change":"2024-05-09T08:23:50.571747+0000","last_active":"2024-05-09T08:23:56.654971+0000","last_peered":"2024-05-09T08:23:56.654971+0000","last_clean":"2024-05-09T08:23:56.654971+0000","last_became_active":"2024-05-09T08:23:50.571460+0000","last_became_peered":"2024-05-09T08:23:50.571460+0000","last_unstale":"2024-05-09T08:23:56.654971+0000","last_undegraded":"2024-05-09T08:23:56.654971+0000","last_fullsized":"2024-05-09T08:23:56.654971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:15.519879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022701+0000","last_change":"2024-05-09T08:23:51.574336+0000","last_active":"2024-05-09T08:23:58.022701+0000","last_peered":"2024-05-09T08:23:58.022701+0000","last_clean":"2024-05-09T08:23:58.022701+0000","last_became_active":"2024-05-09T08:23:51.573801+0000","last_became_peered":"2024-05-09T08:23:51.573801+0000","last_unstale":"2024-05-09T08:23:58.022701+0000","last_undegraded":"2024-05-09T08:23:58.022701+0000","last_fullsized":"2024-05-09T08:23:58.022701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:47.389762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921579+0000","last_change":"2024-05-09T08:23:55.670194+0000","last_active":"2024-05-09T08:23:56.921579+0000","last_peered":"2024-05-09T08:23:56.921579+0000","last_clean":"2024-05-09T08:23:56.921579+0000","last_became_active":"2024-05-09T08:23:55.669762+0000","last_became_peered":"2024-05-09T08:23:55.669762+0000","last_unstale":"2024-05-09T08:23:56.921579+0000","last_undegraded":"2024-05-09T08:23:56.921579+0000","last_fullsized":"2024-05-09T08:23:56.921579+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:22.879635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022754+0000","last_change":"2024-05-09T08:23:50.574421+0000","last_active":"2024-05-09T08:23:58.022754+0000","last_peered":"2024-05-09T08:23:58.022754+0000","last_clean":"2024-05-09T08:23:58.022754+0000","last_became_active":"2024-05-09T08:23:50.574070+0000","last_became_peered":"2024-05-09T08:23:50.574070+0000","last_unstale":"2024-05-09T08:23:58.022754+0000","last_undegraded":"2024-05-09T08:23:58.022754+0000","last_fullsized":"2024-05-09T08:23:58.022754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:22.996034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300934+0000","last_change":"2024-05-09T08:23:51.583275+0000","last_active":"2024-05-09T08:23:58.300934+0000","last_peered":"2024-05-09T08:23:58.300934+0000","last_clean":"2024-05-09T08:23:58.300934+0000","last_became_active":"2024-05-09T08:23:51.582847+0000","last_became_peered":"2024-05-09T08:23:51.582847+0000","last_unstale":"2024-05-09T08:23:58.300934+0000","last_undegraded":"2024-05-09T08:23:58.300934+0000","last_fullsized":"2024-05-09T08:23:58.300934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:03:03.512852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921663+0000","last_change":"2024-05-09T08:23:55.672256+0000","last_active":"2024-05-09T08:23:56.921663+0000","last_peered":"2024-05-09T08:23:56.921663+0000","last_clean":"2024-05-09T08:23:56.921663+0000","last_became_active":"2024-05-09T08:23:55.671855+0000","last_became_peered":"2024-05-09T08:23:55.671855+0000","last_unstale":"2024-05-09T08:23:56.921663+0000","last_undegraded":"2024-05-09T08:23:56.921663+0000","last_fullsized":"2024-05-09T08:23:56.921663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:45:12.199359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.393140+0000","last_change":"2024-05-09T08:23:50.566686+0000","last_active":"2024-05-09T08:24:17.393140+0000","last_peered":"2024-05-09T08:24:17.393140+0000","last_clean":"2024-05-09T08:24:17.393140+0000","last_became_active":"2024-05-09T08:23:50.565910+0000","last_became_peered":"2024-05-09T08:23:50.565910+0000","last_unstale":"2024-05-09T08:24:17.393140+0000","last_undegraded":"2024-05-09T08:24:17.393140+0000","last_fullsized":"2024-05-09T08:24:17.393140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:02.211551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488098+0000","last_change":"2024-05-09T08:23:51.570972+0000","last_active":"2024-05-09T08:23:57.488098+0000","last_peered":"2024-05-09T08:23:57.488098+0000","last_clean":"2024-05-09T08:23:57.488098+0000","last_became_active":"2024-05-09T08:23:51.570638+0000","last_became_peered":"2024-05-09T08:23:51.570638+0000","last_unstale":"2024-05-09T08:23:57.488098+0000","last_undegraded":"2024-05-09T08:23:57.488098+0000","last_fullsized":"2024-05-09T08:23:57.488098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:38.738820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921800+0000","last_change":"2024-05-09T08:23:55.671929+0000","last_active":"2024-05-09T08:23:56.921800+0000","last_peered":"2024-05-09T08:23:56.921800+0000","last_clean":"2024-05-09T08:23:56.921800+0000","last_became_active":"2024-05-09T08:23:55.671483+0000","last_became_peered":"2024-05-09T08:23:55.671483+0000","last_unstale":"2024-05-09T08:23:56.921800+0000","last_undegraded":"2024-05-09T08:23:56.921800+0000","last_fullsized":"2024-05-09T08:23:56.921800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:21.527266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469748+0000","last_change":"2024-05-09T08:23:50.565644+0000","last_active":"2024-05-09T08:23:57.469748+0000","last_peered":"2024-05-09T08:23:57.469748+0000","last_clean":"2024-05-09T08:23:57.469748+0000","last_became_active":"2024-05-09T08:23:50.565265+0000","last_became_peered":"2024-05-09T08:23:50.565265+0000","last_unstale":"2024-05-09T08:23:57.469748+0000","last_undegraded":"2024-05-09T08:23:57.469748+0000","last_fullsized":"2024-05-09T08:23:57.469748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:51:57.972535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023089+0000","last_change":"2024-05-09T08:23:51.571732+0000","last_active":"2024-05-09T08:23:58.023089+0000","last_peered":"2024-05-09T08:23:58.023089+0000","last_clean":"2024-05-09T08:23:58.023089+0000","last_became_active":"2024-05-09T08:23:51.570783+0000","last_became_peered":"2024-05-09T08:23:51.570783+0000","last_unstale":"2024-05-09T08:23:58.023089+0000","last_undegraded":"2024-05-09T08:23:58.023089+0000","last_fullsized":"2024-05-09T08:23:58.023089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:18.515905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.655000+0000","last_change":"2024-05-09T08:23:55.666727+0000","last_active":"2024-05-09T08:23:56.655000+0000","last_peered":"2024-05-09T08:23:56.655000+0000","last_clean":"2024-05-09T08:23:56.655000+0000","last_became_active":"2024-05-09T08:23:55.666284+0000","last_became_peered":"2024-05-09T08:23:55.666284+0000","last_unstale":"2024-05-09T08:23:56.655000+0000","last_undegraded":"2024-05-09T08:23:56.655000+0000","last_fullsized":"2024-05-09T08:23:56.655000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:05.355404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378806+0000","last_change":"2024-05-09T08:23:50.573308+0000","last_active":"2024-05-09T08:24:15.378806+0000","last_peered":"2024-05-09T08:24:15.378806+0000","last_clean":"2024-05-09T08:24:15.378806+0000","last_became_active":"2024-05-09T08:23:50.572946+0000","last_became_peered":"2024-05-09T08:23:50.572946+0000","last_unstale":"2024-05-09T08:24:15.378806+0000","last_undegraded":"2024-05-09T08:24:15.378806+0000","last_fullsized":"2024-05-09T08:24:15.378806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:44:35.816620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469752+0000","last_change":"2024-05-09T08:23:51.571748+0000","last_active":"2024-05-09T08:23:57.469752+0000","last_peered":"2024-05-09T08:23:57.469752+0000","last_clean":"2024-05-09T08:23:57.469752+0000","last_became_active":"2024-05-09T08:23:51.571388+0000","last_became_peered":"2024-05-09T08:23:51.571388+0000","last_unstale":"2024-05-09T08:23:57.469752+0000","last_undegraded":"2024-05-09T08:23:57.469752+0000","last_fullsized":"2024-05-09T08:23:57.469752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:34.916491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654957+0000","last_change":"2024-05-09T08:23:55.674022+0000","last_active":"2024-05-09T08:23:56.654957+0000","last_peered":"2024-05-09T08:23:56.654957+0000","last_clean":"2024-05-09T08:23:56.654957+0000","last_became_active":"2024-05-09T08:23:55.673322+0000","last_became_peered":"2024-05-09T08:23:55.673322+0000","last_unstale":"2024-05-09T08:23:56.654957+0000","last_undegraded":"2024-05-09T08:23:56.654957+0000","last_fullsized":"2024-05-09T08:23:56.654957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:59:14.299886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654276+0000","last_change":"2024-05-09T08:23:50.571235+0000","last_active":"2024-05-09T08:23:56.654276+0000","last_peered":"2024-05-09T08:23:56.654276+0000","last_clean":"2024-05-09T08:23:56.654276+0000","last_became_active":"2024-05-09T08:23:50.570694+0000","last_became_peered":"2024-05-09T08:23:50.570694+0000","last_unstale":"2024-05-09T08:23:56.654276+0000","last_undegraded":"2024-05-09T08:23:56.654276+0000","last_fullsized":"2024-05-09T08:23:56.654276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:22.850248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022898+0000","last_change":"2024-05-09T08:23:51.572387+0000","last_active":"2024-05-09T08:23:58.022898+0000","last_peered":"2024-05-09T08:23:58.022898+0000","last_clean":"2024-05-09T08:23:58.022898+0000","last_became_active":"2024-05-09T08:23:51.572038+0000","last_became_peered":"2024-05-09T08:23:51.572038+0000","last_unstale":"2024-05-09T08:23:58.022898+0000","last_undegraded":"2024-05-09T08:23:58.022898+0000","last_fullsized":"2024-05-09T08:23:58.022898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:38.118125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329864+0000","last_change":"2024-05-09T08:23:55.665885+0000","last_active":"2024-05-09T08:23:57.329864+0000","last_peered":"2024-05-09T08:23:57.329864+0000","last_clean":"2024-05-09T08:23:57.329864+0000","last_became_active":"2024-05-09T08:23:55.665422+0000","last_became_peered":"2024-05-09T08:23:55.665422+0000","last_unstale":"2024-05-09T08:23:57.329864+0000","last_undegraded":"2024-05-09T08:23:57.329864+0000","last_fullsized":"2024-05-09T08:23:57.329864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:12:37.952612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.388806+0000","last_change":"2024-05-09T08:23:50.570040+0000","last_active":"2024-05-09T08:24:16.388806+0000","last_peered":"2024-05-09T08:24:16.388806+0000","last_clean":"2024-05-09T08:24:16.388806+0000","last_became_active":"2024-05-09T08:23:50.569665+0000","last_became_peered":"2024-05-09T08:23:50.569665+0000","last_unstale":"2024-05-09T08:24:16.388806+0000","last_undegraded":"2024-05-09T08:24:16.388806+0000","last_fullsized":"2024-05-09T08:24:16.388806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:22:29.639712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022076+0000","last_change":"2024-05-09T08:23:51.568561+0000","last_active":"2024-05-09T08:23:58.022076+0000","last_peered":"2024-05-09T08:23:58.022076+0000","last_clean":"2024-05-09T08:23:58.022076+0000","last_became_active":"2024-05-09T08:23:51.568196+0000","last_became_peered":"2024-05-09T08:23:51.568196+0000","last_unstale":"2024-05-09T08:23:58.022076+0000","last_undegraded":"2024-05-09T08:23:58.022076+0000","last_fullsized":"2024-05-09T08:23:58.022076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:52.132837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.301020+0000","last_change":"2024-05-09T08:23:55.674960+0000","last_active":"2024-05-09T08:23:58.301020+0000","last_peered":"2024-05-09T08:23:58.301020+0000","last_clean":"2024-05-09T08:23:58.301020+0000","last_became_active":"2024-05-09T08:23:55.674509+0000","last_became_peered":"2024-05-09T08:23:55.674509+0000","last_unstale":"2024-05-09T08:23:58.301020+0000","last_undegraded":"2024-05-09T08:23:58.301020+0000","last_fullsized":"2024-05-09T08:23:58.301020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:22.479867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022713+0000","last_change":"2024-05-09T08:23:50.569896+0000","last_active":"2024-05-09T08:23:58.022713+0000","last_peered":"2024-05-09T08:23:58.022713+0000","last_clean":"2024-05-09T08:23:58.022713+0000","last_became_active":"2024-05-09T08:23:50.569562+0000","last_became_peered":"2024-05-09T08:23:50.569562+0000","last_unstale":"2024-05-09T08:23:58.022713+0000","last_undegraded":"2024-05-09T08:23:58.022713+0000","last_fullsized":"2024-05-09T08:23:58.022713+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:25:20.278726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654129+0000","last_change":"2024-05-09T08:23:51.570260+0000","last_active":"2024-05-09T08:23:56.654129+0000","last_peered":"2024-05-09T08:23:56.654129+0000","last_clean":"2024-05-09T08:23:56.654129+0000","last_became_active":"2024-05-09T08:23:51.569837+0000","last_became_peered":"2024-05-09T08:23:51.569837+0000","last_unstale":"2024-05-09T08:23:56.654129+0000","last_undegraded":"2024-05-09T08:23:56.654129+0000","last_fullsized":"2024-05-09T08:23:56.654129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:01:44.805271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023027+0000","last_change":"2024-05-09T08:23:55.676639+0000","last_active":"2024-05-09T08:23:58.023027+0000","last_peered":"2024-05-09T08:23:58.023027+0000","last_clean":"2024-05-09T08:23:58.023027+0000","last_became_active":"2024-05-09T08:23:55.673643+0000","last_became_peered":"2024-05-09T08:23:55.673643+0000","last_unstale":"2024-05-09T08:23:58.023027+0000","last_undegraded":"2024-05-09T08:23:58.023027+0000","last_fullsized":"2024-05-09T08:23:58.023027+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:02.377184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378337+0000","last_change":"2024-05-09T08:23:50.565745+0000","last_active":"2024-05-09T08:24:15.378337+0000","last_peered":"2024-05-09T08:24:15.378337+0000","last_clean":"2024-05-09T08:24:15.378337+0000","last_became_active":"2024-05-09T08:23:50.565385+0000","last_became_peered":"2024-05-09T08:23:50.565385+0000","last_unstale":"2024-05-09T08:24:15.378337+0000","last_undegraded":"2024-05-09T08:24:15.378337+0000","last_fullsized":"2024-05-09T08:24:15.378337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:05:00.818398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488401+0000","last_change":"2024-05-09T08:23:51.572341+0000","last_active":"2024-05-09T08:23:57.488401+0000","last_peered":"2024-05-09T08:23:57.488401+0000","last_clean":"2024-05-09T08:23:57.488401+0000","last_became_active":"2024-05-09T08:23:51.571583+0000","last_became_peered":"2024-05-09T08:23:51.571583+0000","last_unstale":"2024-05-09T08:23:57.488401+0000","last_undegraded":"2024-05-09T08:23:57.488401+0000","last_fullsized":"2024-05-09T08:23:57.488401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:20.695995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654079+0000","last_change":"2024-05-09T08:23:55.677603+0000","last_active":"2024-05-09T08:23:56.654079+0000","last_peered":"2024-05-09T08:23:56.654079+0000","last_clean":"2024-05-09T08:23:56.654079+0000","last_became_active":"2024-05-09T08:23:55.677143+0000","last_became_peered":"2024-05-09T08:23:55.677143+0000","last_unstale":"2024-05-09T08:23:56.654079+0000","last_undegraded":"2024-05-09T08:23:56.654079+0000","last_fullsized":"2024-05-09T08:23:56.654079+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:27.850997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378182+0000","last_change":"2024-05-09T08:23:50.565497+0000","last_active":"2024-05-09T08:24:15.378182+0000","last_peered":"2024-05-09T08:24:15.378182+0000","last_clean":"2024-05-09T08:24:15.378182+0000","last_became_active":"2024-05-09T08:23:50.565120+0000","last_became_peered":"2024-05-09T08:23:50.565120+0000","last_unstale":"2024-05-09T08:24:15.378182+0000","last_undegraded":"2024-05-09T08:24:15.378182+0000","last_fullsized":"2024-05-09T08:24:15.378182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:56.141997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.301086+0000","last_change":"2024-05-09T08:23:51.576489+0000","last_active":"2024-05-09T08:23:58.301086+0000","last_peered":"2024-05-09T08:23:58.301086+0000","last_clean":"2024-05-09T08:23:58.301086+0000","last_became_active":"2024-05-09T08:23:51.576145+0000","last_became_peered":"2024-05-09T08:23:51.576145+0000","last_unstale":"2024-05-09T08:23:58.301086+0000","last_undegraded":"2024-05-09T08:23:58.301086+0000","last_fullsized":"2024-05-09T08:23:58.301086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:37.650308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.922006+0000","last_change":"2024-05-09T08:23:55.666181+0000","last_active":"2024-05-09T08:23:56.922006+0000","last_peered":"2024-05-09T08:23:56.922006+0000","last_clean":"2024-05-09T08:23:56.922006+0000","last_became_active":"2024-05-09T08:23:55.665795+0000","last_became_peered":"2024-05-09T08:23:55.665795+0000","last_unstale":"2024-05-09T08:23:56.922006+0000","last_undegraded":"2024-05-09T08:23:56.922006+0000","last_fullsized":"2024-05-09T08:23:56.922006+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:48.659719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023098+0000","last_change":"2024-05-09T08:23:50.567182+0000","last_active":"2024-05-09T08:23:58.023098+0000","last_peered":"2024-05-09T08:23:58.023098+0000","last_clean":"2024-05-09T08:23:58.023098+0000","last_became_active":"2024-05-09T08:23:50.566851+0000","last_became_peered":"2024-05-09T08:23:50.566851+0000","last_unstale":"2024-05-09T08:23:58.023098+0000","last_undegraded":"2024-05-09T08:23:58.023098+0000","last_fullsized":"2024-05-09T08:23:58.023098+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:26.540456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488226+0000","last_change":"2024-05-09T08:23:51.574074+0000","last_active":"2024-05-09T08:23:57.488226+0000","last_peered":"2024-05-09T08:23:57.488226+0000","last_clean":"2024-05-09T08:23:57.488226+0000","last_became_active":"2024-05-09T08:23:51.572587+0000","last_became_peered":"2024-05-09T08:23:51.572587+0000","last_unstale":"2024-05-09T08:23:57.488226+0000","last_undegraded":"2024-05-09T08:23:57.488226+0000","last_fullsized":"2024-05-09T08:23:57.488226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:46.186358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654670+0000","last_change":"2024-05-09T08:23:55.667170+0000","last_active":"2024-05-09T08:23:56.654670+0000","last_peered":"2024-05-09T08:23:56.654670+0000","last_clean":"2024-05-09T08:23:56.654670+0000","last_became_active":"2024-05-09T08:23:55.666707+0000","last_became_peered":"2024-05-09T08:23:55.666707+0000","last_unstale":"2024-05-09T08:23:56.654670+0000","last_undegraded":"2024-05-09T08:23:56.654670+0000","last_fullsized":"2024-05-09T08:23:56.654670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:48.332207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.377419+0000","last_change":"2024-05-09T08:23:50.572694+0000","last_active":"2024-05-09T08:24:15.377419+0000","last_peered":"2024-05-09T08:24:15.377419+0000","last_clean":"2024-05-09T08:24:15.377419+0000","last_became_active":"2024-05-09T08:23:50.572335+0000","last_became_peered":"2024-05-09T08:23:50.572335+0000","last_unstale":"2024-05-09T08:24:15.377419+0000","last_undegraded":"2024-05-09T08:24:15.377419+0000","last_fullsized":"2024-05-09T08:24:15.377419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:28:24.028910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469990+0000","last_change":"2024-05-09T08:23:51.577595+0000","last_active":"2024-05-09T08:23:57.469990+0000","last_peered":"2024-05-09T08:23:57.469990+0000","last_clean":"2024-05-09T08:23:57.469990+0000","last_became_active":"2024-05-09T08:23:51.577271+0000","last_became_peered":"2024-05-09T08:23:51.577271+0000","last_unstale":"2024-05-09T08:23:57.469990+0000","last_undegraded":"2024-05-09T08:23:57.469990+0000","last_fullsized":"2024-05-09T08:23:57.469990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:34.679111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023394+0000","last_change":"2024-05-09T08:23:55.671128+0000","last_active":"2024-05-09T08:23:58.023394+0000","last_peered":"2024-05-09T08:23:58.023394+0000","last_clean":"2024-05-09T08:23:58.023394+0000","last_became_active":"2024-05-09T08:23:55.670414+0000","last_became_peered":"2024-05-09T08:23:55.670414+0000","last_unstale":"2024-05-09T08:23:58.023394+0000","last_undegraded":"2024-05-09T08:23:58.023394+0000","last_fullsized":"2024-05-09T08:23:58.023394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:31:55.307412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.388632+0000","last_change":"2024-05-09T08:23:50.568503+0000","last_active":"2024-05-09T08:24:16.388632+0000","last_peered":"2024-05-09T08:24:16.388632+0000","last_clean":"2024-05-09T08:24:16.388632+0000","last_became_active":"2024-05-09T08:23:50.568181+0000","last_became_peered":"2024-05-09T08:23:50.568181+0000","last_unstale":"2024-05-09T08:24:16.388632+0000","last_undegraded":"2024-05-09T08:24:16.388632+0000","last_fullsized":"2024-05-09T08:24:16.388632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:40.883792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022063+0000","last_change":"2024-05-09T08:23:51.573602+0000","last_active":"2024-05-09T08:23:58.022063+0000","last_peered":"2024-05-09T08:23:58.022063+0000","last_clean":"2024-05-09T08:23:58.022063+0000","last_became_active":"2024-05-09T08:23:51.572557+0000","last_became_peered":"2024-05-09T08:23:51.572557+0000","last_unstale":"2024-05-09T08:23:58.022063+0000","last_undegraded":"2024-05-09T08:23:58.022063+0000","last_fullsized":"2024-05-09T08:23:58.022063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.170962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329640+0000","last_change":"2024-05-09T08:23:55.673773+0000","last_active":"2024-05-09T08:23:57.329640+0000","last_peered":"2024-05-09T08:23:57.329640+0000","last_clean":"2024-05-09T08:23:57.329640+0000","last_became_active":"2024-05-09T08:23:55.673276+0000","last_became_peered":"2024-05-09T08:23:55.673276+0000","last_unstale":"2024-05-09T08:23:57.329640+0000","last_undegraded":"2024-05-09T08:23:57.329640+0000","last_fullsized":"2024-05-09T08:23:57.329640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:59:00.629339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378394+0000","last_change":"2024-05-09T08:23:50.575894+0000","last_active":"2024-05-09T08:24:15.378394+0000","last_peered":"2024-05-09T08:24:15.378394+0000","last_clean":"2024-05-09T08:24:15.378394+0000","last_became_active":"2024-05-09T08:23:50.575555+0000","last_became_peered":"2024-05-09T08:23:50.575555+0000","last_unstale":"2024-05-09T08:24:15.378394+0000","last_undegraded":"2024-05-09T08:24:15.378394+0000","last_fullsized":"2024-05-09T08:24:15.378394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:19.070764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022381+0000","last_change":"2024-05-09T08:23:51.574125+0000","last_active":"2024-05-09T08:23:58.022381+0000","last_peered":"2024-05-09T08:23:58.022381+0000","last_clean":"2024-05-09T08:23:58.022381+0000","last_became_active":"2024-05-09T08:23:51.573794+0000","last_became_peered":"2024-05-09T08:23:51.573794+0000","last_unstale":"2024-05-09T08:23:58.022381+0000","last_undegraded":"2024-05-09T08:23:58.022381+0000","last_fullsized":"2024-05-09T08:23:58.022381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:19:26.634208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654685+0000","last_change":"2024-05-09T08:23:55.677711+0000","last_active":"2024-05-09T08:23:56.654685+0000","last_peered":"2024-05-09T08:23:56.654685+0000","last_clean":"2024-05-09T08:23:56.654685+0000","last_became_active":"2024-05-09T08:23:55.677058+0000","last_became_peered":"2024-05-09T08:23:55.677058+0000","last_unstale":"2024-05-09T08:23:56.654685+0000","last_undegraded":"2024-05-09T08:23:56.654685+0000","last_fullsized":"2024-05-09T08:23:56.654685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:26.842145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.393470+0000","last_change":"2024-05-09T08:23:50.566413+0000","last_active":"2024-05-09T08:24:17.393470+0000","last_peered":"2024-05-09T08:24:17.393470+0000","last_clean":"2024-05-09T08:24:17.393470+0000","last_became_active":"2024-05-09T08:23:50.566047+0000","last_became_peered":"2024-05-09T08:23:50.566047+0000","last_unstale":"2024-05-09T08:24:17.393470+0000","last_undegraded":"2024-05-09T08:24:17.393470+0000","last_fullsized":"2024-05-09T08:24:17.393470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:15.340947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022166+0000","last_change":"2024-05-09T08:23:51.576779+0000","last_active":"2024-05-09T08:23:58.022166+0000","last_peered":"2024-05-09T08:23:58.022166+0000","last_clean":"2024-05-09T08:23:58.022166+0000","last_became_active":"2024-05-09T08:23:51.576119+0000","last_became_peered":"2024-05-09T08:23:51.576119+0000","last_unstale":"2024-05-09T08:23:58.022166+0000","last_undegraded":"2024-05-09T08:23:58.022166+0000","last_fullsized":"2024-05-09T08:23:58.022166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:02.250135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022592+0000","last_change":"2024-05-09T08:23:55.678564+0000","last_active":"2024-05-09T08:23:58.022592+0000","last_peered":"2024-05-09T08:23:58.022592+0000","last_clean":"2024-05-09T08:23:58.022592+0000","last_became_active":"2024-05-09T08:23:55.678128+0000","last_became_peered":"2024-05-09T08:23:55.678128+0000","last_unstale":"2024-05-09T08:23:58.022592+0000","last_undegraded":"2024-05-09T08:23:58.022592+0000","last_fullsized":"2024-05-09T08:23:58.022592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:13:21.868267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.389391+0000","last_change":"2024-05-09T08:23:50.565062+0000","last_active":"2024-05-09T08:24:16.389391+0000","last_peered":"2024-05-09T08:24:16.389391+0000","last_clean":"2024-05-09T08:24:16.389391+0000","last_became_active":"2024-05-09T08:23:50.564393+0000","last_became_peered":"2024-05-09T08:23:50.564393+0000","last_unstale":"2024-05-09T08:24:16.389391+0000","last_undegraded":"2024-05-09T08:24:16.389391+0000","last_fullsized":"2024-05-09T08:24:16.389391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:08:35.060335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022608+0000","last_change":"2024-05-09T08:23:51.574914+0000","last_active":"2024-05-09T08:23:58.022608+0000","last_peered":"2024-05-09T08:23:58.022608+0000","last_clean":"2024-05-09T08:23:58.022608+0000","last_became_active":"2024-05-09T08:23:51.574380+0000","last_became_peered":"2024-05-09T08:23:51.574380+0000","last_unstale":"2024-05-09T08:23:58.022608+0000","last_undegraded":"2024-05-09T08:23:58.022608+0000","last_fullsized":"2024-05-09T08:23:58.022608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:34.698328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022830+0000","last_change":"2024-05-09T08:23:55.670198+0000","last_active":"2024-05-09T08:23:58.022830+0000","last_peered":"2024-05-09T08:23:58.022830+0000","last_clean":"2024-05-09T08:23:58.022830+0000","last_became_active":"2024-05-09T08:23:55.669798+0000","last_became_peered":"2024-05-09T08:23:55.669798+0000","last_unstale":"2024-05-09T08:23:58.022830+0000","last_undegraded":"2024-05-09T08:23:58.022830+0000","last_fullsized":"2024-05-09T08:23:58.022830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:09.474264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.470141+0000","last_change":"2024-05-09T08:23:50.565061+0000","last_active":"2024-05-09T08:23:57.470141+0000","last_peered":"2024-05-09T08:23:57.470141+0000","last_clean":"2024-05-09T08:23:57.470141+0000","last_became_active":"2024-05-09T08:23:50.564736+0000","last_became_peered":"2024-05-09T08:23:50.564736+0000","last_unstale":"2024-05-09T08:23:57.470141+0000","last_undegraded":"2024-05-09T08:23:57.470141+0000","last_fullsized":"2024-05-09T08:23:57.470141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:33.593543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329780+0000","last_change":"2024-05-09T08:23:51.573434+0000","last_active":"2024-05-09T08:23:57.329780+0000","last_peered":"2024-05-09T08:23:57.329780+0000","last_clean":"2024-05-09T08:23:57.329780+0000","last_became_active":"2024-05-09T08:23:51.573081+0000","last_became_peered":"2024-05-09T08:23:51.573081+0000","last_unstale":"2024-05-09T08:23:57.329780+0000","last_undegraded":"2024-05-09T08:23:57.329780+0000","last_fullsized":"2024-05-09T08:23:57.329780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:49.270355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469927+0000","last_change":"2024-05-09T08:23:55.853114+0000","last_active":"2024-05-09T08:23:57.469927+0000","last_peered":"2024-05-09T08:23:57.469927+0000","last_clean":"2024-05-09T08:23:57.469927+0000","last_became_active":"2024-05-09T08:23:55.852640+0000","last_became_peered":"2024-05-09T08:23:55.852640+0000","last_unstale":"2024-05-09T08:23:57.469927+0000","last_undegraded":"2024-05-09T08:23:57.469927+0000","last_fullsized":"2024-05-09T08:23:57.469927+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:55:17.823585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654431+0000","last_change":"2024-05-09T08:23:50.567834+0000","last_active":"2024-05-09T08:23:56.654431+0000","last_peered":"2024-05-09T08:23:56.654431+0000","last_clean":"2024-05-09T08:23:56.654431+0000","last_became_active":"2024-05-09T08:23:50.567538+0000","last_became_peered":"2024-05-09T08:23:50.567538+0000","last_unstale":"2024-05-09T08:23:56.654431+0000","last_undegraded":"2024-05-09T08:23:56.654431+0000","last_fullsized":"2024-05-09T08:23:56.654431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.674352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654617+0000","last_change":"2024-05-09T08:23:51.580219+0000","last_active":"2024-05-09T08:23:56.654617+0000","last_peered":"2024-05-09T08:23:56.654617+0000","last_clean":"2024-05-09T08:23:56.654617+0000","last_became_active":"2024-05-09T08:23:51.579835+0000","last_became_peered":"2024-05-09T08:23:51.579835+0000","last_unstale":"2024-05-09T08:23:56.654617+0000","last_undegraded":"2024-05-09T08:23:56.654617+0000","last_fullsized":"2024-05-09T08:23:56.654617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:15.232318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.021951+0000","last_change":"2024-05-09T08:23:55.674324+0000","last_active":"2024-05-09T08:23:58.021951+0000","last_peered":"2024-05-09T08:23:58.021951+0000","last_clean":"2024-05-09T08:23:58.021951+0000","last_became_active":"2024-05-09T08:23:55.673254+0000","last_became_peered":"2024-05-09T08:23:55.673254+0000","last_unstale":"2024-05-09T08:23:58.021951+0000","last_undegraded":"2024-05-09T08:23:58.021951+0000","last_fullsized":"2024-05-09T08:23:58.021951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:05:23.303987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022220+0000","last_change":"2024-05-09T08:23:50.571682+0000","last_active":"2024-05-09T08:23:58.022220+0000","last_peered":"2024-05-09T08:23:58.022220+0000","last_clean":"2024-05-09T08:23:58.022220+0000","last_became_active":"2024-05-09T08:23:50.571338+0000","last_became_peered":"2024-05-09T08:23:50.571338+0000","last_unstale":"2024-05-09T08:23:58.022220+0000","last_undegraded":"2024-05-09T08:23:58.022220+0000","last_fullsized":"2024-05-09T08:23:58.022220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:17.578071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654373+0000","last_change":"2024-05-09T08:23:51.580048+0000","last_active":"2024-05-09T08:23:56.654373+0000","last_peered":"2024-05-09T08:23:56.654373+0000","last_clean":"2024-05-09T08:23:56.654373+0000","last_became_active":"2024-05-09T08:23:51.579725+0000","last_became_peered":"2024-05-09T08:23:51.579725+0000","last_unstale":"2024-05-09T08:23:56.654373+0000","last_undegraded":"2024-05-09T08:23:56.654373+0000","last_fullsized":"2024-05-09T08:23:56.654373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:14:47.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022365+0000","last_change":"2024-05-09T08:23:55.853086+0000","last_active":"2024-05-09T08:23:58.022365+0000","last_peered":"2024-05-09T08:23:58.022365+0000","last_clean":"2024-05-09T08:23:58.022365+0000","last_became_active":"2024-05-09T08:23:55.852302+0000","last_became_peered":"2024-05-09T08:23:55.852302+0000","last_unstale":"2024-05-09T08:23:58.022365+0000","last_undegraded":"2024-05-09T08:23:58.022365+0000","last_fullsized":"2024-05-09T08:23:58.022365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:42:18.547623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022780+0000","last_change":"2024-05-09T08:23:51.575522+0000","last_active":"2024-05-09T08:23:58.022780+0000","last_peered":"2024-05-09T08:23:58.022780+0000","last_clean":"2024-05-09T08:23:58.022780+0000","last_became_active":"2024-05-09T08:23:51.575204+0000","last_became_peered":"2024-05-09T08:23:51.575204+0000","last_unstale":"2024-05-09T08:23:58.022780+0000","last_undegraded":"2024-05-09T08:23:58.022780+0000","last_fullsized":"2024-05-09T08:23:58.022780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:57.257435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.388636+0000","last_change":"2024-05-09T08:23:50.574996+0000","last_active":"2024-05-09T08:24:16.388636+0000","last_peered":"2024-05-09T08:24:16.388636+0000","last_clean":"2024-05-09T08:24:16.388636+0000","last_became_active":"2024-05-09T08:23:50.574619+0000","last_became_peered":"2024-05-09T08:23:50.574619+0000","last_unstale":"2024-05-09T08:24:16.388636+0000","last_undegraded":"2024-05-09T08:24:16.388636+0000","last_fullsized":"2024-05-09T08:24:16.388636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:44.681314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022504+0000","last_change":"2024-05-09T08:23:55.667214+0000","last_active":"2024-05-09T08:23:58.022504+0000","last_peered":"2024-05-09T08:23:58.022504+0000","last_clean":"2024-05-09T08:23:58.022504+0000","last_became_active":"2024-05-09T08:23:55.666716+0000","last_became_peered":"2024-05-09T08:23:55.666716+0000","last_unstale":"2024-05-09T08:23:58.022504+0000","last_undegraded":"2024-05-09T08:23:58.022504+0000","last_fullsized":"2024-05-09T08:23:58.022504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:32:05.332938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022096+0000","last_change":"2024-05-09T08:23:51.573944+0000","last_active":"2024-05-09T08:23:58.022096+0000","last_peered":"2024-05-09T08:23:58.022096+0000","last_clean":"2024-05-09T08:23:58.022096+0000","last_became_active":"2024-05-09T08:23:51.573229+0000","last_became_peered":"2024-05-09T08:23:51.573229+0000","last_unstale":"2024-05-09T08:23:58.022096+0000","last_undegraded":"2024-05-09T08:23:58.022096+0000","last_fullsized":"2024-05-09T08:23:58.022096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:53.828582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.393399+0000","last_change":"2024-05-09T08:23:50.566415+0000","last_active":"2024-05-09T08:24:17.393399+0000","last_peered":"2024-05-09T08:24:17.393399+0000","last_clean":"2024-05-09T08:24:17.393399+0000","last_became_active":"2024-05-09T08:23:50.565506+0000","last_became_peered":"2024-05-09T08:23:50.565506+0000","last_unstale":"2024-05-09T08:24:17.393399+0000","last_undegraded":"2024-05-09T08:24:17.393399+0000","last_fullsized":"2024-05-09T08:24:17.393399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:15.741565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022001+0000","last_change":"2024-05-09T08:22:38.366126+0000","last_active":"2024-05-09T08:23:58.022001+0000","last_peered":"2024-05-09T08:23:58.022001+0000","last_clean":"2024-05-09T08:23:58.022001+0000","last_became_active":"2024-05-09T08:22:38.365683+0000","last_became_peered":"2024-05-09T08:22:38.365683+0000","last_unstale":"2024-05-09T08:23:58.022001+0000","last_undegraded":"2024-05-09T08:23:58.022001+0000","last_fullsized":"2024-05-09T08:23:58.022001+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_clean_scrub_stamp":"2024-05-09T08:19:12.435445+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:04.744060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023282+0000","last_change":"2024-05-09T08:23:55.853248+0000","last_active":"2024-05-09T08:23:58.023282+0000","last_peered":"2024-05-09T08:23:58.023282+0000","last_clean":"2024-05-09T08:23:58.023282+0000","last_became_active":"2024-05-09T08:23:55.852646+0000","last_became_peered":"2024-05-09T08:23:55.852646+0000","last_unstale":"2024-05-09T08:23:58.023282+0000","last_undegraded":"2024-05-09T08:23:58.023282+0000","last_fullsized":"2024-05-09T08:23:58.023282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:19:54.378185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.021652+0000","last_change":"2024-05-09T08:23:51.574967+0000","last_active":"2024-05-09T08:23:58.021652+0000","last_peered":"2024-05-09T08:23:58.021652+0000","last_clean":"2024-05-09T08:23:58.021652+0000","last_became_active":"2024-05-09T08:23:51.574600+0000","last_became_peered":"2024-05-09T08:23:51.574600+0000","last_unstale":"2024-05-09T08:23:58.021652+0000","last_undegraded":"2024-05-09T08:23:58.021652+0000","last_fullsized":"2024-05-09T08:23:58.021652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:03.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.387812+0000","last_change":"2024-05-09T08:23:50.569886+0000","last_active":"2024-05-09T08:24:16.387812+0000","last_peered":"2024-05-09T08:24:16.387812+0000","last_clean":"2024-05-09T08:24:16.387812+0000","last_became_active":"2024-05-09T08:23:50.569018+0000","last_became_peered":"2024-05-09T08:23:50.569018+0000","last_unstale":"2024-05-09T08:24:16.387812+0000","last_undegraded":"2024-05-09T08:24:16.387812+0000","last_fullsized":"2024-05-09T08:24:16.387812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:20.711485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329690+0000","last_change":"2024-05-09T08:23:55.676408+0000","last_active":"2024-05-09T08:23:57.329690+0000","last_peered":"2024-05-09T08:23:57.329690+0000","last_clean":"2024-05-09T08:23:57.329690+0000","last_became_active":"2024-05-09T08:23:55.675964+0000","last_became_peered":"2024-05-09T08:23:55.675964+0000","last_unstale":"2024-05-09T08:23:57.329690+0000","last_undegraded":"2024-05-09T08:23:57.329690+0000","last_fullsized":"2024-05-09T08:23:57.329690+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:29.210602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.392980+0000","last_change":"2024-05-09T08:23:50.566946+0000","last_active":"2024-05-09T08:24:17.392980+0000","last_peered":"2024-05-09T08:24:17.392980+0000","last_clean":"2024-05-09T08:24:17.392980+0000","last_became_active":"2024-05-09T08:23:50.566366+0000","last_became_peered":"2024-05-09T08:23:50.566366+0000","last_unstale":"2024-05-09T08:24:17.392980+0000","last_undegraded":"2024-05-09T08:24:17.392980+0000","last_fullsized":"2024-05-09T08:24:17.392980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:37:30.720879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022521+0000","last_change":"2024-05-09T08:23:51.571520+0000","last_active":"2024-05-09T08:23:58.022521+0000","last_peered":"2024-05-09T08:23:58.022521+0000","last_clean":"2024-05-09T08:23:58.022521+0000","last_became_active":"2024-05-09T08:23:51.571166+0000","last_became_peered":"2024-05-09T08:23:51.571166+0000","last_unstale":"2024-05-09T08:23:58.022521+0000","last_undegraded":"2024-05-09T08:23:58.022521+0000","last_fullsized":"2024-05-09T08:23:58.022521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:02:12.560238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.021947+0000","last_change":"2024-05-09T08:23:55.852649+0000","last_active":"2024-05-09T08:23:58.021947+0000","last_peered":"2024-05-09T08:23:58.021947+0000","last_clean":"2024-05-09T08:23:58.021947+0000","last_became_active":"2024-05-09T08:23:55.852180+0000","last_became_peered":"2024-05-09T08:23:55.852180+0000","last_unstale":"2024-05-09T08:23:58.021947+0000","last_undegraded":"2024-05-09T08:23:58.021947+0000","last_fullsized":"2024-05-09T08:23:58.021947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:33.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654852+0000","last_change":"2024-05-09T08:23:50.561639+0000","last_active":"2024-05-09T08:23:56.654852+0000","last_peered":"2024-05-09T08:23:56.654852+0000","last_clean":"2024-05-09T08:23:56.654852+0000","last_became_active":"2024-05-09T08:23:50.561306+0000","last_became_peered":"2024-05-09T08:23:50.561306+0000","last_unstale":"2024-05-09T08:23:56.654852+0000","last_undegraded":"2024-05-09T08:23:56.654852+0000","last_fullsized":"2024-05-09T08:23:56.654852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:46.897965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022842+0000","last_change":"2024-05-09T08:23:51.573765+0000","last_active":"2024-05-09T08:23:58.022842+0000","last_peered":"2024-05-09T08:23:58.022842+0000","last_clean":"2024-05-09T08:23:58.022842+0000","last_became_active":"2024-05-09T08:23:51.573134+0000","last_became_peered":"2024-05-09T08:23:51.573134+0000","last_unstale":"2024-05-09T08:23:58.022842+0000","last_undegraded":"2024-05-09T08:23:58.022842+0000","last_fullsized":"2024-05-09T08:23:58.022842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:25.334148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654463+0000","last_change":"2024-05-09T08:23:55.663480+0000","last_active":"2024-05-09T08:23:56.654463+0000","last_peered":"2024-05-09T08:23:56.654463+0000","last_clean":"2024-05-09T08:23:56.654463+0000","last_became_active":"2024-05-09T08:23:55.662823+0000","last_became_peered":"2024-05-09T08:23:55.662823+0000","last_unstale":"2024-05-09T08:23:56.654463+0000","last_undegraded":"2024-05-09T08:23:56.654463+0000","last_fullsized":"2024-05-09T08:23:56.654463+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:18.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378366+0000","last_change":"2024-05-09T08:23:50.574641+0000","last_active":"2024-05-09T08:24:15.378366+0000","last_peered":"2024-05-09T08:24:15.378366+0000","last_clean":"2024-05-09T08:24:15.378366+0000","last_became_active":"2024-05-09T08:23:50.574266+0000","last_became_peered":"2024-05-09T08:23:50.574266+0000","last_unstale":"2024-05-09T08:24:15.378366+0000","last_undegraded":"2024-05-09T08:24:15.378366+0000","last_fullsized":"2024-05-09T08:24:15.378366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.575211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.301162+0000","last_change":"2024-05-09T08:23:51.583698+0000","last_active":"2024-05-09T08:23:58.301162+0000","last_peered":"2024-05-09T08:23:58.301162+0000","last_clean":"2024-05-09T08:23:58.301162+0000","last_became_active":"2024-05-09T08:23:51.583342+0000","last_became_peered":"2024-05-09T08:23:51.583342+0000","last_unstale":"2024-05-09T08:23:58.301162+0000","last_undegraded":"2024-05-09T08:23:58.301162+0000","last_fullsized":"2024-05-09T08:23:58.301162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:37.444163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921948+0000","last_change":"2024-05-09T08:23:55.853759+0000","last_active":"2024-05-09T08:23:56.921948+0000","last_peered":"2024-05-09T08:23:56.921948+0000","last_clean":"2024-05-09T08:23:56.921948+0000","last_became_active":"2024-05-09T08:23:55.853045+0000","last_became_peered":"2024-05-09T08:23:55.853045+0000","last_unstale":"2024-05-09T08:23:56.921948+0000","last_undegraded":"2024-05-09T08:23:56.921948+0000","last_fullsized":"2024-05-09T08:23:56.921948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:53.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.377837+0000","last_change":"2024-05-09T08:23:50.568373+0000","last_active":"2024-05-09T08:24:15.377837+0000","last_peered":"2024-05-09T08:24:15.377837+0000","last_clean":"2024-05-09T08:24:15.377837+0000","last_became_active":"2024-05-09T08:23:50.568015+0000","last_became_peered":"2024-05-09T08:23:50.568015+0000","last_unstale":"2024-05-09T08:24:15.377837+0000","last_undegraded":"2024-05-09T08:24:15.377837+0000","last_fullsized":"2024-05-09T08:24:15.377837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:05:16.391673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469519+0000","last_change":"2024-05-09T08:23:51.576825+0000","last_active":"2024-05-09T08:23:57.469519+0000","last_peered":"2024-05-09T08:23:57.469519+0000","last_clean":"2024-05-09T08:23:57.469519+0000","last_became_active":"2024-05-09T08:23:51.573350+0000","last_became_peered":"2024-05-09T08:23:51.573350+0000","last_unstale":"2024-05-09T08:23:57.469519+0000","last_undegraded":"2024-05-09T08:23:57.469519+0000","last_fullsized":"2024-05-09T08:23:57.469519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:00.963281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654788+0000","last_change":"2024-05-09T08:23:55.672145+0000","last_active":"2024-05-09T08:23:56.654788+0000","last_peered":"2024-05-09T08:23:56.654788+0000","last_clean":"2024-05-09T08:23:56.654788+0000","last_became_active":"2024-05-09T08:23:55.671743+0000","last_became_peered":"2024-05-09T08:23:55.671743+0000","last_unstale":"2024-05-09T08:23:56.654788+0000","last_undegraded":"2024-05-09T08:23:56.654788+0000","last_fullsized":"2024-05-09T08:23:56.654788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:09:23.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378677+0000","last_change":"2024-05-09T08:23:50.568908+0000","last_active":"2024-05-09T08:24:15.378677+0000","last_peered":"2024-05-09T08:24:15.378677+0000","last_clean":"2024-05-09T08:24:15.378677+0000","last_became_active":"2024-05-09T08:23:50.567009+0000","last_became_peered":"2024-05-09T08:23:50.567009+0000","last_unstale":"2024-05-09T08:24:15.378677+0000","last_undegraded":"2024-05-09T08:24:15.378677+0000","last_fullsized":"2024-05-09T08:24:15.378677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:35.190156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329478+0000","last_change":"2024-05-09T08:23:51.580121+0000","last_active":"2024-05-09T08:23:57.329478+0000","last_peered":"2024-05-09T08:23:57.329478+0000","last_clean":"2024-05-09T08:23:57.329478+0000","last_became_active":"2024-05-09T08:23:51.579402+0000","last_became_peered":"2024-05-09T08:23:51.579402+0000","last_unstale":"2024-05-09T08:23:57.329478+0000","last_undegraded":"2024-05-09T08:23:57.329478+0000","last_fullsized":"2024-05-09T08:23:57.329478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:11:26.144344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921851+0000","last_change":"2024-05-09T08:23:55.664017+0000","last_active":"2024-05-09T08:23:56.921851+0000","last_peered":"2024-05-09T08:23:56.921851+0000","last_clean":"2024-05-09T08:23:56.921851+0000","last_became_active":"2024-05-09T08:23:55.663546+0000","last_became_peered":"2024-05-09T08:23:55.663546+0000","last_unstale":"2024-05-09T08:23:56.921851+0000","last_undegraded":"2024-05-09T08:23:56.921851+0000","last_fullsized":"2024-05-09T08:23:56.921851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:13:14.705971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.373758+0000","last_change":"2024-05-09T08:23:50.568472+0000","last_active":"2024-05-09T08:24:17.373758+0000","last_peered":"2024-05-09T08:24:17.373758+0000","last_clean":"2024-05-09T08:24:17.373758+0000","last_became_active":"2024-05-09T08:23:50.568132+0000","last_became_peered":"2024-05-09T08:23:50.568132+0000","last_unstale":"2024-05-09T08:24:17.373758+0000","last_undegraded":"2024-05-09T08:24:17.373758+0000","last_fullsized":"2024-05-09T08:24:17.373758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:34.468411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022722+0000","last_change":"2024-05-09T08:23:51.571414+0000","last_active":"2024-05-09T08:23:58.022722+0000","last_peered":"2024-05-09T08:23:58.022722+0000","last_clean":"2024-05-09T08:23:58.022722+0000","last_became_active":"2024-05-09T08:23:51.571085+0000","last_became_peered":"2024-05-09T08:23:51.571085+0000","last_unstale":"2024-05-09T08:23:58.022722+0000","last_undegraded":"2024-05-09T08:23:58.022722+0000","last_fullsized":"2024-05-09T08:23:58.022722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:37.513584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921812+0000","last_change":"2024-05-09T08:23:55.852785+0000","last_active":"2024-05-09T08:23:56.921812+0000","last_peered":"2024-05-09T08:23:56.921812+0000","last_clean":"2024-05-09T08:23:56.921812+0000","last_became_active":"2024-05-09T08:23:55.852252+0000","last_became_peered":"2024-05-09T08:23:55.852252+0000","last_unstale":"2024-05-09T08:23:56.921812+0000","last_undegraded":"2024-05-09T08:23:56.921812+0000","last_fullsized":"2024-05-09T08:23:56.921812+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:44.503295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378953+0000","last_change":"2024-05-09T08:23:50.564787+0000","last_active":"2024-05-09T08:24:15.378953+0000","last_peered":"2024-05-09T08:24:15.378953+0000","last_clean":"2024-05-09T08:24:15.378953+0000","last_became_active":"2024-05-09T08:23:50.564409+0000","last_became_peered":"2024-05-09T08:23:50.564409+0000","last_unstale":"2024-05-09T08:24:15.378953+0000","last_undegraded":"2024-05-09T08:24:15.378953+0000","last_fullsized":"2024-05-09T08:24:15.378953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:23.051221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022284+0000","last_change":"2024-05-09T08:23:51.576830+0000","last_active":"2024-05-09T08:23:58.022284+0000","last_peered":"2024-05-09T08:23:58.022284+0000","last_clean":"2024-05-09T08:23:58.022284+0000","last_became_active":"2024-05-09T08:23:51.576311+0000","last_became_peered":"2024-05-09T08:23:51.576311+0000","last_unstale":"2024-05-09T08:23:58.022284+0000","last_undegraded":"2024-05-09T08:23:58.022284+0000","last_fullsized":"2024-05-09T08:23:58.022284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:26:08.651061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488040+0000","last_change":"2024-05-09T08:23:55.672309+0000","last_active":"2024-05-09T08:23:57.488040+0000","last_peered":"2024-05-09T08:23:57.488040+0000","last_clean":"2024-05-09T08:23:57.488040+0000","last_became_active":"2024-05-09T08:23:55.671811+0000","last_became_peered":"2024-05-09T08:23:55.671811+0000","last_unstale":"2024-05-09T08:23:57.488040+0000","last_undegraded":"2024-05-09T08:23:57.488040+0000","last_fullsized":"2024-05-09T08:23:57.488040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:19.400949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.395073+0000","last_change":"2024-05-09T08:23:50.575317+0000","last_active":"2024-05-09T08:24:17.395073+0000","last_peered":"2024-05-09T08:24:17.395073+0000","last_clean":"2024-05-09T08:24:17.395073+0000","last_became_active":"2024-05-09T08:23:50.574995+0000","last_became_peered":"2024-05-09T08:23:50.574995+0000","last_unstale":"2024-05-09T08:24:17.395073+0000","last_undegraded":"2024-05-09T08:24:17.395073+0000","last_fullsized":"2024-05-09T08:24:17.395073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:32:54.801799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022672+0000","last_change":"2024-05-09T08:23:51.575904+0000","last_active":"2024-05-09T08:23:58.022672+0000","last_peered":"2024-05-09T08:23:58.022672+0000","last_clean":"2024-05-09T08:23:58.022672+0000","last_became_active":"2024-05-09T08:23:51.575599+0000","last_became_peered":"2024-05-09T08:23:51.575599+0000","last_unstale":"2024-05-09T08:23:58.022672+0000","last_undegraded":"2024-05-09T08:23:58.022672+0000","last_fullsized":"2024-05-09T08:23:58.022672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:20:13.087096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300776+0000","last_change":"2024-05-09T08:23:55.680683+0000","last_active":"2024-05-09T08:23:58.300776+0000","last_peered":"2024-05-09T08:23:58.300776+0000","last_clean":"2024-05-09T08:23:58.300776+0000","last_became_active":"2024-05-09T08:23:55.680232+0000","last_became_peered":"2024-05-09T08:23:55.680232+0000","last_unstale":"2024-05-09T08:23:58.300776+0000","last_undegraded":"2024-05-09T08:23:58.300776+0000","last_fullsized":"2024-05-09T08:23:58.300776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:33.583552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023199+0000","last_change":"2024-05-09T08:23:50.575802+0000","last_active":"2024-05-09T08:23:58.023199+0000","last_peered":"2024-05-09T08:23:58.023199+0000","last_clean":"2024-05-09T08:23:58.023199+0000","last_became_active":"2024-05-09T08:23:50.575416+0000","last_became_peered":"2024-05-09T08:23:50.575416+0000","last_unstale":"2024-05-09T08:23:58.023199+0000","last_undegraded":"2024-05-09T08:23:58.023199+0000","last_fullsized":"2024-05-09T08:23:58.023199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:24:04.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022441+0000","last_change":"2024-05-09T08:23:51.578189+0000","last_active":"2024-05-09T08:23:58.022441+0000","last_peered":"2024-05-09T08:23:58.022441+0000","last_clean":"2024-05-09T08:23:58.022441+0000","last_became_active":"2024-05-09T08:23:51.577500+0000","last_became_peered":"2024-05-09T08:23:51.577500+0000","last_unstale":"2024-05-09T08:23:58.022441+0000","last_undegraded":"2024-05-09T08:23:58.022441+0000","last_fullsized":"2024-05-09T08:23:58.022441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:26:24.718791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022440+0000","last_change":"2024-05-09T08:23:55.670859+0000","last_active":"2024-05-09T08:23:58.022440+0000","last_peered":"2024-05-09T08:23:58.022440+0000","last_clean":"2024-05-09T08:23:58.022440+0000","last_became_active":"2024-05-09T08:23:55.670447+0000","last_became_peered":"2024-05-09T08:23:55.670447+0000","last_unstale":"2024-05-09T08:23:58.022440+0000","last_undegraded":"2024-05-09T08:23:58.022440+0000","last_fullsized":"2024-05-09T08:23:58.022440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:25:58.018733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022748+0000","last_change":"2024-05-09T08:23:50.578438+0000","last_active":"2024-05-09T08:23:58.022748+0000","last_peered":"2024-05-09T08:23:58.022748+0000","last_clean":"2024-05-09T08:23:58.022748+0000","last_became_active":"2024-05-09T08:23:50.578039+0000","last_became_peered":"2024-05-09T08:23:50.578039+0000","last_unstale":"2024-05-09T08:23:58.022748+0000","last_undegraded":"2024-05-09T08:23:58.022748+0000","last_fullsized":"2024-05-09T08:23:58.022748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:59.420193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022732+0000","last_change":"2024-05-09T08:23:51.568037+0000","last_active":"2024-05-09T08:23:58.022732+0000","last_peered":"2024-05-09T08:23:58.022732+0000","last_clean":"2024-05-09T08:23:58.022732+0000","last_became_active":"2024-05-09T08:23:51.567706+0000","last_became_peered":"2024-05-09T08:23:51.567706+0000","last_unstale":"2024-05-09T08:23:58.022732+0000","last_undegraded":"2024-05-09T08:23:58.022732+0000","last_fullsized":"2024-05-09T08:23:58.022732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:15:19.623805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329571+0000","last_change":"2024-05-09T08:23:51.579885+0000","last_active":"2024-05-09T08:23:57.329571+0000","last_peered":"2024-05-09T08:23:57.329571+0000","last_clean":"2024-05-09T08:23:57.329571+0000","last_became_active":"2024-05-09T08:23:51.578953+0000","last_became_peered":"2024-05-09T08:23:51.578953+0000","last_unstale":"2024-05-09T08:23:57.329571+0000","last_undegraded":"2024-05-09T08:23:57.329571+0000","last_fullsized":"2024-05-09T08:23:57.329571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:57:23.793119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654492+0000","last_change":"2024-05-09T08:23:51.583423+0000","last_active":"2024-05-09T08:23:56.654492+0000","last_peered":"2024-05-09T08:23:56.654492+0000","last_clean":"2024-05-09T08:23:56.654492+0000","last_became_active":"2024-05-09T08:23:51.583092+0000","last_became_peered":"2024-05-09T08:23:51.583092+0000","last_unstale":"2024-05-09T08:23:56.654492+0000","last_undegraded":"2024-05-09T08:23:56.654492+0000","last_fullsized":"2024-05-09T08:23:56.654492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:44:47.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654349+0000","last_change":"2024-05-09T08:23:51.582116+0000","last_active":"2024-05-09T08:23:56.654349+0000","last_peered":"2024-05-09T08:23:56.654349+0000","last_clean":"2024-05-09T08:23:56.654349+0000","last_became_active":"2024-05-09T08:23:51.581715+0000","last_became_peered":"2024-05-09T08:23:51.581715+0000","last_unstale":"2024-05-09T08:23:56.654349+0000","last_undegraded":"2024-05-09T08:23:56.654349+0000","last_fullsized":"2024-05-09T08:23:56.654349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:17.528758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300902+0000","last_change":"2024-05-09T08:23:51.575959+0000","last_active":"2024-05-09T08:23:58.300902+0000","last_peered":"2024-05-09T08:23:58.300902+0000","last_clean":"2024-05-09T08:23:58.300902+0000","last_became_active":"2024-05-09T08:23:51.575592+0000","last_became_peered":"2024-05-09T08:23:51.575592+0000","last_unstale":"2024-05-09T08:23:58.300902+0000","last_undegraded":"2024-05-09T08:23:58.300902+0000","last_fullsized":"2024-05-09T08:23:58.300902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:26.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022639+0000","last_change":"2024-05-09T08:23:51.570756+0000","last_active":"2024-05-09T08:23:58.022639+0000","last_peered":"2024-05-09T08:23:58.022639+0000","last_clean":"2024-05-09T08:23:58.022639+0000","last_became_active":"2024-05-09T08:23:51.570412+0000","last_became_peered":"2024-05-09T08:23:51.570412+0000","last_unstale":"2024-05-09T08:23:58.022639+0000","last_undegraded":"2024-05-09T08:23:58.022639+0000","last_fullsized":"2024-05-09T08:23:58.022639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:11:53.959559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329974+0000","last_change":"2024-05-09T08:23:51.572612+0000","last_active":"2024-05-09T08:23:57.329974+0000","last_peered":"2024-05-09T08:23:57.329974+0000","last_clean":"2024-05-09T08:23:57.329974+0000","last_became_active":"2024-05-09T08:23:51.572227+0000","last_became_peered":"2024-05-09T08:23:51.572227+0000","last_unstale":"2024-05-09T08:23:57.329974+0000","last_undegraded":"2024-05-09T08:23:57.329974+0000","last_fullsized":"2024-05-09T08:23:57.329974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:18.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.655079+0000","last_change":"2024-05-09T08:23:51.573021+0000","last_active":"2024-05-09T08:23:56.655079+0000","last_peered":"2024-05-09T08:23:56.655079+0000","last_clean":"2024-05-09T08:23:56.655079+0000","last_became_active":"2024-05-09T08:23:51.572716+0000","last_became_peered":"2024-05-09T08:23:51.572716+0000","last_unstale":"2024-05-09T08:23:56.655079+0000","last_undegraded":"2024-05-09T08:23:56.655079+0000","last_fullsized":"2024-05-09T08:23:56.655079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:31.494329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022950+0000","last_change":"2024-05-09T08:23:51.572687+0000","last_active":"2024-05-09T08:23:58.022950+0000","last_peered":"2024-05-09T08:23:58.022950+0000","last_clean":"2024-05-09T08:23:58.022950+0000","last_became_active":"2024-05-09T08:23:51.572339+0000","last_became_peered":"2024-05-09T08:23:51.572339+0000","last_unstale":"2024-05-09T08:23:58.022950+0000","last_undegraded":"2024-05-09T08:23:58.022950+0000","last_fullsized":"2024-05-09T08:23:58.022950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:24.012360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022673+0000","last_change":"2024-05-09T08:23:51.569404+0000","last_active":"2024-05-09T08:23:58.022673+0000","last_peered":"2024-05-09T08:23:58.022673+0000","last_clean":"2024-05-09T08:23:58.022673+0000","last_became_active":"2024-05-09T08:23:51.569044+0000","last_became_peered":"2024-05-09T08:23:51.569044+0000","last_unstale":"2024-05-09T08:23:58.022673+0000","last_undegraded":"2024-05-09T08:23:58.022673+0000","last_fullsized":"2024-05-09T08:23:58.022673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:26:01.343606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023141+0000","last_change":"2024-05-09T08:23:51.577483+0000","last_active":"2024-05-09T08:23:58.023141+0000","last_peered":"2024-05-09T08:23:58.023141+0000","last_clean":"2024-05-09T08:23:58.023141+0000","last_became_active":"2024-05-09T08:23:51.577078+0000","last_became_peered":"2024-05-09T08:23:51.577078+0000","last_unstale":"2024-05-09T08:23:58.023141+0000","last_undegraded":"2024-05-09T08:23:58.023141+0000","last_fullsized":"2024-05-09T08:23:58.023141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:29.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488149+0000","last_change":"2024-05-09T08:23:51.579736+0000","last_active":"2024-05-09T08:23:57.488149+0000","last_peered":"2024-05-09T08:23:57.488149+0000","last_clean":"2024-05-09T08:23:57.488149+0000","last_became_active":"2024-05-09T08:23:51.579409+0000","last_became_peered":"2024-05-09T08:23:51.579409+0000","last_unstale":"2024-05-09T08:23:57.488149+0000","last_undegraded":"2024-05-09T08:23:57.488149+0000","last_fullsized":"2024-05-09T08:23:57.488149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:05.697642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022353+0000","last_change":"2024-05-09T08:23:51.569461+0000","last_active":"2024-05-09T08:23:58.022353+0000","last_peered":"2024-05-09T08:23:58.022353+0000","last_clean":"2024-05-09T08:23:58.022353+0000","last_became_active":"2024-05-09T08:23:51.569085+0000","last_became_peered":"2024-05-09T08:23:51.569085+0000","last_unstale":"2024-05-09T08:23:58.022353+0000","last_undegraded":"2024-05-09T08:23:58.022353+0000","last_fullsized":"2024-05-09T08:23:58.022353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:54.554949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329662+0000","last_change":"2024-05-09T08:23:51.570593+0000","last_active":"2024-05-09T08:23:57.329662+0000","last_peered":"2024-05-09T08:23:57.329662+0000","last_clean":"2024-05-09T08:23:57.329662+0000","last_became_active":"2024-05-09T08:23:51.570195+0000","last_became_peered":"2024-05-09T08:23:51.570195+0000","last_unstale":"2024-05-09T08:23:57.329662+0000","last_undegraded":"2024-05-09T08:23:57.329662+0000","last_fullsized":"2024-05-09T08:23:57.329662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:18.376268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283814,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8487,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":61417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50773,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50773,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":488239,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482407,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":478272,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50773,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":42223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:24:20.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:24:20.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.396+0000 7f457c3d5640 1 -- 172.21.15.28:0/810513162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574104c30 msgr2=0x7f4574107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.396+0000 7f457c3d5640 1 --2- 172.21.15.28:0/810513162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574104c30 0x7f4574107020 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4564009900 tx=0x7f456402a410 comp rx=0 tx=0).stop 2024-05-09T08:24:20.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.396+0000 7f457c3d5640 1 -- 172.21.15.28:0/810513162 shutdown_connections 2024-05-09T08:24:20.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.396+0000 7f457c3d5640 1 --2- 172.21.15.28:0/810513162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4574107560 0x7f45741099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.396+0000 7f457c3d5640 1 --2- 172.21.15.28:0/810513162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574104c30 0x7f4574107020 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.396+0000 7f457c3d5640 1 --2- 172.21.15.28:0/810513162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4574102300 0x7f45741046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.396+0000 7f457c3d5640 1 -- 172.21.15.28:0/810513162 >> 172.21.15.28:0/810513162 conn(0x7f45740fbd30 msgr2=0x7f45740fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.396+0000 7f457c3d5640 1 -- 172.21.15.28:0/810513162 shutdown_connections 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.396+0000 7f457c3d5640 1 -- 172.21.15.28:0/810513162 wait complete. 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457c3d5640 1 Processor -- start 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457c3d5640 1 -- start start 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457c3d5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4574102300 0x7f4574197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457c3d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4574104c30 0x7f45741981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457c3d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574107560 0x7f457419f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457c3d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f457410d5b0 con 0x7f4574107560 2024-05-09T08:24:20.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457c3d5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f457410d430 con 0x7f4574102300 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457c3d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f457410d730 con 0x7f4574104c30 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a972640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574107560 0x7f457419f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a171640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4574102300 0x7f4574197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a171640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4574102300 0x7f4574197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:38424/0 (socket says 172.21.15.28:38424) 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a972640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574107560 0x7f457419f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35742/0 (socket says 172.21.15.28:35742) 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a171640 1 -- 172.21.15.28:0/200695187 learned_addr learned my addr 172.21.15.28:0/200695187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f4579970640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4574104c30 0x7f45741981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a972640 1 -- 172.21.15.28:0/200695187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4574104c30 msgr2=0x7f45741981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a972640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4574104c30 0x7f45741981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a972640 1 -- 172.21.15.28:0/200695187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4574102300 msgr2=0x7f4574197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a972640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4574102300 0x7f4574197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a972640 1 -- 172.21.15.28:0/200695187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457419f880 con 0x7f4574107560 2024-05-09T08:24:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f4579970640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4574104c30 0x7f45741981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.397+0000 7f457a972640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574107560 0x7f457419f1e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f456c009fd0 tx=0x7f456c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.398+0000 7f456b7fe640 1 -- 172.21.15.28:0/200695187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c00ce20 con 0x7f4574107560 2024-05-09T08:24:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.398+0000 7f456b7fe640 1 -- 172.21.15.28:0/200695187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f456c005020 con 0x7f4574107560 2024-05-09T08:24:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.398+0000 7f456b7fe640 1 -- 172.21.15.28:0/200695187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c010720 con 0x7f4574107560 2024-05-09T08:24:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.398+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f457419fb70 con 0x7f4574107560 2024-05-09T08:24:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.398+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45741a0070 con 0x7f4574107560 2024-05-09T08:24:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.399+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45740fd820 con 0x7f4574107560 2024-05-09T08:24:20.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.400+0000 7f456b7fe640 1 -- 172.21.15.28:0/200695187 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f456c005540 con 0x7f4574107560 2024-05-09T08:24:20.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.401+0000 7f456b7fe640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f45600a2b90 0x7f45600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:20.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.401+0000 7f457a171640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f45600a2b90 0x7f45600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:20.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.401+0000 7f456b7fe640 1 -- 172.21.15.28:0/200695187 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f456c0cfad0 con 0x7f4574107560 2024-05-09T08:24:20.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.401+0000 7f457a171640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f45600a2b90 0x7f45600a5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4574199140 tx=0x7f45700091c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:20.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.405+0000 7f456b7fe640 1 -- 172.21.15.28:0/200695187 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f456c09c7f0 con 0x7f4574107560 2024-05-09T08:24:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.423+0000 7f456b7fe640 1 -- 172.21.15.28:0/200695187 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f456c0986d0 con 0x7f4574107560 2024-05-09T08:24:20.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.592+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4574061c70 con 0x7f45600a2b90 2024-05-09T08:24:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.602+0000 7f456b7fe640 1 -- 172.21.15.28:0/200695187 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340623 (secure 0 0 0) 0x7f4574061c70 con 0x7f45600a2b90 2024-05-09T08:24:20.610 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:20.612 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-09T08:24:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f45600a2b90 msgr2=0x7f45600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f45600a2b90 0x7f45600a5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4574199140 tx=0x7f45700091c0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574107560 msgr2=0x7f457419f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574107560 0x7f457419f1e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f456c009fd0 tx=0x7f456c00ef10 comp rx=0 tx=0).stop 2024-05-09T08:24:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 shutdown_connections 2024-05-09T08:24:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f45600a2b90 0x7f45600a5050 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4574107560 0x7f457419f1e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4574104c30 0x7f45741981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 --2- 172.21.15.28:0/200695187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4574102300 0x7f4574197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 >> 172.21.15.28:0/200695187 conn(0x7f45740fbd30 msgr2=0x7f4574102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 shutdown_connections 2024-05-09T08:24:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:20.606+0000 7f457c3d5640 1 -- 172.21.15.28:0/200695187 wait complete. 2024-05-09T08:24:20.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:20 smithi028 ceph-mon[30119]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:20.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:20 smithi028 ceph-mon[30119]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:20.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:20 smithi028 ceph-mon[30119]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:20.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:20 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] up:active 2024-05-09T08:24:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:20 smithi028 ceph-mon[30119]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-09T08:24:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:20 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:24:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:20 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2509947952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:24:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:20 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2852229510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:20.619 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-05-09T08:24:19.643870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598855,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":315,"num_write_kb":2985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":258,"ondisk_log_size":258,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373268,"kb_used_data":11612,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568556,"statfs":{"total":1151940427776,"available":1151558201344,"internally_reserved":0,"allocated":11890688,"data_stored":7523513,"data_compressed":284457,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3988,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003355"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.330072+0000","last_change":"2024-05-09T08:23:51.579564+0000","last_active":"2024-05-09T08:23:57.330072+0000","last_peered":"2024-05-09T08:23:57.330072+0000","last_clean":"2024-05-09T08:23:57.330072+0000","last_became_active":"2024-05-09T08:23:51.578884+0000","last_became_peered":"2024-05-09T08:23:51.578884+0000","last_unstale":"2024-05-09T08:23:57.330072+0000","last_undegraded":"2024-05-09T08:23:57.330072+0000","last_fullsized":"2024-05-09T08:23:57.330072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:40.473483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.301084+0000","last_change":"2024-05-09T08:23:51.574187+0000","last_active":"2024-05-09T08:23:58.301084+0000","last_peered":"2024-05-09T08:23:58.301084+0000","last_clean":"2024-05-09T08:23:58.301084+0000","last_became_active":"2024-05-09T08:23:51.573837+0000","last_became_peered":"2024-05-09T08:23:51.573837+0000","last_unstale":"2024-05-09T08:23:58.301084+0000","last_undegraded":"2024-05-09T08:23:58.301084+0000","last_fullsized":"2024-05-09T08:23:58.301084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:31.448038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300979+0000","last_change":"2024-05-09T08:23:51.577224+0000","last_active":"2024-05-09T08:23:58.300979+0000","last_peered":"2024-05-09T08:23:58.300979+0000","last_clean":"2024-05-09T08:23:58.300979+0000","last_became_active":"2024-05-09T08:23:51.576822+0000","last_became_peered":"2024-05-09T08:23:51.576822+0000","last_unstale":"2024-05-09T08:23:58.300979+0000","last_undegraded":"2024-05-09T08:23:58.300979+0000","last_fullsized":"2024-05-09T08:23:58.300979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:04.902530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921941+0000","last_change":"2024-05-09T08:23:51.577172+0000","last_active":"2024-05-09T08:23:56.921941+0000","last_peered":"2024-05-09T08:23:56.921941+0000","last_clean":"2024-05-09T08:23:56.921941+0000","last_became_active":"2024-05-09T08:23:51.576861+0000","last_became_peered":"2024-05-09T08:23:51.576861+0000","last_unstale":"2024-05-09T08:23:56.921941+0000","last_undegraded":"2024-05-09T08:23:56.921941+0000","last_fullsized":"2024-05-09T08:23:56.921941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:10.925739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469607+0000","last_change":"2024-05-09T08:23:51.579408+0000","last_active":"2024-05-09T08:23:57.469607+0000","last_peered":"2024-05-09T08:23:57.469607+0000","last_clean":"2024-05-09T08:23:57.469607+0000","last_became_active":"2024-05-09T08:23:51.578229+0000","last_became_peered":"2024-05-09T08:23:51.578229+0000","last_unstale":"2024-05-09T08:23:57.469607+0000","last_undegraded":"2024-05-09T08:23:57.469607+0000","last_fullsized":"2024-05-09T08:23:57.469607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:47:27.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654282+0000","last_change":"2024-05-09T08:23:51.576924+0000","last_active":"2024-05-09T08:23:56.654282+0000","last_peered":"2024-05-09T08:23:56.654282+0000","last_clean":"2024-05-09T08:23:56.654282+0000","last_became_active":"2024-05-09T08:23:51.576601+0000","last_became_peered":"2024-05-09T08:23:51.576601+0000","last_unstale":"2024-05-09T08:23:56.654282+0000","last_undegraded":"2024-05-09T08:23:56.654282+0000","last_fullsized":"2024-05-09T08:23:56.654282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:07:16.910271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022073+0000","last_change":"2024-05-09T08:23:51.574843+0000","last_active":"2024-05-09T08:23:58.022073+0000","last_peered":"2024-05-09T08:23:58.022073+0000","last_clean":"2024-05-09T08:23:58.022073+0000","last_became_active":"2024-05-09T08:23:51.574498+0000","last_became_peered":"2024-05-09T08:23:51.574498+0000","last_unstale":"2024-05-09T08:23:58.022073+0000","last_undegraded":"2024-05-09T08:23:58.022073+0000","last_fullsized":"2024-05-09T08:23:58.022073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:59:50.974491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022179+0000","last_change":"2024-05-09T08:23:51.574001+0000","last_active":"2024-05-09T08:23:58.022179+0000","last_peered":"2024-05-09T08:23:58.022179+0000","last_clean":"2024-05-09T08:23:58.022179+0000","last_became_active":"2024-05-09T08:23:51.573632+0000","last_became_peered":"2024-05-09T08:23:51.573632+0000","last_unstale":"2024-05-09T08:23:58.022179+0000","last_undegraded":"2024-05-09T08:23:58.022179+0000","last_fullsized":"2024-05-09T08:23:58.022179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:55:16.418951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488452+0000","last_change":"2024-05-09T08:23:51.567635+0000","last_active":"2024-05-09T08:23:57.488452+0000","last_peered":"2024-05-09T08:23:57.488452+0000","last_clean":"2024-05-09T08:23:57.488452+0000","last_became_active":"2024-05-09T08:23:51.567283+0000","last_became_peered":"2024-05-09T08:23:51.567283+0000","last_unstale":"2024-05-09T08:23:57.488452+0000","last_undegraded":"2024-05-09T08:23:57.488452+0000","last_fullsized":"2024-05-09T08:23:57.488452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:42.346856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654839+0000","last_change":"2024-05-09T08:23:51.576090+0000","last_active":"2024-05-09T08:23:56.654839+0000","last_peered":"2024-05-09T08:23:56.654839+0000","last_clean":"2024-05-09T08:23:56.654839+0000","last_became_active":"2024-05-09T08:23:51.575768+0000","last_became_peered":"2024-05-09T08:23:51.575768+0000","last_unstale":"2024-05-09T08:23:56.654839+0000","last_undegraded":"2024-05-09T08:23:56.654839+0000","last_fullsized":"2024-05-09T08:23:56.654839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:00:50.016924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022633+0000","last_change":"2024-05-09T08:23:51.578957+0000","last_active":"2024-05-09T08:23:58.022633+0000","last_peered":"2024-05-09T08:23:58.022633+0000","last_clean":"2024-05-09T08:23:58.022633+0000","last_became_active":"2024-05-09T08:23:51.578210+0000","last_became_peered":"2024-05-09T08:23:51.578210+0000","last_unstale":"2024-05-09T08:23:58.022633+0000","last_undegraded":"2024-05-09T08:23:58.022633+0000","last_fullsized":"2024-05-09T08:23:58.022633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:28:28.953434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921864+0000","last_change":"2024-05-09T08:23:51.575389+0000","last_active":"2024-05-09T08:23:56.921864+0000","last_peered":"2024-05-09T08:23:56.921864+0000","last_clean":"2024-05-09T08:23:56.921864+0000","last_became_active":"2024-05-09T08:23:51.575071+0000","last_became_peered":"2024-05-09T08:23:51.575071+0000","last_unstale":"2024-05-09T08:23:56.921864+0000","last_undegraded":"2024-05-09T08:23:56.921864+0000","last_fullsized":"2024-05-09T08:23:56.921864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:34.844611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022544+0000","last_change":"2024-05-09T08:23:51.572574+0000","last_active":"2024-05-09T08:23:58.022544+0000","last_peered":"2024-05-09T08:23:58.022544+0000","last_clean":"2024-05-09T08:23:58.022544+0000","last_became_active":"2024-05-09T08:23:51.572243+0000","last_became_peered":"2024-05-09T08:23:51.572243+0000","last_unstale":"2024-05-09T08:23:58.022544+0000","last_undegraded":"2024-05-09T08:23:58.022544+0000","last_fullsized":"2024-05-09T08:23:58.022544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:52.534365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300681+0000","last_change":"2024-05-09T08:23:51.574963+0000","last_active":"2024-05-09T08:23:58.300681+0000","last_peered":"2024-05-09T08:23:58.300681+0000","last_clean":"2024-05-09T08:23:58.300681+0000","last_became_active":"2024-05-09T08:23:51.574597+0000","last_became_peered":"2024-05-09T08:23:51.574597+0000","last_unstale":"2024-05-09T08:23:58.300681+0000","last_undegraded":"2024-05-09T08:23:58.300681+0000","last_fullsized":"2024-05-09T08:23:58.300681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:33.348828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300607+0000","last_change":"2024-05-09T08:23:51.577777+0000","last_active":"2024-05-09T08:23:58.300607+0000","last_peered":"2024-05-09T08:23:58.300607+0000","last_clean":"2024-05-09T08:23:58.300607+0000","last_became_active":"2024-05-09T08:23:51.577073+0000","last_became_peered":"2024-05-09T08:23:51.577073+0000","last_unstale":"2024-05-09T08:23:58.300607+0000","last_undegraded":"2024-05-09T08:23:58.300607+0000","last_fullsized":"2024-05-09T08:23:58.300607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:26.249488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.487831+0000","last_change":"2024-05-09T08:23:51.574476+0000","last_active":"2024-05-09T08:23:57.487831+0000","last_peered":"2024-05-09T08:23:57.487831+0000","last_clean":"2024-05-09T08:23:57.487831+0000","last_became_active":"2024-05-09T08:23:51.572748+0000","last_became_peered":"2024-05-09T08:23:51.572748+0000","last_unstale":"2024-05-09T08:23:57.487831+0000","last_undegraded":"2024-05-09T08:23:57.487831+0000","last_fullsized":"2024-05-09T08:23:57.487831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:43.515419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329835+0000","last_change":"2024-05-09T08:23:51.580568+0000","last_active":"2024-05-09T08:23:57.329835+0000","last_peered":"2024-05-09T08:23:57.329835+0000","last_clean":"2024-05-09T08:23:57.329835+0000","last_became_active":"2024-05-09T08:23:51.579626+0000","last_became_peered":"2024-05-09T08:23:51.579626+0000","last_unstale":"2024-05-09T08:23:57.329835+0000","last_undegraded":"2024-05-09T08:23:57.329835+0000","last_fullsized":"2024-05-09T08:23:57.329835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:28.756079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654855+0000","last_change":"2024-05-09T08:23:51.574363+0000","last_active":"2024-05-09T08:23:56.654855+0000","last_peered":"2024-05-09T08:23:56.654855+0000","last_clean":"2024-05-09T08:23:56.654855+0000","last_became_active":"2024-05-09T08:23:51.573858+0000","last_became_peered":"2024-05-09T08:23:51.573858+0000","last_unstale":"2024-05-09T08:23:56.654855+0000","last_undegraded":"2024-05-09T08:23:56.654855+0000","last_fullsized":"2024-05-09T08:23:56.654855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:30.752939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654980+0000","last_change":"2024-05-09T08:23:51.579361+0000","last_active":"2024-05-09T08:23:56.654980+0000","last_peered":"2024-05-09T08:23:56.654980+0000","last_clean":"2024-05-09T08:23:56.654980+0000","last_became_active":"2024-05-09T08:23:51.579025+0000","last_became_peered":"2024-05-09T08:23:51.579025+0000","last_unstale":"2024-05-09T08:23:56.654980+0000","last_undegraded":"2024-05-09T08:23:56.654980+0000","last_fullsized":"2024-05-09T08:23:56.654980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:02.273999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654896+0000","last_change":"2024-05-09T08:23:55.670107+0000","last_active":"2024-05-09T08:23:56.654896+0000","last_peered":"2024-05-09T08:23:56.654896+0000","last_clean":"2024-05-09T08:23:56.654896+0000","last_became_active":"2024-05-09T08:23:55.669621+0000","last_became_peered":"2024-05-09T08:23:55.669621+0000","last_unstale":"2024-05-09T08:23:56.654896+0000","last_undegraded":"2024-05-09T08:23:56.654896+0000","last_fullsized":"2024-05-09T08:23:56.654896+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:00.547137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654971+0000","last_change":"2024-05-09T08:23:50.571747+0000","last_active":"2024-05-09T08:23:56.654971+0000","last_peered":"2024-05-09T08:23:56.654971+0000","last_clean":"2024-05-09T08:23:56.654971+0000","last_became_active":"2024-05-09T08:23:50.571460+0000","last_became_peered":"2024-05-09T08:23:50.571460+0000","last_unstale":"2024-05-09T08:23:56.654971+0000","last_undegraded":"2024-05-09T08:23:56.654971+0000","last_fullsized":"2024-05-09T08:23:56.654971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:15.519879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022701+0000","last_change":"2024-05-09T08:23:51.574336+0000","last_active":"2024-05-09T08:23:58.022701+0000","last_peered":"2024-05-09T08:23:58.022701+0000","last_clean":"2024-05-09T08:23:58.022701+0000","last_became_active":"2024-05-09T08:23:51.573801+0000","last_became_peered":"2024-05-09T08:23:51.573801+0000","last_unstale":"2024-05-09T08:23:58.022701+0000","last_undegraded":"2024-05-09T08:23:58.022701+0000","last_fullsized":"2024-05-09T08:23:58.022701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:47.389762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921579+0000","last_change":"2024-05-09T08:23:55.670194+0000","last_active":"2024-05-09T08:23:56.921579+0000","last_peered":"2024-05-09T08:23:56.921579+0000","last_clean":"2024-05-09T08:23:56.921579+0000","last_became_active":"2024-05-09T08:23:55.669762+0000","last_became_peered":"2024-05-09T08:23:55.669762+0000","last_unstale":"2024-05-09T08:23:56.921579+0000","last_undegraded":"2024-05-09T08:23:56.921579+0000","last_fullsized":"2024-05-09T08:23:56.921579+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:22.879635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022754+0000","last_change":"2024-05-09T08:23:50.574421+0000","last_active":"2024-05-09T08:23:58.022754+0000","last_peered":"2024-05-09T08:23:58.022754+0000","last_clean":"2024-05-09T08:23:58.022754+0000","last_became_active":"2024-05-09T08:23:50.574070+0000","last_became_peered":"2024-05-09T08:23:50.574070+0000","last_unstale":"2024-05-09T08:23:58.022754+0000","last_undegraded":"2024-05-09T08:23:58.022754+0000","last_fullsized":"2024-05-09T08:23:58.022754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:22.996034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300934+0000","last_change":"2024-05-09T08:23:51.583275+0000","last_active":"2024-05-09T08:23:58.300934+0000","last_peered":"2024-05-09T08:23:58.300934+0000","last_clean":"2024-05-09T08:23:58.300934+0000","last_became_active":"2024-05-09T08:23:51.582847+0000","last_became_peered":"2024-05-09T08:23:51.582847+0000","last_unstale":"2024-05-09T08:23:58.300934+0000","last_undegraded":"2024-05-09T08:23:58.300934+0000","last_fullsized":"2024-05-09T08:23:58.300934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:03:03.512852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921663+0000","last_change":"2024-05-09T08:23:55.672256+0000","last_active":"2024-05-09T08:23:56.921663+0000","last_peered":"2024-05-09T08:23:56.921663+0000","last_clean":"2024-05-09T08:23:56.921663+0000","last_became_active":"2024-05-09T08:23:55.671855+0000","last_became_peered":"2024-05-09T08:23:55.671855+0000","last_unstale":"2024-05-09T08:23:56.921663+0000","last_undegraded":"2024-05-09T08:23:56.921663+0000","last_fullsized":"2024-05-09T08:23:56.921663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:45:12.199359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.393140+0000","last_change":"2024-05-09T08:23:50.566686+0000","last_active":"2024-05-09T08:24:17.393140+0000","last_peered":"2024-05-09T08:24:17.393140+0000","last_clean":"2024-05-09T08:24:17.393140+0000","last_became_active":"2024-05-09T08:23:50.565910+0000","last_became_peered":"2024-05-09T08:23:50.565910+0000","last_unstale":"2024-05-09T08:24:17.393140+0000","last_undegraded":"2024-05-09T08:24:17.393140+0000","last_fullsized":"2024-05-09T08:24:17.393140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:02.211551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488098+0000","last_change":"2024-05-09T08:23:51.570972+0000","last_active":"2024-05-09T08:23:57.488098+0000","last_peered":"2024-05-09T08:23:57.488098+0000","last_clean":"2024-05-09T08:23:57.488098+0000","last_became_active":"2024-05-09T08:23:51.570638+0000","last_became_peered":"2024-05-09T08:23:51.570638+0000","last_unstale":"2024-05-09T08:23:57.488098+0000","last_undegraded":"2024-05-09T08:23:57.488098+0000","last_fullsized":"2024-05-09T08:23:57.488098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:38.738820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921800+0000","last_change":"2024-05-09T08:23:55.671929+0000","last_active":"2024-05-09T08:23:56.921800+0000","last_peered":"2024-05-09T08:23:56.921800+0000","last_clean":"2024-05-09T08:23:56.921800+0000","last_became_active":"2024-05-09T08:23:55.671483+0000","last_became_peered":"2024-05-09T08:23:55.671483+0000","last_unstale":"2024-05-09T08:23:56.921800+0000","last_undegraded":"2024-05-09T08:23:56.921800+0000","last_fullsized":"2024-05-09T08:23:56.921800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:21.527266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469748+0000","last_change":"2024-05-09T08:23:50.565644+0000","last_active":"2024-05-09T08:23:57.469748+0000","last_peered":"2024-05-09T08:23:57.469748+0000","last_clean":"2024-05-09T08:23:57.469748+0000","last_became_active":"2024-05-09T08:23:50.565265+0000","last_became_peered":"2024-05-09T08:23:50.565265+0000","last_unstale":"2024-05-09T08:23:57.469748+0000","last_undegraded":"2024-05-09T08:23:57.469748+0000","last_fullsized":"2024-05-09T08:23:57.469748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:51:57.972535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023089+0000","last_change":"2024-05-09T08:23:51.571732+0000","last_active":"2024-05-09T08:23:58.023089+0000","last_peered":"2024-05-09T08:23:58.023089+0000","last_clean":"2024-05-09T08:23:58.023089+0000","last_became_active":"2024-05-09T08:23:51.570783+0000","last_became_peered":"2024-05-09T08:23:51.570783+0000","last_unstale":"2024-05-09T08:23:58.023089+0000","last_undegraded":"2024-05-09T08:23:58.023089+0000","last_fullsized":"2024-05-09T08:23:58.023089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:18.515905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.655000+0000","last_change":"2024-05-09T08:23:55.666727+0000","last_active":"2024-05-09T08:23:56.655000+0000","last_peered":"2024-05-09T08:23:56.655000+0000","last_clean":"2024-05-09T08:23:56.655000+0000","last_became_active":"2024-05-09T08:23:55.666284+0000","last_became_peered":"2024-05-09T08:23:55.666284+0000","last_unstale":"2024-05-09T08:23:56.655000+0000","last_undegraded":"2024-05-09T08:23:56.655000+0000","last_fullsized":"2024-05-09T08:23:56.655000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:05.355404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378806+0000","last_change":"2024-05-09T08:23:50.573308+0000","last_active":"2024-05-09T08:24:15.378806+0000","last_peered":"2024-05-09T08:24:15.378806+0000","last_clean":"2024-05-09T08:24:15.378806+0000","last_became_active":"2024-05-09T08:23:50.572946+0000","last_became_peered":"2024-05-09T08:23:50.572946+0000","last_unstale":"2024-05-09T08:24:15.378806+0000","last_undegraded":"2024-05-09T08:24:15.378806+0000","last_fullsized":"2024-05-09T08:24:15.378806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:44:35.816620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469752+0000","last_change":"2024-05-09T08:23:51.571748+0000","last_active":"2024-05-09T08:23:57.469752+0000","last_peered":"2024-05-09T08:23:57.469752+0000","last_clean":"2024-05-09T08:23:57.469752+0000","last_became_active":"2024-05-09T08:23:51.571388+0000","last_became_peered":"2024-05-09T08:23:51.571388+0000","last_unstale":"2024-05-09T08:23:57.469752+0000","last_undegraded":"2024-05-09T08:23:57.469752+0000","last_fullsized":"2024-05-09T08:23:57.469752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:34.916491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654957+0000","last_change":"2024-05-09T08:23:55.674022+0000","last_active":"2024-05-09T08:23:56.654957+0000","last_peered":"2024-05-09T08:23:56.654957+0000","last_clean":"2024-05-09T08:23:56.654957+0000","last_became_active":"2024-05-09T08:23:55.673322+0000","last_became_peered":"2024-05-09T08:23:55.673322+0000","last_unstale":"2024-05-09T08:23:56.654957+0000","last_undegraded":"2024-05-09T08:23:56.654957+0000","last_fullsized":"2024-05-09T08:23:56.654957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:59:14.299886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654276+0000","last_change":"2024-05-09T08:23:50.571235+0000","last_active":"2024-05-09T08:23:56.654276+0000","last_peered":"2024-05-09T08:23:56.654276+0000","last_clean":"2024-05-09T08:23:56.654276+0000","last_became_active":"2024-05-09T08:23:50.570694+0000","last_became_peered":"2024-05-09T08:23:50.570694+0000","last_unstale":"2024-05-09T08:23:56.654276+0000","last_undegraded":"2024-05-09T08:23:56.654276+0000","last_fullsized":"2024-05-09T08:23:56.654276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:22.850248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022898+0000","last_change":"2024-05-09T08:23:51.572387+0000","last_active":"2024-05-09T08:23:58.022898+0000","last_peered":"2024-05-09T08:23:58.022898+0000","last_clean":"2024-05-09T08:23:58.022898+0000","last_became_active":"2024-05-09T08:23:51.572038+0000","last_became_peered":"2024-05-09T08:23:51.572038+0000","last_unstale":"2024-05-09T08:23:58.022898+0000","last_undegraded":"2024-05-09T08:23:58.022898+0000","last_fullsized":"2024-05-09T08:23:58.022898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:38.118125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329864+0000","last_change":"2024-05-09T08:23:55.665885+0000","last_active":"2024-05-09T08:23:57.329864+0000","last_peered":"2024-05-09T08:23:57.329864+0000","last_clean":"2024-05-09T08:23:57.329864+0000","last_became_active":"2024-05-09T08:23:55.665422+0000","last_became_peered":"2024-05-09T08:23:55.665422+0000","last_unstale":"2024-05-09T08:23:57.329864+0000","last_undegraded":"2024-05-09T08:23:57.329864+0000","last_fullsized":"2024-05-09T08:23:57.329864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:12:37.952612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.388806+0000","last_change":"2024-05-09T08:23:50.570040+0000","last_active":"2024-05-09T08:24:16.388806+0000","last_peered":"2024-05-09T08:24:16.388806+0000","last_clean":"2024-05-09T08:24:16.388806+0000","last_became_active":"2024-05-09T08:23:50.569665+0000","last_became_peered":"2024-05-09T08:23:50.569665+0000","last_unstale":"2024-05-09T08:24:16.388806+0000","last_undegraded":"2024-05-09T08:24:16.388806+0000","last_fullsized":"2024-05-09T08:24:16.388806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:22:29.639712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022076+0000","last_change":"2024-05-09T08:23:51.568561+0000","last_active":"2024-05-09T08:23:58.022076+0000","last_peered":"2024-05-09T08:23:58.022076+0000","last_clean":"2024-05-09T08:23:58.022076+0000","last_became_active":"2024-05-09T08:23:51.568196+0000","last_became_peered":"2024-05-09T08:23:51.568196+0000","last_unstale":"2024-05-09T08:23:58.022076+0000","last_undegraded":"2024-05-09T08:23:58.022076+0000","last_fullsized":"2024-05-09T08:23:58.022076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:52.132837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.301020+0000","last_change":"2024-05-09T08:23:55.674960+0000","last_active":"2024-05-09T08:23:58.301020+0000","last_peered":"2024-05-09T08:23:58.301020+0000","last_clean":"2024-05-09T08:23:58.301020+0000","last_became_active":"2024-05-09T08:23:55.674509+0000","last_became_peered":"2024-05-09T08:23:55.674509+0000","last_unstale":"2024-05-09T08:23:58.301020+0000","last_undegraded":"2024-05-09T08:23:58.301020+0000","last_fullsized":"2024-05-09T08:23:58.301020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:22.479867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022713+0000","last_change":"2024-05-09T08:23:50.569896+0000","last_active":"2024-05-09T08:23:58.022713+0000","last_peered":"2024-05-09T08:23:58.022713+0000","last_clean":"2024-05-09T08:23:58.022713+0000","last_became_active":"2024-05-09T08:23:50.569562+0000","last_became_peered":"2024-05-09T08:23:50.569562+0000","last_unstale":"2024-05-09T08:23:58.022713+0000","last_undegraded":"2024-05-09T08:23:58.022713+0000","last_fullsized":"2024-05-09T08:23:58.022713+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:25:20.278726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654129+0000","last_change":"2024-05-09T08:23:51.570260+0000","last_active":"2024-05-09T08:23:56.654129+0000","last_peered":"2024-05-09T08:23:56.654129+0000","last_clean":"2024-05-09T08:23:56.654129+0000","last_became_active":"2024-05-09T08:23:51.569837+0000","last_became_peered":"2024-05-09T08:23:51.569837+0000","last_unstale":"2024-05-09T08:23:56.654129+0000","last_undegraded":"2024-05-09T08:23:56.654129+0000","last_fullsized":"2024-05-09T08:23:56.654129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:01:44.805271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023027+0000","last_change":"2024-05-09T08:23:55.676639+0000","last_active":"2024-05-09T08:23:58.023027+0000","last_peered":"2024-05-09T08:23:58.023027+0000","last_clean":"2024-05-09T08:23:58.023027+0000","last_became_active":"2024-05-09T08:23:55.673643+0000","last_became_peered":"2024-05-09T08:23:55.673643+0000","last_unstale":"2024-05-09T08:23:58.023027+0000","last_undegraded":"2024-05-09T08:23:58.023027+0000","last_fullsized":"2024-05-09T08:23:58.023027+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:02.377184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378337+0000","last_change":"2024-05-09T08:23:50.565745+0000","last_active":"2024-05-09T08:24:15.378337+0000","last_peered":"2024-05-09T08:24:15.378337+0000","last_clean":"2024-05-09T08:24:15.378337+0000","last_became_active":"2024-05-09T08:23:50.565385+0000","last_became_peered":"2024-05-09T08:23:50.565385+0000","last_unstale":"2024-05-09T08:24:15.378337+0000","last_undegraded":"2024-05-09T08:24:15.378337+0000","last_fullsized":"2024-05-09T08:24:15.378337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:05:00.818398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488401+0000","last_change":"2024-05-09T08:23:51.572341+0000","last_active":"2024-05-09T08:23:57.488401+0000","last_peered":"2024-05-09T08:23:57.488401+0000","last_clean":"2024-05-09T08:23:57.488401+0000","last_became_active":"2024-05-09T08:23:51.571583+0000","last_became_peered":"2024-05-09T08:23:51.571583+0000","last_unstale":"2024-05-09T08:23:57.488401+0000","last_undegraded":"2024-05-09T08:23:57.488401+0000","last_fullsized":"2024-05-09T08:23:57.488401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:20.695995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654079+0000","last_change":"2024-05-09T08:23:55.677603+0000","last_active":"2024-05-09T08:23:56.654079+0000","last_peered":"2024-05-09T08:23:56.654079+0000","last_clean":"2024-05-09T08:23:56.654079+0000","last_became_active":"2024-05-09T08:23:55.677143+0000","last_became_peered":"2024-05-09T08:23:55.677143+0000","last_unstale":"2024-05-09T08:23:56.654079+0000","last_undegraded":"2024-05-09T08:23:56.654079+0000","last_fullsized":"2024-05-09T08:23:56.654079+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:27.850997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378182+0000","last_change":"2024-05-09T08:23:50.565497+0000","last_active":"2024-05-09T08:24:15.378182+0000","last_peered":"2024-05-09T08:24:15.378182+0000","last_clean":"2024-05-09T08:24:15.378182+0000","last_became_active":"2024-05-09T08:23:50.565120+0000","last_became_peered":"2024-05-09T08:23:50.565120+0000","last_unstale":"2024-05-09T08:24:15.378182+0000","last_undegraded":"2024-05-09T08:24:15.378182+0000","last_fullsized":"2024-05-09T08:24:15.378182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:56.141997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.301086+0000","last_change":"2024-05-09T08:23:51.576489+0000","last_active":"2024-05-09T08:23:58.301086+0000","last_peered":"2024-05-09T08:23:58.301086+0000","last_clean":"2024-05-09T08:23:58.301086+0000","last_became_active":"2024-05-09T08:23:51.576145+0000","last_became_peered":"2024-05-09T08:23:51.576145+0000","last_unstale":"2024-05-09T08:23:58.301086+0000","last_undegraded":"2024-05-09T08:23:58.301086+0000","last_fullsized":"2024-05-09T08:23:58.301086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:37.650308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.922006+0000","last_change":"2024-05-09T08:23:55.666181+0000","last_active":"2024-05-09T08:23:56.922006+0000","last_peered":"2024-05-09T08:23:56.922006+0000","last_clean":"2024-05-09T08:23:56.922006+0000","last_became_active":"2024-05-09T08:23:55.665795+0000","last_became_peered":"2024-05-09T08:23:55.665795+0000","last_unstale":"2024-05-09T08:23:56.922006+0000","last_undegraded":"2024-05-09T08:23:56.922006+0000","last_fullsized":"2024-05-09T08:23:56.922006+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:48.659719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023098+0000","last_change":"2024-05-09T08:23:50.567182+0000","last_active":"2024-05-09T08:23:58.023098+0000","last_peered":"2024-05-09T08:23:58.023098+0000","last_clean":"2024-05-09T08:23:58.023098+0000","last_became_active":"2024-05-09T08:23:50.566851+0000","last_became_peered":"2024-05-09T08:23:50.566851+0000","last_unstale":"2024-05-09T08:23:58.023098+0000","last_undegraded":"2024-05-09T08:23:58.023098+0000","last_fullsized":"2024-05-09T08:23:58.023098+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:26.540456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488226+0000","last_change":"2024-05-09T08:23:51.574074+0000","last_active":"2024-05-09T08:23:57.488226+0000","last_peered":"2024-05-09T08:23:57.488226+0000","last_clean":"2024-05-09T08:23:57.488226+0000","last_became_active":"2024-05-09T08:23:51.572587+0000","last_became_peered":"2024-05-09T08:23:51.572587+0000","last_unstale":"2024-05-09T08:23:57.488226+0000","last_undegraded":"2024-05-09T08:23:57.488226+0000","last_fullsized":"2024-05-09T08:23:57.488226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:46.186358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654670+0000","last_change":"2024-05-09T08:23:55.667170+0000","last_active":"2024-05-09T08:23:56.654670+0000","last_peered":"2024-05-09T08:23:56.654670+0000","last_clean":"2024-05-09T08:23:56.654670+0000","last_became_active":"2024-05-09T08:23:55.666707+0000","last_became_peered":"2024-05-09T08:23:55.666707+0000","last_unstale":"2024-05-09T08:23:56.654670+0000","last_undegraded":"2024-05-09T08:23:56.654670+0000","last_fullsized":"2024-05-09T08:23:56.654670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:48.332207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.377419+0000","last_change":"2024-05-09T08:23:50.572694+0000","last_active":"2024-05-09T08:24:15.377419+0000","last_peered":"2024-05-09T08:24:15.377419+0000","last_clean":"2024-05-09T08:24:15.377419+0000","last_became_active":"2024-05-09T08:23:50.572335+0000","last_became_peered":"2024-05-09T08:23:50.572335+0000","last_unstale":"2024-05-09T08:24:15.377419+0000","last_undegraded":"2024-05-09T08:24:15.377419+0000","last_fullsized":"2024-05-09T08:24:15.377419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:28:24.028910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469990+0000","last_change":"2024-05-09T08:23:51.577595+0000","last_active":"2024-05-09T08:23:57.469990+0000","last_peered":"2024-05-09T08:23:57.469990+0000","last_clean":"2024-05-09T08:23:57.469990+0000","last_became_active":"2024-05-09T08:23:51.577271+0000","last_became_peered":"2024-05-09T08:23:51.577271+0000","last_unstale":"2024-05-09T08:23:57.469990+0000","last_undegraded":"2024-05-09T08:23:57.469990+0000","last_fullsized":"2024-05-09T08:23:57.469990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:34.679111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023394+0000","last_change":"2024-05-09T08:23:55.671128+0000","last_active":"2024-05-09T08:23:58.023394+0000","last_peered":"2024-05-09T08:23:58.023394+0000","last_clean":"2024-05-09T08:23:58.023394+0000","last_became_active":"2024-05-09T08:23:55.670414+0000","last_became_peered":"2024-05-09T08:23:55.670414+0000","last_unstale":"2024-05-09T08:23:58.023394+0000","last_undegraded":"2024-05-09T08:23:58.023394+0000","last_fullsized":"2024-05-09T08:23:58.023394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:31:55.307412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.388632+0000","last_change":"2024-05-09T08:23:50.568503+0000","last_active":"2024-05-09T08:24:16.388632+0000","last_peered":"2024-05-09T08:24:16.388632+0000","last_clean":"2024-05-09T08:24:16.388632+0000","last_became_active":"2024-05-09T08:23:50.568181+0000","last_became_peered":"2024-05-09T08:23:50.568181+0000","last_unstale":"2024-05-09T08:24:16.388632+0000","last_undegraded":"2024-05-09T08:24:16.388632+0000","last_fullsized":"2024-05-09T08:24:16.388632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:40.883792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022063+0000","last_change":"2024-05-09T08:23:51.573602+0000","last_active":"2024-05-09T08:23:58.022063+0000","last_peered":"2024-05-09T08:23:58.022063+0000","last_clean":"2024-05-09T08:23:58.022063+0000","last_became_active":"2024-05-09T08:23:51.572557+0000","last_became_peered":"2024-05-09T08:23:51.572557+0000","last_unstale":"2024-05-09T08:23:58.022063+0000","last_undegraded":"2024-05-09T08:23:58.022063+0000","last_fullsized":"2024-05-09T08:23:58.022063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.170962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329640+0000","last_change":"2024-05-09T08:23:55.673773+0000","last_active":"2024-05-09T08:23:57.329640+0000","last_peered":"2024-05-09T08:23:57.329640+0000","last_clean":"2024-05-09T08:23:57.329640+0000","last_became_active":"2024-05-09T08:23:55.673276+0000","last_became_peered":"2024-05-09T08:23:55.673276+0000","last_unstale":"2024-05-09T08:23:57.329640+0000","last_undegraded":"2024-05-09T08:23:57.329640+0000","last_fullsized":"2024-05-09T08:23:57.329640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:59:00.629339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378394+0000","last_change":"2024-05-09T08:23:50.575894+0000","last_active":"2024-05-09T08:24:15.378394+0000","last_peered":"2024-05-09T08:24:15.378394+0000","last_clean":"2024-05-09T08:24:15.378394+0000","last_became_active":"2024-05-09T08:23:50.575555+0000","last_became_peered":"2024-05-09T08:23:50.575555+0000","last_unstale":"2024-05-09T08:24:15.378394+0000","last_undegraded":"2024-05-09T08:24:15.378394+0000","last_fullsized":"2024-05-09T08:24:15.378394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:19.070764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022381+0000","last_change":"2024-05-09T08:23:51.574125+0000","last_active":"2024-05-09T08:23:58.022381+0000","last_peered":"2024-05-09T08:23:58.022381+0000","last_clean":"2024-05-09T08:23:58.022381+0000","last_became_active":"2024-05-09T08:23:51.573794+0000","last_became_peered":"2024-05-09T08:23:51.573794+0000","last_unstale":"2024-05-09T08:23:58.022381+0000","last_undegraded":"2024-05-09T08:23:58.022381+0000","last_fullsized":"2024-05-09T08:23:58.022381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:19:26.634208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654685+0000","last_change":"2024-05-09T08:23:55.677711+0000","last_active":"2024-05-09T08:23:56.654685+0000","last_peered":"2024-05-09T08:23:56.654685+0000","last_clean":"2024-05-09T08:23:56.654685+0000","last_became_active":"2024-05-09T08:23:55.677058+0000","last_became_peered":"2024-05-09T08:23:55.677058+0000","last_unstale":"2024-05-09T08:23:56.654685+0000","last_undegraded":"2024-05-09T08:23:56.654685+0000","last_fullsized":"2024-05-09T08:23:56.654685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:26.842145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.393470+0000","last_change":"2024-05-09T08:23:50.566413+0000","last_active":"2024-05-09T08:24:17.393470+0000","last_peered":"2024-05-09T08:24:17.393470+0000","last_clean":"2024-05-09T08:24:17.393470+0000","last_became_active":"2024-05-09T08:23:50.566047+0000","last_became_peered":"2024-05-09T08:23:50.566047+0000","last_unstale":"2024-05-09T08:24:17.393470+0000","last_undegraded":"2024-05-09T08:24:17.393470+0000","last_fullsized":"2024-05-09T08:24:17.393470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:15.340947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022166+0000","last_change":"2024-05-09T08:23:51.576779+0000","last_active":"2024-05-09T08:23:58.022166+0000","last_peered":"2024-05-09T08:23:58.022166+0000","last_clean":"2024-05-09T08:23:58.022166+0000","last_became_active":"2024-05-09T08:23:51.576119+0000","last_became_peered":"2024-05-09T08:23:51.576119+0000","last_unstale":"2024-05-09T08:23:58.022166+0000","last_undegraded":"2024-05-09T08:23:58.022166+0000","last_fullsized":"2024-05-09T08:23:58.022166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:02.250135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022592+0000","last_change":"2024-05-09T08:23:55.678564+0000","last_active":"2024-05-09T08:23:58.022592+0000","last_peered":"2024-05-09T08:23:58.022592+0000","last_clean":"2024-05-09T08:23:58.022592+0000","last_became_active":"2024-05-09T08:23:55.678128+0000","last_became_peered":"2024-05-09T08:23:55.678128+0000","last_unstale":"2024-05-09T08:23:58.022592+0000","last_undegraded":"2024-05-09T08:23:58.022592+0000","last_fullsized":"2024-05-09T08:23:58.022592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:13:21.868267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.389391+0000","last_change":"2024-05-09T08:23:50.565062+0000","last_active":"2024-05-09T08:24:16.389391+0000","last_peered":"2024-05-09T08:24:16.389391+0000","last_clean":"2024-05-09T08:24:16.389391+0000","last_became_active":"2024-05-09T08:23:50.564393+0000","last_became_peered":"2024-05-09T08:23:50.564393+0000","last_unstale":"2024-05-09T08:24:16.389391+0000","last_undegraded":"2024-05-09T08:24:16.389391+0000","last_fullsized":"2024-05-09T08:24:16.389391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:08:35.060335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022608+0000","last_change":"2024-05-09T08:23:51.574914+0000","last_active":"2024-05-09T08:23:58.022608+0000","last_peered":"2024-05-09T08:23:58.022608+0000","last_clean":"2024-05-09T08:23:58.022608+0000","last_became_active":"2024-05-09T08:23:51.574380+0000","last_became_peered":"2024-05-09T08:23:51.574380+0000","last_unstale":"2024-05-09T08:23:58.022608+0000","last_undegraded":"2024-05-09T08:23:58.022608+0000","last_fullsized":"2024-05-09T08:23:58.022608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:34.698328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022830+0000","last_change":"2024-05-09T08:23:55.670198+0000","last_active":"2024-05-09T08:23:58.022830+0000","last_peered":"2024-05-09T08:23:58.022830+0000","last_clean":"2024-05-09T08:23:58.022830+0000","last_became_active":"2024-05-09T08:23:55.669798+0000","last_became_peered":"2024-05-09T08:23:55.669798+0000","last_unstale":"2024-05-09T08:23:58.022830+0000","last_undegraded":"2024-05-09T08:23:58.022830+0000","last_fullsized":"2024-05-09T08:23:58.022830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:09.474264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.470141+0000","last_change":"2024-05-09T08:23:50.565061+0000","last_active":"2024-05-09T08:23:57.470141+0000","last_peered":"2024-05-09T08:23:57.470141+0000","last_clean":"2024-05-09T08:23:57.470141+0000","last_became_active":"2024-05-09T08:23:50.564736+0000","last_became_peered":"2024-05-09T08:23:50.564736+0000","last_unstale":"2024-05-09T08:23:57.470141+0000","last_undegraded":"2024-05-09T08:23:57.470141+0000","last_fullsized":"2024-05-09T08:23:57.470141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:33.593543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329780+0000","last_change":"2024-05-09T08:23:51.573434+0000","last_active":"2024-05-09T08:23:57.329780+0000","last_peered":"2024-05-09T08:23:57.329780+0000","last_clean":"2024-05-09T08:23:57.329780+0000","last_became_active":"2024-05-09T08:23:51.573081+0000","last_became_peered":"2024-05-09T08:23:51.573081+0000","last_unstale":"2024-05-09T08:23:57.329780+0000","last_undegraded":"2024-05-09T08:23:57.329780+0000","last_fullsized":"2024-05-09T08:23:57.329780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:49.270355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469927+0000","last_change":"2024-05-09T08:23:55.853114+0000","last_active":"2024-05-09T08:23:57.469927+0000","last_peered":"2024-05-09T08:23:57.469927+0000","last_clean":"2024-05-09T08:23:57.469927+0000","last_became_active":"2024-05-09T08:23:55.852640+0000","last_became_peered":"2024-05-09T08:23:55.852640+0000","last_unstale":"2024-05-09T08:23:57.469927+0000","last_undegraded":"2024-05-09T08:23:57.469927+0000","last_fullsized":"2024-05-09T08:23:57.469927+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:55:17.823585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654431+0000","last_change":"2024-05-09T08:23:50.567834+0000","last_active":"2024-05-09T08:23:56.654431+0000","last_peered":"2024-05-09T08:23:56.654431+0000","last_clean":"2024-05-09T08:23:56.654431+0000","last_became_active":"2024-05-09T08:23:50.567538+0000","last_became_peered":"2024-05-09T08:23:50.567538+0000","last_unstale":"2024-05-09T08:23:56.654431+0000","last_undegraded":"2024-05-09T08:23:56.654431+0000","last_fullsized":"2024-05-09T08:23:56.654431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.674352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654617+0000","last_change":"2024-05-09T08:23:51.580219+0000","last_active":"2024-05-09T08:23:56.654617+0000","last_peered":"2024-05-09T08:23:56.654617+0000","last_clean":"2024-05-09T08:23:56.654617+0000","last_became_active":"2024-05-09T08:23:51.579835+0000","last_became_peered":"2024-05-09T08:23:51.579835+0000","last_unstale":"2024-05-09T08:23:56.654617+0000","last_undegraded":"2024-05-09T08:23:56.654617+0000","last_fullsized":"2024-05-09T08:23:56.654617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:15.232318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.021951+0000","last_change":"2024-05-09T08:23:55.674324+0000","last_active":"2024-05-09T08:23:58.021951+0000","last_peered":"2024-05-09T08:23:58.021951+0000","last_clean":"2024-05-09T08:23:58.021951+0000","last_became_active":"2024-05-09T08:23:55.673254+0000","last_became_peered":"2024-05-09T08:23:55.673254+0000","last_unstale":"2024-05-09T08:23:58.021951+0000","last_undegraded":"2024-05-09T08:23:58.021951+0000","last_fullsized":"2024-05-09T08:23:58.021951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:05:23.303987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022220+0000","last_change":"2024-05-09T08:23:50.571682+0000","last_active":"2024-05-09T08:23:58.022220+0000","last_peered":"2024-05-09T08:23:58.022220+0000","last_clean":"2024-05-09T08:23:58.022220+0000","last_became_active":"2024-05-09T08:23:50.571338+0000","last_became_peered":"2024-05-09T08:23:50.571338+0000","last_unstale":"2024-05-09T08:23:58.022220+0000","last_undegraded":"2024-05-09T08:23:58.022220+0000","last_fullsized":"2024-05-09T08:23:58.022220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:17.578071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654373+0000","last_change":"2024-05-09T08:23:51.580048+0000","last_active":"2024-05-09T08:23:56.654373+0000","last_peered":"2024-05-09T08:23:56.654373+0000","last_clean":"2024-05-09T08:23:56.654373+0000","last_became_active":"2024-05-09T08:23:51.579725+0000","last_became_peered":"2024-05-09T08:23:51.579725+0000","last_unstale":"2024-05-09T08:23:56.654373+0000","last_undegraded":"2024-05-09T08:23:56.654373+0000","last_fullsized":"2024-05-09T08:23:56.654373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:14:47.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022365+0000","last_change":"2024-05-09T08:23:55.853086+0000","last_active":"2024-05-09T08:23:58.022365+0000","last_peered":"2024-05-09T08:23:58.022365+0000","last_clean":"2024-05-09T08:23:58.022365+0000","last_became_active":"2024-05-09T08:23:55.852302+0000","last_became_peered":"2024-05-09T08:23:55.852302+0000","last_unstale":"2024-05-09T08:23:58.022365+0000","last_undegraded":"2024-05-09T08:23:58.022365+0000","last_fullsized":"2024-05-09T08:23:58.022365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:42:18.547623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022780+0000","last_change":"2024-05-09T08:23:51.575522+0000","last_active":"2024-05-09T08:23:58.022780+0000","last_peered":"2024-05-09T08:23:58.022780+0000","last_clean":"2024-05-09T08:23:58.022780+0000","last_became_active":"2024-05-09T08:23:51.575204+0000","last_became_peered":"2024-05-09T08:23:51.575204+0000","last_unstale":"2024-05-09T08:23:58.022780+0000","last_undegraded":"2024-05-09T08:23:58.022780+0000","last_fullsized":"2024-05-09T08:23:58.022780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:57.257435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.388636+0000","last_change":"2024-05-09T08:23:50.574996+0000","last_active":"2024-05-09T08:24:16.388636+0000","last_peered":"2024-05-09T08:24:16.388636+0000","last_clean":"2024-05-09T08:24:16.388636+0000","last_became_active":"2024-05-09T08:23:50.574619+0000","last_became_peered":"2024-05-09T08:23:50.574619+0000","last_unstale":"2024-05-09T08:24:16.388636+0000","last_undegraded":"2024-05-09T08:24:16.388636+0000","last_fullsized":"2024-05-09T08:24:16.388636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:44.681314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022504+0000","last_change":"2024-05-09T08:23:55.667214+0000","last_active":"2024-05-09T08:23:58.022504+0000","last_peered":"2024-05-09T08:23:58.022504+0000","last_clean":"2024-05-09T08:23:58.022504+0000","last_became_active":"2024-05-09T08:23:55.666716+0000","last_became_peered":"2024-05-09T08:23:55.666716+0000","last_unstale":"2024-05-09T08:23:58.022504+0000","last_undegraded":"2024-05-09T08:23:58.022504+0000","last_fullsized":"2024-05-09T08:23:58.022504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:32:05.332938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022096+0000","last_change":"2024-05-09T08:23:51.573944+0000","last_active":"2024-05-09T08:23:58.022096+0000","last_peered":"2024-05-09T08:23:58.022096+0000","last_clean":"2024-05-09T08:23:58.022096+0000","last_became_active":"2024-05-09T08:23:51.573229+0000","last_became_peered":"2024-05-09T08:23:51.573229+0000","last_unstale":"2024-05-09T08:23:58.022096+0000","last_undegraded":"2024-05-09T08:23:58.022096+0000","last_fullsized":"2024-05-09T08:23:58.022096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:53.828582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.393399+0000","last_change":"2024-05-09T08:23:50.566415+0000","last_active":"2024-05-09T08:24:17.393399+0000","last_peered":"2024-05-09T08:24:17.393399+0000","last_clean":"2024-05-09T08:24:17.393399+0000","last_became_active":"2024-05-09T08:23:50.565506+0000","last_became_peered":"2024-05-09T08:23:50.565506+0000","last_unstale":"2024-05-09T08:24:17.393399+0000","last_undegraded":"2024-05-09T08:24:17.393399+0000","last_fullsized":"2024-05-09T08:24:17.393399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:15.741565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022001+0000","last_change":"2024-05-09T08:22:38.366126+0000","last_active":"2024-05-09T08:23:58.022001+0000","last_peered":"2024-05-09T08:23:58.022001+0000","last_clean":"2024-05-09T08:23:58.022001+0000","last_became_active":"2024-05-09T08:22:38.365683+0000","last_became_peered":"2024-05-09T08:22:38.365683+0000","last_unstale":"2024-05-09T08:23:58.022001+0000","last_undegraded":"2024-05-09T08:23:58.022001+0000","last_fullsized":"2024-05-09T08:23:58.022001+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_clean_scrub_stamp":"2024-05-09T08:19:12.435445+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:04.744060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023282+0000","last_change":"2024-05-09T08:23:55.853248+0000","last_active":"2024-05-09T08:23:58.023282+0000","last_peered":"2024-05-09T08:23:58.023282+0000","last_clean":"2024-05-09T08:23:58.023282+0000","last_became_active":"2024-05-09T08:23:55.852646+0000","last_became_peered":"2024-05-09T08:23:55.852646+0000","last_unstale":"2024-05-09T08:23:58.023282+0000","last_undegraded":"2024-05-09T08:23:58.023282+0000","last_fullsized":"2024-05-09T08:23:58.023282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:19:54.378185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.021652+0000","last_change":"2024-05-09T08:23:51.574967+0000","last_active":"2024-05-09T08:23:58.021652+0000","last_peered":"2024-05-09T08:23:58.021652+0000","last_clean":"2024-05-09T08:23:58.021652+0000","last_became_active":"2024-05-09T08:23:51.574600+0000","last_became_peered":"2024-05-09T08:23:51.574600+0000","last_unstale":"2024-05-09T08:23:58.021652+0000","last_undegraded":"2024-05-09T08:23:58.021652+0000","last_fullsized":"2024-05-09T08:23:58.021652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:03.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:16.387812+0000","last_change":"2024-05-09T08:23:50.569886+0000","last_active":"2024-05-09T08:24:16.387812+0000","last_peered":"2024-05-09T08:24:16.387812+0000","last_clean":"2024-05-09T08:24:16.387812+0000","last_became_active":"2024-05-09T08:23:50.569018+0000","last_became_peered":"2024-05-09T08:23:50.569018+0000","last_unstale":"2024-05-09T08:24:16.387812+0000","last_undegraded":"2024-05-09T08:24:16.387812+0000","last_fullsized":"2024-05-09T08:24:16.387812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:20.711485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329690+0000","last_change":"2024-05-09T08:23:55.676408+0000","last_active":"2024-05-09T08:23:57.329690+0000","last_peered":"2024-05-09T08:23:57.329690+0000","last_clean":"2024-05-09T08:23:57.329690+0000","last_became_active":"2024-05-09T08:23:55.675964+0000","last_became_peered":"2024-05-09T08:23:55.675964+0000","last_unstale":"2024-05-09T08:23:57.329690+0000","last_undegraded":"2024-05-09T08:23:57.329690+0000","last_fullsized":"2024-05-09T08:23:57.329690+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:29.210602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.392980+0000","last_change":"2024-05-09T08:23:50.566946+0000","last_active":"2024-05-09T08:24:17.392980+0000","last_peered":"2024-05-09T08:24:17.392980+0000","last_clean":"2024-05-09T08:24:17.392980+0000","last_became_active":"2024-05-09T08:23:50.566366+0000","last_became_peered":"2024-05-09T08:23:50.566366+0000","last_unstale":"2024-05-09T08:24:17.392980+0000","last_undegraded":"2024-05-09T08:24:17.392980+0000","last_fullsized":"2024-05-09T08:24:17.392980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:37:30.720879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022521+0000","last_change":"2024-05-09T08:23:51.571520+0000","last_active":"2024-05-09T08:23:58.022521+0000","last_peered":"2024-05-09T08:23:58.022521+0000","last_clean":"2024-05-09T08:23:58.022521+0000","last_became_active":"2024-05-09T08:23:51.571166+0000","last_became_peered":"2024-05-09T08:23:51.571166+0000","last_unstale":"2024-05-09T08:23:58.022521+0000","last_undegraded":"2024-05-09T08:23:58.022521+0000","last_fullsized":"2024-05-09T08:23:58.022521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:02:12.560238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.021947+0000","last_change":"2024-05-09T08:23:55.852649+0000","last_active":"2024-05-09T08:23:58.021947+0000","last_peered":"2024-05-09T08:23:58.021947+0000","last_clean":"2024-05-09T08:23:58.021947+0000","last_became_active":"2024-05-09T08:23:55.852180+0000","last_became_peered":"2024-05-09T08:23:55.852180+0000","last_unstale":"2024-05-09T08:23:58.021947+0000","last_undegraded":"2024-05-09T08:23:58.021947+0000","last_fullsized":"2024-05-09T08:23:58.021947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:33.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654852+0000","last_change":"2024-05-09T08:23:50.561639+0000","last_active":"2024-05-09T08:23:56.654852+0000","last_peered":"2024-05-09T08:23:56.654852+0000","last_clean":"2024-05-09T08:23:56.654852+0000","last_became_active":"2024-05-09T08:23:50.561306+0000","last_became_peered":"2024-05-09T08:23:50.561306+0000","last_unstale":"2024-05-09T08:23:56.654852+0000","last_undegraded":"2024-05-09T08:23:56.654852+0000","last_fullsized":"2024-05-09T08:23:56.654852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:46.897965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022842+0000","last_change":"2024-05-09T08:23:51.573765+0000","last_active":"2024-05-09T08:23:58.022842+0000","last_peered":"2024-05-09T08:23:58.022842+0000","last_clean":"2024-05-09T08:23:58.022842+0000","last_became_active":"2024-05-09T08:23:51.573134+0000","last_became_peered":"2024-05-09T08:23:51.573134+0000","last_unstale":"2024-05-09T08:23:58.022842+0000","last_undegraded":"2024-05-09T08:23:58.022842+0000","last_fullsized":"2024-05-09T08:23:58.022842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:25.334148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654463+0000","last_change":"2024-05-09T08:23:55.663480+0000","last_active":"2024-05-09T08:23:56.654463+0000","last_peered":"2024-05-09T08:23:56.654463+0000","last_clean":"2024-05-09T08:23:56.654463+0000","last_became_active":"2024-05-09T08:23:55.662823+0000","last_became_peered":"2024-05-09T08:23:55.662823+0000","last_unstale":"2024-05-09T08:23:56.654463+0000","last_undegraded":"2024-05-09T08:23:56.654463+0000","last_fullsized":"2024-05-09T08:23:56.654463+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:18.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378366+0000","last_change":"2024-05-09T08:23:50.574641+0000","last_active":"2024-05-09T08:24:15.378366+0000","last_peered":"2024-05-09T08:24:15.378366+0000","last_clean":"2024-05-09T08:24:15.378366+0000","last_became_active":"2024-05-09T08:23:50.574266+0000","last_became_peered":"2024-05-09T08:23:50.574266+0000","last_unstale":"2024-05-09T08:24:15.378366+0000","last_undegraded":"2024-05-09T08:24:15.378366+0000","last_fullsized":"2024-05-09T08:24:15.378366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.575211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.301162+0000","last_change":"2024-05-09T08:23:51.583698+0000","last_active":"2024-05-09T08:23:58.301162+0000","last_peered":"2024-05-09T08:23:58.301162+0000","last_clean":"2024-05-09T08:23:58.301162+0000","last_became_active":"2024-05-09T08:23:51.583342+0000","last_became_peered":"2024-05-09T08:23:51.583342+0000","last_unstale":"2024-05-09T08:23:58.301162+0000","last_undegraded":"2024-05-09T08:23:58.301162+0000","last_fullsized":"2024-05-09T08:23:58.301162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:37.444163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921948+0000","last_change":"2024-05-09T08:23:55.853759+0000","last_active":"2024-05-09T08:23:56.921948+0000","last_peered":"2024-05-09T08:23:56.921948+0000","last_clean":"2024-05-09T08:23:56.921948+0000","last_became_active":"2024-05-09T08:23:55.853045+0000","last_became_peered":"2024-05-09T08:23:55.853045+0000","last_unstale":"2024-05-09T08:23:56.921948+0000","last_undegraded":"2024-05-09T08:23:56.921948+0000","last_fullsized":"2024-05-09T08:23:56.921948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:53.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.377837+0000","last_change":"2024-05-09T08:23:50.568373+0000","last_active":"2024-05-09T08:24:15.377837+0000","last_peered":"2024-05-09T08:24:15.377837+0000","last_clean":"2024-05-09T08:24:15.377837+0000","last_became_active":"2024-05-09T08:23:50.568015+0000","last_became_peered":"2024-05-09T08:23:50.568015+0000","last_unstale":"2024-05-09T08:24:15.377837+0000","last_undegraded":"2024-05-09T08:24:15.377837+0000","last_fullsized":"2024-05-09T08:24:15.377837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:05:16.391673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.469519+0000","last_change":"2024-05-09T08:23:51.576825+0000","last_active":"2024-05-09T08:23:57.469519+0000","last_peered":"2024-05-09T08:23:57.469519+0000","last_clean":"2024-05-09T08:23:57.469519+0000","last_became_active":"2024-05-09T08:23:51.573350+0000","last_became_peered":"2024-05-09T08:23:51.573350+0000","last_unstale":"2024-05-09T08:23:57.469519+0000","last_undegraded":"2024-05-09T08:23:57.469519+0000","last_fullsized":"2024-05-09T08:23:57.469519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:00.963281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654788+0000","last_change":"2024-05-09T08:23:55.672145+0000","last_active":"2024-05-09T08:23:56.654788+0000","last_peered":"2024-05-09T08:23:56.654788+0000","last_clean":"2024-05-09T08:23:56.654788+0000","last_became_active":"2024-05-09T08:23:55.671743+0000","last_became_peered":"2024-05-09T08:23:55.671743+0000","last_unstale":"2024-05-09T08:23:56.654788+0000","last_undegraded":"2024-05-09T08:23:56.654788+0000","last_fullsized":"2024-05-09T08:23:56.654788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:09:23.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378677+0000","last_change":"2024-05-09T08:23:50.568908+0000","last_active":"2024-05-09T08:24:15.378677+0000","last_peered":"2024-05-09T08:24:15.378677+0000","last_clean":"2024-05-09T08:24:15.378677+0000","last_became_active":"2024-05-09T08:23:50.567009+0000","last_became_peered":"2024-05-09T08:23:50.567009+0000","last_unstale":"2024-05-09T08:24:15.378677+0000","last_undegraded":"2024-05-09T08:24:15.378677+0000","last_fullsized":"2024-05-09T08:24:15.378677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:35.190156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329478+0000","last_change":"2024-05-09T08:23:51.580121+0000","last_active":"2024-05-09T08:23:57.329478+0000","last_peered":"2024-05-09T08:23:57.329478+0000","last_clean":"2024-05-09T08:23:57.329478+0000","last_became_active":"2024-05-09T08:23:51.579402+0000","last_became_peered":"2024-05-09T08:23:51.579402+0000","last_unstale":"2024-05-09T08:23:57.329478+0000","last_undegraded":"2024-05-09T08:23:57.329478+0000","last_fullsized":"2024-05-09T08:23:57.329478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:11:26.144344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921851+0000","last_change":"2024-05-09T08:23:55.664017+0000","last_active":"2024-05-09T08:23:56.921851+0000","last_peered":"2024-05-09T08:23:56.921851+0000","last_clean":"2024-05-09T08:23:56.921851+0000","last_became_active":"2024-05-09T08:23:55.663546+0000","last_became_peered":"2024-05-09T08:23:55.663546+0000","last_unstale":"2024-05-09T08:23:56.921851+0000","last_undegraded":"2024-05-09T08:23:56.921851+0000","last_fullsized":"2024-05-09T08:23:56.921851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:13:14.705971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.373758+0000","last_change":"2024-05-09T08:23:50.568472+0000","last_active":"2024-05-09T08:24:17.373758+0000","last_peered":"2024-05-09T08:24:17.373758+0000","last_clean":"2024-05-09T08:24:17.373758+0000","last_became_active":"2024-05-09T08:23:50.568132+0000","last_became_peered":"2024-05-09T08:23:50.568132+0000","last_unstale":"2024-05-09T08:24:17.373758+0000","last_undegraded":"2024-05-09T08:24:17.373758+0000","last_fullsized":"2024-05-09T08:24:17.373758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:34.468411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022722+0000","last_change":"2024-05-09T08:23:51.571414+0000","last_active":"2024-05-09T08:23:58.022722+0000","last_peered":"2024-05-09T08:23:58.022722+0000","last_clean":"2024-05-09T08:23:58.022722+0000","last_became_active":"2024-05-09T08:23:51.571085+0000","last_became_peered":"2024-05-09T08:23:51.571085+0000","last_unstale":"2024-05-09T08:23:58.022722+0000","last_undegraded":"2024-05-09T08:23:58.022722+0000","last_fullsized":"2024-05-09T08:23:58.022722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:37.513584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.921812+0000","last_change":"2024-05-09T08:23:55.852785+0000","last_active":"2024-05-09T08:23:56.921812+0000","last_peered":"2024-05-09T08:23:56.921812+0000","last_clean":"2024-05-09T08:23:56.921812+0000","last_became_active":"2024-05-09T08:23:55.852252+0000","last_became_peered":"2024-05-09T08:23:55.852252+0000","last_unstale":"2024-05-09T08:23:56.921812+0000","last_undegraded":"2024-05-09T08:23:56.921812+0000","last_fullsized":"2024-05-09T08:23:56.921812+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:44.503295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:15.378953+0000","last_change":"2024-05-09T08:23:50.564787+0000","last_active":"2024-05-09T08:24:15.378953+0000","last_peered":"2024-05-09T08:24:15.378953+0000","last_clean":"2024-05-09T08:24:15.378953+0000","last_became_active":"2024-05-09T08:23:50.564409+0000","last_became_peered":"2024-05-09T08:23:50.564409+0000","last_unstale":"2024-05-09T08:24:15.378953+0000","last_undegraded":"2024-05-09T08:24:15.378953+0000","last_fullsized":"2024-05-09T08:24:15.378953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:23.051221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022284+0000","last_change":"2024-05-09T08:23:51.576830+0000","last_active":"2024-05-09T08:23:58.022284+0000","last_peered":"2024-05-09T08:23:58.022284+0000","last_clean":"2024-05-09T08:23:58.022284+0000","last_became_active":"2024-05-09T08:23:51.576311+0000","last_became_peered":"2024-05-09T08:23:51.576311+0000","last_unstale":"2024-05-09T08:23:58.022284+0000","last_undegraded":"2024-05-09T08:23:58.022284+0000","last_fullsized":"2024-05-09T08:23:58.022284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:26:08.651061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488040+0000","last_change":"2024-05-09T08:23:55.672309+0000","last_active":"2024-05-09T08:23:57.488040+0000","last_peered":"2024-05-09T08:23:57.488040+0000","last_clean":"2024-05-09T08:23:57.488040+0000","last_became_active":"2024-05-09T08:23:55.671811+0000","last_became_peered":"2024-05-09T08:23:55.671811+0000","last_unstale":"2024-05-09T08:23:57.488040+0000","last_undegraded":"2024-05-09T08:23:57.488040+0000","last_fullsized":"2024-05-09T08:23:57.488040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:19.400949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:24:17.395073+0000","last_change":"2024-05-09T08:23:50.575317+0000","last_active":"2024-05-09T08:24:17.395073+0000","last_peered":"2024-05-09T08:24:17.395073+0000","last_clean":"2024-05-09T08:24:17.395073+0000","last_became_active":"2024-05-09T08:23:50.574995+0000","last_became_peered":"2024-05-09T08:23:50.574995+0000","last_unstale":"2024-05-09T08:24:17.395073+0000","last_undegraded":"2024-05-09T08:24:17.395073+0000","last_fullsized":"2024-05-09T08:24:17.395073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:32:54.801799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022672+0000","last_change":"2024-05-09T08:23:51.575904+0000","last_active":"2024-05-09T08:23:58.022672+0000","last_peered":"2024-05-09T08:23:58.022672+0000","last_clean":"2024-05-09T08:23:58.022672+0000","last_became_active":"2024-05-09T08:23:51.575599+0000","last_became_peered":"2024-05-09T08:23:51.575599+0000","last_unstale":"2024-05-09T08:23:58.022672+0000","last_undegraded":"2024-05-09T08:23:58.022672+0000","last_fullsized":"2024-05-09T08:23:58.022672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:20:13.087096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300776+0000","last_change":"2024-05-09T08:23:55.680683+0000","last_active":"2024-05-09T08:23:58.300776+0000","last_peered":"2024-05-09T08:23:58.300776+0000","last_clean":"2024-05-09T08:23:58.300776+0000","last_became_active":"2024-05-09T08:23:55.680232+0000","last_became_peered":"2024-05-09T08:23:55.680232+0000","last_unstale":"2024-05-09T08:23:58.300776+0000","last_undegraded":"2024-05-09T08:23:58.300776+0000","last_fullsized":"2024-05-09T08:23:58.300776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:33.583552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023199+0000","last_change":"2024-05-09T08:23:50.575802+0000","last_active":"2024-05-09T08:23:58.023199+0000","last_peered":"2024-05-09T08:23:58.023199+0000","last_clean":"2024-05-09T08:23:58.023199+0000","last_became_active":"2024-05-09T08:23:50.575416+0000","last_became_peered":"2024-05-09T08:23:50.575416+0000","last_unstale":"2024-05-09T08:23:58.023199+0000","last_undegraded":"2024-05-09T08:23:58.023199+0000","last_fullsized":"2024-05-09T08:23:58.023199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:24:04.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022441+0000","last_change":"2024-05-09T08:23:51.578189+0000","last_active":"2024-05-09T08:23:58.022441+0000","last_peered":"2024-05-09T08:23:58.022441+0000","last_clean":"2024-05-09T08:23:58.022441+0000","last_became_active":"2024-05-09T08:23:51.577500+0000","last_became_peered":"2024-05-09T08:23:51.577500+0000","last_unstale":"2024-05-09T08:23:58.022441+0000","last_undegraded":"2024-05-09T08:23:58.022441+0000","last_fullsized":"2024-05-09T08:23:58.022441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:26:24.718791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022440+0000","last_change":"2024-05-09T08:23:55.670859+0000","last_active":"2024-05-09T08:23:58.022440+0000","last_peered":"2024-05-09T08:23:58.022440+0000","last_clean":"2024-05-09T08:23:58.022440+0000","last_became_active":"2024-05-09T08:23:55.670447+0000","last_became_peered":"2024-05-09T08:23:55.670447+0000","last_unstale":"2024-05-09T08:23:58.022440+0000","last_undegraded":"2024-05-09T08:23:58.022440+0000","last_fullsized":"2024-05-09T08:23:58.022440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:25:58.018733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022748+0000","last_change":"2024-05-09T08:23:50.578438+0000","last_active":"2024-05-09T08:23:58.022748+0000","last_peered":"2024-05-09T08:23:58.022748+0000","last_clean":"2024-05-09T08:23:58.022748+0000","last_became_active":"2024-05-09T08:23:50.578039+0000","last_became_peered":"2024-05-09T08:23:50.578039+0000","last_unstale":"2024-05-09T08:23:58.022748+0000","last_undegraded":"2024-05-09T08:23:58.022748+0000","last_fullsized":"2024-05-09T08:23:58.022748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:59.420193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022732+0000","last_change":"2024-05-09T08:23:51.568037+0000","last_active":"2024-05-09T08:23:58.022732+0000","last_peered":"2024-05-09T08:23:58.022732+0000","last_clean":"2024-05-09T08:23:58.022732+0000","last_became_active":"2024-05-09T08:23:51.567706+0000","last_became_peered":"2024-05-09T08:23:51.567706+0000","last_unstale":"2024-05-09T08:23:58.022732+0000","last_undegraded":"2024-05-09T08:23:58.022732+0000","last_fullsized":"2024-05-09T08:23:58.022732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:15:19.623805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329571+0000","last_change":"2024-05-09T08:23:51.579885+0000","last_active":"2024-05-09T08:23:57.329571+0000","last_peered":"2024-05-09T08:23:57.329571+0000","last_clean":"2024-05-09T08:23:57.329571+0000","last_became_active":"2024-05-09T08:23:51.578953+0000","last_became_peered":"2024-05-09T08:23:51.578953+0000","last_unstale":"2024-05-09T08:23:57.329571+0000","last_undegraded":"2024-05-09T08:23:57.329571+0000","last_fullsized":"2024-05-09T08:23:57.329571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:57:23.793119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654492+0000","last_change":"2024-05-09T08:23:51.583423+0000","last_active":"2024-05-09T08:23:56.654492+0000","last_peered":"2024-05-09T08:23:56.654492+0000","last_clean":"2024-05-09T08:23:56.654492+0000","last_became_active":"2024-05-09T08:23:51.583092+0000","last_became_peered":"2024-05-09T08:23:51.583092+0000","last_unstale":"2024-05-09T08:23:56.654492+0000","last_undegraded":"2024-05-09T08:23:56.654492+0000","last_fullsized":"2024-05-09T08:23:56.654492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:44:47.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.654349+0000","last_change":"2024-05-09T08:23:51.582116+0000","last_active":"2024-05-09T08:23:56.654349+0000","last_peered":"2024-05-09T08:23:56.654349+0000","last_clean":"2024-05-09T08:23:56.654349+0000","last_became_active":"2024-05-09T08:23:51.581715+0000","last_became_peered":"2024-05-09T08:23:51.581715+0000","last_unstale":"2024-05-09T08:23:56.654349+0000","last_undegraded":"2024-05-09T08:23:56.654349+0000","last_fullsized":"2024-05-09T08:23:56.654349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:17.528758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.300902+0000","last_change":"2024-05-09T08:23:51.575959+0000","last_active":"2024-05-09T08:23:58.300902+0000","last_peered":"2024-05-09T08:23:58.300902+0000","last_clean":"2024-05-09T08:23:58.300902+0000","last_became_active":"2024-05-09T08:23:51.575592+0000","last_became_peered":"2024-05-09T08:23:51.575592+0000","last_unstale":"2024-05-09T08:23:58.300902+0000","last_undegraded":"2024-05-09T08:23:58.300902+0000","last_fullsized":"2024-05-09T08:23:58.300902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:26.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022639+0000","last_change":"2024-05-09T08:23:51.570756+0000","last_active":"2024-05-09T08:23:58.022639+0000","last_peered":"2024-05-09T08:23:58.022639+0000","last_clean":"2024-05-09T08:23:58.022639+0000","last_became_active":"2024-05-09T08:23:51.570412+0000","last_became_peered":"2024-05-09T08:23:51.570412+0000","last_unstale":"2024-05-09T08:23:58.022639+0000","last_undegraded":"2024-05-09T08:23:58.022639+0000","last_fullsized":"2024-05-09T08:23:58.022639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:11:53.959559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329974+0000","last_change":"2024-05-09T08:23:51.572612+0000","last_active":"2024-05-09T08:23:57.329974+0000","last_peered":"2024-05-09T08:23:57.329974+0000","last_clean":"2024-05-09T08:23:57.329974+0000","last_became_active":"2024-05-09T08:23:51.572227+0000","last_became_peered":"2024-05-09T08:23:51.572227+0000","last_unstale":"2024-05-09T08:23:57.329974+0000","last_undegraded":"2024-05-09T08:23:57.329974+0000","last_fullsized":"2024-05-09T08:23:57.329974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:18.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:56.655079+0000","last_change":"2024-05-09T08:23:51.573021+0000","last_active":"2024-05-09T08:23:56.655079+0000","last_peered":"2024-05-09T08:23:56.655079+0000","last_clean":"2024-05-09T08:23:56.655079+0000","last_became_active":"2024-05-09T08:23:51.572716+0000","last_became_peered":"2024-05-09T08:23:51.572716+0000","last_unstale":"2024-05-09T08:23:56.655079+0000","last_undegraded":"2024-05-09T08:23:56.655079+0000","last_fullsized":"2024-05-09T08:23:56.655079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:31.494329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022950+0000","last_change":"2024-05-09T08:23:51.572687+0000","last_active":"2024-05-09T08:23:58.022950+0000","last_peered":"2024-05-09T08:23:58.022950+0000","last_clean":"2024-05-09T08:23:58.022950+0000","last_became_active":"2024-05-09T08:23:51.572339+0000","last_became_peered":"2024-05-09T08:23:51.572339+0000","last_unstale":"2024-05-09T08:23:58.022950+0000","last_undegraded":"2024-05-09T08:23:58.022950+0000","last_fullsized":"2024-05-09T08:23:58.022950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:24.012360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022673+0000","last_change":"2024-05-09T08:23:51.569404+0000","last_active":"2024-05-09T08:23:58.022673+0000","last_peered":"2024-05-09T08:23:58.022673+0000","last_clean":"2024-05-09T08:23:58.022673+0000","last_became_active":"2024-05-09T08:23:51.569044+0000","last_became_peered":"2024-05-09T08:23:51.569044+0000","last_unstale":"2024-05-09T08:23:58.022673+0000","last_undegraded":"2024-05-09T08:23:58.022673+0000","last_fullsized":"2024-05-09T08:23:58.022673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:26:01.343606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.023141+0000","last_change":"2024-05-09T08:23:51.577483+0000","last_active":"2024-05-09T08:23:58.023141+0000","last_peered":"2024-05-09T08:23:58.023141+0000","last_clean":"2024-05-09T08:23:58.023141+0000","last_became_active":"2024-05-09T08:23:51.577078+0000","last_became_peered":"2024-05-09T08:23:51.577078+0000","last_unstale":"2024-05-09T08:23:58.023141+0000","last_undegraded":"2024-05-09T08:23:58.023141+0000","last_fullsized":"2024-05-09T08:23:58.023141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:29.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.488149+0000","last_change":"2024-05-09T08:23:51.579736+0000","last_active":"2024-05-09T08:23:57.488149+0000","last_peered":"2024-05-09T08:23:57.488149+0000","last_clean":"2024-05-09T08:23:57.488149+0000","last_became_active":"2024-05-09T08:23:51.579409+0000","last_became_peered":"2024-05-09T08:23:51.579409+0000","last_unstale":"2024-05-09T08:23:57.488149+0000","last_undegraded":"2024-05-09T08:23:57.488149+0000","last_fullsized":"2024-05-09T08:23:57.488149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:05.697642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:58.022353+0000","last_change":"2024-05-09T08:23:51.569461+0000","last_active":"2024-05-09T08:23:58.022353+0000","last_peered":"2024-05-09T08:23:58.022353+0000","last_clean":"2024-05-09T08:23:58.022353+0000","last_became_active":"2024-05-09T08:23:51.569085+0000","last_became_peered":"2024-05-09T08:23:51.569085+0000","last_unstale":"2024-05-09T08:23:58.022353+0000","last_undegraded":"2024-05-09T08:23:58.022353+0000","last_fullsized":"2024-05-09T08:23:58.022353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:54.554949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:23:57.329662+0000","last_change":"2024-05-09T08:23:51.570593+0000","last_active":"2024-05-09T08:23:57.329662+0000","last_peered":"2024-05-09T08:23:57.329662+0000","last_clean":"2024-05-09T08:23:57.329662+0000","last_became_active":"2024-05-09T08:23:51.570195+0000","last_became_peered":"2024-05-09T08:23:51.570195+0000","last_unstale":"2024-05-09T08:23:57.329662+0000","last_undegraded":"2024-05-09T08:23:57.329662+0000","last_fullsized":"2024-05-09T08:23:57.329662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:18.376268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283814,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8487,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":61417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50773,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50773,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":488239,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482407,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":478272,"data_compressed":14682,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50773,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":42223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:24:20.624 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-09T08:24:20.624 INFO:tasks.cephadm:Setting up client nodes... 2024-05-09T08:24:20.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-09T08:24:20.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-mon[34540]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:20.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-mon[34540]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:20.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-mon[34540]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:20.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] up:active 2024-05-09T08:24:20.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-mon[34540]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-09T08:24:20.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:24:20.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2509947952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:24:20.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2852229510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:20.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:20 smithi184 ceph-mon[34509]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:20.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:20 smithi184 ceph-mon[34509]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:20.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:20 smithi184 ceph-mon[34509]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:20.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:20 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] up:active 2024-05-09T08:24:20.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:20 smithi184 ceph-mon[34509]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-09T08:24:20.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:20 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:24:20.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:20 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2509947952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:24:20.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:20 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2852229510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:20.913 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:21.102 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b[64615]: 2024-05-09T08:24:20.743+0000 7fd5675ca700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-05-09T08:24:21.102 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:24:20 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b[64615]: 2024-05-09T08:24:20.743+0000 7fd5675ca700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-05-09T08:24:21.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:21 smithi028 ceph-mon[30119]: pgmap v304: 129 pgs: 129 active+clean; 585 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2024-05-09T08:24:21.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:21 smithi028 ceph-mon[30119]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:21.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:21 smithi028 ceph-mon[30119]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T08:24:21.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:21 smithi165 ceph-mon[34540]: pgmap v304: 129 pgs: 129 active+clean; 585 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2024-05-09T08:24:21.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:21 smithi165 ceph-mon[34540]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:21.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:21 smithi165 ceph-mon[34540]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T08:24:21.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:21 smithi184 ceph-mon[34509]: pgmap v304: 129 pgs: 129 active+clean; 585 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2024-05-09T08:24:21.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:21 smithi184 ceph-mon[34509]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:21.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:21 smithi184 ceph-mon[34509]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T08:24:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 -- 172.21.15.28:0/209947811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8101f70 msgr2=0x7f60c8109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 --2- 172.21.15.28:0/209947811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8101f70 0x7f60c8109810 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f60c000b0a0 tx=0x7f60c000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 -- 172.21.15.28:0/209947811 shutdown_connections 2024-05-09T08:24:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 --2- 172.21.15.28:0/209947811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8101f70 0x7f60c8109810 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 --2- 172.21.15.28:0/209947811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c81029e0 0x7f60c8102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 --2- 172.21.15.28:0/209947811 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60c81015b0 0x7f60c8101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 -- 172.21.15.28:0/209947811 >> 172.21.15.28:0/209947811 conn(0x7f60c80fc080 msgr2=0x7f60c80fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:22.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 -- 172.21.15.28:0/209947811 shutdown_connections 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 -- 172.21.15.28:0/209947811 wait complete. 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 Processor -- start 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.642+0000 7f60cede6700 1 -- start start 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60cede6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c81015b0 0x7f60c8101000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60cede6700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60c8101f70 0x7f60c819d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60cede6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c81029e0 0x7f60c819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60cede6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810cf80 con 0x7f60c81015b0 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60cede6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810ce00 con 0x7f60c81029e0 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60cede6700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810d100 con 0x7f60c8101f70 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60ccb82700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c81015b0 0x7f60c8101000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60ccb82700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c81015b0 0x7f60c8101000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35760/0 (socket says 172.21.15.28:35760) 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60ccb82700 1 -- 172.21.15.28:0/1854537457 learned_addr learned my addr 172.21.15.28:0/1854537457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60cd383700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c81029e0 0x7f60c819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60c7fff700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60c8101f70 0x7f60c819d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60ccb82700 1 -- 172.21.15.28:0/1854537457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60c8101f70 msgr2=0x7f60c819d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60ccb82700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60c8101f70 0x7f60c819d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60ccb82700 1 -- 172.21.15.28:0/1854537457 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c81029e0 msgr2=0x7f60c819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60ccb82700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c81029e0 0x7f60c819f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60ccb82700 1 -- 172.21.15.28:0/1854537457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c81a0110 con 0x7f60c81015b0 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.643+0000 7f60ccb82700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c81015b0 0x7f60c8101000 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00e9a0 tx=0x7f60bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.644+0000 7f60c5ffb700 1 -- 172.21.15.28:0/1854537457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00cbd0 con 0x7f60c81015b0 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.644+0000 7f60c5ffb700 1 -- 172.21.15.28:0/1854537457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60bc00cd70 con 0x7f60c81015b0 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.644+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c81a0400 con 0x7f60c81015b0 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.644+0000 7f60c5ffb700 1 -- 172.21.15.28:0/1854537457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc018750 con 0x7f60c81015b0 2024-05-09T08:24:22.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.644+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c81a0940 con 0x7f60c81015b0 2024-05-09T08:24:22.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.645+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c8050890 con 0x7f60c81015b0 2024-05-09T08:24:22.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.646+0000 7f60c5ffb700 1 -- 172.21.15.28:0/1854537457 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f60bc010430 con 0x7f60c81015b0 2024-05-09T08:24:22.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.647+0000 7f60c5ffb700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f60b80a2d60 0x7f60b80a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:22.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.647+0000 7f60c7fff700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f60b80a2d60 0x7f60b80a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:22.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.647+0000 7f60c5ffb700 1 -- 172.21.15.28:0/1854537457 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f60bc0cf730 con 0x7f60c81015b0 2024-05-09T08:24:22.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.647+0000 7f60c7fff700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f60b80a2d60 0x7f60b80a5220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f60c8101580 tx=0x7f60b4005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:22.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.651+0000 7f60c5ffb700 1 -- 172.21.15.28:0/1854537457 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f60bc09c290 con 0x7f60c81015b0 2024-05-09T08:24:22.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:22 smithi028 ceph-mon[30119]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:22.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:22 smithi165 ceph-mon[34540]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:22.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:22 smithi184 ceph-mon[34509]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:22.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.944+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f60c8062f30 con 0x7f60c81015b0 2024-05-09T08:24:22.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.946+0000 7f60c5ffb700 1 -- 172.21.15.28:0/1854537457 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f60bc09bb90 con 0x7f60c81015b0 2024-05-09T08:24:22.947 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-05-09T08:24:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQA2iDxm9MpgOBAA6/Kg9DGCTCw3EG/eziNJrw== 2024-05-09T08:24:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f60b80a2d60 msgr2=0x7f60b80a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f60b80a2d60 0x7f60b80a5220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f60c8101580 tx=0x7f60b4005eb0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c81015b0 msgr2=0x7f60c8101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c81015b0 0x7f60c8101000 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00e9a0 tx=0x7f60bc00ecb0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 shutdown_connections 2024-05-09T08:24:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f60b80a2d60 0x7f60b80a5220 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c81029e0 0x7f60c819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c81015b0 0x7f60c8101000 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 --2- 172.21.15.28:0/1854537457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f60c8101f70 0x7f60c819d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 >> 172.21.15.28:0/1854537457 conn(0x7f60c80fc080 msgr2=0x7f60c81057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 shutdown_connections 2024-05-09T08:24:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:22.949+0000 7f60cede6700 1 -- 172.21.15.28:0/1854537457 wait complete. 2024-05-09T08:24:23.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:23 smithi028 ceph-mon[30119]: pgmap v305: 129 pgs: 129 active+clean; 626 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.4 KiB/s wr, 13 op/s 2024-05-09T08:24:23.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:23 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1854537457' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T08:24:23.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:23 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1854537457' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T08:24:23.663 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:24:23.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-09T08:24:23.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-09T08:24:23.708 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-09T08:24:23.708 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-09T08:24:23.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph mgr dump --format=json 2024-05-09T08:24:23.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:23 smithi165 ceph-mon[34540]: pgmap v305: 129 pgs: 129 active+clean; 626 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.4 KiB/s wr, 13 op/s 2024-05-09T08:24:23.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:23 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1854537457' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T08:24:23.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:23 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1854537457' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T08:24:23.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:23 smithi184 ceph-mon[34509]: pgmap v305: 129 pgs: 129 active+clean; 626 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.4 KiB/s wr, 13 op/s 2024-05-09T08:24:23.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:23 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1854537457' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T08:24:23.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:23 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1854537457' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T08:24:24.005 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:25.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:25 smithi028 ceph-mon[30119]: pgmap v306: 129 pgs: 129 active+clean; 628 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-05-09T08:24:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.790+0000 7fd541f52700 1 -- 172.21.15.28:0/129026313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c1083a0 msgr2=0x7fd53c108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.790+0000 7fd541f52700 1 --2- 172.21.15.28:0/129026313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c1083a0 0x7fd53c108780 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0099c0 tx=0x7fd52c009cd0 comp rx=0 tx=0).stop 2024-05-09T08:24:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.790+0000 7fd541f52700 1 -- 172.21.15.28:0/129026313 shutdown_connections 2024-05-09T08:24:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.790+0000 7fd541f52700 1 --2- 172.21.15.28:0/129026313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd53c06b3f0 0x7fd53c10f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.790+0000 7fd541f52700 1 --2- 172.21.15.28:0/129026313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c1083a0 0x7fd53c108780 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.790+0000 7fd541f52700 1 --2- 172.21.15.28:0/129026313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd53c06aa50 0x7fd53c06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.790+0000 7fd541f52700 1 -- 172.21.15.28:0/129026313 >> 172.21.15.28:0/129026313 conn(0x7fd53c0781c0 msgr2=0x7fd53c0785d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.790+0000 7fd541f52700 1 -- 172.21.15.28:0/129026313 shutdown_connections 2024-05-09T08:24:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.790+0000 7fd541f52700 1 -- 172.21.15.28:0/129026313 wait complete. 2024-05-09T08:24:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd541f52700 1 Processor -- start 2024-05-09T08:24:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd541f52700 1 -- start start 2024-05-09T08:24:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd541f52700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd53c06aa50 0x7fd53c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd541f52700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c06b3f0 0x7fd53c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd541f52700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd53c1083a0 0x7fd53c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd541f52700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c1125e0 con 0x7fd53c06b3f0 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd541f52700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c112460 con 0x7fd53c1083a0 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd541f52700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c112760 con 0x7fd53c06aa50 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd53affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c06b3f0 0x7fd53c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd53affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c06b3f0 0x7fd53c19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48598/0 (socket says 172.21.15.28:48598) 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd53affd700 1 -- 172.21.15.28:0/488612012 learned_addr learned my addr 172.21.15.28:0/488612012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd53bfff700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd53c1083a0 0x7fd53c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.791+0000 7fd53b7fe700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd53c06aa50 0x7fd53c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd53affd700 1 -- 172.21.15.28:0/488612012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd53c06aa50 msgr2=0x7fd53c19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd53affd700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd53c06aa50 0x7fd53c19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd53affd700 1 -- 172.21.15.28:0/488612012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd53c1083a0 msgr2=0x7fd53c198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd53affd700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd53c1083a0 0x7fd53c198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd53affd700 1 -- 172.21.15.28:0/488612012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53c198fe0 con 0x7fd53c06b3f0 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd53b7fe700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd53c06aa50 0x7fd53c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd53affd700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c06b3f0 0x7fd53c19eb70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd53000c9b0 tx=0x7fd53000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd538ff9700 1 -- 172.21.15.28:0/488612012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd530007b00 con 0x7fd53c06b3f0 2024-05-09T08:24:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd53c1992d0 con 0x7fd53c06b3f0 2024-05-09T08:24:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd538ff9700 1 -- 172.21.15.28:0/488612012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd530007ca0 con 0x7fd53c06b3f0 2024-05-09T08:24:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd538ff9700 1 -- 172.21.15.28:0/488612012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd530018730 con 0x7fd53c06b3f0 2024-05-09T08:24:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.792+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd53c1a5490 con 0x7fd53c06b3f0 2024-05-09T08:24:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.793+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd53c103a50 con 0x7fd53c06b3f0 2024-05-09T08:24:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.794+0000 7fd538ff9700 1 -- 172.21.15.28:0/488612012 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd5300188d0 con 0x7fd53c06b3f0 2024-05-09T08:24:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.795+0000 7fd538ff9700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5240a2cb0 0x7fd5240a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.795+0000 7fd53b7fe700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5240a2cb0 0x7fd5240a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.795+0000 7fd538ff9700 1 -- 172.21.15.28:0/488612012 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd5300cf420 con 0x7fd53c06b3f0 2024-05-09T08:24:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.796+0000 7fd53b7fe700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5240a2cb0 0x7fd5240a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd52c005b40 tx=0x7fd52c005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:25.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:25.800+0000 7fd538ff9700 1 -- 172.21.15.28:0/488612012 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd530014030 con 0x7fd53c06b3f0 2024-05-09T08:24:25.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:25 smithi165 ceph-mon[34540]: pgmap v306: 129 pgs: 129 active+clean; 628 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-05-09T08:24:25.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:25 smithi184 ceph-mon[34509]: pgmap v306: 129 pgs: 129 active+clean; 628 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-05-09T08:24:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.066+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd53c199ce0 con 0x7fd53c06b3f0 2024-05-09T08:24:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.071+0000 7fd538ff9700 1 -- 172.21.15.28:0/488612012 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263197 (secure 0 0 0) 0x7fd53009bfe0 con 0x7fd53c06b3f0 2024-05-09T08:24:26.075 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5240a2cb0 msgr2=0x7fd5240a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5240a2cb0 0x7fd5240a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd52c005b40 tx=0x7fd52c005ab0 comp rx=0 tx=0).stop 2024-05-09T08:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c06b3f0 msgr2=0x7fd53c19eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c06b3f0 0x7fd53c19eb70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd53000c9b0 tx=0x7fd53000ccc0 comp rx=0 tx=0).stop 2024-05-09T08:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 shutdown_connections 2024-05-09T08:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd5240a2cb0 0x7fd5240a5170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd53c06aa50 0x7fd53c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd53c1083a0 0x7fd53c198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 --2- 172.21.15.28:0/488612012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd53c06b3f0 0x7fd53c19eb70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:26.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 >> 172.21.15.28:0/488612012 conn(0x7fd53c0781c0 msgr2=0x7fd53c0ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:26.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 shutdown_connections 2024-05-09T08:24:26.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:26.074+0000 7fd541f52700 1 -- 172.21.15.28:0/488612012 wait complete. 2024-05-09T08:24:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:26 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/488612012' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T08:24:26.846 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":953899001},{"type":"v1","addr":"172.21.15.28:6801","nonce":953899001}]},"active_addr":"172.21.15.28:6801/953899001","active_change":"2024-05-09T08:16:49.552788+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3330063293}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":273816117}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3745037139}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2130261041}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2247538520}]}]} 2024-05-09T08:24:26.852 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-09T08:24:26.852 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-09T08:24:26.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T08:24:26.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:26 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/488612012' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T08:24:26.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:26 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/488612012' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T08:24:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:27.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:27 smithi028 ceph-mon[30119]: pgmap v307: 129 pgs: 129 active+clean; 628 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-05-09T08:24:27.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:27 smithi165 ceph-mon[34540]: pgmap v307: 129 pgs: 129 active+clean; 628 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-05-09T08:24:27.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:27 smithi184 ceph-mon[34509]: pgmap v307: 129 pgs: 129 active+clean; 628 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-05-09T08:24:28.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.903+0000 7f1151ef1700 1 -- 172.21.15.28:0/3442700393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c107680 msgr2=0x7f114c10f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:28.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.903+0000 7f1151ef1700 1 --2- 172.21.15.28:0/3442700393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c107680 0x7f114c10f250 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f114400b0d0 tx=0x7f114400b3e0 comp rx=0 tx=0).stop 2024-05-09T08:24:28.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.903+0000 7f1151ef1700 1 -- 172.21.15.28:0/3442700393 shutdown_connections 2024-05-09T08:24:28.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.903+0000 7f1151ef1700 1 --2- 172.21.15.28:0/3442700393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c107680 0x7f114c10f250 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:28.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.903+0000 7f1151ef1700 1 --2- 172.21.15.28:0/3442700393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f114c06ad30 0x7f114c06b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:28.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.903+0000 7f1151ef1700 1 --2- 172.21.15.28:0/3442700393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114c06b6e0 0x7f114c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:28.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.903+0000 7f1151ef1700 1 -- 172.21.15.28:0/3442700393 >> 172.21.15.28:0/3442700393 conn(0x7f114c0782c0 msgr2=0x7f114c0786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:28.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.903+0000 7f1151ef1700 1 -- 172.21.15.28:0/3442700393 shutdown_connections 2024-05-09T08:24:28.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.903+0000 7f1151ef1700 1 -- 172.21.15.28:0/3442700393 wait complete. 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f1151ef1700 1 Processor -- start 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f1151ef1700 1 -- start start 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f1151ef1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c06ad30 0x7f114c101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f1151ef1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f114c06b6e0 0x7f114c1017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f1151ef1700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114c107680 0x7f114c101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f1151ef1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f114c112680 con 0x7f114c06ad30 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f1151ef1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f114c112500 con 0x7f114c06b6e0 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f1151ef1700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f114c112800 con 0x7f114c107680 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f114b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c06ad30 0x7f114c101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:28.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f114b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c06ad30 0x7f114c101290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48624/0 (socket says 172.21.15.28:48624) 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f114b7fe700 1 -- 172.21.15.28:0/4052363104 learned_addr learned my addr 172.21.15.28:0/4052363104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.904+0000 7f114bfff700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114c107680 0x7f114c101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f114affd700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f114c06b6e0 0x7f114c1017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f114b7fe700 1 -- 172.21.15.28:0/4052363104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114c107680 msgr2=0x7f114c101d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f114b7fe700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114c107680 0x7f114c101d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f114b7fe700 1 -- 172.21.15.28:0/4052363104 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f114c06b6e0 msgr2=0x7f114c1017d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f114b7fe700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f114c06b6e0 0x7f114c1017d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f114b7fe700 1 -- 172.21.15.28:0/4052363104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114c1a6fe0 con 0x7f114c06ad30 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f114b7fe700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c06ad30 0x7f114c101290 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f113c00d950 tx=0x7f113c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f1148ff9700 1 -- 172.21.15.28:0/4052363104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c016070 con 0x7f114c06ad30 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f1148ff9700 1 -- 172.21.15.28:0/4052363104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f113c009e20 con 0x7f114c06ad30 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f1148ff9700 1 -- 172.21.15.28:0/4052363104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c00f620 con 0x7f114c06ad30 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f114c1a72d0 con 0x7f114c06ad30 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.905+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f114c1a7860 con 0x7f114c06ad30 2024-05-09T08:24:28.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.906+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f114c069010 con 0x7f114c06ad30 2024-05-09T08:24:28.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.907+0000 7f1148ff9700 1 -- 172.21.15.28:0/4052363104 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f113c019070 con 0x7f114c06ad30 2024-05-09T08:24:28.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.908+0000 7f1148ff9700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11340a2d30 0x7f11340a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:28.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.908+0000 7f114affd700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11340a2d30 0x7f11340a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:28.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.908+0000 7f1148ff9700 1 -- 172.21.15.28:0/4052363104 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f113c0cefd0 con 0x7f114c06ad30 2024-05-09T08:24:28.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.909+0000 7f114affd700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11340a2d30 0x7f11340a51f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1140005950 tx=0x7f11400058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:28.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:28.911+0000 7f1148ff9700 1 -- 172.21.15.28:0/4052363104 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f113c09bb30 con 0x7f114c06ad30 2024-05-09T08:24:29.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.120+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f114c02db80 con 0x7f114c06ad30 2024-05-09T08:24:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.121+0000 7f1148ff9700 1 -- 172.21.15.28:0/4052363104 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7f113c09b430 con 0x7f114c06ad30 2024-05-09T08:24:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:29.123 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:24:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11340a2d30 msgr2=0x7f11340a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11340a2d30 0x7f11340a51f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1140005950 tx=0x7f11400058e0 comp rx=0 tx=0).stop 2024-05-09T08:24:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c06ad30 msgr2=0x7f114c101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c06ad30 0x7f114c101290 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f113c00d950 tx=0x7f113c00dc60 comp rx=0 tx=0).stop 2024-05-09T08:24:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 shutdown_connections 2024-05-09T08:24:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11340a2d30 0x7f11340a51f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114c107680 0x7f114c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114c06ad30 0x7f114c101290 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 --2- 172.21.15.28:0/4052363104 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f114c06b6e0 0x7f114c1017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:29.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 >> 172.21.15.28:0/4052363104 conn(0x7f114c0782c0 msgr2=0x7f114c0ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:29.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 shutdown_connections 2024-05-09T08:24:29.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:29.124+0000 7f1151ef1700 1 -- 172.21.15.28:0/4052363104 wait complete. 2024-05-09T08:24:29.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:29 smithi028 ceph-mon[30119]: pgmap v308: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.7 KiB/s wr, 23 op/s 2024-05-09T08:24:29.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:29 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4052363104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:29.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:29 smithi165 ceph-mon[34540]: pgmap v308: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.7 KiB/s wr, 23 op/s 2024-05-09T08:24:29.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:29 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4052363104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:29.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:29 smithi184 ceph-mon[34509]: pgmap v308: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.7 KiB/s wr, 23 op/s 2024-05-09T08:24:29.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:29 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4052363104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:29.943 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-09T08:24:29.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T08:24:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:31.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:31 smithi028 ceph-mon[30119]: pgmap v309: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.0 KiB/s wr, 22 op/s 2024-05-09T08:24:31.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:31 smithi165 ceph-mon[34540]: pgmap v309: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.0 KiB/s wr, 22 op/s 2024-05-09T08:24:31.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:31 smithi184 ceph-mon[34509]: pgmap v309: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.0 KiB/s wr, 22 op/s 2024-05-09T08:24:31.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.968+0000 7efcb1aac700 1 -- 172.21.15.28:0/3654814027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac107660 msgr2=0x7efcac10f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:31.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.968+0000 7efcb1aac700 1 --2- 172.21.15.28:0/3654814027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac107660 0x7efcac10f230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efca400b0a0 tx=0x7efca400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.968+0000 7efcb1aac700 1 -- 172.21.15.28:0/3654814027 shutdown_connections 2024-05-09T08:24:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.968+0000 7efcb1aac700 1 --2- 172.21.15.28:0/3654814027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac107660 0x7efcac10f230 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.968+0000 7efcb1aac700 1 --2- 172.21.15.28:0/3654814027 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac1025d0 0x7efcac1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.968+0000 7efcb1aac700 1 --2- 172.21.15.28:0/3654814027 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcac102f80 0x7efcac106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.968+0000 7efcb1aac700 1 -- 172.21.15.28:0/3654814027 >> 172.21.15.28:0/3654814027 conn(0x7efcac079480 msgr2=0x7efcac079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.968+0000 7efcb1aac700 1 -- 172.21.15.28:0/3654814027 shutdown_connections 2024-05-09T08:24:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.968+0000 7efcb1aac700 1 -- 172.21.15.28:0/3654814027 wait complete. 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcb1aac700 1 Processor -- start 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcb1aac700 1 -- start start 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcb1aac700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac1025d0 0x7efcac19e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcb1aac700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac102f80 0x7efcac19ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcb1aac700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcac107660 0x7efcac198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcb1aac700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcac112690 con 0x7efcac1025d0 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcb1aac700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efcac112510 con 0x7efcac102f80 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcb1aac700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efcac112810 con 0x7efcac107660 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcab7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac1025d0 0x7efcac19e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcab7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac1025d0 0x7efcac19e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48640/0 (socket says 172.21.15.28:48640) 2024-05-09T08:24:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efcab7fe700 1 -- 172.21.15.28:0/2935605862 learned_addr learned my addr 172.21.15.28:0/2935605862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.969+0000 7efca3fff700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac102f80 0x7efcac19ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcabfff700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcac107660 0x7efcac198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcab7fe700 1 -- 172.21.15.28:0/2935605862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcac107660 msgr2=0x7efcac198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcab7fe700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcac107660 0x7efcac198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcab7fe700 1 -- 172.21.15.28:0/2935605862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac102f80 msgr2=0x7efcac19ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcab7fe700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac102f80 0x7efcac19ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcab7fe700 1 -- 172.21.15.28:0/2935605862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcac199070 con 0x7efcac1025d0 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcabfff700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcac107660 0x7efcac198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcab7fe700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac1025d0 0x7efcac19e750 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00d950 tx=0x7efc9c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efca97fa700 1 -- 172.21.15.28:0/2935605862 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c016070 con 0x7efcac1025d0 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcac199360 con 0x7efcac1025d0 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efca97fa700 1 -- 172.21.15.28:0/2935605862 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc9c009e20 con 0x7efcac1025d0 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcac1a5620 con 0x7efcac1025d0 2024-05-09T08:24:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.970+0000 7efca97fa700 1 -- 172.21.15.28:0/2935605862 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c00f620 con 0x7efcac1025d0 2024-05-09T08:24:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.971+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc8c005330 con 0x7efcac1025d0 2024-05-09T08:24:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.972+0000 7efca97fa700 1 -- 172.21.15.28:0/2935605862 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7efc9c019070 con 0x7efcac1025d0 2024-05-09T08:24:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.973+0000 7efca97fa700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efc980a2cb0 0x7efc980a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.973+0000 7efca3fff700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efc980a2cb0 0x7efc980a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.973+0000 7efca97fa700 1 -- 172.21.15.28:0/2935605862 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efc9c0cf310 con 0x7efcac1025d0 2024-05-09T08:24:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.974+0000 7efca3fff700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efc980a2cb0 0x7efc980a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efcac108ac0 tx=0x7efc9400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:31.978+0000 7efca97fa700 1 -- 172.21.15.28:0/2935605862 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efc9c09bed0 con 0x7efcac1025d0 2024-05-09T08:24:32.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.182+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc8c0058f0 con 0x7efcac1025d0 2024-05-09T08:24:32.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.184+0000 7efca97fa700 1 -- 172.21.15.28:0/2935605862 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24140 (secure 0 0 0) 0x7efc9c09b7d0 con 0x7efcac1025d0 2024-05-09T08:24:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:24:32.185 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:23:56.027023+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:24:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efc980a2cb0 msgr2=0x7efc980a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efc980a2cb0 0x7efc980a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efcac108ac0 tx=0x7efc9400b3c0 comp rx=0 tx=0).stop 2024-05-09T08:24:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac1025d0 msgr2=0x7efcac19e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac1025d0 0x7efcac19e750 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00d950 tx=0x7efc9c00dc60 comp rx=0 tx=0).stop 2024-05-09T08:24:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 shutdown_connections 2024-05-09T08:24:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efc980a2cb0 0x7efc980a5170 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcac107660 0x7efcac198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcac1025d0 0x7efcac19e750 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 --2- 172.21.15.28:0/2935605862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac102f80 0x7efcac19ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 >> 172.21.15.28:0/2935605862 conn(0x7efcac079480 msgr2=0x7efcac1003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 shutdown_connections 2024-05-09T08:24:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:32.186+0000 7efcb1aac700 1 -- 172.21.15.28:0/2935605862 wait complete. 2024-05-09T08:24:32.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:32 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2935605862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:32.759 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:32 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2935605862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:32.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:32 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2935605862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:24:32.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-09T08:24:32.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-09T08:24:32.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-09T08:24:32.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-09T08:24:32.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-09T08:24:32.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-09T08:24:32.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-09T08:24:32.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-09T08:24:32.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-09T08:24:32.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-09T08:24:32.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-09T08:24:32.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-09T08:24:33.485 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:33.718 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:33 smithi028 ceph-mon[30119]: pgmap v310: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 KiB/s wr, 20 op/s 2024-05-09T08:24:33.845 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:33.848 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:33.848 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:33.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:33 smithi165 ceph-mon[34540]: pgmap v310: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 KiB/s wr, 20 op/s 2024-05-09T08:24:33.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:33 smithi184 ceph-mon[34509]: pgmap v310: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 KiB/s wr, 20 op/s 2024-05-09T08:24:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:34.252 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:35.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:35 smithi028 ceph-mon[30119]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.4 KiB/s wr, 15 op/s 2024-05-09T08:24:35.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:35 smithi165 ceph-mon[34540]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.4 KiB/s wr, 15 op/s 2024-05-09T08:24:35.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:35 smithi184 ceph-mon[34509]: pgmap v311: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.4 KiB/s wr, 15 op/s 2024-05-09T08:24:37.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:37 smithi028 ceph-mon[30119]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-05-09T08:24:37.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:37 smithi165 ceph-mon[34540]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-05-09T08:24:37.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:37 smithi184 ceph-mon[34509]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-05-09T08:24:39.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:39 smithi028 ceph-mon[30119]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-05-09T08:24:39.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:39 smithi165 ceph-mon[34540]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-05-09T08:24:39.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:39 smithi184 ceph-mon[34509]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-05-09T08:24:41.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:41 smithi028 ceph-mon[30119]: pgmap v314: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 9 op/s 2024-05-09T08:24:41.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:41 smithi165 ceph-mon[34540]: pgmap v314: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 9 op/s 2024-05-09T08:24:41.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:41 smithi184 ceph-mon[34509]: pgmap v314: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 9 op/s 2024-05-09T08:24:43.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:43 smithi028 ceph-mon[30119]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-05-09T08:24:43.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:43 smithi165 ceph-mon[34540]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-05-09T08:24:43.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:43 smithi184 ceph-mon[34509]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-05-09T08:24:44.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.484+0000 7f824759e700 1 -- 172.21.15.28:0/4182692893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248101420 msgr2=0x7f8248101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.484+0000 7f824759e700 1 --2- 172.21.15.28:0/4182692893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248101420 0x7f8248101800 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f82380099c0 tx=0x7f8238009cd0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.484+0000 7f824759e700 1 -- 172.21.15.28:0/4182692893 shutdown_connections 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.484+0000 7f824759e700 1 --2- 172.21.15.28:0/4182692893 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8248106500 0x7f824810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.484+0000 7f824759e700 1 --2- 172.21.15.28:0/4182692893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248101420 0x7f8248101800 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.484+0000 7f824759e700 1 --2- 172.21.15.28:0/4182692893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8248101dd0 0x7f8248105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.484+0000 7f824759e700 1 -- 172.21.15.28:0/4182692893 >> 172.21.15.28:0/4182692893 conn(0x7f82480fccd0 msgr2=0x7f82480ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.484+0000 7f824759e700 1 -- 172.21.15.28:0/4182692893 shutdown_connections 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.484+0000 7f824759e700 1 -- 172.21.15.28:0/4182692893 wait complete. 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.485+0000 7f824759e700 1 Processor -- start 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.485+0000 7f824759e700 1 -- start start 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f824759e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8248101420 0x7f824819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f824759e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248101dd0 0x7f824819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f824759e700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8248106500 0x7f8248198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f824759e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82481114e0 con 0x7f8248101dd0 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f824759e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8248111360 con 0x7f8248101420 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f824759e700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8248111660 con 0x7f8248106500 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8245d9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248101dd0 0x7f824819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8245d9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248101dd0 0x7f824819ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46762/0 (socket says 172.21.15.28:46762) 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8245d9b700 1 -- 172.21.15.28:0/1699356808 learned_addr learned my addr 172.21.15.28:0/1699356808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f824659c700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8248101420 0x7f824819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8246d9d700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8248106500 0x7f8248198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8245d9b700 1 -- 172.21.15.28:0/1699356808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8248106500 msgr2=0x7f8248198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8245d9b700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8248106500 0x7f8248198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8245d9b700 1 -- 172.21.15.28:0/1699356808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8248101420 msgr2=0x7f824819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8245d9b700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8248101420 0x7f824819e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8245d9b700 1 -- 172.21.15.28:0/1699356808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8248199020 con 0x7f8248101dd0 2024-05-09T08:24:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f824659c700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8248101420 0x7f824819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f8245d9b700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248101dd0 0x7f824819ec00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f823c00c9e0 tx=0x7f823c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.486+0000 7f82377fe700 1 -- 172.21.15.28:0/1699356808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c007aa0 con 0x7f8248101dd0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.487+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8248199310 con 0x7f8248101dd0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.487+0000 7f82377fe700 1 -- 172.21.15.28:0/1699356808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f823c007c40 con 0x7f8248101dd0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.487+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82481a5520 con 0x7f8248101dd0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.487+0000 7f82377fe700 1 -- 172.21.15.28:0/1699356808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c004a30 con 0x7f8248101dd0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.487+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8248062a30 con 0x7f8248101dd0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.489+0000 7f82377fe700 1 -- 172.21.15.28:0/1699356808 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f823c00f450 con 0x7f8248101dd0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.489+0000 7f82377fe700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f82300a2cb0 0x7f82300a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.489+0000 7f824659c700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f82300a2cb0 0x7f82300a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.490+0000 7f82377fe700 1 -- 172.21.15.28:0/1699356808 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f823c0cf640 con 0x7f8248101dd0 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.490+0000 7f82377fe700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] conn(0x7f82300ad970 0x7f82300afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.490+0000 7f82377fe700 1 -- 172.21.15.28:0/1699356808 --> [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f823c023e50 con 0x7f82300ad970 2024-05-09T08:24:44.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.490+0000 7f82377fe700 1 -- 172.21.15.28:0/1699356808 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f823c0cfa90 con 0x7f8248101dd0 2024-05-09T08:24:44.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.490+0000 7f824659c700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f82300a2cb0 0x7f82300a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8238009fd0 tx=0x7f8238005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:44.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.490+0000 7f8246d9d700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] conn(0x7f82300ad970 0x7f82300afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:44.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.490+0000 7f8246d9d700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] conn(0x7f82300ad970 0x7f82300afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:44.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.491+0000 7f82377fe700 1 -- 172.21.15.28:0/1699356808 <== osd.7 v2:172.21.15.165:6816/1526551047 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f823c023e50 con 0x7f82300ad970 2024-05-09T08:24:44.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.510+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 --> [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8248062e30 con 0x7f82300ad970 2024-05-09T08:24:44.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f82377fe700 1 -- 172.21.15.28:0/1699356808 <== osd.7 v2:172.21.15.165:6816/1526551047 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8248062e30 con 0x7f82300ad970 2024-05-09T08:24:44.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 >> [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] conn(0x7f82300ad970 msgr2=0x7f82300afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] conn(0x7f82300ad970 0x7f82300afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f82300a2cb0 msgr2=0x7f82300a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f82300a2cb0 0x7f82300a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8238009fd0 tx=0x7f8238005fb0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248101dd0 msgr2=0x7f824819ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 --2- 172.21.15.28:0/1699356808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8248101dd0 0x7f824819ec00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f823c00c9e0 tx=0x7f823c00ccf0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f8246d9d700 1 -- 172.21.15.28:0/1699356808 reap_dead start 2024-05-09T08:24:44.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 shutdown_connections 2024-05-09T08:24:44.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 >> 172.21.15.28:0/1699356808 conn(0x7f82480fccd0 msgr2=0x7f824806ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:44.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 shutdown_connections 2024-05-09T08:24:44.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:44.511+0000 7f824759e700 1 -- 172.21.15.28:0/1699356808 wait complete. 2024-05-09T08:24:44.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.299+0000 7f5a221f0700 1 -- 172.21.15.28:0/2917944340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c107690 msgr2=0x7f5a1c10f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.299+0000 7f5a221f0700 1 --2- 172.21.15.28:0/2917944340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c107690 0x7f5a1c10f260 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400b0a0 tx=0x7f5a1400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.299+0000 7f5a221f0700 1 -- 172.21.15.28:0/2917944340 shutdown_connections 2024-05-09T08:24:44.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.299+0000 7f5a221f0700 1 --2- 172.21.15.28:0/2917944340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c1025d0 0x7f5a1c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.299+0000 7f5a221f0700 1 --2- 172.21.15.28:0/2917944340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c107690 0x7f5a1c10f260 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.299+0000 7f5a221f0700 1 --2- 172.21.15.28:0/2917944340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a1c102f80 0x7f5a1c106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.299+0000 7f5a221f0700 1 -- 172.21.15.28:0/2917944340 >> 172.21.15.28:0/2917944340 conn(0x7f5a1c0794d0 msgr2=0x7f5a1c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:44.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.299+0000 7f5a221f0700 1 -- 172.21.15.28:0/2917944340 shutdown_connections 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.299+0000 7f5a221f0700 1 -- 172.21.15.28:0/2917944340 wait complete. 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.300+0000 7f5a221f0700 1 Processor -- start 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.300+0000 7f5a221f0700 1 -- start start 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.300+0000 7f5a221f0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c1025d0 0x7f5a1c19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.300+0000 7f5a221f0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a1c102f80 0x7f5a1c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.300+0000 7f5a221f0700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c107690 0x7f5a1c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.300+0000 7f5a221f0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1c112670 con 0x7f5a1c1025d0 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.300+0000 7f5a221f0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1c1124f0 con 0x7f5a1c102f80 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.300+0000 7f5a221f0700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1c1127f0 con 0x7f5a1c107690 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c1025d0 0x7f5a1c19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c1025d0 0x7f5a1c19e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46730/0 (socket says 172.21.15.28:46730) 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1b7fe700 1 -- 172.21.15.28:0/1420910343 learned_addr learned my addr 172.21.15.28:0/1420910343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:44.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1bfff700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c107690 0x7f5a1c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1b7fe700 1 -- 172.21.15.28:0/1420910343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c107690 msgr2=0x7f5a1c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1b7fe700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a1c107690 0x7f5a1c198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1b7fe700 1 -- 172.21.15.28:0/1420910343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a1c102f80 msgr2=0x7f5a1c19ebf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1b7fe700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a1c102f80 0x7f5a1c19ebf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1b7fe700 1 -- 172.21.15.28:0/1420910343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1c198fd0 con 0x7f5a1c1025d0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.301+0000 7f5a1b7fe700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c1025d0 0x7f5a1c19e6b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00d980 tx=0x7f5a0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.302+0000 7f5a197fa700 1 -- 172.21.15.28:0/1420910343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c017070 con 0x7f5a1c1025d0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.302+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1c1992c0 con 0x7f5a1c1025d0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.302+0000 7f5a197fa700 1 -- 172.21.15.28:0/1420910343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a0c005020 con 0x7f5a1c1025d0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.302+0000 7f5a197fa700 1 -- 172.21.15.28:0/1420910343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c009bc0 con 0x7f5a1c1025d0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.302+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a1c1a5580 con 0x7f5a1c1025d0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.303+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f59fc001030 con 0x7f5a1c1025d0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.304+0000 7f5a197fa700 1 -- 172.21.15.28:0/1420910343 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5a0c004a30 con 0x7f5a1c1025d0 2024-05-09T08:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.305+0000 7f5a197fa700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a080a2d30 0x7f5a080a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.305+0000 7f5a12ffd700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a080a2d30 0x7f5a080a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:44.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.305+0000 7f5a197fa700 1 -- 172.21.15.28:0/1420910343 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5a0c0cf830 con 0x7f5a1c1025d0 2024-05-09T08:24:44.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.305+0000 7f5a197fa700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] conn(0x7f5a080ad9f0 0x7f5a080afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.305+0000 7f5a197fa700 1 -- 172.21.15.28:0/1420910343 --> [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a080b0500 con 0x7f5a080ad9f0 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.305+0000 7f5a197fa700 1 -- 172.21.15.28:0/1420910343 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f5a0c0cfc80 con 0x7f5a1c1025d0 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.305+0000 7f5a12ffd700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a080a2d30 0x7f5a080a51f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c108ae0 tx=0x7f5a0400b370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.306+0000 7f5a1bfff700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] conn(0x7f5a080ad9f0 0x7f5a080afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.306+0000 7f5a1bfff700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] conn(0x7f5a080ad9f0 0x7f5a080afe30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.307+0000 7f5a197fa700 1 -- 172.21.15.28:0/1420910343 <== osd.2 v2:172.21.15.184:6800/3751267842 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a080b0500 con 0x7f5a080ad9f0 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.324+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 --> [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f59fc002de0 con 0x7f5a080ad9f0 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.324+0000 7f5a197fa700 1 -- 172.21.15.28:0/1420910343 <== osd.2 v2:172.21.15.184:6800/3751267842 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f59fc002de0 con 0x7f5a080ad9f0 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 >> [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] conn(0x7f5a080ad9f0 msgr2=0x7f5a080afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] conn(0x7f5a080ad9f0 0x7f5a080afe30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a080a2d30 msgr2=0x7f5a080a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a080a2d30 0x7f5a080a51f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c108ae0 tx=0x7f5a0400b370 comp rx=0 tx=0).stop 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c1025d0 msgr2=0x7f5a1c19e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 --2- 172.21.15.28:0/1420910343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a1c1025d0 0x7f5a1c19e6b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00d980 tx=0x7f5a0c00dc90 comp rx=0 tx=0).stop 2024-05-09T08:24:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a1bfff700 1 -- 172.21.15.28:0/1420910343 reap_dead start 2024-05-09T08:24:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 shutdown_connections 2024-05-09T08:24:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 >> 172.21.15.28:0/1420910343 conn(0x7f5a1c0794d0 msgr2=0x7f5a1c100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 shutdown_connections 2024-05-09T08:24:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:43.325+0000 7f5a221f0700 1 -- 172.21.15.28:0/1420910343 wait complete. 2024-05-09T08:24:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:45.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:45 smithi165 ceph-mon[34540]: pgmap v316: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-05-09T08:24:45.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:45 smithi184 ceph-mon[34509]: pgmap v316: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-05-09T08:24:45.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:45 smithi028 ceph-mon[30119]: pgmap v316: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-05-09T08:24:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.781+0000 7f5c09d4b700 1 -- 172.21.15.28:0/1655029147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c04107280 msgr2=0x7f5c04107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.781+0000 7f5c09d4b700 1 --2- 172.21.15.28:0/1655029147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c04107280 0x7f5c04107660 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400b0a0 tx=0x7f5bf400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.781+0000 7f5c09d4b700 1 -- 172.21.15.28:0/1655029147 shutdown_connections 2024-05-09T08:24:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.781+0000 7f5c09d4b700 1 --2- 172.21.15.28:0/1655029147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c04101bd0 0x7f5c0410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.781+0000 7f5c09d4b700 1 --2- 172.21.15.28:0/1655029147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c04107280 0x7f5c04107660 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.781+0000 7f5c09d4b700 1 --2- 172.21.15.28:0/1655029147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c04101210 0x7f5c04101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.781+0000 7f5c09d4b700 1 -- 172.21.15.28:0/1655029147 >> 172.21.15.28:0/1655029147 conn(0x7f5c04078190 msgr2=0x7f5c040785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.781+0000 7f5c09d4b700 1 -- 172.21.15.28:0/1655029147 shutdown_connections 2024-05-09T08:24:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.781+0000 7f5c09d4b700 1 -- 172.21.15.28:0/1655029147 wait complete. 2024-05-09T08:24:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.782+0000 7f5c09d4b700 1 Processor -- start 2024-05-09T08:24:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.782+0000 7f5c09d4b700 1 -- start start 2024-05-09T08:24:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.782+0000 7f5c09d4b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c04101210 0x7f5c04102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.782+0000 7f5c09d4b700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c04101bd0 0x7f5c04103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.782+0000 7f5c09d4b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c04107280 0x7f5c04103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.782+0000 7f5c09d4b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041114f0 con 0x7f5c04107280 2024-05-09T08:24:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.782+0000 7f5c09d4b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c04111370 con 0x7f5c04101210 2024-05-09T08:24:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.782+0000 7f5c09d4b700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5c04111670 con 0x7f5c04101bd0 2024-05-09T08:24:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c03fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c04107280 0x7f5c04103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c03fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c04107280 0x7f5c04103970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33062/0 (socket says 172.21.15.28:33062) 2024-05-09T08:24:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c03fff700 1 -- 172.21.15.28:0/2754430183 learned_addr learned my addr 172.21.15.28:0/2754430183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c037fe700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c04101210 0x7f5c04102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c02ffd700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c04101bd0 0x7f5c04103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c03fff700 1 -- 172.21.15.28:0/2754430183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c04101bd0 msgr2=0x7f5c04103430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c03fff700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c04101bd0 0x7f5c04103430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c03fff700 1 -- 172.21.15.28:0/2754430183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c04101210 msgr2=0x7f5c04102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c03fff700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c04101210 0x7f5c04102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c03fff700 1 -- 172.21.15.28:0/2754430183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c041a6ff0 con 0x7f5c04107280 2024-05-09T08:24:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c03fff700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c04107280 0x7f5c04103970 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00d980 tx=0x7f5bfc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c00ff9700 1 -- 172.21.15.28:0/2754430183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc017070 con 0x7f5c04107280 2024-05-09T08:24:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.783+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c041a72e0 con 0x7f5c04107280 2024-05-09T08:24:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.784+0000 7f5c00ff9700 1 -- 172.21.15.28:0/2754430183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bfc005020 con 0x7f5c04107280 2024-05-09T08:24:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.784+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c041a7820 con 0x7f5c04107280 2024-05-09T08:24:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.784+0000 7f5c00ff9700 1 -- 172.21.15.28:0/2754430183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc00fb90 con 0x7f5c04107280 2024-05-09T08:24:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.784+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5c04105300 con 0x7f5c04107280 2024-05-09T08:24:46.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.786+0000 7f5c00ff9700 1 -- 172.21.15.28:0/2754430183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5bfc00fd30 con 0x7f5c04107280 2024-05-09T08:24:46.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.786+0000 7f5c00ff9700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5bec0a2cb0 0x7f5bec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.787+0000 7f5c037fe700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5bec0a2cb0 0x7f5bec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.787+0000 7f5c00ff9700 1 -- 172.21.15.28:0/2754430183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5bfc0cf750 con 0x7f5c04107280 2024-05-09T08:24:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.787+0000 7f5c00ff9700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] conn(0x7f5bec0ad970 0x7f5bec0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.787+0000 7f5c00ff9700 1 -- 172.21.15.28:0/2754430183 --> [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5bfc016ec0 con 0x7f5bec0ad970 2024-05-09T08:24:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.787+0000 7f5c00ff9700 1 -- 172.21.15.28:0/2754430183 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f5bfc0cfba0 con 0x7f5c04107280 2024-05-09T08:24:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.787+0000 7f5c037fe700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5bec0a2cb0 0x7f5bec0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009e60 tx=0x7f5bf400b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.787+0000 7f5c02ffd700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] conn(0x7f5bec0ad970 0x7f5bec0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.787+0000 7f5c02ffd700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] conn(0x7f5bec0ad970 0x7f5bec0afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.788+0000 7f5c00ff9700 1 -- 172.21.15.28:0/2754430183 <== osd.11 v2:172.21.15.184:6824/1865031750 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5bfc016ec0 con 0x7f5bec0ad970 2024-05-09T08:24:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.805+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 --> [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5c04102930 con 0x7f5bec0ad970 2024-05-09T08:24:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.805+0000 7f5c00ff9700 1 -- 172.21.15.28:0/2754430183 <== osd.11 v2:172.21.15.184:6824/1865031750 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5c04102930 con 0x7f5bec0ad970 2024-05-09T08:24:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 >> [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] conn(0x7f5bec0ad970 msgr2=0x7f5bec0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] conn(0x7f5bec0ad970 0x7f5bec0afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5bec0a2cb0 msgr2=0x7f5bec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5bec0a2cb0 0x7f5bec0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009e60 tx=0x7f5bf400b7b0 comp rx=0 tx=0).stop 2024-05-09T08:24:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c04107280 msgr2=0x7f5c04103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 --2- 172.21.15.28:0/2754430183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c04107280 0x7f5c04103970 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00d980 tx=0x7f5bfc00dc90 comp rx=0 tx=0).stop 2024-05-09T08:24:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c03fff700 1 -- 172.21.15.28:0/2754430183 reap_dead start 2024-05-09T08:24:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 shutdown_connections 2024-05-09T08:24:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 >> 172.21.15.28:0/2754430183 conn(0x7f5c04078190 msgr2=0x7f5c04100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 shutdown_connections 2024-05-09T08:24:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.806+0000 7f5c09d4b700 1 -- 172.21.15.28:0/2754430183 wait complete. 2024-05-09T08:24:47.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.991+0000 7fde738c5700 1 -- 172.21.15.28:0/4045990787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c101220 msgr2=0x7fde6c1016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.991+0000 7fde738c5700 1 --2- 172.21.15.28:0/4045990787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c101220 0x7fde6c1016a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009990 tx=0x7fde5c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.991+0000 7fde738c5700 1 -- 172.21.15.28:0/4045990787 shutdown_connections 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.991+0000 7fde738c5700 1 --2- 172.21.15.28:0/4045990787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6c107290 0x7fde6c107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.991+0000 7fde738c5700 1 --2- 172.21.15.28:0/4045990787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde6c101be0 0x7fde6c10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.991+0000 7fde738c5700 1 --2- 172.21.15.28:0/4045990787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c101220 0x7fde6c1016a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.991+0000 7fde738c5700 1 -- 172.21.15.28:0/4045990787 >> 172.21.15.28:0/4045990787 conn(0x7fde6c0781d0 msgr2=0x7fde6c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.991+0000 7fde738c5700 1 -- 172.21.15.28:0/4045990787 shutdown_connections 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.991+0000 7fde738c5700 1 -- 172.21.15.28:0/4045990787 wait complete. 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde738c5700 1 Processor -- start 2024-05-09T08:24:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde738c5700 1 -- start start 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde738c5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde6c101220 0x7fde6c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde738c5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c101be0 0x7fde6c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde738c5700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6c107290 0x7fde6c1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde738c5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c111520 con 0x7fde6c101be0 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde738c5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c1113a0 con 0x7fde6c101220 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde738c5700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c1116a0 con 0x7fde6c107290 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde70e60700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c101be0 0x7fde6c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde70e60700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c101be0 0x7fde6c19ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33092/0 (socket says 172.21.15.28:33092) 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde70e60700 1 -- 172.21.15.28:0/2023906900 learned_addr learned my addr 172.21.15.28:0/2023906900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.992+0000 7fde71661700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde6c101220 0x7fde6c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde71e62700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6c107290 0x7fde6c1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde70e60700 1 -- 172.21.15.28:0/2023906900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6c107290 msgr2=0x7fde6c1988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde70e60700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6c107290 0x7fde6c1988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde70e60700 1 -- 172.21.15.28:0/2023906900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde6c101220 msgr2=0x7fde6c19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde70e60700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde6c101220 0x7fde6c19e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde70e60700 1 -- 172.21.15.28:0/2023906900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6c199050 con 0x7fde6c101be0 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde70e60700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c101be0 0x7fde6c19ec70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0038b0 tx=0x7fde5c009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde627fc700 1 -- 172.21.15.28:0/2023906900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c027070 con 0x7fde6c101be0 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6c1992e0 con 0x7fde6c101be0 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde627fc700 1 -- 172.21.15.28:0/2023906900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde5c004050 con 0x7fde6c101be0 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde627fc700 1 -- 172.21.15.28:0/2023906900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c022cd0 con 0x7fde6c101be0 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.993+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde6c1a5600 con 0x7fde6c101be0 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.994+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fde6c105390 con 0x7fde6c101be0 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.995+0000 7fde627fc700 1 -- 172.21.15.28:0/2023906900 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fde5c02a070 con 0x7fde6c101be0 2024-05-09T08:24:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.996+0000 7fde627fc700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fde580a2cb0 0x7fde580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.996+0000 7fde627fc700 1 -- 172.21.15.28:0/2023906900 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fde5c0d0de0 con 0x7fde6c101be0 2024-05-09T08:24:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.996+0000 7fde71661700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fde580a2cb0 0x7fde580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.996+0000 7fde627fc700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] conn(0x7fde580ad970 0x7fde580afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.996+0000 7fde627fc700 1 -- 172.21.15.28:0/2023906900 --> [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde5c010ec0 con 0x7fde580ad970 2024-05-09T08:24:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.996+0000 7fde627fc700 1 -- 172.21.15.28:0/2023906900 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fde5c0d1190 con 0x7fde6c101be0 2024-05-09T08:24:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.996+0000 7fde71e62700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] conn(0x7fde580ad970 0x7fde580afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.997+0000 7fde71661700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fde580a2cb0 0x7fde580a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fde68007fd0 tx=0x7fde6800c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.997+0000 7fde71e62700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] conn(0x7fde580ad970 0x7fde580afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:46.998+0000 7fde627fc700 1 -- 172.21.15.28:0/2023906900 <== osd.3 v2:172.21.15.28:6810/3526420147 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde5c010ec0 con 0x7fde580ad970 2024-05-09T08:24:47.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.014+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 --> [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fde6c102940 con 0x7fde580ad970 2024-05-09T08:24:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.015+0000 7fde627fc700 1 -- 172.21.15.28:0/2023906900 <== osd.3 v2:172.21.15.28:6810/3526420147 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fde6c102940 con 0x7fde580ad970 2024-05-09T08:24:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.015+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] conn(0x7fde580ad970 msgr2=0x7fde580afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.015+0000 7fde738c5700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] conn(0x7fde580ad970 0x7fde580afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.016+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fde580a2cb0 msgr2=0x7fde580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.016+0000 7fde738c5700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fde580a2cb0 0x7fde580a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fde68007fd0 tx=0x7fde6800c040 comp rx=0 tx=0).stop 2024-05-09T08:24:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.016+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c101be0 msgr2=0x7fde6c19ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.016+0000 7fde738c5700 1 --2- 172.21.15.28:0/2023906900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c101be0 0x7fde6c19ec70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0038b0 tx=0x7fde5c009f50 comp rx=0 tx=0).stop 2024-05-09T08:24:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.016+0000 7fde71e62700 1 -- 172.21.15.28:0/2023906900 reap_dead start 2024-05-09T08:24:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.016+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 shutdown_connections 2024-05-09T08:24:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.016+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 >> 172.21.15.28:0/2023906900 conn(0x7fde6c0781d0 msgr2=0x7fde6c1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:47.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.016+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 shutdown_connections 2024-05-09T08:24:47.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.016+0000 7fde738c5700 1 -- 172.21.15.28:0/2023906900 wait complete. 2024-05-09T08:24:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 -- 172.21.15.28:0/2087213375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d4101210 msgr2=0x7fb1d4101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 --2- 172.21.15.28:0/2087213375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d4101210 0x7fb1d4101690 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009990 tx=0x7fb1bc009ca0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 -- 172.21.15.28:0/2087213375 shutdown_connections 2024-05-09T08:24:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 --2- 172.21.15.28:0/2087213375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1d4107280 0x7fb1d4107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 --2- 172.21.15.28:0/2087213375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1d4101bd0 0x7fb1d410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 --2- 172.21.15.28:0/2087213375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d4101210 0x7fb1d4101690 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 -- 172.21.15.28:0/2087213375 >> 172.21.15.28:0/2087213375 conn(0x7fb1d4078190 msgr2=0x7fb1d40785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 -- 172.21.15.28:0/2087213375 shutdown_connections 2024-05-09T08:24:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 -- 172.21.15.28:0/2087213375 wait complete. 2024-05-09T08:24:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 Processor -- start 2024-05-09T08:24:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.607+0000 7fb1da9d4700 1 -- start start 2024-05-09T08:24:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1da9d4700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1d4101210 0x7fb1d4102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1da9d4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d4101bd0 0x7fb1d4103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1da9d4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1d4107280 0x7fb1d4103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1da9d4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d4111570 con 0x7fb1d4101bd0 2024-05-09T08:24:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1da9d4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d41113f0 con 0x7fb1d4107280 2024-05-09T08:24:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1da9d4700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d41116f0 con 0x7fb1d4101210 2024-05-09T08:24:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1cbfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d4101bd0 0x7fb1d4103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1cbfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d4101bd0 0x7fb1d4103430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33106/0 (socket says 172.21.15.28:33106) 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1cbfff700 1 -- 172.21.15.28:0/944552957 learned_addr learned my addr 172.21.15.28:0/944552957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1d8f71700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1d4107280 0x7fb1d4103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1d3fff700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1d4101210 0x7fb1d4102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1cbfff700 1 -- 172.21.15.28:0/944552957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1d4101210 msgr2=0x7fb1d4102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1cbfff700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1d4101210 0x7fb1d4102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1cbfff700 1 -- 172.21.15.28:0/944552957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1d4107280 msgr2=0x7fb1d4103970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1cbfff700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1d4107280 0x7fb1d4103970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1cbfff700 1 -- 172.21.15.28:0/944552957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d41a7000 con 0x7fb1d4101bd0 2024-05-09T08:24:47.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1cbfff700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d4101bd0 0x7fb1d4103430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc000c00 tx=0x7fb1bc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.608+0000 7fb1d1ffb700 1 -- 172.21.15.28:0/944552957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc027070 con 0x7fb1d4101bd0 2024-05-09T08:24:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.609+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d41a7290 con 0x7fb1d4101bd0 2024-05-09T08:24:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.609+0000 7fb1d1ffb700 1 -- 172.21.15.28:0/944552957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1bc003780 con 0x7fb1d4101bd0 2024-05-09T08:24:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.609+0000 7fb1d1ffb700 1 -- 172.21.15.28:0/944552957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc022dd0 con 0x7fb1d4101bd0 2024-05-09T08:24:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.609+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d41a7740 con 0x7fb1d4101bd0 2024-05-09T08:24:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.609+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb1d406ab40 con 0x7fb1d4101bd0 2024-05-09T08:24:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.611+0000 7fb1d1ffb700 1 -- 172.21.15.28:0/944552957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb1bc02a070 con 0x7fb1d4101bd0 2024-05-09T08:24:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.612+0000 7fb1d1ffb700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb1c00a2cb0 0x7fb1c00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.612+0000 7fb1d3fff700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb1c00a2cb0 0x7fb1c00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.612+0000 7fb1d1ffb700 1 -- 172.21.15.28:0/944552957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb1bc0d0740 con 0x7fb1d4101bd0 2024-05-09T08:24:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.612+0000 7fb1d1ffb700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] conn(0x7fb1c00ad970 0x7fb1c00afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.612+0000 7fb1d1ffb700 1 -- 172.21.15.28:0/944552957 --> [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1bc001d30 con 0x7fb1c00ad970 2024-05-09T08:24:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.612+0000 7fb1d1ffb700 1 -- 172.21.15.28:0/944552957 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fb1bc0d0af0 con 0x7fb1d4101bd0 2024-05-09T08:24:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.612+0000 7fb1d8f71700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] conn(0x7fb1c00ad970 0x7fb1c00afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.612+0000 7fb1d3fff700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb1c00a2cb0 0x7fb1c00a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4006fd0 tx=0x7fb1c4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.612+0000 7fb1d8f71700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] conn(0x7fb1c00ad970 0x7fb1c00afdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.613+0000 7fb1d1ffb700 1 -- 172.21.15.28:0/944552957 <== osd.0 v2:172.21.15.28:6802/1756971731 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb1bc001d30 con 0x7fb1c00ad970 2024-05-09T08:24:47.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 --> [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb1d4102930 con 0x7fb1c00ad970 2024-05-09T08:24:47.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1d1ffb700 1 -- 172.21.15.28:0/944552957 <== osd.0 v2:172.21.15.28:6802/1756971731 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb1d4102930 con 0x7fb1c00ad970 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] conn(0x7fb1c00ad970 msgr2=0x7fb1c00afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] conn(0x7fb1c00ad970 0x7fb1c00afdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb1c00a2cb0 msgr2=0x7fb1c00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb1c00a2cb0 0x7fb1c00a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4006fd0 tx=0x7fb1c4008040 comp rx=0 tx=0).stop 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d4101bd0 msgr2=0x7fb1d4103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 --2- 172.21.15.28:0/944552957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d4101bd0 0x7fb1d4103430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc000c00 tx=0x7fb1bc0047b0 comp rx=0 tx=0).stop 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1d8f71700 1 -- 172.21.15.28:0/944552957 reap_dead start 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 shutdown_connections 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 >> 172.21.15.28:0/944552957 conn(0x7fb1d4078190 msgr2=0x7fb1d4100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 shutdown_connections 2024-05-09T08:24:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:47.630+0000 7fb1da9d4700 1 -- 172.21.15.28:0/944552957 wait complete. 2024-05-09T08:24:47.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:47 smithi028 ceph-mon[30119]: pgmap v317: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 341 B/s wr, 8 op/s 2024-05-09T08:24:47.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:47 smithi165 ceph-mon[34540]: pgmap v317: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 341 B/s wr, 8 op/s 2024-05-09T08:24:47.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:47 smithi184 ceph-mon[34509]: pgmap v317: 129 pgs: 129 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 341 B/s wr, 8 op/s 2024-05-09T08:24:48.316 INFO:teuthology.orchestra.run.smithi028.stdout:34359738448 2024-05-09T08:24:48.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-09T08:24:49.004 INFO:teuthology.orchestra.run.smithi028.stdout:343597383705 2024-05-09T08:24:49.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-09T08:24:49.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:49 smithi028 ceph-mon[30119]: pgmap v318: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-05-09T08:24:49.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:49 smithi165 ceph-mon[34540]: pgmap v318: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-05-09T08:24:49.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:49 smithi184 ceph-mon[34509]: pgmap v318: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-05-09T08:24:50.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.118+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/2188748198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0101be0 msgr2=0x7f1cf010e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.118+0000 7f1cf5b3e700 1 --2- 172.21.15.28:0/2188748198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0101be0 0x7f1cf010e0d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800b0a0 tx=0x7f1ce800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.118+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/2188748198 shutdown_connections 2024-05-09T08:24:50.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.118+0000 7f1cf5b3e700 1 --2- 172.21.15.28:0/2188748198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf0107290 0x7f1cf0107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.118+0000 7f1cf5b3e700 1 --2- 172.21.15.28:0/2188748198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0101be0 0x7f1cf010e0d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.118+0000 7f1cf5b3e700 1 --2- 172.21.15.28:0/2188748198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1cf0101220 0x7f1cf01016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.118+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/2188748198 >> 172.21.15.28:0/2188748198 conn(0x7f1cf00781d0 msgr2=0x7f1cf00785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.118+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/2188748198 shutdown_connections 2024-05-09T08:24:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.119+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/2188748198 wait complete. 2024-05-09T08:24:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.119+0000 7f1cf5b3e700 1 Processor -- start 2024-05-09T08:24:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.119+0000 7f1cf5b3e700 1 -- start start 2024-05-09T08:24:50.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1cf5b3e700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf0101220 0x7f1cf0102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1cf5b3e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1cf0101be0 0x7f1cf0103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1cf5b3e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0107290 0x7f1cf0103980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1cf5b3e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf0111520 con 0x7f1cf0107290 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1cf5b3e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf01113a0 con 0x7f1cf0101be0 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1cf5b3e700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf01116a0 con 0x7f1cf0101220 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0107290 0x7f1cf0103980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0107290 0x7f1cf0103980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33120/0 (socket says 172.21.15.28:33120) 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceffff700 1 -- 172.21.15.28:0/1808992428 learned_addr learned my addr 172.21.15.28:0/1808992428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceeffd700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1cf0101be0 0x7f1cf0103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1cef7fe700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf0101220 0x7f1cf0102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceffff700 1 -- 172.21.15.28:0/1808992428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf0101220 msgr2=0x7f1cf0102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceffff700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1cf0101220 0x7f1cf0102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceffff700 1 -- 172.21.15.28:0/1808992428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1cf0101be0 msgr2=0x7f1cf0103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceffff700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1cf0101be0 0x7f1cf0103440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceffff700 1 -- 172.21.15.28:0/1808992428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf01a6fb0 con 0x7f1cf0107290 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceeffd700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1cf0101be0 0x7f1cf0103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.120+0000 7f1ceffff700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0107290 0x7f1cf0103980 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80062a0 tx=0x7f1ce8004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.121+0000 7f1cecff9700 1 -- 172.21.15.28:0/1808992428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8021070 con 0x7f1cf0107290 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.121+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf01a7240 con 0x7f1cf0107290 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.121+0000 7f1cecff9700 1 -- 172.21.15.28:0/1808992428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce8003c00 con 0x7f1cf0107290 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.121+0000 7f1cecff9700 1 -- 172.21.15.28:0/1808992428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8004880 con 0x7f1cf0107290 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.121+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf01a7740 con 0x7f1cf0107290 2024-05-09T08:24:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.122+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1cf0105310 con 0x7f1cf0107290 2024-05-09T08:24:50.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.123+0000 7f1cecff9700 1 -- 172.21.15.28:0/1808992428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1ce8003db0 con 0x7f1cf0107290 2024-05-09T08:24:50.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.124+0000 7f1cecff9700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1cd80a2cb0 0x7f1cd80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.124+0000 7f1cef7fe700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1cd80a2cb0 0x7f1cd80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.124+0000 7f1cecff9700 1 -- 172.21.15.28:0/1808992428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1ce800f020 con 0x7f1cf0107290 2024-05-09T08:24:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.124+0000 7f1cecff9700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] conn(0x7f1cd80ad970 0x7f1cd80afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.124+0000 7f1cecff9700 1 -- 172.21.15.28:0/1808992428 --> [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1cd80b0480 con 0x7f1cd80ad970 2024-05-09T08:24:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.124+0000 7f1cecff9700 1 -- 172.21.15.28:0/1808992428 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f1ce80d0f10 con 0x7f1cf0107290 2024-05-09T08:24:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.124+0000 7f1cef7fe700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1cd80a2cb0 0x7f1cd80a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00045b0 tx=0x7f1ce000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.124+0000 7f1ceeffd700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] conn(0x7f1cd80ad970 0x7f1cd80afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.125+0000 7f1ceeffd700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] conn(0x7f1cd80ad970 0x7f1cd80afdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.126+0000 7f1cecff9700 1 -- 172.21.15.28:0/1808992428 <== osd.5 v2:172.21.15.184:6808/2066099940 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cd80b0480 con 0x7f1cd80ad970 2024-05-09T08:24:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.145+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 --> [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1cf0102940 con 0x7f1cd80ad970 2024-05-09T08:24:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.146+0000 7f1cecff9700 1 -- 172.21.15.28:0/1808992428 <== osd.5 v2:172.21.15.184:6808/2066099940 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1cf0102940 con 0x7f1cd80ad970 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.146+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 >> [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] conn(0x7f1cd80ad970 msgr2=0x7f1cd80afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.146+0000 7f1cf5b3e700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] conn(0x7f1cd80ad970 0x7f1cd80afdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.146+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1cd80a2cb0 msgr2=0x7f1cd80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.146+0000 7f1cf5b3e700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1cd80a2cb0 0x7f1cd80a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00045b0 tx=0x7f1ce000b3c0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.146+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0107290 msgr2=0x7f1cf0103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.146+0000 7f1cf5b3e700 1 --2- 172.21.15.28:0/1808992428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0107290 0x7f1cf0103980 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80062a0 tx=0x7f1ce8004d70 comp rx=0 tx=0).stop 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.146+0000 7f1ceffff700 1 -- 172.21.15.28:0/1808992428 reap_dead start 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.147+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 shutdown_connections 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.147+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 >> 172.21.15.28:0/1808992428 conn(0x7f1cf00781d0 msgr2=0x7f1cf0100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.147+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 shutdown_connections 2024-05-09T08:24:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.147+0000 7f1cf5b3e700 1 -- 172.21.15.28:0/1808992428 wait complete. 2024-05-09T08:24:50.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-05-09T08:24:50.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-09T08:24:50.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-05-09T08:24:50.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-09T08:24:50.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-05-09T08:24:50.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-05-09T08:24:50.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-05-09T08:24:50.737 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-05-09T08:24:50.737 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-05-09T08:24:50.737 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-09T08:24:50.737 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:50 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:50.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-05-09T08:24:50.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-09T08:24:50.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-05-09T08:24:50.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-09T08:24:50.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-05-09T08:24:50.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-05-09T08:24:50.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-05-09T08:24:50.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-05-09T08:24:50.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-05-09T08:24:50.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-09T08:24:50.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:50 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:50.760 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-09T08:24:50.761 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:50 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 -- 172.21.15.28:0/1346675329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95841023a0 msgr2=0x7f9584102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 --2- 172.21.15.28:0/1346675329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95841023a0 0x7f9584102800 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f956c009990 tx=0x7f956c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 -- 172.21.15.28:0/1346675329 shutdown_connections 2024-05-09T08:24:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 --2- 172.21.15.28:0/1346675329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9584102d40 0x7f958410f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 --2- 172.21.15.28:0/1346675329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95841083e0 0x7f95841087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 --2- 172.21.15.28:0/1346675329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95841023a0 0x7f9584102800 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 -- 172.21.15.28:0/1346675329 >> 172.21.15.28:0/1346675329 conn(0x7f95840793f0 msgr2=0x7f9584079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 -- 172.21.15.28:0/1346675329 shutdown_connections 2024-05-09T08:24:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 -- 172.21.15.28:0/1346675329 wait complete. 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.861+0000 7f958a531700 1 Processor -- start 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f958a531700 1 -- start start 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f958a531700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95841023a0 0x7f9584074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f958a531700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9584102d40 0x7f9584074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f958a531700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95841083e0 0x7f95840750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f958a531700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9584112660 con 0x7f95841023a0 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f958a531700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f95841124e0 con 0x7f9584102d40 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f958a531700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f95841127e0 con 0x7f95841083e0 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f9583fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95841023a0 0x7f9584074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f9588ace700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95841083e0 0x7f95840750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f9583fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95841023a0 0x7f9584074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33130/0 (socket says 172.21.15.28:33130) 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f9583fff700 1 -- 172.21.15.28:0/1984774409 learned_addr learned my addr 172.21.15.28:0/1984774409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f95837fe700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9584102d40 0x7f9584074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f9583fff700 1 -- 172.21.15.28:0/1984774409 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95841083e0 msgr2=0x7f95840750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f9583fff700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95841083e0 0x7f95840750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f9583fff700 1 -- 172.21.15.28:0/1984774409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9584102d40 msgr2=0x7f9584074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f9583fff700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9584102d40 0x7f9584074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.862+0000 7f9583fff700 1 -- 172.21.15.28:0/1984774409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95841a6fc0 con 0x7f95841023a0 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.863+0000 7f9583fff700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95841023a0 0x7f9584074620 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f957400d950 tx=0x7f957400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.863+0000 7f95817fa700 1 -- 172.21.15.28:0/1984774409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9574016070 con 0x7f95841023a0 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.863+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95841a72b0 con 0x7f95841023a0 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.863+0000 7f95817fa700 1 -- 172.21.15.28:0/1984774409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f957400fd70 con 0x7f95841023a0 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.863+0000 7f95817fa700 1 -- 172.21.15.28:0/1984774409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957400e610 con 0x7f95841023a0 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.863+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95841a7810 con 0x7f95841023a0 2024-05-09T08:24:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.863+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9584062a30 con 0x7f95841023a0 2024-05-09T08:24:50.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.865+0000 7f95817fa700 1 -- 172.21.15.28:0/1984774409 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9574019070 con 0x7f95841023a0 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.866+0000 7f95817fa700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95700a2cb0 0x7f95700a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.866+0000 7f95837fe700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95700a2cb0 0x7f95700a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.866+0000 7f95817fa700 1 -- 172.21.15.28:0/1984774409 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f95740cf4d0 con 0x7f95841023a0 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.866+0000 7f95817fa700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] conn(0x7f95700ade10 0x7f95700b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.866+0000 7f95817fa700 1 -- 172.21.15.28:0/1984774409 --> [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95700b0980 con 0x7f95700ade10 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.867+0000 7f95817fa700 1 -- 172.21.15.28:0/1984774409 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f95740cf880 con 0x7f95841023a0 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.867+0000 7f9588ace700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] conn(0x7f95700ade10 0x7f95700b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.867+0000 7f95837fe700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95700a2cb0 0x7f95700a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f956c0098e0 tx=0x7f956c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.867+0000 7f9588ace700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] conn(0x7f95700ade10 0x7f95700b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.868+0000 7f95817fa700 1 -- 172.21.15.28:0/1984774409 <== osd.6 v2:172.21.15.28:6818/899794462 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f95700b0980 con 0x7f95700ade10 2024-05-09T08:24:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.884+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 --> [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9584062e30 con 0x7f95700ade10 2024-05-09T08:24:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f95817fa700 1 -- 172.21.15.28:0/1984774409 <== osd.6 v2:172.21.15.28:6818/899794462 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9584062e30 con 0x7f95700ade10 2024-05-09T08:24:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] conn(0x7f95700ade10 msgr2=0x7f95700b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f958a531700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] conn(0x7f95700ade10 0x7f95700b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95700a2cb0 msgr2=0x7f95700a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f958a531700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f95700a2cb0 0x7f95700a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f956c0098e0 tx=0x7f956c005960 comp rx=0 tx=0).stop 2024-05-09T08:24:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95841023a0 msgr2=0x7f9584074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f958a531700 1 --2- 172.21.15.28:0/1984774409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95841023a0 0x7f9584074620 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f957400d950 tx=0x7f957400dc60 comp rx=0 tx=0).stop 2024-05-09T08:24:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f9588ace700 1 -- 172.21.15.28:0/1984774409 reap_dead start 2024-05-09T08:24:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 shutdown_connections 2024-05-09T08:24:50.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.885+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 >> 172.21.15.28:0/1984774409 conn(0x7f95840793f0 msgr2=0x7f9584100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:50.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.886+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 shutdown_connections 2024-05-09T08:24:50.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:50.886+0000 7f958a531700 1 -- 172.21.15.28:0/1984774409 wait complete. 2024-05-09T08:24:51.434 INFO:teuthology.orchestra.run.smithi028.stdout:107374182463 2024-05-09T08:24:51.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-09T08:24:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: pgmap v319: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:24:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-05-09T08:24:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-09T08:24:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-05-09T08:24:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-09T08:24:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-05-09T08:24:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-05-09T08:24:51.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-05-09T08:24:51.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-05-09T08:24:51.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-05-09T08:24:51.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-09T08:24:51.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T08:24:51.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: pgmap v319: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:24:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-05-09T08:24:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-09T08:24:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-05-09T08:24:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-09T08:24:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-05-09T08:24:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-05-09T08:24:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-05-09T08:24:51.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-05-09T08:24:51.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-05-09T08:24:51.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-09T08:24:51.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T08:24:51.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: pgmap v319: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:24:51.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-05-09T08:24:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-09T08:24:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-05-09T08:24:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-09T08:24:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-05-09T08:24:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-05-09T08:24:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-05-09T08:24:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-05-09T08:24:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-05-09T08:24:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-09T08:24:51.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T08:24:51.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:51.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:24:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.219+0000 7f37a5b0b700 1 -- 172.21.15.28:0/875247672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37a0108390 msgr2=0x7f37a0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.219+0000 7f37a5b0b700 1 --2- 172.21.15.28:0/875247672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37a0108390 0x7f37a0108770 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3790009990 tx=0x7f3790009ca0 comp rx=0 tx=0).stop 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.219+0000 7f37a5b0b700 1 -- 172.21.15.28:0/875247672 shutdown_connections 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.219+0000 7f37a5b0b700 1 --2- 172.21.15.28:0/875247672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37a006b260 0x7f37a010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.219+0000 7f37a5b0b700 1 --2- 172.21.15.28:0/875247672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37a0108390 0x7f37a0108770 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.219+0000 7f37a5b0b700 1 --2- 172.21.15.28:0/875247672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37a006a8c0 0x7f37a006ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.219+0000 7f37a5b0b700 1 -- 172.21.15.28:0/875247672 >> 172.21.15.28:0/875247672 conn(0x7f37a0078190 msgr2=0x7f37a00785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.219+0000 7f37a5b0b700 1 -- 172.21.15.28:0/875247672 shutdown_connections 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.219+0000 7f37a5b0b700 1 -- 172.21.15.28:0/875247672 wait complete. 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f37a5b0b700 1 Processor -- start 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f37a5b0b700 1 -- start start 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f37a5b0b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37a006a8c0 0x7f37a019e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f37a5b0b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37a006b260 0x7f37a019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f37a5b0b700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37a0108390 0x7f37a0198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f37a5b0b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37a01126d0 con 0x7f37a006b260 2024-05-09T08:24:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f37a5b0b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f37a0112550 con 0x7f37a006a8c0 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f37a5b0b700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f37a0112850 con 0x7f37a0108390 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f379effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37a006b260 0x7f37a019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f379effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37a006b260 0x7f37a019ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33142/0 (socket says 172.21.15.28:33142) 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f379effd700 1 -- 172.21.15.28:0/910503340 learned_addr learned my addr 172.21.15.28:0/910503340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f379f7fe700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37a006a8c0 0x7f37a019e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.220+0000 7f379ffff700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37a0108390 0x7f37a0198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379effd700 1 -- 172.21.15.28:0/910503340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37a0108390 msgr2=0x7f37a0198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379effd700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37a0108390 0x7f37a0198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379effd700 1 -- 172.21.15.28:0/910503340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37a006a8c0 msgr2=0x7f37a019e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379effd700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37a006a8c0 0x7f37a019e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379effd700 1 -- 172.21.15.28:0/910503340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a01991b0 con 0x7f37a006b260 2024-05-09T08:24:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379f7fe700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37a006a8c0 0x7f37a019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379effd700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37a006b260 0x7f37a019ed80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f378800b530 tx=0x7f378800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379cff9700 1 -- 172.21.15.28:0/910503340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788015020 con 0x7f37a006b260 2024-05-09T08:24:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a01994a0 con 0x7f37a006b260 2024-05-09T08:24:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a01a5710 con 0x7f37a006b260 2024-05-09T08:24:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379cff9700 1 -- 172.21.15.28:0/910503340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3788011d10 con 0x7f37a006b260 2024-05-09T08:24:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.221+0000 7f379cff9700 1 -- 172.21.15.28:0/910503340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788010e50 con 0x7f37a006b260 2024-05-09T08:24:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.222+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3780001030 con 0x7f37a006b260 2024-05-09T08:24:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.224+0000 7f379cff9700 1 -- 172.21.15.28:0/910503340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3788018020 con 0x7f37a006b260 2024-05-09T08:24:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.224+0000 7f379cff9700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f378c0a2e50 0x7f378c0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.224+0000 7f379f7fe700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f378c0a2e50 0x7f378c0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.224+0000 7f379cff9700 1 -- 172.21.15.28:0/910503340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f37880cf1a0 con 0x7f37a006b260 2024-05-09T08:24:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.225+0000 7f379cff9700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] conn(0x7f378c0adfb0 0x7f378c0b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.225+0000 7f379cff9700 1 -- 172.21.15.28:0/910503340 --> [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37880044e0 con 0x7f378c0adfb0 2024-05-09T08:24:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.225+0000 7f379cff9700 1 -- 172.21.15.28:0/910503340 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f37880cf5f0 con 0x7f37a006b260 2024-05-09T08:24:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.225+0000 7f379f7fe700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f378c0a2e50 0x7f378c0a5310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f37900098e0 tx=0x7f379000b560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.225+0000 7f379ffff700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] conn(0x7f378c0adfb0 0x7f378c0b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.225+0000 7f379ffff700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] conn(0x7f378c0adfb0 0x7f378c0b0450 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.226+0000 7f379cff9700 1 -- 172.21.15.28:0/910503340 <== osd.1 v2:172.21.15.165:6800/3539015661 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37880044e0 con 0x7f378c0adfb0 2024-05-09T08:24:52.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.243+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 --> [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3780002e60 con 0x7f378c0adfb0 2024-05-09T08:24:52.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.243+0000 7f379cff9700 1 -- 172.21.15.28:0/910503340 <== osd.1 v2:172.21.15.165:6800/3539015661 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3780002e60 con 0x7f378c0adfb0 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 >> [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] conn(0x7f378c0adfb0 msgr2=0x7f378c0b0450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] conn(0x7f378c0adfb0 0x7f378c0b0450 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f378c0a2e50 msgr2=0x7f378c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f378c0a2e50 0x7f378c0a5310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f37900098e0 tx=0x7f379000b560 comp rx=0 tx=0).stop 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37a006b260 msgr2=0x7f37a019ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 --2- 172.21.15.28:0/910503340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37a006b260 0x7f37a019ed80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f378800b530 tx=0x7f378800b8f0 comp rx=0 tx=0).stop 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f379ffff700 1 -- 172.21.15.28:0/910503340 reap_dead start 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 shutdown_connections 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 >> 172.21.15.28:0/910503340 conn(0x7f37a0078190 msgr2=0x7f37a00fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 shutdown_connections 2024-05-09T08:24:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:52.244+0000 7f37a5b0b700 1 -- 172.21.15.28:0/910503340 wait complete. 2024-05-09T08:24:52.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:52 smithi028 ceph-mon[30119]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T08:24:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:52 smithi165 ceph-mon[34540]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T08:24:52.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:52 smithi184 ceph-mon[34509]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T08:24:52.893 INFO:teuthology.orchestra.run.smithi028.stdout:77309411398 2024-05-09T08:24:52.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-09T08:24:53.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:53 smithi028 ceph-mon[30119]: pgmap v322: 129 pgs: 6 peering, 123 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 383 B/s wr, 9 op/s 2024-05-09T08:24:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:53 smithi165 ceph-mon[34540]: pgmap v322: 129 pgs: 6 peering, 123 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 383 B/s wr, 9 op/s 2024-05-09T08:24:53.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:53 smithi184 ceph-mon[34509]: pgmap v322: 129 pgs: 6 peering, 123 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 383 B/s wr, 9 op/s 2024-05-09T08:24:54.091 INFO:teuthology.orchestra.run.smithi028.stdout:219043332140 2024-05-09T08:24:54.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-09T08:24:54.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.279+0000 7fa77283e700 1 -- 172.21.15.28:0/4277151419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c06a9d0 msgr2=0x7fa76c10f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:54.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.279+0000 7fa77283e700 1 --2- 172.21.15.28:0/4277151419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c06a9d0 0x7fa76c10f1e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa76400b0a0 tx=0x7fa76400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.279+0000 7fa77283e700 1 -- 172.21.15.28:0/4277151419 shutdown_connections 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.279+0000 7fa77283e700 1 --2- 172.21.15.28:0/4277151419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c06a9d0 0x7fa76c10f1e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.279+0000 7fa77283e700 1 --2- 172.21.15.28:0/4277151419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa76c1083a0 0x7fa76c108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.279+0000 7fa77283e700 1 --2- 172.21.15.28:0/4277151419 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa76c06a030 0x7fa76c06a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.279+0000 7fa77283e700 1 -- 172.21.15.28:0/4277151419 >> 172.21.15.28:0/4277151419 conn(0x7fa76c0781d0 msgr2=0x7fa76c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.279+0000 7fa77283e700 1 -- 172.21.15.28:0/4277151419 shutdown_connections 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.279+0000 7fa77283e700 1 -- 172.21.15.28:0/4277151419 wait complete. 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa77283e700 1 Processor -- start 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa77283e700 1 -- start start 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa77283e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c06a030 0x7fa76c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa77283e700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa76c06a9d0 0x7fa76c073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa77283e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa76c1083a0 0x7fa76c06cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:54.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa77283e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c112620 con 0x7fa76c06a030 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa77283e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1124a0 con 0x7fa76c1083a0 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa77283e700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1127a0 con 0x7fa76c06a9d0 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa76bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c06a030 0x7fa76c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa76bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c06a030 0x7fa76c073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33154/0 (socket says 172.21.15.28:33154) 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.280+0000 7fa76bfff700 1 -- 172.21.15.28:0/3179583552 learned_addr learned my addr 172.21.15.28:0/3179583552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa770ddb700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa76c1083a0 0x7fa76c06cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa76b7fe700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa76c06a9d0 0x7fa76c073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa76bfff700 1 -- 172.21.15.28:0/3179583552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa76c06a9d0 msgr2=0x7fa76c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa76bfff700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa76c06a9d0 0x7fa76c073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa76bfff700 1 -- 172.21.15.28:0/3179583552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa76c1083a0 msgr2=0x7fa76c06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa76bfff700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa76c1083a0 0x7fa76c06cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa76bfff700 1 -- 172.21.15.28:0/3179583552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c06d630 con 0x7fa76c06a030 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa76bfff700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c06a030 0x7fa76c073a50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00e9a0 tx=0x7fa75c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa7697fa700 1 -- 172.21.15.28:0/3179583552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c00cbd0 con 0x7fa76c06a030 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa7697fa700 1 -- 172.21.15.28:0/3179583552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75c00cd70 con 0x7fa76c06a030 2024-05-09T08:24:54.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76c06d920 con 0x7fa76c06a030 2024-05-09T08:24:54.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa7697fa700 1 -- 172.21.15.28:0/3179583552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c018750 con 0x7fa76c06a030 2024-05-09T08:24:54.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.281+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76c071890 con 0x7fa76c06a030 2024-05-09T08:24:54.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.282+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa76c071e80 con 0x7fa76c06a030 2024-05-09T08:24:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.284+0000 7fa7697fa700 1 -- 172.21.15.28:0/3179583552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa75c010430 con 0x7fa76c06a030 2024-05-09T08:24:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.285+0000 7fa7697fa700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa7540a2d30 0x7fa7540a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.285+0000 7fa76b7fe700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa7540a2d30 0x7fa7540a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.285+0000 7fa7697fa700 1 -- 172.21.15.28:0/3179583552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa75c0d0270 con 0x7fa76c06a030 2024-05-09T08:24:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.286+0000 7fa7697fa700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] conn(0x7fa7540ade90 0x7fa7540b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.286+0000 7fa7697fa700 1 -- 172.21.15.28:0/3179583552 --> [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa75c01bec0 con 0x7fa7540ade90 2024-05-09T08:24:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.286+0000 7fa7697fa700 1 -- 172.21.15.28:0/3179583552 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa75c0d06b0 con 0x7fa76c06a030 2024-05-09T08:24:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.286+0000 7fa76b7fe700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa7540a2d30 0x7fa7540a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa76c06e2a0 tx=0x7fa760009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.286+0000 7fa770ddb700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] conn(0x7fa7540ade90 0x7fa7540b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:54.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.287+0000 7fa770ddb700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] conn(0x7fa7540ade90 0x7fa7540b0330 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:54.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.288+0000 7fa7697fa700 1 -- 172.21.15.28:0/3179583552 <== osd.8 v2:172.21.15.184:6816/2287150908 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa75c01bec0 con 0x7fa7540ade90 2024-05-09T08:24:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.308+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 --> [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa76c072280 con 0x7fa7540ade90 2024-05-09T08:24:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.308+0000 7fa7697fa700 1 -- 172.21.15.28:0/3179583552 <== osd.8 v2:172.21.15.184:6816/2287150908 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa76c072280 con 0x7fa7540ade90 2024-05-09T08:24:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.309+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 >> [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] conn(0x7fa7540ade90 msgr2=0x7fa7540b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.309+0000 7fa77283e700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] conn(0x7fa7540ade90 0x7fa7540b0330 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.309+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa7540a2d30 msgr2=0x7fa7540a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.309+0000 7fa77283e700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fa7540a2d30 0x7fa7540a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa76c06e2a0 tx=0x7fa760009260 comp rx=0 tx=0).stop 2024-05-09T08:24:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.309+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c06a030 msgr2=0x7fa76c073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.309+0000 7fa77283e700 1 --2- 172.21.15.28:0/3179583552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c06a030 0x7fa76c073a50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00e9a0 tx=0x7fa75c00ecb0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.309+0000 7fa770ddb700 1 -- 172.21.15.28:0/3179583552 reap_dead start 2024-05-09T08:24:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.309+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 shutdown_connections 2024-05-09T08:24:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.309+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 >> 172.21.15.28:0/3179583552 conn(0x7fa76c0781d0 msgr2=0x7fa76c0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.310+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 shutdown_connections 2024-05-09T08:24:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:54.310+0000 7fa77283e700 1 -- 172.21.15.28:0/3179583552 wait complete. 2024-05-09T08:24:54.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:54 smithi028 ceph-mon[30119]: pgmap v323: 129 pgs: 6 peering, 123 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 383 B/s wr, 10 op/s; 28 B/s, 1 objects/s recovering 2024-05-09T08:24:54.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:54 smithi165 ceph-mon[34540]: pgmap v323: 129 pgs: 6 peering, 123 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 383 B/s wr, 10 op/s; 28 B/s, 1 objects/s recovering 2024-05-09T08:24:54.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:54 smithi184 ceph-mon[34509]: pgmap v323: 129 pgs: 6 peering, 123 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 383 B/s wr, 10 op/s; 28 B/s, 1 objects/s recovering 2024-05-09T08:24:54.996 INFO:teuthology.orchestra.run.smithi028.stdout:163208757303 2024-05-09T08:24:54.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-09T08:24:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.251+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/2916284791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec1023a0 msgr2=0x7ff8ec102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.251+0000 7ff8f1d0a700 1 --2- 172.21.15.28:0/2916284791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec1023a0 0x7ff8ec102800 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009990 tx=0x7ff8e0009ca0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.251+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/2916284791 shutdown_connections 2024-05-09T08:24:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.251+0000 7ff8f1d0a700 1 --2- 172.21.15.28:0/2916284791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8ec1083e0 0x7ff8ec1087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.251+0000 7ff8f1d0a700 1 --2- 172.21.15.28:0/2916284791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8ec102d40 0x7ff8ec10f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.251+0000 7ff8f1d0a700 1 --2- 172.21.15.28:0/2916284791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec1023a0 0x7ff8ec102800 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.251+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/2916284791 >> 172.21.15.28:0/2916284791 conn(0x7ff8ec0793f0 msgr2=0x7ff8ec079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.251+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/2916284791 shutdown_connections 2024-05-09T08:24:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.251+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/2916284791 wait complete. 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.252+0000 7ff8f1d0a700 1 Processor -- start 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8f1d0a700 1 -- start start 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8f1d0a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec1023a0 0x7ff8ec19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8f1d0a700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8ec102d40 0x7ff8ec19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8f1d0a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8ec1083e0 0x7ff8ec1988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8f1d0a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec112640 con 0x7ff8ec1023a0 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8f1d0a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec1124c0 con 0x7ff8ec1083e0 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8f1d0a700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec1127c0 con 0x7ff8ec102d40 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8eb7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec1023a0 0x7ff8ec19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8eb7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec1023a0 0x7ff8ec19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34524/0 (socket says 172.21.15.28:34524) 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8eb7fe700 1 -- 172.21.15.28:0/3301875570 learned_addr learned my addr 172.21.15.28:0/3301875570 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8eaffd700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8ec102d40 0x7ff8ec19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8ebfff700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8ec1083e0 0x7ff8ec1988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8eb7fe700 1 -- 172.21.15.28:0/3301875570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8ec102d40 msgr2=0x7ff8ec19ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8eb7fe700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff8ec102d40 0x7ff8ec19ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8eb7fe700 1 -- 172.21.15.28:0/3301875570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8ec1083e0 msgr2=0x7ff8ec1988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8eb7fe700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8ec1083e0 0x7ff8ec1988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.253+0000 7ff8eb7fe700 1 -- 172.21.15.28:0/3301875570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ec199000 con 0x7ff8ec1023a0 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.254+0000 7ff8eb7fe700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec1023a0 0x7ff8ec19e6e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00d950 tx=0x7ff8dc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.254+0000 7ff8e8ff9700 1 -- 172.21.15.28:0/3301875570 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc016070 con 0x7ff8ec1023a0 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.254+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ec1992f0 con 0x7ff8ec1023a0 2024-05-09T08:24:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.254+0000 7ff8e8ff9700 1 -- 172.21.15.28:0/3301875570 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8dc00fd70 con 0x7ff8ec1023a0 2024-05-09T08:24:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.254+0000 7ff8e8ff9700 1 -- 172.21.15.28:0/3301875570 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00e610 con 0x7ff8ec1023a0 2024-05-09T08:24:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.254+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ec1a55b0 con 0x7ff8ec1023a0 2024-05-09T08:24:55.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.255+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff8ec062ab0 con 0x7ff8ec1023a0 2024-05-09T08:24:55.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.256+0000 7ff8e8ff9700 1 -- 172.21.15.28:0/3301875570 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff8dc019070 con 0x7ff8ec1023a0 2024-05-09T08:24:55.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.257+0000 7ff8e8ff9700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff8d40a2cb0 0x7ff8d40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:55.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.257+0000 7ff8eaffd700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff8d40a2cb0 0x7ff8d40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:55.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.257+0000 7ff8e8ff9700 1 -- 172.21.15.28:0/3301875570 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff8dc0cef40 con 0x7ff8ec1023a0 2024-05-09T08:24:55.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.257+0000 7ff8e8ff9700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] conn(0x7ff8d40ade10 0x7ff8d40b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:55.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.257+0000 7ff8e8ff9700 1 -- 172.21.15.28:0/3301875570 --> [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8d40b0980 con 0x7ff8d40ade10 2024-05-09T08:24:55.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.257+0000 7ff8e8ff9700 1 -- 172.21.15.28:0/3301875570 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7ff8dc0d1600 con 0x7ff8ec1023a0 2024-05-09T08:24:55.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.257+0000 7ff8eaffd700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff8d40a2cb0 0x7ff8d40a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000b5c0 tx=0x7ff8e0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:55.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.258+0000 7ff8ebfff700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] conn(0x7ff8d40ade10 0x7ff8d40b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:55.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.258+0000 7ff8ebfff700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] conn(0x7ff8d40ade10 0x7ff8d40b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:55.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.259+0000 7ff8e8ff9700 1 -- 172.21.15.28:0/3301875570 <== osd.4 v2:172.21.15.165:6808/3527398709 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8d40b0980 con 0x7ff8d40ade10 2024-05-09T08:24:55.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.275+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 --> [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff8ec062f30 con 0x7ff8d40ade10 2024-05-09T08:24:55.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8e8ff9700 1 -- 172.21.15.28:0/3301875570 <== osd.4 v2:172.21.15.165:6808/3527398709 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff8ec062f30 con 0x7ff8d40ade10 2024-05-09T08:24:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 >> [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] conn(0x7ff8d40ade10 msgr2=0x7ff8d40b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8f1d0a700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] conn(0x7ff8d40ade10 0x7ff8d40b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff8d40a2cb0 msgr2=0x7ff8d40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8f1d0a700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff8d40a2cb0 0x7ff8d40a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000b5c0 tx=0x7ff8e0005960 comp rx=0 tx=0).stop 2024-05-09T08:24:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec1023a0 msgr2=0x7ff8ec19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8f1d0a700 1 --2- 172.21.15.28:0/3301875570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec1023a0 0x7ff8ec19e6e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00d950 tx=0x7ff8dc00dc60 comp rx=0 tx=0).stop 2024-05-09T08:24:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8ebfff700 1 -- 172.21.15.28:0/3301875570 reap_dead start 2024-05-09T08:24:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 shutdown_connections 2024-05-09T08:24:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.276+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 >> 172.21.15.28:0/3301875570 conn(0x7ff8ec0793f0 msgr2=0x7ff8ec101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.277+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 shutdown_connections 2024-05-09T08:24:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:55.277+0000 7ff8f1d0a700 1 -- 172.21.15.28:0/3301875570 wait complete. 2024-05-09T08:24:55.955 INFO:teuthology.orchestra.run.smithi028.stdout:253403070506 2024-05-09T08:24:55.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-09T08:24:56.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:56 smithi028 ceph-mon[30119]: pgmap v324: 129 pgs: 6 peering, 123 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s; 28 B/s, 1 objects/s recovering 2024-05-09T08:24:57.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:56 smithi165 ceph-mon[34540]: pgmap v324: 129 pgs: 6 peering, 123 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s; 28 B/s, 1 objects/s recovering 2024-05-09T08:24:57.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:56 smithi184 ceph-mon[34509]: pgmap v324: 129 pgs: 6 peering, 123 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s; 28 B/s, 1 objects/s recovering 2024-05-09T08:24:57.716 INFO:teuthology.orchestra.run.smithi028.stdout:55834574924 2024-05-09T08:24:57.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.601+0000 7f3c069c9700 1 -- 172.21.15.28:0/2862279789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c00101be0 msgr2=0x7f3c0010e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.601+0000 7f3c069c9700 1 --2- 172.21.15.28:0/2862279789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c00101be0 0x7f3c0010e0d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800b0a0 tx=0x7f3bf800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.601+0000 7f3c069c9700 1 -- 172.21.15.28:0/2862279789 shutdown_connections 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.601+0000 7f3c069c9700 1 --2- 172.21.15.28:0/2862279789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c00101be0 0x7f3c0010e0d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.601+0000 7f3c069c9700 1 --2- 172.21.15.28:0/2862279789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c00107290 0x7f3c00107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.601+0000 7f3c069c9700 1 --2- 172.21.15.28:0/2862279789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c00101220 0x7f3c001016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.601+0000 7f3c069c9700 1 -- 172.21.15.28:0/2862279789 >> 172.21.15.28:0/2862279789 conn(0x7f3c000781d0 msgr2=0x7f3c000785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.601+0000 7f3c069c9700 1 -- 172.21.15.28:0/2862279789 shutdown_connections 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.601+0000 7f3c069c9700 1 -- 172.21.15.28:0/2862279789 wait complete. 2024-05-09T08:24:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.602+0000 7f3c069c9700 1 Processor -- start 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.602+0000 7f3c069c9700 1 -- start start 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.602+0000 7f3c069c9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c00101220 0x7f3c00199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.602+0000 7f3c069c9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c00101be0 0x7f3c00199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.602+0000 7f3c069c9700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c00107290 0x7f3c001a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.602+0000 7f3c069c9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c00111550 con 0x7f3c00101220 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.602+0000 7f3c069c9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3c001113d0 con 0x7f3c00101be0 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.602+0000 7f3c069c9700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3c001116d0 con 0x7f3c00107290 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c00101220 0x7f3c00199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c00101220 0x7f3c00199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34574/0 (socket says 172.21.15.28:34574) 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfffff700 1 -- 172.21.15.28:0/1291071921 learned_addr learned my addr 172.21.15.28:0/1291071921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3c04f66700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c00107290 0x7f3c001a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bf7fff700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c00101be0 0x7f3c00199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfffff700 1 -- 172.21.15.28:0/1291071921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c00107290 msgr2=0x7f3c001a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfffff700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3c00107290 0x7f3c001a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfffff700 1 -- 172.21.15.28:0/1291071921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c00101be0 msgr2=0x7f3c00199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfffff700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c00101be0 0x7f3c00199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfffff700 1 -- 172.21.15.28:0/1291071921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c001a4e50 con 0x7f3c00101220 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfffff700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c00101220 0x7f3c00199890 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c9b0 tx=0x7f3bf000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfdffb700 1 -- 172.21.15.28:0/1291071921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0007b00 con 0x7f3c00101220 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c001a5080 con 0x7f3c00101220 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfdffb700 1 -- 172.21.15.28:0/1291071921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bf0007ca0 con 0x7f3c00101220 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c001a55e0 con 0x7f3c00101220 2024-05-09T08:24:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.603+0000 7f3bfdffb700 1 -- 172.21.15.28:0/1291071921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0018730 con 0x7f3c00101220 2024-05-09T08:24:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.604+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3c00100570 con 0x7f3c00101220 2024-05-09T08:24:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.606+0000 7f3bfdffb700 1 -- 172.21.15.28:0/1291071921 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3bf00188d0 con 0x7f3c00101220 2024-05-09T08:24:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.606+0000 7f3bfdffb700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3bec0a2cb0 0x7f3bec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.606+0000 7f3bfdffb700 1 -- 172.21.15.28:0/1291071921 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3bf00cf8a0 con 0x7f3c00101220 2024-05-09T08:24:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.606+0000 7f3bf7fff700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3bec0a2cb0 0x7f3bec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.607+0000 7f3bfdffb700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] conn(0x7f3bec0ade10 0x7f3bec0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.607+0000 7f3bfdffb700 1 -- 172.21.15.28:0/1291071921 --> [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3bf0010e70 con 0x7f3bec0ade10 2024-05-09T08:24:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.607+0000 7f3bfdffb700 1 -- 172.21.15.28:0/1291071921 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3bf00cfcf0 con 0x7f3c00101220 2024-05-09T08:24:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.607+0000 7f3bf7fff700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3bec0a2cb0 0x7f3bec0a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3c00078680 tx=0x7f3be8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.607+0000 7f3c04f66700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] conn(0x7f3bec0ade10 0x7f3bec0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.607+0000 7f3c04f66700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] conn(0x7f3bec0ade10 0x7f3bec0b02b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.608+0000 7f3bfdffb700 1 -- 172.21.15.28:0/1291071921 <== osd.10 v2:172.21.15.165:6824/3106810724 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bf0010e70 con 0x7f3bec0ade10 2024-05-09T08:24:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.625+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 --> [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3c00102940 con 0x7f3bec0ade10 2024-05-09T08:24:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.625+0000 7f3bfdffb700 1 -- 172.21.15.28:0/1291071921 <== osd.10 v2:172.21.15.165:6824/3106810724 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3c00102940 con 0x7f3bec0ade10 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.625+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 >> [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] conn(0x7f3bec0ade10 msgr2=0x7f3bec0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.625+0000 7f3c069c9700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] conn(0x7f3bec0ade10 0x7f3bec0b02b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.626+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3bec0a2cb0 msgr2=0x7f3bec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.626+0000 7f3c069c9700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3bec0a2cb0 0x7f3bec0a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3c00078680 tx=0x7f3be8009260 comp rx=0 tx=0).stop 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.626+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c00101220 msgr2=0x7f3c00199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.626+0000 7f3c069c9700 1 --2- 172.21.15.28:0/1291071921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c00101220 0x7f3c00199890 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c9b0 tx=0x7f3bf000ccc0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.626+0000 7f3c04f66700 1 -- 172.21.15.28:0/1291071921 reap_dead start 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.626+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 shutdown_connections 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.626+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 >> 172.21.15.28:0/1291071921 conn(0x7f3c000781d0 msgr2=0x7f3c000ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.626+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 shutdown_connections 2024-05-09T08:24:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.626+0000 7f3c069c9700 1 -- 172.21.15.28:0/1291071921 wait complete. 2024-05-09T08:24:58.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:24:58 smithi028 ceph-mon[30119]: pgmap v325: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 7.8 KiB/s, 3 objects/s recovering 2024-05-09T08:24:59.048 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:59.074 INFO:teuthology.orchestra.run.smithi028.stdout:197568495666 2024-05-09T08:24:59.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-09T08:24:59.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:24:58 smithi165 ceph-mon[34540]: pgmap v325: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 7.8 KiB/s, 3 objects/s recovering 2024-05-09T08:24:59.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:24:58 smithi184 ceph-mon[34509]: pgmap v325: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 7.8 KiB/s, 3 objects/s recovering 2024-05-09T08:24:59.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 -- 172.21.15.28:0/3499259496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e901134f0 msgr2=0x7f5e901158e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:59.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 --2- 172.21.15.28:0/3499259496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e901134f0 0x7f5e901158e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800b0a0 tx=0x7f5e8800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:59.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 -- 172.21.15.28:0/3499259496 shutdown_connections 2024-05-09T08:24:59.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 --2- 172.21.15.28:0/3499259496 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e90074550 0x7f5e90074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:59.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 --2- 172.21.15.28:0/3499259496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e901134f0 0x7f5e901158e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:59.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 --2- 172.21.15.28:0/3499259496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e90074e70 0x7f5e90112fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 -- 172.21.15.28:0/3499259496 >> 172.21.15.28:0/3499259496 conn(0x7f5e90079480 msgr2=0x7f5e90079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 -- 172.21.15.28:0/3499259496 shutdown_connections 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 -- 172.21.15.28:0/3499259496 wait complete. 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.197+0000 7f5e978ce700 1 Processor -- start 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e978ce700 1 -- start start 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e978ce700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e90074550 0x7f5e9019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e978ce700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e90074e70 0x7f5e9019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e978ce700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e901134f0 0x7f5e901a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e978ce700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e90118c60 con 0x7f5e90074550 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e978ce700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5e90118ae0 con 0x7f5e90074e70 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e978ce700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5e90118de0 con 0x7f5e901134f0 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e9566a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e90074550 0x7f5e9019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e9566a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e90074550 0x7f5e9019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34554/0 (socket says 172.21.15.28:34554) 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e9566a700 1 -- 172.21.15.28:0/1304491178 learned_addr learned my addr 172.21.15.28:0/1304491178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e94e69700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e90074e70 0x7f5e9019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e95e6b700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e901134f0 0x7f5e901a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e9566a700 1 -- 172.21.15.28:0/1304491178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e901134f0 msgr2=0x7f5e901a3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e9566a700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e901134f0 0x7f5e901a3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e9566a700 1 -- 172.21.15.28:0/1304491178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e90074e70 msgr2=0x7f5e9019f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e9566a700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e90074e70 0x7f5e9019f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.198+0000 7f5e9566a700 1 -- 172.21.15.28:0/1304491178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e901a4040 con 0x7f5e90074550 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.199+0000 7f5e9566a700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e90074550 0x7f5e9019f030 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00c9b0 tx=0x7f5e8c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.199+0000 7f5e867fc700 1 -- 172.21.15.28:0/1304491178 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c007b00 con 0x7f5e90074550 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.199+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e901a4330 con 0x7f5e90074550 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.199+0000 7f5e867fc700 1 -- 172.21.15.28:0/1304491178 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e8c00ce50 con 0x7f5e90074550 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.199+0000 7f5e867fc700 1 -- 172.21.15.28:0/1304491178 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c018660 con 0x7f5e90074550 2024-05-09T08:24:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.199+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e901abbb0 con 0x7f5e90074550 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.200+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5e90112c60 con 0x7f5e90074550 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.201+0000 7f5e867fc700 1 -- 172.21.15.28:0/1304491178 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5e8c00f450 con 0x7f5e90074550 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.202+0000 7f5e867fc700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e7c0a2cb0 0x7f5e7c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.202+0000 7f5e94e69700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e7c0a2cb0 0x7f5e7c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.202+0000 7f5e867fc700 1 -- 172.21.15.28:0/1304491178 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5e8c0cf470 con 0x7f5e90074550 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.202+0000 7f5e867fc700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] conn(0x7f5e7c0ade10 0x7f5e7c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.202+0000 7f5e867fc700 1 -- 172.21.15.28:0/1304491178 --> [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e7c0b0980 con 0x7f5e7c0ade10 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.202+0000 7f5e867fc700 1 -- 172.21.15.28:0/1304491178 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f5e8c0d6050 con 0x7f5e90074550 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.202+0000 7f5e95e6b700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] conn(0x7f5e7c0ade10 0x7f5e7c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.203+0000 7f5e94e69700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e7c0a2cb0 0x7f5e7c0a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5e9010fdc0 tx=0x7f5e800073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.203+0000 7f5e95e6b700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] conn(0x7f5e7c0ade10 0x7f5e7c0b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.204+0000 7f5e867fc700 1 -- 172.21.15.28:0/1304491178 <== osd.9 v2:172.21.15.28:6826/712618237 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5e7c0b0980 con 0x7f5e7c0ade10 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.220+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 --> [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5e90075c40 con 0x7f5e7c0ade10 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.220+0000 7f5e867fc700 1 -- 172.21.15.28:0/1304491178 <== osd.9 v2:172.21.15.28:6826/712618237 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5e90075c40 con 0x7f5e7c0ade10 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.220+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] conn(0x7f5e7c0ade10 msgr2=0x7f5e7c0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.220+0000 7f5e978ce700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] conn(0x7f5e7c0ade10 0x7f5e7c0b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.220+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e7c0a2cb0 msgr2=0x7f5e7c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.220+0000 7f5e978ce700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e7c0a2cb0 0x7f5e7c0a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5e9010fdc0 tx=0x7f5e800073d0 comp rx=0 tx=0).stop 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.221+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e90074550 msgr2=0x7f5e9019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.221+0000 7f5e978ce700 1 --2- 172.21.15.28:0/1304491178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e90074550 0x7f5e9019f030 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00c9b0 tx=0x7f5e8c00ccc0 comp rx=0 tx=0).stop 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.221+0000 7f5e95e6b700 1 -- 172.21.15.28:0/1304491178 reap_dead start 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.221+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 shutdown_connections 2024-05-09T08:24:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.221+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 >> 172.21.15.28:0/1304491178 conn(0x7f5e90079480 msgr2=0x7f5e90112240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.221+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 shutdown_connections 2024-05-09T08:24:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:24:58.221+0000 7f5e978ce700 1 -- 172.21.15.28:0/1304491178 wait complete. 2024-05-09T08:24:59.520 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:59.522 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:59.976 INFO:teuthology.orchestra.run.smithi028.stdout:287762808869 2024-05-09T08:24:59.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-09T08:25:00.498 INFO:teuthology.orchestra.run.smithi028.stdout:141733920829 2024-05-09T08:25:00.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-09T08:25:00.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:00 smithi028 ceph-mon[30119]: pgmap v326: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 6.8 KiB/s, 2 objects/s recovering 2024-05-09T08:25:00.933 INFO:teuthology.orchestra.run.smithi028.stdout:309237645344 2024-05-09T08:25:00.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-09T08:25:00.938 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:00.939 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:00.946 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:01.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:00 smithi165 ceph-mon[34540]: pgmap v326: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 6.8 KiB/s, 2 objects/s recovering 2024-05-09T08:25:01.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:00 smithi184 ceph-mon[34509]: pgmap v326: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 6.8 KiB/s, 2 objects/s recovering 2024-05-09T08:25:02.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:02 smithi028 ceph-mon[30119]: pgmap v327: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 6.2 KiB/s, 2 objects/s recovering 2024-05-09T08:25:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:02 smithi165 ceph-mon[34540]: pgmap v327: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 6.2 KiB/s, 2 objects/s recovering 2024-05-09T08:25:03.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:02 smithi184 ceph-mon[34509]: pgmap v327: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 6.2 KiB/s, 2 objects/s recovering 2024-05-09T08:25:04.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:04 smithi028 ceph-mon[30119]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 5.2 KiB/s, 2 objects/s recovering 2024-05-09T08:25:05.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:04 smithi165 ceph-mon[34540]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 5.2 KiB/s, 2 objects/s recovering 2024-05-09T08:25:05.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:04 smithi184 ceph-mon[34509]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 5.2 KiB/s, 2 objects/s recovering 2024-05-09T08:25:06.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:06 smithi028 ceph-mon[30119]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-05-09T08:25:07.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:06 smithi165 ceph-mon[34540]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-05-09T08:25:07.110 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:06 smithi184 ceph-mon[34509]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-05-09T08:25:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.150+0000 7f6f1b431700 1 -- 172.21.15.28:0/948172462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f141025d0 msgr2=0x7f6f141029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.150+0000 7f6f1b431700 1 --2- 172.21.15.28:0/948172462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f141025d0 0x7f6f141029b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400b0a0 tx=0x7f6f0400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.150+0000 7f6f1b431700 1 -- 172.21.15.28:0/948172462 shutdown_connections 2024-05-09T08:25:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.150+0000 7f6f1b431700 1 --2- 172.21.15.28:0/948172462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f14107660 0x7f6f1410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.150+0000 7f6f1b431700 1 --2- 172.21.15.28:0/948172462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f141025d0 0x7f6f141029b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.150+0000 7f6f1b431700 1 --2- 172.21.15.28:0/948172462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f14102f80 0x7f6f14106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.150+0000 7f6f1b431700 1 -- 172.21.15.28:0/948172462 >> 172.21.15.28:0/948172462 conn(0x7f6f14079480 msgr2=0x7f6f14079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.150+0000 7f6f1b431700 1 -- 172.21.15.28:0/948172462 shutdown_connections 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.150+0000 7f6f1b431700 1 -- 172.21.15.28:0/948172462 wait complete. 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f1b431700 1 Processor -- start 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f1b431700 1 -- start start 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f1b431700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f141025d0 0x7f6f14074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f1b431700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f14102f80 0x7f6f14074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f1b431700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f14107660 0x7f6f140750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f1b431700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f14112620 con 0x7f6f14102f80 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f1b431700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141124a0 con 0x7f6f14107660 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f1b431700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141127a0 con 0x7f6f141025d0 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f189cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f14102f80 0x7f6f14074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f189cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f14102f80 0x7f6f14074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51552/0 (socket says 172.21.15.28:51552) 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.151+0000 7f6f189cc700 1 -- 172.21.15.28:0/1863889737 learned_addr learned my addr 172.21.15.28:0/1863889737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f199ce700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f14107660 0x7f6f140750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f191cd700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f141025d0 0x7f6f14074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f189cc700 1 -- 172.21.15.28:0/1863889737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f141025d0 msgr2=0x7f6f14074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f189cc700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f141025d0 0x7f6f14074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f189cc700 1 -- 172.21.15.28:0/1863889737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f14107660 msgr2=0x7f6f140750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f189cc700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f14107660 0x7f6f140750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f189cc700 1 -- 172.21.15.28:0/1863889737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f141a6f90 con 0x7f6f14102f80 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f199ce700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f14107660 0x7f6f140750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f189cc700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f14102f80 0x7f6f14074b60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800d950 tx=0x7f6f0800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f127fc700 1 -- 172.21.15.28:0/1863889737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f08016070 con 0x7f6f14102f80 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f127fc700 1 -- 172.21.15.28:0/1863889737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f0800de30 con 0x7f6f14102f80 2024-05-09T08:25:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f141a7280 con 0x7f6f14102f80 2024-05-09T08:25:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f127fc700 1 -- 172.21.15.28:0/1863889737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f08005420 con 0x7f6f14102f80 2024-05-09T08:25:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.152+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f141a77c0 con 0x7f6f14102f80 2024-05-09T08:25:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.153+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f14069010 con 0x7f6f14102f80 2024-05-09T08:25:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.155+0000 7f6f127fc700 1 -- 172.21.15.28:0/1863889737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6f08004750 con 0x7f6f14102f80 2024-05-09T08:25:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.156+0000 7f6f127fc700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6efc0a2cb0 0x7f6efc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.156+0000 7f6f191cd700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6efc0a2cb0 0x7f6efc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.156+0000 7f6f127fc700 1 -- 172.21.15.28:0/1863889737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6f080cf140 con 0x7f6f14102f80 2024-05-09T08:25:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.157+0000 7f6f191cd700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6efc0a2cb0 0x7f6efc0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400b830 tx=0x7f6f04009be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.159+0000 7f6f127fc700 1 -- 172.21.15.28:0/1863889737 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6f0801f070 con 0x7f6f14102f80 2024-05-09T08:25:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.369+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6f1402db80 con 0x7f6f14102f80 2024-05-09T08:25:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.369+0000 7f6f127fc700 1 -- 172.21.15.28:0/1863889737 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6f0809bbe0 con 0x7f6f14102f80 2024-05-09T08:25:07.370 INFO:teuthology.orchestra.run.smithi028.stdout:163208757306 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6efc0a2cb0 msgr2=0x7f6efc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6efc0a2cb0 0x7f6efc0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400b830 tx=0x7f6f04009be0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f14102f80 msgr2=0x7f6f14074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f14102f80 0x7f6f14074b60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800d950 tx=0x7f6f0800dd10 comp rx=0 tx=0).stop 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 shutdown_connections 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6efc0a2cb0 0x7f6efc0a5170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f141025d0 0x7f6f14074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f14107660 0x7f6f140750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 --2- 172.21.15.28:0/1863889737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f14102f80 0x7f6f14074b60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 >> 172.21.15.28:0/1863889737 conn(0x7f6f14079480 msgr2=0x7f6f14100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:07.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 shutdown_connections 2024-05-09T08:25:07.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.372+0000 7f6f1b431700 1 -- 172.21.15.28:0/1863889737 wait complete. 2024-05-09T08:25:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.485+0000 7f114a3a8700 1 -- 172.21.15.28:0/1163878166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a9d0 msgr2=0x7f114410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.485+0000 7f114a3a8700 1 --2- 172.21.15.28:0/1163878166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a9d0 0x7f114410f1e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f113c00b0a0 tx=0x7f113c00b3b0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.485+0000 7f114a3a8700 1 -- 172.21.15.28:0/1163878166 shutdown_connections 2024-05-09T08:25:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.485+0000 7f114a3a8700 1 --2- 172.21.15.28:0/1163878166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11441083a0 0x7f1144108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.485+0000 7f114a3a8700 1 --2- 172.21.15.28:0/1163878166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a9d0 0x7f114410f1e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.485+0000 7f114a3a8700 1 --2- 172.21.15.28:0/1163878166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f114406a030 0x7f114406a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.485+0000 7f114a3a8700 1 -- 172.21.15.28:0/1163878166 >> 172.21.15.28:0/1163878166 conn(0x7f11440781d0 msgr2=0x7f11440785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.485+0000 7f114a3a8700 1 -- 172.21.15.28:0/1163878166 shutdown_connections 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.485+0000 7f114a3a8700 1 -- 172.21.15.28:0/1163878166 wait complete. 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.486+0000 7f114a3a8700 1 Processor -- start 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.486+0000 7f114a3a8700 1 -- start start 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f114a3a8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a030 0x7f11441012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f114a3a8700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114406a9d0 0x7f1144101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f114a3a8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11441083a0 0x7f1144101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f114a3a8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11441125f0 con 0x7f114406a030 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f114a3a8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1144112470 con 0x7f11441083a0 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f114a3a8700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1144112770 con 0x7f114406a9d0 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1143fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a030 0x7f11441012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1143fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a030 0x7f11441012f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51568/0 (socket says 172.21.15.28:51568) 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1143fff700 1 -- 172.21.15.28:0/2842672397 learned_addr learned my addr 172.21.15.28:0/2842672397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f113bfff700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114406a9d0 0x7f1144101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1148945700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11441083a0 0x7f1144101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1143fff700 1 -- 172.21.15.28:0/2842672397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114406a9d0 msgr2=0x7f1144101830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1143fff700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114406a9d0 0x7f1144101830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1143fff700 1 -- 172.21.15.28:0/2842672397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11441083a0 msgr2=0x7f1144101d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1143fff700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11441083a0 0x7f1144101d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1143fff700 1 -- 172.21.15.28:0/2842672397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11441a6f90 con 0x7f114406a030 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1143fff700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a030 0x7f11441012f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f113400d980 tx=0x7f113400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1141ffb700 1 -- 172.21.15.28:0/2842672397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134016070 con 0x7f114406a030 2024-05-09T08:25:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11441a7280 con 0x7f114406a030 2024-05-09T08:25:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1141ffb700 1 -- 172.21.15.28:0/2842672397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1134004d10 con 0x7f114406a030 2024-05-09T08:25:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11441a7810 con 0x7f114406a030 2024-05-09T08:25:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.487+0000 7f1141ffb700 1 -- 172.21.15.28:0/2842672397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134005670 con 0x7f114406a030 2024-05-09T08:25:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.488+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1144069010 con 0x7f114406a030 2024-05-09T08:25:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.490+0000 7f1141ffb700 1 -- 172.21.15.28:0/2842672397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1134004750 con 0x7f114406a030 2024-05-09T08:25:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.490+0000 7f1141ffb700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11300a2d30 0x7f11300a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.491+0000 7f113bfff700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11300a2d30 0x7f11300a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.491+0000 7f1141ffb700 1 -- 172.21.15.28:0/2842672397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f11340cf200 con 0x7f114406a030 2024-05-09T08:25:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.491+0000 7f113bfff700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11300a2d30 0x7f11300a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f11440787d0 tx=0x7f112c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.495+0000 7f1141ffb700 1 -- 172.21.15.28:0/2842672397 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f113401f020 con 0x7f114406a030 2024-05-09T08:25:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.708+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f1144102820 con 0x7f114406a030 2024-05-09T08:25:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.709+0000 7f1141ffb700 1 -- 172.21.15.28:0/2842672397 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1134022350 con 0x7f114406a030 2024-05-09T08:25:07.710 INFO:teuthology.orchestra.run.smithi028.stdout:55834574927 2024-05-09T08:25:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11300a2d30 msgr2=0x7f11300a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11300a2d30 0x7f11300a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f11440787d0 tx=0x7f112c009260 comp rx=0 tx=0).stop 2024-05-09T08:25:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a030 msgr2=0x7f11441012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a030 0x7f11441012f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f113400d980 tx=0x7f113400dc90 comp rx=0 tx=0).stop 2024-05-09T08:25:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 shutdown_connections 2024-05-09T08:25:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f11300a2d30 0x7f11300a51f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11441083a0 0x7f1144101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f114406a030 0x7f11441012f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 --2- 172.21.15.28:0/2842672397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f114406a9d0 0x7f1144101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 >> 172.21.15.28:0/2842672397 conn(0x7f11440781d0 msgr2=0x7f11440ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 shutdown_connections 2024-05-09T08:25:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:07.711+0000 7f114a3a8700 1 -- 172.21.15.28:0/2842672397 wait complete. 2024-05-09T08:25:08.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1863889737' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T08:25:08.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2842672397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T08:25:08.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1863889737' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T08:25:08.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2842672397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T08:25:08.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1863889737' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T08:25:08.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2842672397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T08:25:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.887+0000 7f72b81a6700 1 -- 172.21.15.28:0/1069515854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074e70 msgr2=0x7f72b0112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.887+0000 7f72b81a6700 1 --2- 172.21.15.28:0/1069515854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074e70 0x7f72b0112fe0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009990 tx=0x7f72a0009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.887+0000 7f72b81a6700 1 -- 172.21.15.28:0/1069515854 shutdown_connections 2024-05-09T08:25:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.887+0000 7f72b81a6700 1 --2- 172.21.15.28:0/1069515854 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72b0113520 0x7f72b0115910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.887+0000 7f72b81a6700 1 --2- 172.21.15.28:0/1069515854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72b0074550 0x7f72b0074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.887+0000 7f72b81a6700 1 --2- 172.21.15.28:0/1069515854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074e70 0x7f72b0112fe0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.887+0000 7f72b81a6700 1 -- 172.21.15.28:0/1069515854 >> 172.21.15.28:0/1069515854 conn(0x7f72b00794d0 msgr2=0x7f72b00798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:08.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.887+0000 7f72b81a6700 1 -- 172.21.15.28:0/1069515854 shutdown_connections 2024-05-09T08:25:08.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.887+0000 7f72b81a6700 1 -- 172.21.15.28:0/1069515854 wait complete. 2024-05-09T08:25:08.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.888+0000 7f72b81a6700 1 Processor -- start 2024-05-09T08:25:08.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.888+0000 7f72b81a6700 1 -- start start 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.888+0000 7f72b81a6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074550 0x7f72b019f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.888+0000 7f72b81a6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72b0074e70 0x7f72b019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.888+0000 7f72b81a6700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72b0113520 0x7f72b01a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.888+0000 7f72b81a6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72b0118ce0 con 0x7f72b0074550 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.888+0000 7f72b81a6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f72b0118b60 con 0x7f72b0074e70 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.888+0000 7f72b81a6700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f72b0118e60 con 0x7f72b0113520 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.888+0000 7f72b5f42700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074550 0x7f72b019f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b5f42700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074550 0x7f72b019f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51590/0 (socket says 172.21.15.28:51590) 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b5f42700 1 -- 172.21.15.28:0/2405796488 learned_addr learned my addr 172.21.15.28:0/2405796488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b6743700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72b0113520 0x7f72b01a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b5f42700 1 -- 172.21.15.28:0/2405796488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72b0113520 msgr2=0x7f72b01a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b5f42700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72b0113520 0x7f72b01a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b5f42700 1 -- 172.21.15.28:0/2405796488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72b0074e70 msgr2=0x7f72b019f770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:25:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b5f42700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72b0074e70 0x7f72b019f770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b5f42700 1 -- 172.21.15.28:0/2405796488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b01a4240 con 0x7f72b0074550 2024-05-09T08:25:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b5f42700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074550 0x7f72b019f230 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00d950 tx=0x7f72ac00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72a6ffd700 1 -- 172.21.15.28:0/2405796488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac016070 con 0x7f72b0074550 2024-05-09T08:25:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b01a4530 con 0x7f72b0074550 2024-05-09T08:25:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72a6ffd700 1 -- 172.21.15.28:0/2405796488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ac009b40 con 0x7f72b0074550 2024-05-09T08:25:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72a6ffd700 1 -- 172.21.15.28:0/2405796488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac00f620 con 0x7f72b0074550 2024-05-09T08:25:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.889+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b01abdb0 con 0x7f72b0074550 2024-05-09T08:25:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.890+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b0069010 con 0x7f72b0074550 2024-05-09T08:25:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.893+0000 7f72a6ffd700 1 -- 172.21.15.28:0/2405796488 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f72ac019070 con 0x7f72b0074550 2024-05-09T08:25:08.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.893+0000 7f72a6ffd700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f729c0a2cb0 0x7f729c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:08.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.893+0000 7f72b5741700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f729c0a2cb0 0x7f729c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:08.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.894+0000 7f72a6ffd700 1 -- 172.21.15.28:0/2405796488 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72ac0cf600 con 0x7f72b0074550 2024-05-09T08:25:08.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.894+0000 7f72b5741700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f729c0a2cb0 0x7f729c0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72a0005b40 tx=0x7f72a0005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:08.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:08.898+0000 7f72a6ffd700 1 -- 172.21.15.28:0/2405796488 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f72ac09c0a0 con 0x7f72b0074550 2024-05-09T08:25:08.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:08 smithi028 ceph-mon[30119]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-05-09T08:25:09.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:08 smithi165 ceph-mon[34540]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-05-09T08:25:09.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:08 smithi184 ceph-mon[34509]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-05-09T08:25:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.110+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f72b01abfe0 con 0x7f72b0074550 2024-05-09T08:25:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.110+0000 7f72a6ffd700 1 -- 172.21.15.28:0/2405796488 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f72ac09b9a0 con 0x7f72b0074550 2024-05-09T08:25:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:77309411403 2024-05-09T08:25:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.112+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f729c0a2cb0 msgr2=0x7f729c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.112+0000 7f72b81a6700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f729c0a2cb0 0x7f729c0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72a0005b40 tx=0x7f72a0005a90 comp rx=0 tx=0).stop 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074550 msgr2=0x7f72b019f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074550 0x7f72b019f230 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00d950 tx=0x7f72ac00dd10 comp rx=0 tx=0).stop 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 shutdown_connections 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f729c0a2cb0 0x7f729c0a5170 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72b0113520 0x7f72b01a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b0074550 0x7f72b019f230 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 --2- 172.21.15.28:0/2405796488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72b0074e70 0x7f72b019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 >> 172.21.15.28:0/2405796488 conn(0x7f72b00794d0 msgr2=0x7f72b01122e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 shutdown_connections 2024-05-09T08:25:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:09.113+0000 7f72b81a6700 1 -- 172.21.15.28:0/2405796488 wait complete. 2024-05-09T08:25:09.359 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757303 got 163208757306 for osd.5 2024-05-09T08:25:09.359 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:10.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:09 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2405796488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T08:25:10.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:09 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2405796488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T08:25:10.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:09 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2405796488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T08:25:10.570 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411403 for osd.2 2024-05-09T08:25:10.570 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:10.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:10 smithi028 ceph-mon[30119]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:11.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:10 smithi165 ceph-mon[34540]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:11.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:10 smithi184 ceph-mon[34509]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:13.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:12 smithi028 ceph-mon[30119]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-09T08:25:13.084 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574924 got 55834574927 for osd.1 2024-05-09T08:25:13.084 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:13.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:12 smithi165 ceph-mon[34540]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-09T08:25:13.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:12 smithi184 ceph-mon[34509]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-09T08:25:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 -- 172.21.15.28:0/2332512241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 msgr2=0x7ff7fc1077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 --2- 172.21.15.28:0/2332512241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 0x7ff7fc1077c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009990 tx=0x7ff7f8009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 -- 172.21.15.28:0/2332512241 shutdown_connections 2024-05-09T08:25:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 --2- 172.21.15.28:0/2332512241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7fc101680 0x7ff7fc10f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 --2- 172.21.15.28:0/2332512241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 0x7ff7fc1077c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 --2- 172.21.15.28:0/2332512241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7fc100cc0 0x7ff7fc101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 -- 172.21.15.28:0/2332512241 >> 172.21.15.28:0/2332512241 conn(0x7ff7fc076540 msgr2=0x7ff7fc076950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 -- 172.21.15.28:0/2332512241 shutdown_connections 2024-05-09T08:25:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 -- 172.21.15.28:0/2332512241 wait complete. 2024-05-09T08:25:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 Processor -- start 2024-05-09T08:25:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.541+0000 7ff803382700 1 -- start start 2024-05-09T08:25:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff803382700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7fc100cc0 0x7ff7fc19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff803382700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7fc101680 0x7ff7fc19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff803382700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 0x7ff7fc1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff803382700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc1126c0 con 0x7ff7fc1073e0 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff803382700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc112540 con 0x7ff7fc101680 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff803382700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc112840 con 0x7ff7fc100cc0 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80191f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 0x7ff7fc1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80191f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 0x7ff7fc1989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51606/0 (socket says 172.21.15.28:51606) 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80191f700 1 -- 172.21.15.28:0/1779428092 learned_addr learned my addr 172.21.15.28:0/1779428092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80111e700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7fc100cc0 0x7ff7fc19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80091d700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7fc101680 0x7ff7fc19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80191f700 1 -- 172.21.15.28:0/1779428092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7fc100cc0 msgr2=0x7ff7fc19e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80191f700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7fc100cc0 0x7ff7fc19e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80191f700 1 -- 172.21.15.28:0/1779428092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7fc101680 msgr2=0x7ff7fc19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80191f700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7fc101680 0x7ff7fc19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80191f700 1 -- 172.21.15.28:0/1779428092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc199120 con 0x7ff7fc1073e0 2024-05-09T08:25:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80091d700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7fc101680 0x7ff7fc19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.542+0000 7ff80191f700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 0x7ff7fc1989c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400b970 tx=0x7ff7f400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.543+0000 7ff7f27fc700 1 -- 172.21.15.28:0/1779428092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4019070 con 0x7ff7fc1073e0 2024-05-09T08:25:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.543+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7fc199410 con 0x7ff7fc1073e0 2024-05-09T08:25:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.543+0000 7ff7f27fc700 1 -- 172.21.15.28:0/1779428092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f400ce10 con 0x7ff7fc1073e0 2024-05-09T08:25:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.543+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7fc1a56d0 con 0x7ff7fc1073e0 2024-05-09T08:25:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.543+0000 7ff7f27fc700 1 -- 172.21.15.28:0/1779428092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4007370 con 0x7ff7fc1073e0 2024-05-09T08:25:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.544+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7e0005330 con 0x7ff7fc1073e0 2024-05-09T08:25:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.545+0000 7ff7f27fc700 1 -- 172.21.15.28:0/1779428092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff7f401c070 con 0x7ff7fc1073e0 2024-05-09T08:25:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.546+0000 7ff7f27fc700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff7e80a2d30 0x7ff7e80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.547+0000 7ff7f27fc700 1 -- 172.21.15.28:0/1779428092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff7f40ceaa0 con 0x7ff7fc1073e0 2024-05-09T08:25:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.547+0000 7ff80111e700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff7e80a2d30 0x7ff7e80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:14.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.547+0000 7ff80111e700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff7e80a2d30 0x7ff7e80a51f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8006010 tx=0x7ff7f8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.550+0000 7ff7f27fc700 1 -- 172.21.15.28:0/1779428092 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff7f409b650 con 0x7ff7fc1073e0 2024-05-09T08:25:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.760+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7ff7e00058f0 con 0x7ff7fc1073e0 2024-05-09T08:25:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.760+0000 7ff7f27fc700 1 -- 172.21.15.28:0/1779428092 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff7f409af50 con 0x7ff7fc1073e0 2024-05-09T08:25:14.761 INFO:teuthology.orchestra.run.smithi028.stdout:343597383710 2024-05-09T08:25:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff7e80a2d30 msgr2=0x7ff7e80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff7e80a2d30 0x7ff7e80a51f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8006010 tx=0x7ff7f8005960 comp rx=0 tx=0).stop 2024-05-09T08:25:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 msgr2=0x7ff7fc1989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 0x7ff7fc1989c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400b970 tx=0x7ff7f400bc80 comp rx=0 tx=0).stop 2024-05-09T08:25:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 shutdown_connections 2024-05-09T08:25:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff7e80a2d30 0x7ff7e80a51f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7fc100cc0 0x7ff7fc19e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc1073e0 0x7ff7fc1989c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 --2- 172.21.15.28:0/1779428092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7fc101680 0x7ff7fc19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 >> 172.21.15.28:0/1779428092 conn(0x7ff7fc076540 msgr2=0x7ff7fc0ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 shutdown_connections 2024-05-09T08:25:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:14.762+0000 7ff803382700 1 -- 172.21.15.28:0/1779428092 wait complete. 2024-05-09T08:25:14.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:14 smithi028 ceph-mon[30119]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:15.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:14 smithi165 ceph-mon[34540]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:15.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:14 smithi184 ceph-mon[34509]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:15.404 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383705 got 343597383710 for osd.11 2024-05-09T08:25:15.404 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:15.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:15 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1779428092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T08:25:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.965+0000 7ff35fa74700 1 -- 172.21.15.28:0/381384618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 msgr2=0x7ff35810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.965+0000 7ff35fa74700 1 --2- 172.21.15.28:0/381384618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 0x7ff35810f1e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff35000b0a0 tx=0x7ff35000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:25:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.965+0000 7ff35fa74700 1 -- 172.21.15.28:0/381384618 shutdown_connections 2024-05-09T08:25:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.965+0000 7ff35fa74700 1 --2- 172.21.15.28:0/381384618 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3581083a0 0x7ff358108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.965+0000 7ff35fa74700 1 --2- 172.21.15.28:0/381384618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 0x7ff35810f1e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.965+0000 7ff35fa74700 1 --2- 172.21.15.28:0/381384618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35806a030 0x7ff35806a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.965+0000 7ff35fa74700 1 -- 172.21.15.28:0/381384618 >> 172.21.15.28:0/381384618 conn(0x7ff3580781d0 msgr2=0x7ff3580785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.965+0000 7ff35fa74700 1 -- 172.21.15.28:0/381384618 shutdown_connections 2024-05-09T08:25:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.965+0000 7ff35fa74700 1 -- 172.21.15.28:0/381384618 wait complete. 2024-05-09T08:25:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.966+0000 7ff35fa74700 1 Processor -- start 2024-05-09T08:25:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.966+0000 7ff35fa74700 1 -- start start 2024-05-09T08:25:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.966+0000 7ff35fa74700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35806a030 0x7ff3581012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.966+0000 7ff35fa74700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 0x7ff358101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.966+0000 7ff35fa74700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3581083a0 0x7ff358101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.966+0000 7ff35fa74700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff358112620 con 0x7ff35806a9d0 2024-05-09T08:25:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.966+0000 7ff35fa74700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581124a0 con 0x7ff3581083a0 2024-05-09T08:25:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.966+0000 7ff35fa74700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581127a0 con 0x7ff35806a030 2024-05-09T08:25:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d00f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 0x7ff358101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d00f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 0x7ff358101830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38250/0 (socket says 172.21.15.28:38250) 2024-05-09T08:25:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d00f700 1 -- 172.21.15.28:0/2880313488 learned_addr learned my addr 172.21.15.28:0/2880313488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d810700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35806a030 0x7ff3581012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d810700 1 -- 172.21.15.28:0/2880313488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3581083a0 msgr2=0x7ff358101d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d810700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3581083a0 0x7ff358101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d810700 1 -- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 msgr2=0x7ff358101830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d810700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 0x7ff358101830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d810700 1 -- 172.21.15.28:0/2880313488 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3581a7030 con 0x7ff35806a030 2024-05-09T08:25:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d00f700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 0x7ff358101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:25:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.967+0000 7ff35d810700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35806a030 0x7ff3581012f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff35400e9f0 tx=0x7ff35400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.968+0000 7ff34effd700 1 -- 172.21.15.28:0/2880313488 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35400cbd0 con 0x7ff35806a030 2024-05-09T08:25:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.968+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3581a7320 con 0x7ff35806a030 2024-05-09T08:25:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.968+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3581a7860 con 0x7ff35806a030 2024-05-09T08:25:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.968+0000 7ff34effd700 1 -- 172.21.15.28:0/2880313488 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff354004500 con 0x7ff35806a030 2024-05-09T08:25:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.968+0000 7ff34effd700 1 -- 172.21.15.28:0/2880313488 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff354005280 con 0x7ff35806a030 2024-05-09T08:25:15.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.971+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff33c005330 con 0x7ff35806a030 2024-05-09T08:25:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.972+0000 7ff34effd700 1 -- 172.21.15.28:0/2880313488 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff354005420 con 0x7ff35806a030 2024-05-09T08:25:15.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.973+0000 7ff34effd700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff3440a2cb0 0x7ff3440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:15.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.973+0000 7ff34effd700 1 -- 172.21.15.28:0/2880313488 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff3540cf300 con 0x7ff35806a030 2024-05-09T08:25:15.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.973+0000 7ff35d00f700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff3440a2cb0 0x7ff3440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:15.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.974+0000 7ff35d00f700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff3440a2cb0 0x7ff3440a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff3580787f0 tx=0x7ff348009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:15.974 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:15.975 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:15.975 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.978+0000 7ff34effd700 1 -- 172.21.15.28:0/2880313488 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff35409cec0 con 0x7ff35806a030 2024-05-09T08:25:15.981 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:16.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:15 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1779428092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T08:25:16.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:15 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1779428092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T08:25:16.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.197+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff33c005620 con 0x7ff35806a030 2024-05-09T08:25:16.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.197+0000 7ff34effd700 1 -- 172.21.15.28:0/2880313488 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff35409cec0 con 0x7ff35806a030 2024-05-09T08:25:16.197 INFO:teuthology.orchestra.run.smithi028.stdout:219043332146 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff3440a2cb0 msgr2=0x7ff3440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff3440a2cb0 0x7ff3440a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff3580787f0 tx=0x7ff348009260 comp rx=0 tx=0).stop 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35806a030 msgr2=0x7ff3581012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35806a030 0x7ff3581012f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff35400e9f0 tx=0x7ff35400ed00 comp rx=0 tx=0).stop 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 shutdown_connections 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff3440a2cb0 0x7ff3440a5170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35806a030 0x7ff3581012f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3581083a0 0x7ff358101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 --2- 172.21.15.28:0/2880313488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806a9d0 0x7ff358101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 >> 172.21.15.28:0/2880313488 conn(0x7ff3580781d0 msgr2=0x7ff3580ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 shutdown_connections 2024-05-09T08:25:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.200+0000 7ff35fa74700 1 -- 172.21.15.28:0/2880313488 wait complete. 2024-05-09T08:25:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.861+0000 7fb916d43700 1 -- 172.21.15.28:0/3911313057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91006a030 msgr2=0x7fb91006a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.861+0000 7fb916d43700 1 --2- 172.21.15.28:0/3911313057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91006a030 0x7fb91006a490 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb90000b0a0 tx=0x7fb90000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.861+0000 7fb916d43700 1 -- 172.21.15.28:0/3911313057 shutdown_connections 2024-05-09T08:25:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.861+0000 7fb916d43700 1 --2- 172.21.15.28:0/3911313057 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9101083a0 0x7fb910108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.861+0000 7fb916d43700 1 --2- 172.21.15.28:0/3911313057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91006a9d0 0x7fb91010f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.861+0000 7fb916d43700 1 --2- 172.21.15.28:0/3911313057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb91006a030 0x7fb91006a490 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.861+0000 7fb916d43700 1 -- 172.21.15.28:0/3911313057 >> 172.21.15.28:0/3911313057 conn(0x7fb9100781d0 msgr2=0x7fb9100785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.861+0000 7fb916d43700 1 -- 172.21.15.28:0/3911313057 shutdown_connections 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.862+0000 7fb916d43700 1 -- 172.21.15.28:0/3911313057 wait complete. 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.862+0000 7fb916d43700 1 Processor -- start 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.862+0000 7fb916d43700 1 -- start start 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb916d43700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91006a030 0x7fb9101012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb916d43700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91006a9d0 0x7fb910101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb916d43700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9101083a0 0x7fb910101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb916d43700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9101126b0 con 0x7fb9101083a0 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb916d43700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb910112530 con 0x7fb91006a030 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb916d43700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb910112830 con 0x7fb91006a9d0 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb9152e0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9101083a0 0x7fb910101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb9152e0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9101083a0 0x7fb910101d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38226/0 (socket says 172.21.15.28:38226) 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb9152e0700 1 -- 172.21.15.28:0/3473993050 learned_addr learned my addr 172.21.15.28:0/3473993050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb90ffff700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91006a9d0 0x7fb910101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb914adf700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91006a030 0x7fb9101012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb9152e0700 1 -- 172.21.15.28:0/3473993050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91006a9d0 msgr2=0x7fb910101830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb9152e0700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91006a9d0 0x7fb910101830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb9152e0700 1 -- 172.21.15.28:0/3473993050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91006a030 msgr2=0x7fb9101012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb9152e0700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91006a030 0x7fb9101012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb9152e0700 1 -- 172.21.15.28:0/3473993050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9101a7130 con 0x7fb9101083a0 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb914adf700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91006a030 0x7fb9101012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb9152e0700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9101083a0 0x7fb910101d70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800d980 tx=0x7fb8f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.863+0000 7fb90dffb700 1 -- 172.21.15.28:0/3473993050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8016070 con 0x7fb9101083a0 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.864+0000 7fb90dffb700 1 -- 172.21.15.28:0/3473993050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8f8004d10 con 0x7fb9101083a0 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.864+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9101a7420 con 0x7fb9101083a0 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.864+0000 7fb90dffb700 1 -- 172.21.15.28:0/3473993050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f80055e0 con 0x7fb9101083a0 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.864+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9101a7930 con 0x7fb9101083a0 2024-05-09T08:25:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.865+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb91010a910 con 0x7fb9101083a0 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.866+0000 7fb90dffb700 1 -- 172.21.15.28:0/3473993050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb8f8004750 con 0x7fb9101083a0 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.867+0000 7fb90dffb700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8fc0a2cb0 0x7fb8fc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.867+0000 7fb914adf700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8fc0a2cb0 0x7fb8fc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.867+0000 7fb90dffb700 1 -- 172.21.15.28:0/3473993050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb8f80cf7d0 con 0x7fb9101083a0 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.868+0000 7fb914adf700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8fc0a2cb0 0x7fb8fc0a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb908007900 tx=0x7fb908008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:15.870+0000 7fb90dffb700 1 -- 172.21.15.28:0/3473993050 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb8f801f020 con 0x7fb9101083a0 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.106+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb91010a2f0 con 0x7fb9101083a0 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.106+0000 7fb90dffb700 1 -- 172.21.15.28:0/3473993050 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb8f809c270 con 0x7fb9101083a0 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.108+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8fc0a2cb0 msgr2=0x7fb8fc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.108+0000 7fb916d43700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8fc0a2cb0 0x7fb8fc0a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb908007900 tx=0x7fb908008040 comp rx=0 tx=0).stop 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.108+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9101083a0 msgr2=0x7fb910101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.108+0000 7fb916d43700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9101083a0 0x7fb910101d70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800d980 tx=0x7fb8f800dc90 comp rx=0 tx=0).stop 2024-05-09T08:25:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.109+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 shutdown_connections 2024-05-09T08:25:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.109+0000 7fb916d43700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8fc0a2cb0 0x7fb8fc0a5170 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.109+0000 7fb916d43700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9101083a0 0x7fb910101d70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.109+0000 7fb916d43700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91006a030 0x7fb9101012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.109+0000 7fb916d43700 1 --2- 172.21.15.28:0/3473993050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91006a9d0 0x7fb910101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.109+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 >> 172.21.15.28:0/3473993050 conn(0x7fb9100781d0 msgr2=0x7fb9100fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.109+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 shutdown_connections 2024-05-09T08:25:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:16.109+0000 7fb916d43700 1 -- 172.21.15.28:0/3473993050 wait complete. 2024-05-09T08:25:16.582 INFO:teuthology.orchestra.run.smithi028.stdout:107374182469 2024-05-09T08:25:16.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:16 smithi028 ceph-mon[30119]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:25:16.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:16 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3473993050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T08:25:16.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:16 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2880313488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T08:25:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:16 smithi165 ceph-mon[34540]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:25:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:16 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3473993050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T08:25:17.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:16 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2880313488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T08:25:17.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:16 smithi184 ceph-mon[34509]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:25:17.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:16 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3473993050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T08:25:17.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:16 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2880313488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T08:25:17.195 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182463 got 107374182469 for osd.3 2024-05-09T08:25:17.196 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:17.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.393+0000 7f3d41e0f700 1 -- 172.21.15.28:0/2577423344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 msgr2=0x7f3d3c10d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.393+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/2577423344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 0x7f3d3c10d8b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400b0a0 tx=0x7f3d3400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.393+0000 7f3d41e0f700 1 -- 172.21.15.28:0/2577423344 shutdown_connections 2024-05-09T08:25:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.393+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/2577423344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d3c106a90 0x7f3d3c106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.393+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/2577423344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 0x7f3d3c10d8b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.393+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/2577423344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d3c100ce0 0x7f3d3c101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.393+0000 7f3d41e0f700 1 -- 172.21.15.28:0/2577423344 >> 172.21.15.28:0/2577423344 conn(0x7f3d3c076540 msgr2=0x7f3d3c076950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.393+0000 7f3d41e0f700 1 -- 172.21.15.28:0/2577423344 shutdown_connections 2024-05-09T08:25:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.394+0000 7f3d41e0f700 1 -- 172.21.15.28:0/2577423344 wait complete. 2024-05-09T08:25:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.394+0000 7f3d41e0f700 1 Processor -- start 2024-05-09T08:25:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.394+0000 7f3d41e0f700 1 -- start start 2024-05-09T08:25:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d41e0f700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d3c100ce0 0x7f3d3c1025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d41e0f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 0x7f3d3c102af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d41e0f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d3c106a90 0x7f3d3c103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d41e0f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c110530 con 0x7f3d3c1016a0 2024-05-09T08:25:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d41e0f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c1103b0 con 0x7f3d3c106a90 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d41e0f700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c1106b0 con 0x7f3d3c100ce0 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 0x7f3d3c102af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 0x7f3d3c102af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38270/0 (socket says 172.21.15.28:38270) 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3affd700 1 -- 172.21.15.28:0/3741964554 learned_addr learned my addr 172.21.15.28:0/3741964554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3b7fe700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d3c100ce0 0x7f3d3c1025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3bfff700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d3c106a90 0x7f3d3c103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3affd700 1 -- 172.21.15.28:0/3741964554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d3c100ce0 msgr2=0x7f3d3c1025b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3affd700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d3c100ce0 0x7f3d3c1025b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3affd700 1 -- 172.21.15.28:0/3741964554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d3c106a90 msgr2=0x7f3d3c103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3affd700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d3c106a90 0x7f3d3c103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3affd700 1 -- 172.21.15.28:0/3741964554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c1a6f30 con 0x7f3d3c1016a0 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.395+0000 7f3d3affd700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 0x7f3d3c102af0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00d950 tx=0x7f3d2c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.396+0000 7f3d38ff9700 1 -- 172.21.15.28:0/3741964554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c016070 con 0x7f3d3c1016a0 2024-05-09T08:25:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.396+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3c1a7220 con 0x7f3d3c1016a0 2024-05-09T08:25:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.396+0000 7f3d38ff9700 1 -- 172.21.15.28:0/3741964554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d2c00fd70 con 0x7f3d3c1016a0 2024-05-09T08:25:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.396+0000 7f3d38ff9700 1 -- 172.21.15.28:0/3741964554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00e610 con 0x7f3d3c1016a0 2024-05-09T08:25:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.396+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3c1a77b0 con 0x7f3d3c1016a0 2024-05-09T08:25:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.397+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3c069010 con 0x7f3d3c1016a0 2024-05-09T08:25:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.398+0000 7f3d38ff9700 1 -- 172.21.15.28:0/3741964554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3d2c019070 con 0x7f3d3c1016a0 2024-05-09T08:25:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.399+0000 7f3d38ff9700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d280a2cb0 0x7f3d280a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.399+0000 7f3d38ff9700 1 -- 172.21.15.28:0/3741964554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3d2c0cefc0 con 0x7f3d3c1016a0 2024-05-09T08:25:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.399+0000 7f3d3b7fe700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d280a2cb0 0x7f3d280a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.400+0000 7f3d3b7fe700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d280a2cb0 0x7f3d280a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400bcf0 tx=0x7f3d2400b4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.404+0000 7f3d38ff9700 1 -- 172.21.15.28:0/3741964554 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3d2c09ba60 con 0x7f3d3c1016a0 2024-05-09T08:25:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.620+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3d3c1a7ad0 con 0x7f3d3c1016a0 2024-05-09T08:25:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.620+0000 7f3d38ff9700 1 -- 172.21.15.28:0/3741964554 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3d2c09b360 con 0x7f3d3c1016a0 2024-05-09T08:25:17.620 INFO:teuthology.orchestra.run.smithi028.stdout:197568495671 2024-05-09T08:25:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d280a2cb0 msgr2=0x7f3d280a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d280a2cb0 0x7f3d280a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400bcf0 tx=0x7f3d2400b4c0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 msgr2=0x7f3d3c102af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 0x7f3d3c102af0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00d950 tx=0x7f3d2c00dc60 comp rx=0 tx=0).stop 2024-05-09T08:25:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 shutdown_connections 2024-05-09T08:25:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3d280a2cb0 0x7f3d280a5170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d3c100ce0 0x7f3d3c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d3c106a90 0x7f3d3c103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 --2- 172.21.15.28:0/3741964554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1016a0 0x7f3d3c102af0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 >> 172.21.15.28:0/3741964554 conn(0x7f3d3c076540 msgr2=0x7f3d3c0fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 shutdown_connections 2024-05-09T08:25:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.623+0000 7f3d41e0f700 1 -- 172.21.15.28:0/3741964554 wait complete. 2024-05-09T08:25:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.634+0000 7f7aa899c700 1 -- 172.21.15.28:0/3375305263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa41072c0 msgr2=0x7f7aa41076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.634+0000 7f7aa899c700 1 --2- 172.21.15.28:0/3375305263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa41072c0 0x7f7aa41076a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009990 tx=0x7f7a94009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.635+0000 7f7aa899c700 1 -- 172.21.15.28:0/3375305263 shutdown_connections 2024-05-09T08:25:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.635+0000 7f7aa899c700 1 --2- 172.21.15.28:0/3375305263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7aa4101c10 0x7f7aa410e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.635+0000 7f7aa899c700 1 --2- 172.21.15.28:0/3375305263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa41072c0 0x7f7aa41076a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.635+0000 7f7aa899c700 1 --2- 172.21.15.28:0/3375305263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7aa4101250 0x7f7aa41016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.635+0000 7f7aa899c700 1 -- 172.21.15.28:0/3375305263 >> 172.21.15.28:0/3375305263 conn(0x7f7aa4078260 msgr2=0x7f7aa4078670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.635+0000 7f7aa899c700 1 -- 172.21.15.28:0/3375305263 shutdown_connections 2024-05-09T08:25:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.635+0000 7f7aa899c700 1 -- 172.21.15.28:0/3375305263 wait complete. 2024-05-09T08:25:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa899c700 1 Processor -- start 2024-05-09T08:25:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa899c700 1 -- start start 2024-05-09T08:25:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa899c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa4101250 0x7f7aa419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa899c700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7aa4101c10 0x7f7aa419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa899c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7aa41072c0 0x7f7aa4198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa899c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa41126b0 con 0x7f7aa4101250 2024-05-09T08:25:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa899c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa4112530 con 0x7f7aa41072c0 2024-05-09T08:25:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa899c700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa4112830 con 0x7f7aa4101c10 2024-05-09T08:25:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa4101250 0x7f7aa419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa4101250 0x7f7aa419e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38274/0 (socket says 172.21.15.28:38274) 2024-05-09T08:25:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa259c700 1 -- 172.21.15.28:0/476901214 learned_addr learned my addr 172.21.15.28:0/476901214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa1d9b700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7aa4101c10 0x7f7aa419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.636+0000 7f7aa2d9d700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7aa41072c0 0x7f7aa4198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7aa259c700 1 -- 172.21.15.28:0/476901214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7aa4101c10 msgr2=0x7f7aa419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7aa259c700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7aa4101c10 0x7f7aa419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7aa259c700 1 -- 172.21.15.28:0/476901214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7aa41072c0 msgr2=0x7f7aa4198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7aa259c700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7aa41072c0 0x7f7aa4198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7aa259c700 1 -- 172.21.15.28:0/476901214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa4199070 con 0x7f7aa4101250 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7aa1d9b700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7aa4101c10 0x7f7aa419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7aa259c700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa4101250 0x7f7aa419e700 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7a94000c00 tx=0x7f7a9400bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7a937fe700 1 -- 172.21.15.28:0/476901214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94003880 con 0x7f7aa4101250 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa4199300 con 0x7f7aa4101250 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7a937fe700 1 -- 172.21.15.28:0/476901214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a9401f4d0 con 0x7f7aa4101250 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7a937fe700 1 -- 172.21.15.28:0/476901214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9400f6c0 con 0x7f7aa4101250 2024-05-09T08:25:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.637+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa410eac0 con 0x7f7aa4101250 2024-05-09T08:25:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.638+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aa4102970 con 0x7f7aa4101250 2024-05-09T08:25:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.640+0000 7f7a937fe700 1 -- 172.21.15.28:0/476901214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7a94009d10 con 0x7f7aa4101250 2024-05-09T08:25:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.640+0000 7f7a937fe700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7a8c0a2cb0 0x7f7a8c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.641+0000 7f7a937fe700 1 -- 172.21.15.28:0/476901214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7a940d4760 con 0x7f7aa4101250 2024-05-09T08:25:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.641+0000 7f7aa1d9b700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7a8c0a2cb0 0x7f7a8c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.642+0000 7f7aa1d9b700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7a8c0a2cb0 0x7f7a8c0a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7a98005950 tx=0x7f7a980058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.647+0000 7f7a937fe700 1 -- 172.21.15.28:0/476901214 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7a94013070 con 0x7f7aa4101250 2024-05-09T08:25:17.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.869+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7aa4199df0 con 0x7f7aa4101250 2024-05-09T08:25:17.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.869+0000 7f7a937fe700 1 -- 172.21.15.28:0/476901214 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7a940a13a0 con 0x7f7aa4101250 2024-05-09T08:25:17.869 INFO:teuthology.orchestra.run.smithi028.stdout:34359738454 2024-05-09T08:25:17.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7a8c0a2cb0 msgr2=0x7f7a8c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7a8c0a2cb0 0x7f7a8c0a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7a98005950 tx=0x7f7a980058e0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa4101250 msgr2=0x7f7aa419e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:17.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa4101250 0x7f7aa419e700 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7a94000c00 tx=0x7f7a9400bab0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 shutdown_connections 2024-05-09T08:25:17.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7a8c0a2cb0 0x7f7a8c0a5170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7aa41072c0 0x7f7aa4198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa4101250 0x7f7aa419e700 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 --2- 172.21.15.28:0/476901214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7aa4101c10 0x7f7aa419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:17.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 >> 172.21.15.28:0/476901214 conn(0x7f7aa4078260 msgr2=0x7f7aa4100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:17.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 shutdown_connections 2024-05-09T08:25:17.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:17.871+0000 7f7aa899c700 1 -- 172.21.15.28:0/476901214 wait complete. 2024-05-09T08:25:18.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:17 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3741964554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T08:25:18.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:17 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3741964554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T08:25:18.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3741964554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T08:25:18.314 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332140 got 219043332146 for osd.7 2024-05-09T08:25:18.314 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:18.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:18 smithi028 ceph-mon[30119]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:25:18.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:18 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/476901214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T08:25:18.999 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495666 got 197568495671 for osd.6 2024-05-09T08:25:18.999 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:19.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:18 smithi165 ceph-mon[34540]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:25:19.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:18 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/476901214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T08:25:19.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:18 smithi184 ceph-mon[34509]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:25:19.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:18 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/476901214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T08:25:21.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:20 smithi028 ceph-mon[30119]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:21.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:20 smithi028 ceph-mon[30119]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T08:25:21.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:20 smithi165 ceph-mon[34540]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:21.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:20 smithi165 ceph-mon[34540]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T08:25:21.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:20 smithi184 ceph-mon[34509]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:21.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:20 smithi184 ceph-mon[34509]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T08:25:21.354 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738448 got 34359738454 for osd.0 2024-05-09T08:25:21.355 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:21.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:21 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:25:21.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:21 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:25:22.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:21 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:25:22.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:21 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:25:22.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:21 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:25:22.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:21 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:25:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.826+0000 7f5067e31700 1 -- 172.21.15.28:0/2985406424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 msgr2=0x7f5060112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.826+0000 7f5067e31700 1 --2- 172.21.15.28:0/2985406424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 0x7f5060112fe0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5050009990 tx=0x7f5050009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.827+0000 7f5067e31700 1 -- 172.21.15.28:0/2985406424 shutdown_connections 2024-05-09T08:25:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.827+0000 7f5067e31700 1 --2- 172.21.15.28:0/2985406424 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5060113520 0x7f5060115910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.827+0000 7f5067e31700 1 --2- 172.21.15.28:0/2985406424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5060074550 0x7f5060074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.827+0000 7f5067e31700 1 --2- 172.21.15.28:0/2985406424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 0x7f5060112fe0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.827+0000 7f5067e31700 1 -- 172.21.15.28:0/2985406424 >> 172.21.15.28:0/2985406424 conn(0x7f5060079490 msgr2=0x7f50600798a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.827+0000 7f5067e31700 1 -- 172.21.15.28:0/2985406424 shutdown_connections 2024-05-09T08:25:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.827+0000 7f5067e31700 1 -- 172.21.15.28:0/2985406424 wait complete. 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f5067e31700 1 Processor -- start 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f5067e31700 1 -- start start 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f5067e31700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5060074550 0x7f506019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f5067e31700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 0x7f506019f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f5067e31700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5060113520 0x7f50601a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f5067e31700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5060118cd0 con 0x7f5060074e70 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f5067e31700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5060118b50 con 0x7f5060074550 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f5067e31700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5060118e50 con 0x7f5060113520 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f50653cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 0x7f506019f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f50653cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 0x7f506019f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38292/0 (socket says 172.21.15.28:38292) 2024-05-09T08:25:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f50653cc700 1 -- 172.21.15.28:0/1241209985 learned_addr learned my addr 172.21.15.28:0/1241209985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f5065bcd700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5060074550 0x7f506019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.828+0000 7f50663ce700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5060113520 0x7f50601a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f50653cc700 1 -- 172.21.15.28:0/1241209985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5060113520 msgr2=0x7f50601a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f50653cc700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5060113520 0x7f50601a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f50653cc700 1 -- 172.21.15.28:0/1241209985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5060074550 msgr2=0x7f506019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f50653cc700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5060074550 0x7f506019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f50653cc700 1 -- 172.21.15.28:0/1241209985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50601a40b0 con 0x7f5060074e70 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f5065bcd700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5060074550 0x7f506019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f50653cc700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 0x7f506019f5e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5050005910 tx=0x7f50500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f5056ffd700 1 -- 172.21.15.28:0/1241209985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050027070 con 0x7f5060074e70 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50601a4340 con 0x7f5060074e70 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f5056ffd700 1 -- 172.21.15.28:0/1241209985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5050003780 con 0x7f5060074e70 2024-05-09T08:25:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f5056ffd700 1 -- 172.21.15.28:0/1241209985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050022da0 con 0x7f5060074e70 2024-05-09T08:25:22.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.829+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50601abc20 con 0x7f5060074e70 2024-05-09T08:25:22.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.830+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5060075c40 con 0x7f5060074e70 2024-05-09T08:25:22.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.831+0000 7f5056ffd700 1 -- 172.21.15.28:0/1241209985 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f505002a070 con 0x7f5060074e70 2024-05-09T08:25:22.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.832+0000 7f5056ffd700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f504c0a2c20 0x7f504c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:22.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.832+0000 7f5065bcd700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f504c0a2c20 0x7f504c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:22.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.832+0000 7f5056ffd700 1 -- 172.21.15.28:0/1241209985 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f50500d0080 con 0x7f5060074e70 2024-05-09T08:25:22.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.832+0000 7f5065bcd700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f504c0a2c20 0x7f504c0a50e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f505c005fd0 tx=0x7f505c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:22.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:22.837+0000 7f5056ffd700 1 -- 172.21.15.28:0/1241209985 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f505005e740 con 0x7f5060074e70 2024-05-09T08:25:23.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:22 smithi028 ceph-mon[30119]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:25:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.051+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f50601abdf0 con 0x7f5060074e70 2024-05-09T08:25:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.051+0000 7f5056ffd700 1 -- 172.21.15.28:0/1241209985 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5050013070 con 0x7f5060074e70 2024-05-09T08:25:23.051 INFO:teuthology.orchestra.run.smithi028.stdout:253403070511 2024-05-09T08:25:23.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f504c0a2c20 msgr2=0x7f504c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:23.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f504c0a2c20 0x7f504c0a50e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f505c005fd0 tx=0x7f505c005e30 comp rx=0 tx=0).stop 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 msgr2=0x7f506019f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 0x7f506019f5e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5050005910 tx=0x7f50500047b0 comp rx=0 tx=0).stop 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 shutdown_connections 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f504c0a2c20 0x7f504c0a50e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5060113520 0x7f50601a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5060074550 0x7f506019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 --2- 172.21.15.28:0/1241209985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5060074e70 0x7f506019f5e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.053+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 >> 172.21.15.28:0/1241209985 conn(0x7f5060079490 msgr2=0x7f5060112210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.054+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 shutdown_connections 2024-05-09T08:25:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:23.054+0000 7f5067e31700 1 -- 172.21.15.28:0/1241209985 wait complete. 2024-05-09T08:25:23.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:22 smithi165 ceph-mon[34540]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:25:23.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:22 smithi184 ceph-mon[34509]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:25:24.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:23 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1241209985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T08:25:24.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:24.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:24.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:23 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:25:24.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:23 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1241209985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T08:25:24.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:24.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:24.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:23 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:25:24.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:23 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1241209985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T08:25:24.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:24.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:24.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:23 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' 2024-05-09T08:25:25.096 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070506 got 253403070511 for osd.8 2024-05-09T08:25:25.096 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:25.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:25 smithi028 ceph-mon[30119]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-09T08:25:25.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:25 smithi165 ceph-mon[34540]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-09T08:25:25.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:25 smithi184 ceph-mon[34509]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-09T08:25:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.966+0000 7f8cdf048700 1 -- 172.21.15.28:0/1333745512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 msgr2=0x7f8cd810f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:25.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.966+0000 7f8cdf048700 1 --2- 172.21.15.28:0/1333745512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 0x7f8cd810f1f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000b0a0 tx=0x7f8cd000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:25:25.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.966+0000 7f8cdf048700 1 -- 172.21.15.28:0/1333745512 shutdown_connections 2024-05-09T08:25:25.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.966+0000 7f8cdf048700 1 --2- 172.21.15.28:0/1333745512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 0x7f8cd810f1f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:25.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.966+0000 7f8cdf048700 1 --2- 172.21.15.28:0/1333745512 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cd81083b0 0x7f8cd8108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.966+0000 7f8cdf048700 1 --2- 172.21.15.28:0/1333745512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cd8100910 0x7f8cd8100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.966+0000 7f8cdf048700 1 -- 172.21.15.28:0/1333745512 >> 172.21.15.28:0/1333745512 conn(0x7f8cd8077890 msgr2=0x7f8cd80fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.966+0000 7f8cdf048700 1 -- 172.21.15.28:0/1333745512 shutdown_connections 2024-05-09T08:25:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.966+0000 7f8cdf048700 1 -- 172.21.15.28:0/1333745512 wait complete. 2024-05-09T08:25:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdf048700 1 Processor -- start 2024-05-09T08:25:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdf048700 1 -- start start 2024-05-09T08:25:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdf048700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cd8100910 0x7f8cd81a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdf048700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 0x7f8cd81a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdf048700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cd81083b0 0x7f8cd819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdf048700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd81148d0 con 0x7f8cd81012b0 2024-05-09T08:25:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdf048700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd8114750 con 0x7f8cd81083b0 2024-05-09T08:25:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdf048700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd8114a50 con 0x7f8cd8100910 2024-05-09T08:25:25.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cd7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 0x7f8cd81a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:25.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cd7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 0x7f8cd81a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35758/0 (socket says 172.21.15.28:35758) 2024-05-09T08:25:25.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cd7fff700 1 -- 172.21.15.28:0/68080173 learned_addr learned my addr 172.21.15.28:0/68080173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:25.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdd5e5700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cd81083b0 0x7f8cd819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:25.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.967+0000 7f8cdcde4700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cd8100910 0x7f8cd81a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:25.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cd7fff700 1 -- 172.21.15.28:0/68080173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cd8100910 msgr2=0x7f8cd81a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:25.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cd7fff700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cd8100910 0x7f8cd81a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:25.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cd7fff700 1 -- 172.21.15.28:0/68080173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cd81083b0 msgr2=0x7f8cd819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:25.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cd7fff700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cd81083b0 0x7f8cd819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:25.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cd7fff700 1 -- 172.21.15.28:0/68080173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd819b230 con 0x7f8cd81012b0 2024-05-09T08:25:25.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cd7fff700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 0x7f8cd81a0e50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400b530 tx=0x7f8cc400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:25.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cd5ffb700 1 -- 172.21.15.28:0/68080173 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4015020 con 0x7f8cd81012b0 2024-05-09T08:25:25.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cd5ffb700 1 -- 172.21.15.28:0/68080173 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cc4011d10 con 0x7f8cd81012b0 2024-05-09T08:25:25.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd819b520 con 0x7f8cd81012b0 2024-05-09T08:25:25.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cd5ffb700 1 -- 172.21.15.28:0/68080173 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc400f2e0 con 0x7f8cd81012b0 2024-05-09T08:25:25.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.968+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd81a77e0 con 0x7f8cd81012b0 2024-05-09T08:25:25.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.969+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cd8103c30 con 0x7f8cd81012b0 2024-05-09T08:25:25.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.970+0000 7f8cd5ffb700 1 -- 172.21.15.28:0/68080173 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8cc4018020 con 0x7f8cd81012b0 2024-05-09T08:25:25.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.971+0000 7f8cd5ffb700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8cc80a2c20 0x7f8cc80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:25.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.971+0000 7f8cdcde4700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8cc80a2c20 0x7f8cc80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:25.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.971+0000 7f8cd5ffb700 1 -- 172.21.15.28:0/68080173 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8cc40ceac0 con 0x7f8cd81012b0 2024-05-09T08:25:25.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.972+0000 7f8cdcde4700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8cc80a2c20 0x7f8cc80a50e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc005950 tx=0x7f8ccc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:25.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:25.976+0000 7f8cd5ffb700 1 -- 172.21.15.28:0/68080173 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8cc401f020 con 0x7f8cd81012b0 2024-05-09T08:25:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.187+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f8cd802db80 con 0x7f8cd81012b0 2024-05-09T08:25:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.188+0000 7f8cd5ffb700 1 -- 172.21.15.28:0/68080173 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f8cc409b6c0 con 0x7f8cd81012b0 2024-05-09T08:25:26.189 INFO:teuthology.orchestra.run.smithi028.stdout:309237645349 2024-05-09T08:25:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8cc80a2c20 msgr2=0x7f8cc80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8cc80a2c20 0x7f8cc80a50e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc005950 tx=0x7f8ccc00b410 comp rx=0 tx=0).stop 2024-05-09T08:25:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 msgr2=0x7f8cd81a0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 0x7f8cd81a0e50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400b530 tx=0x7f8cc400b840 comp rx=0 tx=0).stop 2024-05-09T08:25:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 shutdown_connections 2024-05-09T08:25:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f8cc80a2c20 0x7f8cc80a50e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cd8100910 0x7f8cd81a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cd81083b0 0x7f8cd819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 --2- 172.21.15.28:0/68080173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd81012b0 0x7f8cd81a0e50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 >> 172.21.15.28:0/68080173 conn(0x7f8cd8077890 msgr2=0x7f8cd80fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 shutdown_connections 2024-05-09T08:25:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.190+0000 7f8cdf048700 1 -- 172.21.15.28:0/68080173 wait complete. 2024-05-09T08:25:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.232+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3854532074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 msgr2=0x7f7b08101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.232+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3854532074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 0x7f7b08101690 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009990 tx=0x7f7afc009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.232+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3854532074 shutdown_connections 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.232+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3854532074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b08101bd0 0x7f7b0810e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.232+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3854532074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b08107280 0x7f7b08107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.232+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3854532074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 0x7f7b08101690 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.232+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3854532074 >> 172.21.15.28:0/3854532074 conn(0x7f7b08078190 msgr2=0x7f7b080785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.232+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3854532074 shutdown_connections 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.232+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3854532074 wait complete. 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b0d4c8700 1 Processor -- start 2024-05-09T08:25:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b0d4c8700 1 -- start start 2024-05-09T08:25:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b0d4c8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 0x7f7b0819d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b0d4c8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b08101bd0 0x7f7b0819dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b0d4c8700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b08107280 0x7f7b081978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b0d4c8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b08111570 con 0x7f7b08101210 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b0d4c8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7b081113f0 con 0x7f7b08101bd0 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b0d4c8700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b081116f0 con 0x7f7b08107280 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b06ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 0x7f7b0819d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b06ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 0x7f7b0819d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35772/0 (socket says 172.21.15.28:35772) 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b06ffd700 1 -- 172.21.15.28:0/3696118734 learned_addr learned my addr 172.21.15.28:0/3696118734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b067fc700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b08101bd0 0x7f7b0819dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.233+0000 7f7b077fe700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b08107280 0x7f7b081978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7b06ffd700 1 -- 172.21.15.28:0/3696118734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b08107280 msgr2=0x7f7b081978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7b06ffd700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b08107280 0x7f7b081978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7b06ffd700 1 -- 172.21.15.28:0/3696118734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b08101bd0 msgr2=0x7f7b0819dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7b06ffd700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b08101bd0 0x7f7b0819dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7b06ffd700 1 -- 172.21.15.28:0/3696118734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b08198030 con 0x7f7b08101210 2024-05-09T08:25:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7b06ffd700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 0x7f7b0819d6c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7af800d950 tx=0x7f7af800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7aeffff700 1 -- 172.21.15.28:0/3696118734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af8016070 con 0x7f7b08101210 2024-05-09T08:25:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b08198320 con 0x7f7b08101210 2024-05-09T08:25:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7aeffff700 1 -- 172.21.15.28:0/3696118734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7af8009e20 con 0x7f7b08101210 2024-05-09T08:25:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7aeffff700 1 -- 172.21.15.28:0/3696118734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af800f620 con 0x7f7b08101210 2024-05-09T08:25:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.234+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0806a890 con 0x7f7b08101210 2024-05-09T08:25:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.235+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b08069010 con 0x7f7b08101210 2024-05-09T08:25:26.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.236+0000 7f7aeffff700 1 -- 172.21.15.28:0/3696118734 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7af8019070 con 0x7f7b08101210 2024-05-09T08:25:26.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.237+0000 7f7aeffff700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7af00a2c20 0x7f7af00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:26.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.237+0000 7f7b067fc700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7af00a2c20 0x7f7af00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:26.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.237+0000 7f7aeffff700 1 -- 172.21.15.28:0/3696118734 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7af80cefd0 con 0x7f7b08101210 2024-05-09T08:25:26.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.238+0000 7f7b067fc700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7af00a2c20 0x7f7af00a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0098e0 tx=0x7f7afc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.241+0000 7f7aeffff700 1 -- 172.21.15.28:0/3696118734 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7af809bb70 con 0x7f7b08101210 2024-05-09T08:25:26.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.448+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f7b0806b230 con 0x7f7b08101210 2024-05-09T08:25:26.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.448+0000 7f7aeffff700 1 -- 172.21.15.28:0/3696118734 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7af809b470 con 0x7f7b08101210 2024-05-09T08:25:26.449 INFO:teuthology.orchestra.run.smithi028.stdout:287762808874 2024-05-09T08:25:26.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7af00a2c20 msgr2=0x7f7af00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:26.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7af00a2c20 0x7f7af00a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0098e0 tx=0x7f7afc005960 comp rx=0 tx=0).stop 2024-05-09T08:25:26.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 msgr2=0x7f7b0819d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:26.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 0x7f7b0819d6c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7af800d950 tx=0x7f7af800dc60 comp rx=0 tx=0).stop 2024-05-09T08:25:26.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 shutdown_connections 2024-05-09T08:25:26.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7af00a2c20 0x7f7af00a50e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b08107280 0x7f7b081978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b08101210 0x7f7b0819d6c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 --2- 172.21.15.28:0/3696118734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b08101bd0 0x7f7b0819dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:26.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.450+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 >> 172.21.15.28:0/3696118734 conn(0x7f7b08078190 msgr2=0x7f7b080ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:26.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.451+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 shutdown_connections 2024-05-09T08:25:26.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:26.451+0000 7f7b0d4c8700 1 -- 172.21.15.28:0/3696118734 wait complete. 2024-05-09T08:25:27.248 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808869 got 287762808874 for osd.9 2024-05-09T08:25:27.248 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:27.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:27 smithi028 ceph-mon[30119]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-09T08:25:27.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:27 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/68080173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T08:25:27.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:27 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3696118734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T08:25:27.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:27 smithi165 ceph-mon[34540]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-09T08:25:27.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:27 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/68080173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T08:25:27.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:27 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3696118734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T08:25:27.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:27 smithi184 ceph-mon[34509]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-09T08:25:27.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:27 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/68080173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T08:25:27.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:27 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3696118734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T08:25:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 -- 172.21.15.28:0/2824477893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f680006b6e0 msgr2=0x7f6800106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 --2- 172.21.15.28:0/2824477893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f680006b6e0 0x7f6800106fa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f67e80099c0 tx=0x7f67e8009cd0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 -- 172.21.15.28:0/2824477893 shutdown_connections 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 --2- 172.21.15.28:0/2824477893 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f680006ad30 0x7f680006b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 --2- 172.21.15.28:0/2824477893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6800107680 0x7f680010f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 --2- 172.21.15.28:0/2824477893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f680006b6e0 0x7f6800106fa0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 -- 172.21.15.28:0/2824477893 >> 172.21.15.28:0/2824477893 conn(0x7f68000782c0 msgr2=0x7f68000786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 -- 172.21.15.28:0/2824477893 shutdown_connections 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 -- 172.21.15.28:0/2824477893 wait complete. 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.724+0000 7f680595e700 1 Processor -- start 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f680595e700 1 -- start start 2024-05-09T08:25:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f680595e700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f680006ad30 0x7f6800101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f680595e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680006b6e0 0x7f68001017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f680595e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6800107680 0x7f6800101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f680595e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68001126c0 con 0x7f6800107680 2024-05-09T08:25:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f680595e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6800112540 con 0x7f680006b6e0 2024-05-09T08:25:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f680595e700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6800112840 con 0x7f680006ad30 2024-05-09T08:25:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f67ff7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6800107680 0x7f6800101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f67ff7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6800107680 0x7f6800101d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35788/0 (socket says 172.21.15.28:35788) 2024-05-09T08:25:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f67ff7fe700 1 -- 172.21.15.28:0/1850707724 learned_addr learned my addr 172.21.15.28:0/1850707724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f67feffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f680006ad30 0x7f6800101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.725+0000 7f67f67fc700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680006b6e0 0x7f68001017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f67ff7fe700 1 -- 172.21.15.28:0/1850707724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f680006ad30 msgr2=0x7f6800101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f67ff7fe700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f680006ad30 0x7f6800101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f67ff7fe700 1 -- 172.21.15.28:0/1850707724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680006b6e0 msgr2=0x7f68001017d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f67ff7fe700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680006b6e0 0x7f68001017d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f67ff7fe700 1 -- 172.21.15.28:0/1850707724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68001a7060 con 0x7f6800107680 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f67f67fc700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680006b6e0 0x7f68001017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f67feffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f680006ad30 0x7f6800101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f67ff7fe700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6800107680 0x7f6800101d10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f67f800b970 tx=0x7f67f800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f680495c700 1 -- 172.21.15.28:0/1850707724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f8019070 con 0x7f6800107680 2024-05-09T08:25:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f680495c700 1 -- 172.21.15.28:0/1850707724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67f800ce10 con 0x7f6800107680 2024-05-09T08:25:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f680595e700 1 -- 172.21.15.28:0/1850707724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68001a7350 con 0x7f6800107680 2024-05-09T08:25:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.726+0000 7f680495c700 1 -- 172.21.15.28:0/1850707724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f8007370 con 0x7f6800107680 2024-05-09T08:25:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.727+0000 7f680595e700 1 -- 172.21.15.28:0/1850707724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68001a7840 con 0x7f6800107680 2024-05-09T08:25:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.728+0000 7f67f6ffd700 1 -- 172.21.15.28:0/1850707724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67e4005330 con 0x7f6800107680 2024-05-09T08:25:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.729+0000 7f680495c700 1 -- 172.21.15.28:0/1850707724 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f67f801c070 con 0x7f6800107680 2024-05-09T08:25:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.730+0000 7f680495c700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67ec0a2ca0 0x7f67ec0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.730+0000 7f680495c700 1 -- 172.21.15.28:0/1850707724 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f67f80ce700 con 0x7f6800107680 2024-05-09T08:25:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.730+0000 7f67feffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67ec0a2ca0 0x7f67ec0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.731+0000 7f67feffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67ec0a2ca0 0x7f67ec0a5160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f67f0005950 tx=0x7f67f000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.735+0000 7f680495c700 1 -- 172.21.15.28:0/1850707724 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f67f809b300 con 0x7f6800107680 2024-05-09T08:25:27.944 INFO:teuthology.orchestra.run.smithi028.stdout:141733920835 2024-05-09T08:25:27.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.943+0000 7f67f6ffd700 1 -- 172.21.15.28:0/1850707724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f67e4005620 con 0x7f6800107680 2024-05-09T08:25:27.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.943+0000 7f680495c700 1 -- 172.21.15.28:0/1850707724 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f67f809ac00 con 0x7f6800107680 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 -- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67ec0a2ca0 msgr2=0x7f67ec0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67ec0a2ca0 0x7f67ec0a5160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f67f0005950 tx=0x7f67f000a300 comp rx=0 tx=0).stop 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 -- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6800107680 msgr2=0x7f6800101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6800107680 0x7f6800101d10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f67f800b970 tx=0x7f67f800bc80 comp rx=0 tx=0).stop 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 -- 172.21.15.28:0/1850707724 shutdown_connections 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f67ec0a2ca0 0x7f67ec0a5160 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f680006ad30 0x7f6800101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6800107680 0x7f6800101d10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 --2- 172.21.15.28:0/1850707724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680006b6e0 0x7f68001017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 -- 172.21.15.28:0/1850707724 >> 172.21.15.28:0/1850707724 conn(0x7f68000782c0 msgr2=0x7f68000ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 -- 172.21.15.28:0/1850707724 shutdown_connections 2024-05-09T08:25:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:27.946+0000 7f67f6ffd700 1 -- 172.21.15.28:0/1850707724 wait complete. 2024-05-09T08:25:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:28 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1850707724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T08:25:28.288 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645344 got 309237645349 for osd.10 2024-05-09T08:25:28.288 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:28.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:28 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1850707724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T08:25:28.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:28 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1850707724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T08:25:28.992 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920829 got 141733920835 for osd.4 2024-05-09T08:25:28.993 DEBUG:teuthology.parallel:result is None 2024-05-09T08:25:28.993 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-09T08:25:28.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-09T08:25:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:29.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:29 smithi165 ceph-mon[34540]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-09T08:25:29.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:29 smithi184 ceph-mon[34509]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-09T08:25:29.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:29 smithi028 ceph-mon[30119]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-09T08:25:31.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:31 smithi028 ceph-mon[30119]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-09T08:25:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.202+0000 7f835d2b0700 1 -- 172.21.15.28:0/4074247645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101220 msgr2=0x7f83581016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.202+0000 7f835d2b0700 1 --2- 172.21.15.28:0/4074247645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101220 0x7f83581016a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f834c009990 tx=0x7f834c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.202+0000 7f835d2b0700 1 -- 172.21.15.28:0/4074247645 shutdown_connections 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.202+0000 7f835d2b0700 1 --2- 172.21.15.28:0/4074247645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8358101be0 0x7f835810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.202+0000 7f835d2b0700 1 --2- 172.21.15.28:0/4074247645 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8358107290 0x7f8358107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.202+0000 7f835d2b0700 1 --2- 172.21.15.28:0/4074247645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101220 0x7f83581016a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.202+0000 7f835d2b0700 1 -- 172.21.15.28:0/4074247645 >> 172.21.15.28:0/4074247645 conn(0x7f83580781d0 msgr2=0x7f83580785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.202+0000 7f835d2b0700 1 -- 172.21.15.28:0/4074247645 shutdown_connections 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.202+0000 7f835d2b0700 1 -- 172.21.15.28:0/4074247645 wait complete. 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f835d2b0700 1 Processor -- start 2024-05-09T08:25:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f835d2b0700 1 -- start start 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f835d2b0700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8358101220 0x7f83581a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f835d2b0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101be0 0x7f83581a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f835d2b0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8358107290 0x7f835819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f835d2b0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8358111580 con 0x7f8358101be0 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f835d2b0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8358111400 con 0x7f8358107290 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f835d2b0700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8358111700 con 0x7f8358101220 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f83567fc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101be0 0x7f83581a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f83567fc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101be0 0x7f83581a0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35812/0 (socket says 172.21.15.28:35812) 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.203+0000 7f83567fc700 1 -- 172.21.15.28:0/3501076272 learned_addr learned my addr 172.21.15.28:0/3501076272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f8356ffd700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8358101220 0x7f83581a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f83567fc700 1 -- 172.21.15.28:0/3501076272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8358101220 msgr2=0x7f83581a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f83577fe700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8358107290 0x7f835819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f83567fc700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8358101220 0x7f83581a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f83567fc700 1 -- 172.21.15.28:0/3501076272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8358107290 msgr2=0x7f835819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f83567fc700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8358107290 0x7f835819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f83567fc700 1 -- 172.21.15.28:0/3501076272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835819b210 con 0x7f8358101be0 2024-05-09T08:25:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f8356ffd700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8358101220 0x7f83581a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f83577fe700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8358107290 0x7f835819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f83567fc700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101be0 0x7f83581a0e30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f834c00bff0 tx=0x7f834c005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f833ffff700 1 -- 172.21.15.28:0/3501076272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c027070 con 0x7f8358101be0 2024-05-09T08:25:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f833ffff700 1 -- 172.21.15.28:0/3501076272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f834c004150 con 0x7f8358101be0 2024-05-09T08:25:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835819b4a0 con 0x7f8358101be0 2024-05-09T08:25:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f833ffff700 1 -- 172.21.15.28:0/3501076272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c00f460 con 0x7f8358101be0 2024-05-09T08:25:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.204+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83581a77c0 con 0x7f8358101be0 2024-05-09T08:25:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.205+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8358069010 con 0x7f8358101be0 2024-05-09T08:25:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.206+0000 7f833ffff700 1 -- 172.21.15.28:0/3501076272 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f834c02a070 con 0x7f8358101be0 2024-05-09T08:25:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.207+0000 7f833ffff700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f83400a2c20 0x7f83400a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.207+0000 7f8356ffd700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f83400a2c20 0x7f83400a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.207+0000 7f833ffff700 1 -- 172.21.15.28:0/3501076272 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f834c0d0c70 con 0x7f8358101be0 2024-05-09T08:25:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.208+0000 7f8356ffd700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f83400a2c20 0x7f83400a50e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f835819bbf0 tx=0x7f8348008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:31.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.211+0000 7f833ffff700 1 -- 172.21.15.28:0/3501076272 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f834c013070 con 0x7f8358101be0 2024-05-09T08:25:31.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:31 smithi165 ceph-mon[34540]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-09T08:25:31.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:31 smithi184 ceph-mon[34509]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-09T08:25:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.410+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f835806a730 con 0x7f83400a2c20 2024-05-09T08:25:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.419+0000 7f833ffff700 1 -- 172.21.15.28:0/3501076272 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340860 (secure 0 0 0) 0x7f835806a730 con 0x7f83400a2c20 2024-05-09T08:25:31.425 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f83400a2c20 msgr2=0x7f83400a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f83400a2c20 0x7f83400a50e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f835819bbf0 tx=0x7f8348008040 comp rx=0 tx=0).stop 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101be0 msgr2=0x7f83581a0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101be0 0x7f83581a0e30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f834c00bff0 tx=0x7f834c005640 comp rx=0 tx=0).stop 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 shutdown_connections 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f83400a2c20 0x7f83400a50e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8358101220 0x7f83581a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8358107290 0x7f835819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 --2- 172.21.15.28:0/3501076272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8358101be0 0x7f83581a0e30 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 >> 172.21.15.28:0/3501076272 conn(0x7f83580781d0 msgr2=0x7f835806a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 shutdown_connections 2024-05-09T08:25:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:31.424+0000 7f835d2b0700 1 -- 172.21.15.28:0/3501076272 wait complete. 2024-05-09T08:25:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-09T08:25:32.125 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-05-09T08:25:29.664316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661161,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":838,"num_read_kb":544,"num_write":431,"num_write_kb":3085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1245015,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":855,"ondisk_log_size":855,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373884,"kb_used_data":12252,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567940,"statfs":{"total":1151940427776,"available":1151557570560,"internally_reserved":0,"allocated":12546048,"data_stored":7969193,"data_compressed":369617,"data_compressed_allocated":499712,"data_compressed_original":2535424,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":57,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003347"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.679680+0000","last_change":"2024-05-09T08:23:51.579564+0000","last_active":"2024-05-09T08:25:19.679680+0000","last_peered":"2024-05-09T08:25:19.679680+0000","last_clean":"2024-05-09T08:25:19.679680+0000","last_became_active":"2024-05-09T08:23:51.578884+0000","last_became_peered":"2024-05-09T08:23:51.578884+0000","last_unstale":"2024-05-09T08:25:19.679680+0000","last_undegraded":"2024-05-09T08:25:19.679680+0000","last_fullsized":"2024-05-09T08:25:19.679680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:40.473483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.673389+0000","last_change":"2024-05-09T08:23:51.574187+0000","last_active":"2024-05-09T08:25:19.673389+0000","last_peered":"2024-05-09T08:25:19.673389+0000","last_clean":"2024-05-09T08:25:19.673389+0000","last_became_active":"2024-05-09T08:23:51.573837+0000","last_became_peered":"2024-05-09T08:23:51.573837+0000","last_unstale":"2024-05-09T08:25:19.673389+0000","last_undegraded":"2024-05-09T08:25:19.673389+0000","last_fullsized":"2024-05-09T08:25:19.673389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:31.448038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.685622+0000","last_change":"2024-05-09T08:23:51.577224+0000","last_active":"2024-05-09T08:25:19.685622+0000","last_peered":"2024-05-09T08:25:19.685622+0000","last_clean":"2024-05-09T08:25:19.685622+0000","last_became_active":"2024-05-09T08:23:51.576822+0000","last_became_peered":"2024-05-09T08:23:51.576822+0000","last_unstale":"2024-05-09T08:25:19.685622+0000","last_undegraded":"2024-05-09T08:25:19.685622+0000","last_fullsized":"2024-05-09T08:25:19.685622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:04.902530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.667664+0000","last_change":"2024-05-09T08:23:51.577172+0000","last_active":"2024-05-09T08:25:19.667664+0000","last_peered":"2024-05-09T08:25:19.667664+0000","last_clean":"2024-05-09T08:25:19.667664+0000","last_became_active":"2024-05-09T08:23:51.576861+0000","last_became_peered":"2024-05-09T08:23:51.576861+0000","last_unstale":"2024-05-09T08:25:19.667664+0000","last_undegraded":"2024-05-09T08:25:19.667664+0000","last_fullsized":"2024-05-09T08:25:19.667664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:10.925739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.675025+0000","last_change":"2024-05-09T08:23:51.579408+0000","last_active":"2024-05-09T08:25:19.675025+0000","last_peered":"2024-05-09T08:25:19.675025+0000","last_clean":"2024-05-09T08:25:19.675025+0000","last_became_active":"2024-05-09T08:23:51.578229+0000","last_became_peered":"2024-05-09T08:23:51.578229+0000","last_unstale":"2024-05-09T08:25:19.675025+0000","last_undegraded":"2024-05-09T08:25:19.675025+0000","last_fullsized":"2024-05-09T08:25:19.675025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:47:27.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.669065+0000","last_change":"2024-05-09T08:23:51.576924+0000","last_active":"2024-05-09T08:25:19.669065+0000","last_peered":"2024-05-09T08:25:19.669065+0000","last_clean":"2024-05-09T08:25:19.669065+0000","last_became_active":"2024-05-09T08:23:51.576601+0000","last_became_peered":"2024-05-09T08:23:51.576601+0000","last_unstale":"2024-05-09T08:25:19.669065+0000","last_undegraded":"2024-05-09T08:25:19.669065+0000","last_fullsized":"2024-05-09T08:25:19.669065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:07:16.910271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.678048+0000","last_change":"2024-05-09T08:23:51.574843+0000","last_active":"2024-05-09T08:25:19.678048+0000","last_peered":"2024-05-09T08:25:19.678048+0000","last_clean":"2024-05-09T08:25:19.678048+0000","last_became_active":"2024-05-09T08:23:51.574498+0000","last_became_peered":"2024-05-09T08:23:51.574498+0000","last_unstale":"2024-05-09T08:25:19.678048+0000","last_undegraded":"2024-05-09T08:25:19.678048+0000","last_fullsized":"2024-05-09T08:25:19.678048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:59:50.974491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.684127+0000","last_change":"2024-05-09T08:23:51.574001+0000","last_active":"2024-05-09T08:25:19.684127+0000","last_peered":"2024-05-09T08:25:19.684127+0000","last_clean":"2024-05-09T08:25:19.684127+0000","last_became_active":"2024-05-09T08:23:51.573632+0000","last_became_peered":"2024-05-09T08:23:51.573632+0000","last_unstale":"2024-05-09T08:25:19.684127+0000","last_undegraded":"2024-05-09T08:25:19.684127+0000","last_fullsized":"2024-05-09T08:25:19.684127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:55:16.418951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.678169+0000","last_change":"2024-05-09T08:23:51.567635+0000","last_active":"2024-05-09T08:25:19.678169+0000","last_peered":"2024-05-09T08:25:19.678169+0000","last_clean":"2024-05-09T08:25:19.678169+0000","last_became_active":"2024-05-09T08:23:51.567283+0000","last_became_peered":"2024-05-09T08:23:51.567283+0000","last_unstale":"2024-05-09T08:25:19.678169+0000","last_undegraded":"2024-05-09T08:25:19.678169+0000","last_fullsized":"2024-05-09T08:25:19.678169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:42.346856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.669983+0000","last_change":"2024-05-09T08:23:51.576090+0000","last_active":"2024-05-09T08:25:19.669983+0000","last_peered":"2024-05-09T08:25:19.669983+0000","last_clean":"2024-05-09T08:25:19.669983+0000","last_became_active":"2024-05-09T08:23:51.575768+0000","last_became_peered":"2024-05-09T08:23:51.575768+0000","last_unstale":"2024-05-09T08:25:19.669983+0000","last_undegraded":"2024-05-09T08:25:19.669983+0000","last_fullsized":"2024-05-09T08:25:19.669983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:00:50.016924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.677013+0000","last_change":"2024-05-09T08:23:51.578957+0000","last_active":"2024-05-09T08:25:19.677013+0000","last_peered":"2024-05-09T08:25:19.677013+0000","last_clean":"2024-05-09T08:25:19.677013+0000","last_became_active":"2024-05-09T08:23:51.578210+0000","last_became_peered":"2024-05-09T08:23:51.578210+0000","last_unstale":"2024-05-09T08:25:19.677013+0000","last_undegraded":"2024-05-09T08:25:19.677013+0000","last_fullsized":"2024-05-09T08:25:19.677013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:28:28.953434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.667316+0000","last_change":"2024-05-09T08:23:51.575389+0000","last_active":"2024-05-09T08:25:19.667316+0000","last_peered":"2024-05-09T08:25:19.667316+0000","last_clean":"2024-05-09T08:25:19.667316+0000","last_became_active":"2024-05-09T08:23:51.575071+0000","last_became_peered":"2024-05-09T08:23:51.575071+0000","last_unstale":"2024-05-09T08:25:19.667316+0000","last_undegraded":"2024-05-09T08:25:19.667316+0000","last_fullsized":"2024-05-09T08:25:19.667316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:34.844611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.676541+0000","last_change":"2024-05-09T08:23:51.572574+0000","last_active":"2024-05-09T08:25:19.676541+0000","last_peered":"2024-05-09T08:25:19.676541+0000","last_clean":"2024-05-09T08:25:19.676541+0000","last_became_active":"2024-05-09T08:23:51.572243+0000","last_became_peered":"2024-05-09T08:23:51.572243+0000","last_unstale":"2024-05-09T08:25:19.676541+0000","last_undegraded":"2024-05-09T08:25:19.676541+0000","last_fullsized":"2024-05-09T08:25:19.676541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:52.534365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.677778+0000","last_change":"2024-05-09T08:23:51.574963+0000","last_active":"2024-05-09T08:25:19.677778+0000","last_peered":"2024-05-09T08:25:19.677778+0000","last_clean":"2024-05-09T08:25:19.677778+0000","last_became_active":"2024-05-09T08:23:51.574597+0000","last_became_peered":"2024-05-09T08:23:51.574597+0000","last_unstale":"2024-05-09T08:25:19.677778+0000","last_undegraded":"2024-05-09T08:25:19.677778+0000","last_fullsized":"2024-05-09T08:25:19.677778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:33.348828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.680256+0000","last_change":"2024-05-09T08:23:51.577777+0000","last_active":"2024-05-09T08:25:19.680256+0000","last_peered":"2024-05-09T08:25:19.680256+0000","last_clean":"2024-05-09T08:25:19.680256+0000","last_became_active":"2024-05-09T08:23:51.577073+0000","last_became_peered":"2024-05-09T08:23:51.577073+0000","last_unstale":"2024-05-09T08:25:19.680256+0000","last_undegraded":"2024-05-09T08:25:19.680256+0000","last_fullsized":"2024-05-09T08:25:19.680256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:26.249488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.688063+0000","last_change":"2024-05-09T08:23:51.574476+0000","last_active":"2024-05-09T08:25:19.688063+0000","last_peered":"2024-05-09T08:25:19.688063+0000","last_clean":"2024-05-09T08:25:19.688063+0000","last_became_active":"2024-05-09T08:23:51.572748+0000","last_became_peered":"2024-05-09T08:23:51.572748+0000","last_unstale":"2024-05-09T08:25:19.688063+0000","last_undegraded":"2024-05-09T08:25:19.688063+0000","last_fullsized":"2024-05-09T08:25:19.688063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:43.515419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.679912+0000","last_change":"2024-05-09T08:23:51.580568+0000","last_active":"2024-05-09T08:25:19.679912+0000","last_peered":"2024-05-09T08:25:19.679912+0000","last_clean":"2024-05-09T08:25:19.679912+0000","last_became_active":"2024-05-09T08:23:51.579626+0000","last_became_peered":"2024-05-09T08:23:51.579626+0000","last_unstale":"2024-05-09T08:25:19.679912+0000","last_undegraded":"2024-05-09T08:25:19.679912+0000","last_fullsized":"2024-05-09T08:25:19.679912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:28.756079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.680128+0000","last_change":"2024-05-09T08:23:51.574363+0000","last_active":"2024-05-09T08:25:19.680128+0000","last_peered":"2024-05-09T08:25:19.680128+0000","last_clean":"2024-05-09T08:25:19.680128+0000","last_became_active":"2024-05-09T08:23:51.573858+0000","last_became_peered":"2024-05-09T08:23:51.573858+0000","last_unstale":"2024-05-09T08:25:19.680128+0000","last_undegraded":"2024-05-09T08:25:19.680128+0000","last_fullsized":"2024-05-09T08:25:19.680128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:30.752939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.665192+0000","last_change":"2024-05-09T08:23:51.579361+0000","last_active":"2024-05-09T08:25:19.665192+0000","last_peered":"2024-05-09T08:25:19.665192+0000","last_clean":"2024-05-09T08:25:19.665192+0000","last_became_active":"2024-05-09T08:23:51.579025+0000","last_became_peered":"2024-05-09T08:23:51.579025+0000","last_unstale":"2024-05-09T08:25:19.665192+0000","last_undegraded":"2024-05-09T08:25:19.665192+0000","last_fullsized":"2024-05-09T08:25:19.665192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:02.273999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478882+0000","last_change":"2024-05-09T08:23:55.670107+0000","last_active":"2024-05-09T08:24:51.478882+0000","last_peered":"2024-05-09T08:24:51.478882+0000","last_clean":"2024-05-09T08:24:51.478882+0000","last_became_active":"2024-05-09T08:23:55.669621+0000","last_became_peered":"2024-05-09T08:23:55.669621+0000","last_unstale":"2024-05-09T08:24:51.478882+0000","last_undegraded":"2024-05-09T08:24:51.478882+0000","last_fullsized":"2024-05-09T08:24:51.478882+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:00.547137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":128,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:24.532113+0000","last_change":"2024-05-09T08:24:51.498795+0000","last_active":"2024-05-09T08:25:24.532113+0000","last_peered":"2024-05-09T08:25:24.532113+0000","last_clean":"2024-05-09T08:25:24.532113+0000","last_became_active":"2024-05-09T08:24:51.487098+0000","last_became_peered":"2024-05-09T08:24:51.487098+0000","last_unstale":"2024-05-09T08:25:24.532113+0000","last_undegraded":"2024-05-09T08:25:24.532113+0000","last_fullsized":"2024-05-09T08:25:24.532113+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:10.646049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":64,"num_read_kb":64,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.677654+0000","last_change":"2024-05-09T08:23:51.574336+0000","last_active":"2024-05-09T08:25:19.677654+0000","last_peered":"2024-05-09T08:25:19.677654+0000","last_clean":"2024-05-09T08:25:19.677654+0000","last_became_active":"2024-05-09T08:23:51.573801+0000","last_became_peered":"2024-05-09T08:23:51.573801+0000","last_unstale":"2024-05-09T08:25:19.677654+0000","last_undegraded":"2024-05-09T08:25:19.677654+0000","last_fullsized":"2024-05-09T08:25:19.677654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:47.389762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484058+0000","last_change":"2024-05-09T08:23:55.670194+0000","last_active":"2024-05-09T08:24:51.484058+0000","last_peered":"2024-05-09T08:24:51.484058+0000","last_clean":"2024-05-09T08:24:51.484058+0000","last_became_active":"2024-05-09T08:23:55.669762+0000","last_became_peered":"2024-05-09T08:23:55.669762+0000","last_unstale":"2024-05-09T08:24:51.484058+0000","last_undegraded":"2024-05-09T08:24:51.484058+0000","last_fullsized":"2024-05-09T08:24:51.484058+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:22.879635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484304+0000","last_change":"2024-05-09T08:23:50.574421+0000","last_active":"2024-05-09T08:24:51.484304+0000","last_peered":"2024-05-09T08:24:51.484304+0000","last_clean":"2024-05-09T08:24:51.484304+0000","last_became_active":"2024-05-09T08:23:50.574070+0000","last_became_peered":"2024-05-09T08:23:50.574070+0000","last_unstale":"2024-05-09T08:24:51.484304+0000","last_undegraded":"2024-05-09T08:24:51.484304+0000","last_fullsized":"2024-05-09T08:24:51.484304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:22.996034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.685731+0000","last_change":"2024-05-09T08:23:51.583275+0000","last_active":"2024-05-09T08:25:19.685731+0000","last_peered":"2024-05-09T08:25:19.685731+0000","last_clean":"2024-05-09T08:25:19.685731+0000","last_became_active":"2024-05-09T08:23:51.582847+0000","last_became_peered":"2024-05-09T08:23:51.582847+0000","last_unstale":"2024-05-09T08:25:19.685731+0000","last_undegraded":"2024-05-09T08:25:19.685731+0000","last_fullsized":"2024-05-09T08:25:19.685731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:03:03.512852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485323+0000","last_change":"2024-05-09T08:23:55.672256+0000","last_active":"2024-05-09T08:24:51.485323+0000","last_peered":"2024-05-09T08:24:51.485323+0000","last_clean":"2024-05-09T08:24:51.485323+0000","last_became_active":"2024-05-09T08:23:55.671855+0000","last_became_peered":"2024-05-09T08:23:55.671855+0000","last_unstale":"2024-05-09T08:24:51.485323+0000","last_undegraded":"2024-05-09T08:24:51.485323+0000","last_fullsized":"2024-05-09T08:24:51.485323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:45:12.199359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481664+0000","last_change":"2024-05-09T08:23:50.566686+0000","last_active":"2024-05-09T08:24:51.481664+0000","last_peered":"2024-05-09T08:24:51.481664+0000","last_clean":"2024-05-09T08:24:51.481664+0000","last_became_active":"2024-05-09T08:23:50.565910+0000","last_became_peered":"2024-05-09T08:23:50.565910+0000","last_unstale":"2024-05-09T08:24:51.481664+0000","last_undegraded":"2024-05-09T08:24:51.481664+0000","last_fullsized":"2024-05-09T08:24:51.481664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:02.211551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682187+0000","last_change":"2024-05-09T08:23:51.570972+0000","last_active":"2024-05-09T08:25:19.682187+0000","last_peered":"2024-05-09T08:25:19.682187+0000","last_clean":"2024-05-09T08:25:19.682187+0000","last_became_active":"2024-05-09T08:23:51.570638+0000","last_became_peered":"2024-05-09T08:23:51.570638+0000","last_unstale":"2024-05-09T08:25:19.682187+0000","last_undegraded":"2024-05-09T08:25:19.682187+0000","last_fullsized":"2024-05-09T08:25:19.682187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:38.738820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480699+0000","last_change":"2024-05-09T08:23:55.671929+0000","last_active":"2024-05-09T08:24:51.480699+0000","last_peered":"2024-05-09T08:24:51.480699+0000","last_clean":"2024-05-09T08:24:51.480699+0000","last_became_active":"2024-05-09T08:23:55.671483+0000","last_became_peered":"2024-05-09T08:23:55.671483+0000","last_unstale":"2024-05-09T08:24:51.480699+0000","last_undegraded":"2024-05-09T08:24:51.480699+0000","last_fullsized":"2024-05-09T08:24:51.480699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:21.527266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481776+0000","last_change":"2024-05-09T08:23:50.565644+0000","last_active":"2024-05-09T08:24:51.481776+0000","last_peered":"2024-05-09T08:24:51.481776+0000","last_clean":"2024-05-09T08:24:51.481776+0000","last_became_active":"2024-05-09T08:23:50.565265+0000","last_became_peered":"2024-05-09T08:23:50.565265+0000","last_unstale":"2024-05-09T08:24:51.481776+0000","last_undegraded":"2024-05-09T08:24:51.481776+0000","last_fullsized":"2024-05-09T08:24:51.481776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:51:57.972535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.671448+0000","last_change":"2024-05-09T08:23:51.571732+0000","last_active":"2024-05-09T08:25:19.671448+0000","last_peered":"2024-05-09T08:25:19.671448+0000","last_clean":"2024-05-09T08:25:19.671448+0000","last_became_active":"2024-05-09T08:23:51.570783+0000","last_became_peered":"2024-05-09T08:23:51.570783+0000","last_unstale":"2024-05-09T08:25:19.671448+0000","last_undegraded":"2024-05-09T08:25:19.671448+0000","last_fullsized":"2024-05-09T08:25:19.671448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:18.515905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479347+0000","last_change":"2024-05-09T08:23:55.666727+0000","last_active":"2024-05-09T08:24:51.479347+0000","last_peered":"2024-05-09T08:24:51.479347+0000","last_clean":"2024-05-09T08:24:51.479347+0000","last_became_active":"2024-05-09T08:23:55.666284+0000","last_became_peered":"2024-05-09T08:23:55.666284+0000","last_unstale":"2024-05-09T08:24:51.479347+0000","last_undegraded":"2024-05-09T08:24:51.479347+0000","last_fullsized":"2024-05-09T08:24:51.479347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:05.355404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:24.510936+0000","last_change":"2024-05-09T08:23:50.573308+0000","last_active":"2024-05-09T08:25:24.510936+0000","last_peered":"2024-05-09T08:25:24.510936+0000","last_clean":"2024-05-09T08:25:24.510936+0000","last_became_active":"2024-05-09T08:23:50.572946+0000","last_became_peered":"2024-05-09T08:23:50.572946+0000","last_unstale":"2024-05-09T08:25:24.510936+0000","last_undegraded":"2024-05-09T08:25:24.510936+0000","last_fullsized":"2024-05-09T08:25:24.510936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:44:35.816620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.671963+0000","last_change":"2024-05-09T08:23:51.571748+0000","last_active":"2024-05-09T08:25:19.671963+0000","last_peered":"2024-05-09T08:25:19.671963+0000","last_clean":"2024-05-09T08:25:19.671963+0000","last_became_active":"2024-05-09T08:23:51.571388+0000","last_became_peered":"2024-05-09T08:23:51.571388+0000","last_unstale":"2024-05-09T08:25:19.671963+0000","last_undegraded":"2024-05-09T08:25:19.671963+0000","last_fullsized":"2024-05-09T08:25:19.671963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:34.916491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479250+0000","last_change":"2024-05-09T08:23:55.674022+0000","last_active":"2024-05-09T08:24:51.479250+0000","last_peered":"2024-05-09T08:24:51.479250+0000","last_clean":"2024-05-09T08:24:51.479250+0000","last_became_active":"2024-05-09T08:23:55.673322+0000","last_became_peered":"2024-05-09T08:23:55.673322+0000","last_unstale":"2024-05-09T08:24:51.479250+0000","last_undegraded":"2024-05-09T08:24:51.479250+0000","last_fullsized":"2024-05-09T08:24:51.479250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:59:14.299886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481002+0000","last_change":"2024-05-09T08:23:50.571235+0000","last_active":"2024-05-09T08:24:51.481002+0000","last_peered":"2024-05-09T08:24:51.481002+0000","last_clean":"2024-05-09T08:24:51.481002+0000","last_became_active":"2024-05-09T08:23:50.570694+0000","last_became_peered":"2024-05-09T08:23:50.570694+0000","last_unstale":"2024-05-09T08:24:51.481002+0000","last_undegraded":"2024-05-09T08:24:51.481002+0000","last_fullsized":"2024-05-09T08:24:51.481002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:22.850248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681478+0000","last_change":"2024-05-09T08:23:51.572387+0000","last_active":"2024-05-09T08:25:19.681478+0000","last_peered":"2024-05-09T08:25:19.681478+0000","last_clean":"2024-05-09T08:25:19.681478+0000","last_became_active":"2024-05-09T08:23:51.572038+0000","last_became_peered":"2024-05-09T08:23:51.572038+0000","last_unstale":"2024-05-09T08:25:19.681478+0000","last_undegraded":"2024-05-09T08:25:19.681478+0000","last_fullsized":"2024-05-09T08:25:19.681478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:38.118125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.489278+0000","last_change":"2024-05-09T08:24:51.489278+0000","last_active":"2024-05-09T08:24:51.489278+0000","last_peered":"2024-05-09T08:24:51.489278+0000","last_clean":"2024-05-09T08:24:51.489278+0000","last_became_active":"2024-05-09T08:24:51.488875+0000","last_became_peered":"2024-05-09T08:24:51.488875+0000","last_unstale":"2024-05-09T08:24:51.489278+0000","last_undegraded":"2024-05-09T08:24:51.489278+0000","last_fullsized":"2024-05-09T08:24:51.489278+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:47:45.306525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.499072+0000","last_change":"2024-05-09T08:23:50.570040+0000","last_active":"2024-05-09T08:24:51.499072+0000","last_peered":"2024-05-09T08:24:51.499072+0000","last_clean":"2024-05-09T08:24:51.499072+0000","last_became_active":"2024-05-09T08:23:50.569665+0000","last_became_peered":"2024-05-09T08:23:50.569665+0000","last_unstale":"2024-05-09T08:24:51.499072+0000","last_undegraded":"2024-05-09T08:24:51.499072+0000","last_fullsized":"2024-05-09T08:24:51.499072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:22:29.639712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.683305+0000","last_change":"2024-05-09T08:23:51.568561+0000","last_active":"2024-05-09T08:25:19.683305+0000","last_peered":"2024-05-09T08:25:19.683305+0000","last_clean":"2024-05-09T08:25:19.683305+0000","last_became_active":"2024-05-09T08:23:51.568196+0000","last_became_peered":"2024-05-09T08:23:51.568196+0000","last_unstale":"2024-05-09T08:25:19.683305+0000","last_undegraded":"2024-05-09T08:25:19.683305+0000","last_fullsized":"2024-05-09T08:25:19.683305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:52.132837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.486358+0000","last_change":"2024-05-09T08:23:55.674960+0000","last_active":"2024-05-09T08:24:51.486358+0000","last_peered":"2024-05-09T08:24:51.486358+0000","last_clean":"2024-05-09T08:24:51.486358+0000","last_became_active":"2024-05-09T08:23:55.674509+0000","last_became_peered":"2024-05-09T08:23:55.674509+0000","last_unstale":"2024-05-09T08:24:51.486358+0000","last_undegraded":"2024-05-09T08:24:51.486358+0000","last_fullsized":"2024-05-09T08:24:51.486358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:22.479867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483898+0000","last_change":"2024-05-09T08:23:50.569896+0000","last_active":"2024-05-09T08:24:51.483898+0000","last_peered":"2024-05-09T08:24:51.483898+0000","last_clean":"2024-05-09T08:24:51.483898+0000","last_became_active":"2024-05-09T08:23:50.569562+0000","last_became_peered":"2024-05-09T08:23:50.569562+0000","last_unstale":"2024-05-09T08:24:51.483898+0000","last_undegraded":"2024-05-09T08:24:51.483898+0000","last_fullsized":"2024-05-09T08:24:51.483898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:25:20.278726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682617+0000","last_change":"2024-05-09T08:23:51.570260+0000","last_active":"2024-05-09T08:25:19.682617+0000","last_peered":"2024-05-09T08:25:19.682617+0000","last_clean":"2024-05-09T08:25:19.682617+0000","last_became_active":"2024-05-09T08:23:51.569837+0000","last_became_peered":"2024-05-09T08:23:51.569837+0000","last_unstale":"2024-05-09T08:25:19.682617+0000","last_undegraded":"2024-05-09T08:25:19.682617+0000","last_fullsized":"2024-05-09T08:25:19.682617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:01:44.805271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483875+0000","last_change":"2024-05-09T08:23:55.676639+0000","last_active":"2024-05-09T08:24:51.483875+0000","last_peered":"2024-05-09T08:24:51.483875+0000","last_clean":"2024-05-09T08:24:51.483875+0000","last_became_active":"2024-05-09T08:23:55.673643+0000","last_became_peered":"2024-05-09T08:23:55.673643+0000","last_unstale":"2024-05-09T08:24:51.483875+0000","last_undegraded":"2024-05-09T08:24:51.483875+0000","last_fullsized":"2024-05-09T08:24:51.483875+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:02.377184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":165,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:25.511722+0000","last_change":"2024-05-09T08:23:50.565745+0000","last_active":"2024-05-09T08:25:25.511722+0000","last_peered":"2024-05-09T08:25:25.511722+0000","last_clean":"2024-05-09T08:25:25.511722+0000","last_became_active":"2024-05-09T08:23:50.565385+0000","last_became_peered":"2024-05-09T08:23:50.565385+0000","last_unstale":"2024-05-09T08:25:25.511722+0000","last_undegraded":"2024-05-09T08:25:25.511722+0000","last_fullsized":"2024-05-09T08:25:25.511722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:05:00.818398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":134,"num_read_kb":134,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681736+0000","last_change":"2024-05-09T08:23:51.572341+0000","last_active":"2024-05-09T08:25:19.681736+0000","last_peered":"2024-05-09T08:25:19.681736+0000","last_clean":"2024-05-09T08:25:19.681736+0000","last_became_active":"2024-05-09T08:23:51.571583+0000","last_became_peered":"2024-05-09T08:23:51.571583+0000","last_unstale":"2024-05-09T08:25:19.681736+0000","last_undegraded":"2024-05-09T08:25:19.681736+0000","last_fullsized":"2024-05-09T08:25:19.681736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:20.695995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478270+0000","last_change":"2024-05-09T08:23:55.677603+0000","last_active":"2024-05-09T08:24:51.478270+0000","last_peered":"2024-05-09T08:24:51.478270+0000","last_clean":"2024-05-09T08:24:51.478270+0000","last_became_active":"2024-05-09T08:23:55.677143+0000","last_became_peered":"2024-05-09T08:23:55.677143+0000","last_unstale":"2024-05-09T08:24:51.478270+0000","last_undegraded":"2024-05-09T08:24:51.478270+0000","last_fullsized":"2024-05-09T08:24:51.478270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:27.850997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485842+0000","last_change":"2024-05-09T08:23:50.565497+0000","last_active":"2024-05-09T08:24:51.485842+0000","last_peered":"2024-05-09T08:24:51.485842+0000","last_clean":"2024-05-09T08:24:51.485842+0000","last_became_active":"2024-05-09T08:23:50.565120+0000","last_became_peered":"2024-05-09T08:23:50.565120+0000","last_unstale":"2024-05-09T08:24:51.485842+0000","last_undegraded":"2024-05-09T08:24:51.485842+0000","last_fullsized":"2024-05-09T08:24:51.485842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:56.141997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.687454+0000","last_change":"2024-05-09T08:23:51.576489+0000","last_active":"2024-05-09T08:25:19.687454+0000","last_peered":"2024-05-09T08:25:19.687454+0000","last_clean":"2024-05-09T08:25:19.687454+0000","last_became_active":"2024-05-09T08:23:51.576145+0000","last_became_peered":"2024-05-09T08:23:51.576145+0000","last_unstale":"2024-05-09T08:25:19.687454+0000","last_undegraded":"2024-05-09T08:25:19.687454+0000","last_fullsized":"2024-05-09T08:25:19.687454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:37.650308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.490957+0000","last_change":"2024-05-09T08:24:51.490957+0000","last_active":"2024-05-09T08:24:51.490957+0000","last_peered":"2024-05-09T08:24:51.490957+0000","last_clean":"2024-05-09T08:24:51.490957+0000","last_became_active":"2024-05-09T08:24:51.490541+0000","last_became_peered":"2024-05-09T08:24:51.490541+0000","last_unstale":"2024-05-09T08:24:51.490957+0000","last_undegraded":"2024-05-09T08:24:51.490957+0000","last_fullsized":"2024-05-09T08:24:51.490957+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:03.168547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483916+0000","last_change":"2024-05-09T08:23:50.567182+0000","last_active":"2024-05-09T08:24:51.483916+0000","last_peered":"2024-05-09T08:24:51.483916+0000","last_clean":"2024-05-09T08:24:51.483916+0000","last_became_active":"2024-05-09T08:23:50.566851+0000","last_became_peered":"2024-05-09T08:23:50.566851+0000","last_unstale":"2024-05-09T08:24:51.483916+0000","last_undegraded":"2024-05-09T08:24:51.483916+0000","last_fullsized":"2024-05-09T08:24:51.483916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:26.540456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.683808+0000","last_change":"2024-05-09T08:23:51.574074+0000","last_active":"2024-05-09T08:25:19.683808+0000","last_peered":"2024-05-09T08:25:19.683808+0000","last_clean":"2024-05-09T08:25:19.683808+0000","last_became_active":"2024-05-09T08:23:51.572587+0000","last_became_peered":"2024-05-09T08:23:51.572587+0000","last_unstale":"2024-05-09T08:25:19.683808+0000","last_undegraded":"2024-05-09T08:25:19.683808+0000","last_fullsized":"2024-05-09T08:25:19.683808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:46.186358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478461+0000","last_change":"2024-05-09T08:23:55.667170+0000","last_active":"2024-05-09T08:24:51.478461+0000","last_peered":"2024-05-09T08:24:51.478461+0000","last_clean":"2024-05-09T08:24:51.478461+0000","last_became_active":"2024-05-09T08:23:55.666707+0000","last_became_peered":"2024-05-09T08:23:55.666707+0000","last_unstale":"2024-05-09T08:24:51.478461+0000","last_undegraded":"2024-05-09T08:24:51.478461+0000","last_fullsized":"2024-05-09T08:24:51.478461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:48.332207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":226,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:25.541256+0000","last_change":"2024-05-09T08:24:51.509495+0000","last_active":"2024-05-09T08:25:25.541256+0000","last_peered":"2024-05-09T08:25:25.541256+0000","last_clean":"2024-05-09T08:25:25.541256+0000","last_became_active":"2024-05-09T08:24:51.491187+0000","last_became_peered":"2024-05-09T08:24:51.491187+0000","last_unstale":"2024-05-09T08:25:25.541256+0000","last_undegraded":"2024-05-09T08:25:25.541256+0000","last_fullsized":"2024-05-09T08:25:25.541256+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:55.320000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":133,"num_read_kb":129,"num_write":40,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":63356,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.665403+0000","last_change":"2024-05-09T08:23:51.577595+0000","last_active":"2024-05-09T08:25:19.665403+0000","last_peered":"2024-05-09T08:25:19.665403+0000","last_clean":"2024-05-09T08:25:19.665403+0000","last_became_active":"2024-05-09T08:23:51.577271+0000","last_became_peered":"2024-05-09T08:23:51.577271+0000","last_unstale":"2024-05-09T08:25:19.665403+0000","last_undegraded":"2024-05-09T08:25:19.665403+0000","last_fullsized":"2024-05-09T08:25:19.665403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:34.679111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484410+0000","last_change":"2024-05-09T08:23:55.671128+0000","last_active":"2024-05-09T08:24:51.484410+0000","last_peered":"2024-05-09T08:24:51.484410+0000","last_clean":"2024-05-09T08:24:51.484410+0000","last_became_active":"2024-05-09T08:23:55.670414+0000","last_became_peered":"2024-05-09T08:23:55.670414+0000","last_unstale":"2024-05-09T08:24:51.484410+0000","last_undegraded":"2024-05-09T08:24:51.484410+0000","last_fullsized":"2024-05-09T08:24:51.484410+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:31:55.307412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479102+0000","last_change":"2024-05-09T08:23:50.568503+0000","last_active":"2024-05-09T08:24:51.479102+0000","last_peered":"2024-05-09T08:24:51.479102+0000","last_clean":"2024-05-09T08:24:51.479102+0000","last_became_active":"2024-05-09T08:23:50.568181+0000","last_became_peered":"2024-05-09T08:23:50.568181+0000","last_unstale":"2024-05-09T08:24:51.479102+0000","last_undegraded":"2024-05-09T08:24:51.479102+0000","last_fullsized":"2024-05-09T08:24:51.479102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:40.883792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681063+0000","last_change":"2024-05-09T08:23:51.573602+0000","last_active":"2024-05-09T08:25:19.681063+0000","last_peered":"2024-05-09T08:25:19.681063+0000","last_clean":"2024-05-09T08:25:19.681063+0000","last_became_active":"2024-05-09T08:23:51.572557+0000","last_became_peered":"2024-05-09T08:23:51.572557+0000","last_unstale":"2024-05-09T08:25:19.681063+0000","last_undegraded":"2024-05-09T08:25:19.681063+0000","last_fullsized":"2024-05-09T08:25:19.681063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.170962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481604+0000","last_change":"2024-05-09T08:23:55.673773+0000","last_active":"2024-05-09T08:24:51.481604+0000","last_peered":"2024-05-09T08:24:51.481604+0000","last_clean":"2024-05-09T08:24:51.481604+0000","last_became_active":"2024-05-09T08:23:55.673276+0000","last_became_peered":"2024-05-09T08:23:55.673276+0000","last_unstale":"2024-05-09T08:24:51.481604+0000","last_undegraded":"2024-05-09T08:24:51.481604+0000","last_fullsized":"2024-05-09T08:24:51.481604+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:59:00.629339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484874+0000","last_change":"2024-05-09T08:23:50.575894+0000","last_active":"2024-05-09T08:24:51.484874+0000","last_peered":"2024-05-09T08:24:51.484874+0000","last_clean":"2024-05-09T08:24:51.484874+0000","last_became_active":"2024-05-09T08:23:50.575555+0000","last_became_peered":"2024-05-09T08:23:50.575555+0000","last_unstale":"2024-05-09T08:24:51.484874+0000","last_undegraded":"2024-05-09T08:24:51.484874+0000","last_fullsized":"2024-05-09T08:24:51.484874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:19.070764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.680809+0000","last_change":"2024-05-09T08:23:51.574125+0000","last_active":"2024-05-09T08:25:19.680809+0000","last_peered":"2024-05-09T08:25:19.680809+0000","last_clean":"2024-05-09T08:25:19.680809+0000","last_became_active":"2024-05-09T08:23:51.573794+0000","last_became_peered":"2024-05-09T08:23:51.573794+0000","last_unstale":"2024-05-09T08:25:19.680809+0000","last_undegraded":"2024-05-09T08:25:19.680809+0000","last_fullsized":"2024-05-09T08:25:19.680809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:19:26.634208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479015+0000","last_change":"2024-05-09T08:23:55.677711+0000","last_active":"2024-05-09T08:24:51.479015+0000","last_peered":"2024-05-09T08:24:51.479015+0000","last_clean":"2024-05-09T08:24:51.479015+0000","last_became_active":"2024-05-09T08:23:55.677058+0000","last_became_peered":"2024-05-09T08:23:55.677058+0000","last_unstale":"2024-05-09T08:24:51.479015+0000","last_undegraded":"2024-05-09T08:24:51.479015+0000","last_fullsized":"2024-05-09T08:24:51.479015+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:26.842145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:26.523730+0000","last_change":"2024-05-09T08:23:50.566413+0000","last_active":"2024-05-09T08:25:26.523730+0000","last_peered":"2024-05-09T08:25:26.523730+0000","last_clean":"2024-05-09T08:25:26.523730+0000","last_became_active":"2024-05-09T08:23:50.566047+0000","last_became_peered":"2024-05-09T08:23:50.566047+0000","last_unstale":"2024-05-09T08:25:26.523730+0000","last_undegraded":"2024-05-09T08:25:26.523730+0000","last_fullsized":"2024-05-09T08:25:26.523730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:15.340947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.676256+0000","last_change":"2024-05-09T08:23:51.576779+0000","last_active":"2024-05-09T08:25:19.676256+0000","last_peered":"2024-05-09T08:25:19.676256+0000","last_clean":"2024-05-09T08:25:19.676256+0000","last_became_active":"2024-05-09T08:23:51.576119+0000","last_became_peered":"2024-05-09T08:23:51.576119+0000","last_unstale":"2024-05-09T08:25:19.676256+0000","last_undegraded":"2024-05-09T08:25:19.676256+0000","last_fullsized":"2024-05-09T08:25:19.676256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:02.250135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.499744+0000","last_change":"2024-05-09T08:23:55.678564+0000","last_active":"2024-05-09T08:24:51.499744+0000","last_peered":"2024-05-09T08:24:51.499744+0000","last_clean":"2024-05-09T08:24:51.499744+0000","last_became_active":"2024-05-09T08:23:55.678128+0000","last_became_peered":"2024-05-09T08:23:55.678128+0000","last_unstale":"2024-05-09T08:24:51.499744+0000","last_undegraded":"2024-05-09T08:24:51.499744+0000","last_fullsized":"2024-05-09T08:24:51.499744+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:13:21.868267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.501243+0000","last_change":"2024-05-09T08:24:51.501243+0000","last_active":"2024-05-09T08:24:51.501243+0000","last_peered":"2024-05-09T08:24:51.501243+0000","last_clean":"2024-05-09T08:24:51.501243+0000","last_became_active":"2024-05-09T08:24:51.490110+0000","last_became_peered":"2024-05-09T08:24:51.490110+0000","last_unstale":"2024-05-09T08:24:51.501243+0000","last_undegraded":"2024-05-09T08:24:51.501243+0000","last_fullsized":"2024-05-09T08:24:51.501243+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:03.086570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.666882+0000","last_change":"2024-05-09T08:23:51.574914+0000","last_active":"2024-05-09T08:25:19.666882+0000","last_peered":"2024-05-09T08:25:19.666882+0000","last_clean":"2024-05-09T08:25:19.666882+0000","last_became_active":"2024-05-09T08:23:51.574380+0000","last_became_peered":"2024-05-09T08:23:51.574380+0000","last_unstale":"2024-05-09T08:25:19.666882+0000","last_undegraded":"2024-05-09T08:25:19.666882+0000","last_fullsized":"2024-05-09T08:25:19.666882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:34.698328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484388+0000","last_change":"2024-05-09T08:23:55.670198+0000","last_active":"2024-05-09T08:24:51.484388+0000","last_peered":"2024-05-09T08:24:51.484388+0000","last_clean":"2024-05-09T08:24:51.484388+0000","last_became_active":"2024-05-09T08:23:55.669798+0000","last_became_peered":"2024-05-09T08:23:55.669798+0000","last_unstale":"2024-05-09T08:24:51.484388+0000","last_undegraded":"2024-05-09T08:24:51.484388+0000","last_fullsized":"2024-05-09T08:24:51.484388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:09.474264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.504286+0000","last_change":"2024-05-09T08:24:51.504286+0000","last_active":"2024-05-09T08:24:51.504286+0000","last_peered":"2024-05-09T08:24:51.504286+0000","last_clean":"2024-05-09T08:24:51.504286+0000","last_became_active":"2024-05-09T08:24:51.489111+0000","last_became_peered":"2024-05-09T08:24:51.489111+0000","last_unstale":"2024-05-09T08:24:51.504286+0000","last_undegraded":"2024-05-09T08:24:51.504286+0000","last_fullsized":"2024-05-09T08:24:51.504286+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:14.975959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.665938+0000","last_change":"2024-05-09T08:23:51.573434+0000","last_active":"2024-05-09T08:25:19.665938+0000","last_peered":"2024-05-09T08:25:19.665938+0000","last_clean":"2024-05-09T08:25:19.665938+0000","last_became_active":"2024-05-09T08:23:51.573081+0000","last_became_peered":"2024-05-09T08:23:51.573081+0000","last_unstale":"2024-05-09T08:25:19.665938+0000","last_undegraded":"2024-05-09T08:25:19.665938+0000","last_fullsized":"2024-05-09T08:25:19.665938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:49.270355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481580+0000","last_change":"2024-05-09T08:23:55.853114+0000","last_active":"2024-05-09T08:24:51.481580+0000","last_peered":"2024-05-09T08:24:51.481580+0000","last_clean":"2024-05-09T08:24:51.481580+0000","last_became_active":"2024-05-09T08:23:55.852640+0000","last_became_peered":"2024-05-09T08:23:55.852640+0000","last_unstale":"2024-05-09T08:24:51.481580+0000","last_undegraded":"2024-05-09T08:24:51.481580+0000","last_fullsized":"2024-05-09T08:24:51.481580+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:55:17.823585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478828+0000","last_change":"2024-05-09T08:23:50.567834+0000","last_active":"2024-05-09T08:24:51.478828+0000","last_peered":"2024-05-09T08:24:51.478828+0000","last_clean":"2024-05-09T08:24:51.478828+0000","last_became_active":"2024-05-09T08:23:50.567538+0000","last_became_peered":"2024-05-09T08:23:50.567538+0000","last_unstale":"2024-05-09T08:24:51.478828+0000","last_undegraded":"2024-05-09T08:24:51.478828+0000","last_fullsized":"2024-05-09T08:24:51.478828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.674352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682001+0000","last_change":"2024-05-09T08:23:51.580219+0000","last_active":"2024-05-09T08:25:19.682001+0000","last_peered":"2024-05-09T08:25:19.682001+0000","last_clean":"2024-05-09T08:25:19.682001+0000","last_became_active":"2024-05-09T08:23:51.579835+0000","last_became_peered":"2024-05-09T08:23:51.579835+0000","last_unstale":"2024-05-09T08:25:19.682001+0000","last_undegraded":"2024-05-09T08:25:19.682001+0000","last_fullsized":"2024-05-09T08:25:19.682001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:15.232318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.499482+0000","last_change":"2024-05-09T08:23:55.674324+0000","last_active":"2024-05-09T08:24:51.499482+0000","last_peered":"2024-05-09T08:24:51.499482+0000","last_clean":"2024-05-09T08:24:51.499482+0000","last_became_active":"2024-05-09T08:23:55.673254+0000","last_became_peered":"2024-05-09T08:23:55.673254+0000","last_unstale":"2024-05-09T08:24:51.499482+0000","last_undegraded":"2024-05-09T08:24:51.499482+0000","last_fullsized":"2024-05-09T08:24:51.499482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:05:23.303987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":102,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:26.535670+0000","last_change":"2024-05-09T08:23:50.571682+0000","last_active":"2024-05-09T08:25:26.535670+0000","last_peered":"2024-05-09T08:25:26.535670+0000","last_clean":"2024-05-09T08:25:26.535670+0000","last_became_active":"2024-05-09T08:23:50.571338+0000","last_became_peered":"2024-05-09T08:23:50.571338+0000","last_unstale":"2024-05-09T08:25:26.535670+0000","last_undegraded":"2024-05-09T08:25:26.535670+0000","last_fullsized":"2024-05-09T08:25:26.535670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:17.578071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.672368+0000","last_change":"2024-05-09T08:24:51.487208+0000","last_active":"2024-05-09T08:25:19.672368+0000","last_peered":"2024-05-09T08:25:19.672368+0000","last_clean":"2024-05-09T08:25:19.672368+0000","last_became_active":"2024-05-09T08:24:51.486665+0000","last_became_peered":"2024-05-09T08:24:51.486665+0000","last_unstale":"2024-05-09T08:25:19.672368+0000","last_undegraded":"2024-05-09T08:25:19.672368+0000","last_fullsized":"2024-05-09T08:25:19.672368+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:01:10.050846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.500000+0000","last_change":"2024-05-09T08:23:55.853086+0000","last_active":"2024-05-09T08:24:51.500000+0000","last_peered":"2024-05-09T08:24:51.500000+0000","last_clean":"2024-05-09T08:24:51.500000+0000","last_became_active":"2024-05-09T08:23:55.852302+0000","last_became_peered":"2024-05-09T08:23:55.852302+0000","last_unstale":"2024-05-09T08:24:51.500000+0000","last_undegraded":"2024-05-09T08:24:51.500000+0000","last_fullsized":"2024-05-09T08:24:51.500000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:42:18.547623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.666524+0000","last_change":"2024-05-09T08:23:51.575522+0000","last_active":"2024-05-09T08:25:19.666524+0000","last_peered":"2024-05-09T08:25:19.666524+0000","last_clean":"2024-05-09T08:25:19.666524+0000","last_became_active":"2024-05-09T08:23:51.575204+0000","last_became_peered":"2024-05-09T08:23:51.575204+0000","last_unstale":"2024-05-09T08:25:19.666524+0000","last_undegraded":"2024-05-09T08:25:19.666524+0000","last_fullsized":"2024-05-09T08:25:19.666524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:57.257435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:26.523048+0000","last_change":"2024-05-09T08:23:50.574996+0000","last_active":"2024-05-09T08:25:26.523048+0000","last_peered":"2024-05-09T08:25:26.523048+0000","last_clean":"2024-05-09T08:25:26.523048+0000","last_became_active":"2024-05-09T08:23:50.574619+0000","last_became_peered":"2024-05-09T08:23:50.574619+0000","last_unstale":"2024-05-09T08:25:26.523048+0000","last_undegraded":"2024-05-09T08:25:26.523048+0000","last_fullsized":"2024-05-09T08:25:26.523048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:44.681314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.493390+0000","last_change":"2024-05-09T08:24:51.493390+0000","last_active":"2024-05-09T08:24:51.493390+0000","last_peered":"2024-05-09T08:24:51.493390+0000","last_clean":"2024-05-09T08:24:51.493390+0000","last_became_active":"2024-05-09T08:24:51.487797+0000","last_became_peered":"2024-05-09T08:24:51.487797+0000","last_unstale":"2024-05-09T08:24:51.493390+0000","last_undegraded":"2024-05-09T08:24:51.493390+0000","last_fullsized":"2024-05-09T08:24:51.493390+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:46.646471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.678468+0000","last_change":"2024-05-09T08:23:51.573944+0000","last_active":"2024-05-09T08:25:19.678468+0000","last_peered":"2024-05-09T08:25:19.678468+0000","last_clean":"2024-05-09T08:25:19.678468+0000","last_became_active":"2024-05-09T08:23:51.573229+0000","last_became_peered":"2024-05-09T08:23:51.573229+0000","last_unstale":"2024-05-09T08:25:19.678468+0000","last_undegraded":"2024-05-09T08:25:19.678468+0000","last_fullsized":"2024-05-09T08:25:19.678468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:53.828582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485419+0000","last_change":"2024-05-09T08:23:50.566415+0000","last_active":"2024-05-09T08:24:51.485419+0000","last_peered":"2024-05-09T08:24:51.485419+0000","last_clean":"2024-05-09T08:24:51.485419+0000","last_became_active":"2024-05-09T08:23:50.565506+0000","last_became_peered":"2024-05-09T08:23:50.565506+0000","last_unstale":"2024-05-09T08:24:51.485419+0000","last_undegraded":"2024-05-09T08:24:51.485419+0000","last_fullsized":"2024-05-09T08:24:51.485419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:15.741565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485306+0000","last_change":"2024-05-09T08:22:38.366126+0000","last_active":"2024-05-09T08:24:51.485306+0000","last_peered":"2024-05-09T08:24:51.485306+0000","last_clean":"2024-05-09T08:24:51.485306+0000","last_became_active":"2024-05-09T08:22:38.365683+0000","last_became_peered":"2024-05-09T08:22:38.365683+0000","last_unstale":"2024-05-09T08:24:51.485306+0000","last_undegraded":"2024-05-09T08:24:51.485306+0000","last_fullsized":"2024-05-09T08:24:51.485306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_clean_scrub_stamp":"2024-05-09T08:19:12.435445+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:04.744060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.486182+0000","last_change":"2024-05-09T08:23:55.853248+0000","last_active":"2024-05-09T08:24:51.486182+0000","last_peered":"2024-05-09T08:24:51.486182+0000","last_clean":"2024-05-09T08:24:51.486182+0000","last_became_active":"2024-05-09T08:23:55.852646+0000","last_became_peered":"2024-05-09T08:23:55.852646+0000","last_unstale":"2024-05-09T08:24:51.486182+0000","last_undegraded":"2024-05-09T08:24:51.486182+0000","last_fullsized":"2024-05-09T08:24:51.486182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:19:54.378185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.678368+0000","last_change":"2024-05-09T08:23:51.574967+0000","last_active":"2024-05-09T08:25:19.678368+0000","last_peered":"2024-05-09T08:25:19.678368+0000","last_clean":"2024-05-09T08:25:19.678368+0000","last_became_active":"2024-05-09T08:23:51.574600+0000","last_became_peered":"2024-05-09T08:23:51.574600+0000","last_unstale":"2024-05-09T08:25:19.678368+0000","last_undegraded":"2024-05-09T08:25:19.678368+0000","last_fullsized":"2024-05-09T08:25:19.678368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:03.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485355+0000","last_change":"2024-05-09T08:23:50.569886+0000","last_active":"2024-05-09T08:24:51.485355+0000","last_peered":"2024-05-09T08:24:51.485355+0000","last_clean":"2024-05-09T08:24:51.485355+0000","last_became_active":"2024-05-09T08:23:50.569018+0000","last_became_peered":"2024-05-09T08:23:50.569018+0000","last_unstale":"2024-05-09T08:24:51.485355+0000","last_undegraded":"2024-05-09T08:24:51.485355+0000","last_fullsized":"2024-05-09T08:24:51.485355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:20.711485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480994+0000","last_change":"2024-05-09T08:23:55.676408+0000","last_active":"2024-05-09T08:24:51.480994+0000","last_peered":"2024-05-09T08:24:51.480994+0000","last_clean":"2024-05-09T08:24:51.480994+0000","last_became_active":"2024-05-09T08:23:55.675964+0000","last_became_peered":"2024-05-09T08:23:55.675964+0000","last_unstale":"2024-05-09T08:24:51.480994+0000","last_undegraded":"2024-05-09T08:24:51.480994+0000","last_fullsized":"2024-05-09T08:24:51.480994+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:29.210602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481491+0000","last_change":"2024-05-09T08:23:50.566946+0000","last_active":"2024-05-09T08:24:51.481491+0000","last_peered":"2024-05-09T08:24:51.481491+0000","last_clean":"2024-05-09T08:24:51.481491+0000","last_became_active":"2024-05-09T08:23:50.566366+0000","last_became_peered":"2024-05-09T08:23:50.566366+0000","last_unstale":"2024-05-09T08:24:51.481491+0000","last_undegraded":"2024-05-09T08:24:51.481491+0000","last_fullsized":"2024-05-09T08:24:51.481491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:37:30.720879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.673914+0000","last_change":"2024-05-09T08:23:51.571520+0000","last_active":"2024-05-09T08:25:19.673914+0000","last_peered":"2024-05-09T08:25:19.673914+0000","last_clean":"2024-05-09T08:25:19.673914+0000","last_became_active":"2024-05-09T08:23:51.571166+0000","last_became_peered":"2024-05-09T08:23:51.571166+0000","last_unstale":"2024-05-09T08:25:19.673914+0000","last_undegraded":"2024-05-09T08:25:19.673914+0000","last_fullsized":"2024-05-09T08:25:19.673914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:02:12.560238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483815+0000","last_change":"2024-05-09T08:23:55.852649+0000","last_active":"2024-05-09T08:24:51.483815+0000","last_peered":"2024-05-09T08:24:51.483815+0000","last_clean":"2024-05-09T08:24:51.483815+0000","last_became_active":"2024-05-09T08:23:55.852180+0000","last_became_peered":"2024-05-09T08:23:55.852180+0000","last_unstale":"2024-05-09T08:24:51.483815+0000","last_undegraded":"2024-05-09T08:24:51.483815+0000","last_fullsized":"2024-05-09T08:24:51.483815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:33.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479218+0000","last_change":"2024-05-09T08:23:50.561639+0000","last_active":"2024-05-09T08:24:51.479218+0000","last_peered":"2024-05-09T08:24:51.479218+0000","last_clean":"2024-05-09T08:24:51.479218+0000","last_became_active":"2024-05-09T08:23:50.561306+0000","last_became_peered":"2024-05-09T08:23:50.561306+0000","last_unstale":"2024-05-09T08:24:51.479218+0000","last_undegraded":"2024-05-09T08:24:51.479218+0000","last_fullsized":"2024-05-09T08:24:51.479218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:46.897965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.683808+0000","last_change":"2024-05-09T08:23:51.573765+0000","last_active":"2024-05-09T08:25:19.683808+0000","last_peered":"2024-05-09T08:25:19.683808+0000","last_clean":"2024-05-09T08:25:19.683808+0000","last_became_active":"2024-05-09T08:23:51.573134+0000","last_became_peered":"2024-05-09T08:23:51.573134+0000","last_unstale":"2024-05-09T08:25:19.683808+0000","last_undegraded":"2024-05-09T08:25:19.683808+0000","last_fullsized":"2024-05-09T08:25:19.683808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:25.334148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478699+0000","last_change":"2024-05-09T08:23:55.663480+0000","last_active":"2024-05-09T08:24:51.478699+0000","last_peered":"2024-05-09T08:24:51.478699+0000","last_clean":"2024-05-09T08:24:51.478699+0000","last_became_active":"2024-05-09T08:23:55.662823+0000","last_became_peered":"2024-05-09T08:23:55.662823+0000","last_unstale":"2024-05-09T08:24:51.478699+0000","last_undegraded":"2024-05-09T08:24:51.478699+0000","last_fullsized":"2024-05-09T08:24:51.478699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:18.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481834+0000","last_change":"2024-05-09T08:23:50.574641+0000","last_active":"2024-05-09T08:24:51.481834+0000","last_peered":"2024-05-09T08:24:51.481834+0000","last_clean":"2024-05-09T08:24:51.481834+0000","last_became_active":"2024-05-09T08:23:50.574266+0000","last_became_peered":"2024-05-09T08:23:50.574266+0000","last_unstale":"2024-05-09T08:24:51.481834+0000","last_undegraded":"2024-05-09T08:24:51.481834+0000","last_fullsized":"2024-05-09T08:24:51.481834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.575211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.684985+0000","last_change":"2024-05-09T08:23:51.583698+0000","last_active":"2024-05-09T08:25:19.684985+0000","last_peered":"2024-05-09T08:25:19.684985+0000","last_clean":"2024-05-09T08:25:19.684985+0000","last_became_active":"2024-05-09T08:23:51.583342+0000","last_became_peered":"2024-05-09T08:23:51.583342+0000","last_unstale":"2024-05-09T08:25:19.684985+0000","last_undegraded":"2024-05-09T08:25:19.684985+0000","last_fullsized":"2024-05-09T08:25:19.684985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:37.444163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481329+0000","last_change":"2024-05-09T08:23:55.853759+0000","last_active":"2024-05-09T08:24:51.481329+0000","last_peered":"2024-05-09T08:24:51.481329+0000","last_clean":"2024-05-09T08:24:51.481329+0000","last_became_active":"2024-05-09T08:23:55.853045+0000","last_became_peered":"2024-05-09T08:23:55.853045+0000","last_unstale":"2024-05-09T08:24:51.481329+0000","last_undegraded":"2024-05-09T08:24:51.481329+0000","last_fullsized":"2024-05-09T08:24:51.481329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:53.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480004+0000","last_change":"2024-05-09T08:23:50.568373+0000","last_active":"2024-05-09T08:24:51.480004+0000","last_peered":"2024-05-09T08:24:51.480004+0000","last_clean":"2024-05-09T08:24:51.480004+0000","last_became_active":"2024-05-09T08:23:50.568015+0000","last_became_peered":"2024-05-09T08:23:50.568015+0000","last_unstale":"2024-05-09T08:24:51.480004+0000","last_undegraded":"2024-05-09T08:24:51.480004+0000","last_fullsized":"2024-05-09T08:24:51.480004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:05:16.391673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.630236+0000","last_change":"2024-05-09T08:23:51.576825+0000","last_active":"2024-05-09T08:25:19.630236+0000","last_peered":"2024-05-09T08:25:19.630236+0000","last_clean":"2024-05-09T08:25:19.630236+0000","last_became_active":"2024-05-09T08:23:51.573350+0000","last_became_peered":"2024-05-09T08:23:51.573350+0000","last_unstale":"2024-05-09T08:25:19.630236+0000","last_undegraded":"2024-05-09T08:25:19.630236+0000","last_fullsized":"2024-05-09T08:25:19.630236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:00.963281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479110+0000","last_change":"2024-05-09T08:23:55.672145+0000","last_active":"2024-05-09T08:24:51.479110+0000","last_peered":"2024-05-09T08:24:51.479110+0000","last_clean":"2024-05-09T08:24:51.479110+0000","last_became_active":"2024-05-09T08:23:55.671743+0000","last_became_peered":"2024-05-09T08:23:55.671743+0000","last_unstale":"2024-05-09T08:24:51.479110+0000","last_undegraded":"2024-05-09T08:24:51.479110+0000","last_fullsized":"2024-05-09T08:24:51.479110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:09:23.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483985+0000","last_change":"2024-05-09T08:23:50.568908+0000","last_active":"2024-05-09T08:24:51.483985+0000","last_peered":"2024-05-09T08:24:51.483985+0000","last_clean":"2024-05-09T08:24:51.483985+0000","last_became_active":"2024-05-09T08:23:50.567009+0000","last_became_peered":"2024-05-09T08:23:50.567009+0000","last_unstale":"2024-05-09T08:24:51.483985+0000","last_undegraded":"2024-05-09T08:24:51.483985+0000","last_fullsized":"2024-05-09T08:24:51.483985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:35.190156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.660457+0000","last_change":"2024-05-09T08:23:51.580121+0000","last_active":"2024-05-09T08:25:19.660457+0000","last_peered":"2024-05-09T08:25:19.660457+0000","last_clean":"2024-05-09T08:25:19.660457+0000","last_became_active":"2024-05-09T08:23:51.579402+0000","last_became_peered":"2024-05-09T08:23:51.579402+0000","last_unstale":"2024-05-09T08:25:19.660457+0000","last_undegraded":"2024-05-09T08:25:19.660457+0000","last_fullsized":"2024-05-09T08:25:19.660457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:11:26.144344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481111+0000","last_change":"2024-05-09T08:23:55.664017+0000","last_active":"2024-05-09T08:24:51.481111+0000","last_peered":"2024-05-09T08:24:51.481111+0000","last_clean":"2024-05-09T08:24:51.481111+0000","last_became_active":"2024-05-09T08:23:55.663546+0000","last_became_peered":"2024-05-09T08:23:55.663546+0000","last_unstale":"2024-05-09T08:24:51.481111+0000","last_undegraded":"2024-05-09T08:24:51.481111+0000","last_fullsized":"2024-05-09T08:24:51.481111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:13:14.705971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.493220+0000","last_change":"2024-05-09T08:24:51.493220+0000","last_active":"2024-05-09T08:24:51.493220+0000","last_peered":"2024-05-09T08:24:51.493220+0000","last_clean":"2024-05-09T08:24:51.493220+0000","last_became_active":"2024-05-09T08:24:51.486332+0000","last_became_peered":"2024-05-09T08:24:51.486332+0000","last_unstale":"2024-05-09T08:24:51.493220+0000","last_undegraded":"2024-05-09T08:24:51.493220+0000","last_fullsized":"2024-05-09T08:24:51.493220+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:23.694545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.687747+0000","last_change":"2024-05-09T08:23:51.571414+0000","last_active":"2024-05-09T08:25:19.687747+0000","last_peered":"2024-05-09T08:25:19.687747+0000","last_clean":"2024-05-09T08:25:19.687747+0000","last_became_active":"2024-05-09T08:23:51.571085+0000","last_became_peered":"2024-05-09T08:23:51.571085+0000","last_unstale":"2024-05-09T08:25:19.687747+0000","last_undegraded":"2024-05-09T08:25:19.687747+0000","last_fullsized":"2024-05-09T08:25:19.687747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:37.513584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485240+0000","last_change":"2024-05-09T08:23:55.852785+0000","last_active":"2024-05-09T08:24:51.485240+0000","last_peered":"2024-05-09T08:24:51.485240+0000","last_clean":"2024-05-09T08:24:51.485240+0000","last_became_active":"2024-05-09T08:23:55.852252+0000","last_became_peered":"2024-05-09T08:23:55.852252+0000","last_unstale":"2024-05-09T08:24:51.485240+0000","last_undegraded":"2024-05-09T08:24:51.485240+0000","last_fullsized":"2024-05-09T08:24:51.485240+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:44.503295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480830+0000","last_change":"2024-05-09T08:23:50.564787+0000","last_active":"2024-05-09T08:24:51.480830+0000","last_peered":"2024-05-09T08:24:51.480830+0000","last_clean":"2024-05-09T08:24:51.480830+0000","last_became_active":"2024-05-09T08:23:50.564409+0000","last_became_peered":"2024-05-09T08:23:50.564409+0000","last_unstale":"2024-05-09T08:24:51.480830+0000","last_undegraded":"2024-05-09T08:24:51.480830+0000","last_fullsized":"2024-05-09T08:24:51.480830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:23.051221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681007+0000","last_change":"2024-05-09T08:23:51.576830+0000","last_active":"2024-05-09T08:25:19.681007+0000","last_peered":"2024-05-09T08:25:19.681007+0000","last_clean":"2024-05-09T08:25:19.681007+0000","last_became_active":"2024-05-09T08:23:51.576311+0000","last_became_peered":"2024-05-09T08:23:51.576311+0000","last_unstale":"2024-05-09T08:25:19.681007+0000","last_undegraded":"2024-05-09T08:25:19.681007+0000","last_fullsized":"2024-05-09T08:25:19.681007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:26:08.651061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480322+0000","last_change":"2024-05-09T08:23:55.672309+0000","last_active":"2024-05-09T08:24:51.480322+0000","last_peered":"2024-05-09T08:24:51.480322+0000","last_clean":"2024-05-09T08:24:51.480322+0000","last_became_active":"2024-05-09T08:23:55.671811+0000","last_became_peered":"2024-05-09T08:23:55.671811+0000","last_unstale":"2024-05-09T08:24:51.480322+0000","last_undegraded":"2024-05-09T08:24:51.480322+0000","last_fullsized":"2024-05-09T08:24:51.480322+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:19.400949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":102,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:25.523965+0000","last_change":"2024-05-09T08:23:50.575317+0000","last_active":"2024-05-09T08:25:25.523965+0000","last_peered":"2024-05-09T08:25:25.523965+0000","last_clean":"2024-05-09T08:25:25.523965+0000","last_became_active":"2024-05-09T08:23:50.574995+0000","last_became_peered":"2024-05-09T08:23:50.574995+0000","last_unstale":"2024-05-09T08:25:25.523965+0000","last_undegraded":"2024-05-09T08:25:25.523965+0000","last_fullsized":"2024-05-09T08:25:25.523965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:32:54.801799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.677140+0000","last_change":"2024-05-09T08:23:51.575904+0000","last_active":"2024-05-09T08:25:19.677140+0000","last_peered":"2024-05-09T08:25:19.677140+0000","last_clean":"2024-05-09T08:25:19.677140+0000","last_became_active":"2024-05-09T08:23:51.575599+0000","last_became_peered":"2024-05-09T08:23:51.575599+0000","last_unstale":"2024-05-09T08:25:19.677140+0000","last_undegraded":"2024-05-09T08:25:19.677140+0000","last_fullsized":"2024-05-09T08:25:19.677140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:20:13.087096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.486362+0000","last_change":"2024-05-09T08:23:55.680683+0000","last_active":"2024-05-09T08:24:51.486362+0000","last_peered":"2024-05-09T08:24:51.486362+0000","last_clean":"2024-05-09T08:24:51.486362+0000","last_became_active":"2024-05-09T08:23:55.680232+0000","last_became_peered":"2024-05-09T08:23:55.680232+0000","last_unstale":"2024-05-09T08:24:51.486362+0000","last_undegraded":"2024-05-09T08:24:51.486362+0000","last_fullsized":"2024-05-09T08:24:51.486362+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:33.583552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484104+0000","last_change":"2024-05-09T08:23:50.575802+0000","last_active":"2024-05-09T08:24:51.484104+0000","last_peered":"2024-05-09T08:24:51.484104+0000","last_clean":"2024-05-09T08:24:51.484104+0000","last_became_active":"2024-05-09T08:23:50.575416+0000","last_became_peered":"2024-05-09T08:23:50.575416+0000","last_unstale":"2024-05-09T08:24:51.484104+0000","last_undegraded":"2024-05-09T08:24:51.484104+0000","last_fullsized":"2024-05-09T08:24:51.484104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:24:04.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682257+0000","last_change":"2024-05-09T08:23:51.578189+0000","last_active":"2024-05-09T08:25:19.682257+0000","last_peered":"2024-05-09T08:25:19.682257+0000","last_clean":"2024-05-09T08:25:19.682257+0000","last_became_active":"2024-05-09T08:23:51.577500+0000","last_became_peered":"2024-05-09T08:23:51.577500+0000","last_unstale":"2024-05-09T08:25:19.682257+0000","last_undegraded":"2024-05-09T08:25:19.682257+0000","last_fullsized":"2024-05-09T08:25:19.682257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:26:24.718791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.489679+0000","last_change":"2024-05-09T08:24:51.487659+0000","last_active":"2024-05-09T08:24:51.489679+0000","last_peered":"2024-05-09T08:24:51.489679+0000","last_clean":"2024-05-09T08:24:51.489679+0000","last_became_active":"2024-05-09T08:24:51.487212+0000","last_became_peered":"2024-05-09T08:24:51.487212+0000","last_unstale":"2024-05-09T08:24:51.489679+0000","last_undegraded":"2024-05-09T08:24:51.489679+0000","last_fullsized":"2024-05-09T08:24:51.489679+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:29:57.077744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.499956+0000","last_change":"2024-05-09T08:23:50.578438+0000","last_active":"2024-05-09T08:24:51.499956+0000","last_peered":"2024-05-09T08:24:51.499956+0000","last_clean":"2024-05-09T08:24:51.499956+0000","last_became_active":"2024-05-09T08:23:50.578039+0000","last_became_peered":"2024-05-09T08:23:50.578039+0000","last_unstale":"2024-05-09T08:24:51.499956+0000","last_undegraded":"2024-05-09T08:24:51.499956+0000","last_fullsized":"2024-05-09T08:24:51.499956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:59.420193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.670419+0000","last_change":"2024-05-09T08:23:51.568037+0000","last_active":"2024-05-09T08:25:19.670419+0000","last_peered":"2024-05-09T08:25:19.670419+0000","last_clean":"2024-05-09T08:25:19.670419+0000","last_became_active":"2024-05-09T08:23:51.567706+0000","last_became_peered":"2024-05-09T08:23:51.567706+0000","last_unstale":"2024-05-09T08:25:19.670419+0000","last_undegraded":"2024-05-09T08:25:19.670419+0000","last_fullsized":"2024-05-09T08:25:19.670419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:15:19.623805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.663888+0000","last_change":"2024-05-09T08:23:51.579885+0000","last_active":"2024-05-09T08:25:19.663888+0000","last_peered":"2024-05-09T08:25:19.663888+0000","last_clean":"2024-05-09T08:25:19.663888+0000","last_became_active":"2024-05-09T08:23:51.578953+0000","last_became_peered":"2024-05-09T08:23:51.578953+0000","last_unstale":"2024-05-09T08:25:19.663888+0000","last_undegraded":"2024-05-09T08:25:19.663888+0000","last_fullsized":"2024-05-09T08:25:19.663888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:57:23.793119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.669419+0000","last_change":"2024-05-09T08:23:51.583423+0000","last_active":"2024-05-09T08:25:19.669419+0000","last_peered":"2024-05-09T08:25:19.669419+0000","last_clean":"2024-05-09T08:25:19.669419+0000","last_became_active":"2024-05-09T08:23:51.583092+0000","last_became_peered":"2024-05-09T08:23:51.583092+0000","last_unstale":"2024-05-09T08:25:19.669419+0000","last_undegraded":"2024-05-09T08:25:19.669419+0000","last_fullsized":"2024-05-09T08:25:19.669419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:44:47.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.685762+0000","last_change":"2024-05-09T08:23:51.582116+0000","last_active":"2024-05-09T08:25:19.685762+0000","last_peered":"2024-05-09T08:25:19.685762+0000","last_clean":"2024-05-09T08:25:19.685762+0000","last_became_active":"2024-05-09T08:23:51.581715+0000","last_became_peered":"2024-05-09T08:23:51.581715+0000","last_unstale":"2024-05-09T08:25:19.685762+0000","last_undegraded":"2024-05-09T08:25:19.685762+0000","last_fullsized":"2024-05-09T08:25:19.685762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:17.528758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.684388+0000","last_change":"2024-05-09T08:23:51.575959+0000","last_active":"2024-05-09T08:25:19.684388+0000","last_peered":"2024-05-09T08:25:19.684388+0000","last_clean":"2024-05-09T08:25:19.684388+0000","last_became_active":"2024-05-09T08:23:51.575592+0000","last_became_peered":"2024-05-09T08:23:51.575592+0000","last_unstale":"2024-05-09T08:25:19.684388+0000","last_undegraded":"2024-05-09T08:25:19.684388+0000","last_fullsized":"2024-05-09T08:25:19.684388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:26.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.686361+0000","last_change":"2024-05-09T08:23:51.570756+0000","last_active":"2024-05-09T08:25:19.686361+0000","last_peered":"2024-05-09T08:25:19.686361+0000","last_clean":"2024-05-09T08:25:19.686361+0000","last_became_active":"2024-05-09T08:23:51.570412+0000","last_became_peered":"2024-05-09T08:23:51.570412+0000","last_unstale":"2024-05-09T08:25:19.686361+0000","last_undegraded":"2024-05-09T08:25:19.686361+0000","last_fullsized":"2024-05-09T08:25:19.686361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:11:53.959559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.663117+0000","last_change":"2024-05-09T08:23:51.572612+0000","last_active":"2024-05-09T08:25:19.663117+0000","last_peered":"2024-05-09T08:25:19.663117+0000","last_clean":"2024-05-09T08:25:19.663117+0000","last_became_active":"2024-05-09T08:23:51.572227+0000","last_became_peered":"2024-05-09T08:23:51.572227+0000","last_unstale":"2024-05-09T08:25:19.663117+0000","last_undegraded":"2024-05-09T08:25:19.663117+0000","last_fullsized":"2024-05-09T08:25:19.663117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:18.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.661992+0000","last_change":"2024-05-09T08:23:51.573021+0000","last_active":"2024-05-09T08:25:19.661992+0000","last_peered":"2024-05-09T08:25:19.661992+0000","last_clean":"2024-05-09T08:25:19.661992+0000","last_became_active":"2024-05-09T08:23:51.572716+0000","last_became_peered":"2024-05-09T08:23:51.572716+0000","last_unstale":"2024-05-09T08:25:19.661992+0000","last_undegraded":"2024-05-09T08:25:19.661992+0000","last_fullsized":"2024-05-09T08:25:19.661992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:31.494329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.683102+0000","last_change":"2024-05-09T08:23:51.572687+0000","last_active":"2024-05-09T08:25:19.683102+0000","last_peered":"2024-05-09T08:25:19.683102+0000","last_clean":"2024-05-09T08:25:19.683102+0000","last_became_active":"2024-05-09T08:23:51.572339+0000","last_became_peered":"2024-05-09T08:23:51.572339+0000","last_unstale":"2024-05-09T08:25:19.683102+0000","last_undegraded":"2024-05-09T08:25:19.683102+0000","last_fullsized":"2024-05-09T08:25:19.683102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:24.012360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681218+0000","last_change":"2024-05-09T08:23:51.569404+0000","last_active":"2024-05-09T08:25:19.681218+0000","last_peered":"2024-05-09T08:25:19.681218+0000","last_clean":"2024-05-09T08:25:19.681218+0000","last_became_active":"2024-05-09T08:23:51.569044+0000","last_became_peered":"2024-05-09T08:23:51.569044+0000","last_unstale":"2024-05-09T08:25:19.681218+0000","last_undegraded":"2024-05-09T08:25:19.681218+0000","last_fullsized":"2024-05-09T08:25:19.681218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:26:01.343606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.680609+0000","last_change":"2024-05-09T08:23:51.577483+0000","last_active":"2024-05-09T08:25:19.680609+0000","last_peered":"2024-05-09T08:25:19.680609+0000","last_clean":"2024-05-09T08:25:19.680609+0000","last_became_active":"2024-05-09T08:23:51.577078+0000","last_became_peered":"2024-05-09T08:23:51.577078+0000","last_unstale":"2024-05-09T08:25:19.680609+0000","last_undegraded":"2024-05-09T08:25:19.680609+0000","last_fullsized":"2024-05-09T08:25:19.680609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:29.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682945+0000","last_change":"2024-05-09T08:23:51.579736+0000","last_active":"2024-05-09T08:25:19.682945+0000","last_peered":"2024-05-09T08:25:19.682945+0000","last_clean":"2024-05-09T08:25:19.682945+0000","last_became_active":"2024-05-09T08:23:51.579409+0000","last_became_peered":"2024-05-09T08:23:51.579409+0000","last_unstale":"2024-05-09T08:25:19.682945+0000","last_undegraded":"2024-05-09T08:25:19.682945+0000","last_fullsized":"2024-05-09T08:25:19.682945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:05.697642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.686998+0000","last_change":"2024-05-09T08:23:51.569461+0000","last_active":"2024-05-09T08:25:19.686998+0000","last_peered":"2024-05-09T08:25:19.686998+0000","last_clean":"2024-05-09T08:25:19.686998+0000","last_became_active":"2024-05-09T08:23:51.569085+0000","last_became_peered":"2024-05-09T08:23:51.569085+0000","last_unstale":"2024-05-09T08:25:19.686998+0000","last_undegraded":"2024-05-09T08:25:19.686998+0000","last_fullsized":"2024-05-09T08:25:19.686998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:54.554949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.666790+0000","last_change":"2024-05-09T08:23:51.570593+0000","last_active":"2024-05-09T08:25:19.666790+0000","last_peered":"2024-05-09T08:25:19.666790+0000","last_clean":"2024-05-09T08:25:19.666790+0000","last_became_active":"2024-05-09T08:23:51.570195+0000","last_became_peered":"2024-05-09T08:23:51.570195+0000","last_unstale":"2024-05-09T08:25:19.666790+0000","last_undegraded":"2024-05-09T08:25:19.666790+0000","last_fullsized":"2024-05-09T08:25:19.666790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:18.376268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283814,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70559,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":702,"num_read_kb":425,"num_write":248,"num_write_kb":145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":64162,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":211677,"data_compressed":13616,"data_compressed_allocated":20480,"data_compressed_original":77824,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":501447,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32304,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712848,"statfs":{"total":95995035648,"available":95961956352,"internally_reserved":0,"allocated":1155072,"data_stored":1156474,"data_compressed":58336,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":501355,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":501039,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1098371,"data_compressed":56735,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496859,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":562166,"data_compressed":22245,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":498315,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":497997,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":496567,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":565182,"data_compressed":31058,"data_compressed_allocated":40960,"data_compressed_original":118784,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1085440,"data_stored":1093421,"data_compressed":56735,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66029,"data_compressed":1601,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":64949,"data_compressed":10414,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":65873,"data_compressed":1601,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:25:32.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-09T08:25:32.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:32 smithi165 ceph-mon[34540]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:25:32.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:32 smithi184 ceph-mon[34509]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:25:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:32.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:32 smithi028 ceph-mon[30119]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:25:33.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:33 smithi165 ceph-mon[34540]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-09T08:25:33.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:33 smithi184 ceph-mon[34509]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-09T08:25:33.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:33 smithi028 ceph-mon[30119]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-09T08:25:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.139+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/3845201432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e781083e0 msgr2=0x7f5e781087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.139+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/3845201432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e781083e0 0x7f5e781087c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800b0d0 tx=0x7f5e6800b3e0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.139+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/3845201432 shutdown_connections 2024-05-09T08:25:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.139+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/3845201432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e78102d90 0x7f5e7810f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.139+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/3845201432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e781083e0 0x7f5e781087c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.139+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/3845201432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e781023d0 0x7f5e78102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.139+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/3845201432 >> 172.21.15.28:0/3845201432 conn(0x7f5e78079480 msgr2=0x7f5e78079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.139+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/3845201432 shutdown_connections 2024-05-09T08:25:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.139+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/3845201432 wait complete. 2024-05-09T08:25:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.140+0000 7f5e7ec5f700 1 Processor -- start 2024-05-09T08:25:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.140+0000 7f5e7ec5f700 1 -- start start 2024-05-09T08:25:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e7ec5f700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e781023d0 0x7f5e7819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e7ec5f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e78102d90 0x7f5e7819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e7ec5f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e781083e0 0x7f5e781988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e7ec5f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e78112710 con 0x7f5e78102d90 2024-05-09T08:25:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e7ec5f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5e78112590 con 0x7f5e781083e0 2024-05-09T08:25:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e7ec5f700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5e78112890 con 0x7f5e781023d0 2024-05-09T08:25:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e77fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e78102d90 0x7f5e7819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e77fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e78102d90 0x7f5e7819ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35832/0 (socket says 172.21.15.28:35832) 2024-05-09T08:25:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e77fff700 1 -- 172.21.15.28:0/2919541498 learned_addr learned my addr 172.21.15.28:0/2919541498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e7d1fc700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e781083e0 0x7f5e781988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e7c9fb700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e781023d0 0x7f5e7819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e77fff700 1 -- 172.21.15.28:0/2919541498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e781023d0 msgr2=0x7f5e7819e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e77fff700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e781023d0 0x7f5e7819e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e77fff700 1 -- 172.21.15.28:0/2919541498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e781083e0 msgr2=0x7f5e781988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e77fff700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e781083e0 0x7f5e781988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e77fff700 1 -- 172.21.15.28:0/2919541498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e78199010 con 0x7f5e78102d90 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e7d1fc700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e781083e0 0x7f5e781988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.141+0000 7f5e77fff700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e78102d90 0x7f5e7819ec30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000b530 tx=0x7f5e6000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.142+0000 7f5e75ffb700 1 -- 172.21.15.28:0/2919541498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60015020 con 0x7f5e78102d90 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.142+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e78199300 con 0x7f5e78102d90 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.142+0000 7f5e75ffb700 1 -- 172.21.15.28:0/2919541498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e60011d10 con 0x7f5e78102d90 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.142+0000 7f5e75ffb700 1 -- 172.21.15.28:0/2919541498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60010e50 con 0x7f5e78102d90 2024-05-09T08:25:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.142+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e781a55c0 con 0x7f5e78102d90 2024-05-09T08:25:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.143+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e78069010 con 0x7f5e78102d90 2024-05-09T08:25:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.144+0000 7f5e75ffb700 1 -- 172.21.15.28:0/2919541498 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5e60018020 con 0x7f5e78102d90 2024-05-09T08:25:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.145+0000 7f5e75ffb700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e640a2dc0 0x7f5e640a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.145+0000 7f5e7c9fb700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e640a2dc0 0x7f5e640a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.145+0000 7f5e75ffb700 1 -- 172.21.15.28:0/2919541498 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5e600cf140 con 0x7f5e78102d90 2024-05-09T08:25:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.145+0000 7f5e7c9fb700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e640a2dc0 0x7f5e640a5280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009850 tx=0x7f5e68009780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:34.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.148+0000 7f5e75ffb700 1 -- 172.21.15.28:0/2919541498 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5e6001f020 con 0x7f5e78102d90 2024-05-09T08:25:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.350+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e781014a0 con 0x7f5e640a2dc0 2024-05-09T08:25:34.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.359+0000 7f5e75ffb700 1 -- 172.21.15.28:0/2919541498 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340860 (secure 0 0 0) 0x7f5e781014a0 con 0x7f5e640a2dc0 2024-05-09T08:25:34.364 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e640a2dc0 msgr2=0x7f5e640a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e640a2dc0 0x7f5e640a5280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009850 tx=0x7f5e68009780 comp rx=0 tx=0).stop 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e78102d90 msgr2=0x7f5e7819ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e78102d90 0x7f5e7819ec30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000b530 tx=0x7f5e6000b8f0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 shutdown_connections 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5e640a2dc0 0x7f5e640a5280 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e781023d0 0x7f5e7819e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e781083e0 0x7f5e781988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 --2- 172.21.15.28:0/2919541498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e78102d90 0x7f5e7819ec30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 >> 172.21.15.28:0/2919541498 conn(0x7f5e78079480 msgr2=0x7f5e78100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:34.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 shutdown_connections 2024-05-09T08:25:34.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:34.363+0000 7f5e7ec5f700 1 -- 172.21.15.28:0/2919541498 wait complete. 2024-05-09T08:25:34.368 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-09T08:25:35.072 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-05-09T08:25:33.665351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661161,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":888,"num_read_kb":569,"num_write":431,"num_write_kb":3085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1245015,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":855,"ondisk_log_size":855,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373884,"kb_used_data":12252,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567940,"statfs":{"total":1151940427776,"available":1151557570560,"internally_reserved":0,"allocated":12546048,"data_stored":7969193,"data_compressed":369617,"data_compressed_allocated":499712,"data_compressed_original":2535424,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":121,"num_read_kb":56,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.679680+0000","last_change":"2024-05-09T08:23:51.579564+0000","last_active":"2024-05-09T08:25:19.679680+0000","last_peered":"2024-05-09T08:25:19.679680+0000","last_clean":"2024-05-09T08:25:19.679680+0000","last_became_active":"2024-05-09T08:23:51.578884+0000","last_became_peered":"2024-05-09T08:23:51.578884+0000","last_unstale":"2024-05-09T08:25:19.679680+0000","last_undegraded":"2024-05-09T08:25:19.679680+0000","last_fullsized":"2024-05-09T08:25:19.679680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:40.473483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.673389+0000","last_change":"2024-05-09T08:23:51.574187+0000","last_active":"2024-05-09T08:25:19.673389+0000","last_peered":"2024-05-09T08:25:19.673389+0000","last_clean":"2024-05-09T08:25:19.673389+0000","last_became_active":"2024-05-09T08:23:51.573837+0000","last_became_peered":"2024-05-09T08:23:51.573837+0000","last_unstale":"2024-05-09T08:25:19.673389+0000","last_undegraded":"2024-05-09T08:25:19.673389+0000","last_fullsized":"2024-05-09T08:25:19.673389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:31.448038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.685622+0000","last_change":"2024-05-09T08:23:51.577224+0000","last_active":"2024-05-09T08:25:19.685622+0000","last_peered":"2024-05-09T08:25:19.685622+0000","last_clean":"2024-05-09T08:25:19.685622+0000","last_became_active":"2024-05-09T08:23:51.576822+0000","last_became_peered":"2024-05-09T08:23:51.576822+0000","last_unstale":"2024-05-09T08:25:19.685622+0000","last_undegraded":"2024-05-09T08:25:19.685622+0000","last_fullsized":"2024-05-09T08:25:19.685622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:04.902530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.667664+0000","last_change":"2024-05-09T08:23:51.577172+0000","last_active":"2024-05-09T08:25:19.667664+0000","last_peered":"2024-05-09T08:25:19.667664+0000","last_clean":"2024-05-09T08:25:19.667664+0000","last_became_active":"2024-05-09T08:23:51.576861+0000","last_became_peered":"2024-05-09T08:23:51.576861+0000","last_unstale":"2024-05-09T08:25:19.667664+0000","last_undegraded":"2024-05-09T08:25:19.667664+0000","last_fullsized":"2024-05-09T08:25:19.667664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:10.925739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.675025+0000","last_change":"2024-05-09T08:23:51.579408+0000","last_active":"2024-05-09T08:25:19.675025+0000","last_peered":"2024-05-09T08:25:19.675025+0000","last_clean":"2024-05-09T08:25:19.675025+0000","last_became_active":"2024-05-09T08:23:51.578229+0000","last_became_peered":"2024-05-09T08:23:51.578229+0000","last_unstale":"2024-05-09T08:25:19.675025+0000","last_undegraded":"2024-05-09T08:25:19.675025+0000","last_fullsized":"2024-05-09T08:25:19.675025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:47:27.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.669065+0000","last_change":"2024-05-09T08:23:51.576924+0000","last_active":"2024-05-09T08:25:19.669065+0000","last_peered":"2024-05-09T08:25:19.669065+0000","last_clean":"2024-05-09T08:25:19.669065+0000","last_became_active":"2024-05-09T08:23:51.576601+0000","last_became_peered":"2024-05-09T08:23:51.576601+0000","last_unstale":"2024-05-09T08:25:19.669065+0000","last_undegraded":"2024-05-09T08:25:19.669065+0000","last_fullsized":"2024-05-09T08:25:19.669065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:07:16.910271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.678048+0000","last_change":"2024-05-09T08:23:51.574843+0000","last_active":"2024-05-09T08:25:19.678048+0000","last_peered":"2024-05-09T08:25:19.678048+0000","last_clean":"2024-05-09T08:25:19.678048+0000","last_became_active":"2024-05-09T08:23:51.574498+0000","last_became_peered":"2024-05-09T08:23:51.574498+0000","last_unstale":"2024-05-09T08:25:19.678048+0000","last_undegraded":"2024-05-09T08:25:19.678048+0000","last_fullsized":"2024-05-09T08:25:19.678048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:59:50.974491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.684127+0000","last_change":"2024-05-09T08:23:51.574001+0000","last_active":"2024-05-09T08:25:19.684127+0000","last_peered":"2024-05-09T08:25:19.684127+0000","last_clean":"2024-05-09T08:25:19.684127+0000","last_became_active":"2024-05-09T08:23:51.573632+0000","last_became_peered":"2024-05-09T08:23:51.573632+0000","last_unstale":"2024-05-09T08:25:19.684127+0000","last_undegraded":"2024-05-09T08:25:19.684127+0000","last_fullsized":"2024-05-09T08:25:19.684127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:55:16.418951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.678169+0000","last_change":"2024-05-09T08:23:51.567635+0000","last_active":"2024-05-09T08:25:19.678169+0000","last_peered":"2024-05-09T08:25:19.678169+0000","last_clean":"2024-05-09T08:25:19.678169+0000","last_became_active":"2024-05-09T08:23:51.567283+0000","last_became_peered":"2024-05-09T08:23:51.567283+0000","last_unstale":"2024-05-09T08:25:19.678169+0000","last_undegraded":"2024-05-09T08:25:19.678169+0000","last_fullsized":"2024-05-09T08:25:19.678169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:42.346856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.669983+0000","last_change":"2024-05-09T08:23:51.576090+0000","last_active":"2024-05-09T08:25:19.669983+0000","last_peered":"2024-05-09T08:25:19.669983+0000","last_clean":"2024-05-09T08:25:19.669983+0000","last_became_active":"2024-05-09T08:23:51.575768+0000","last_became_peered":"2024-05-09T08:23:51.575768+0000","last_unstale":"2024-05-09T08:25:19.669983+0000","last_undegraded":"2024-05-09T08:25:19.669983+0000","last_fullsized":"2024-05-09T08:25:19.669983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:00:50.016924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.677013+0000","last_change":"2024-05-09T08:23:51.578957+0000","last_active":"2024-05-09T08:25:19.677013+0000","last_peered":"2024-05-09T08:25:19.677013+0000","last_clean":"2024-05-09T08:25:19.677013+0000","last_became_active":"2024-05-09T08:23:51.578210+0000","last_became_peered":"2024-05-09T08:23:51.578210+0000","last_unstale":"2024-05-09T08:25:19.677013+0000","last_undegraded":"2024-05-09T08:25:19.677013+0000","last_fullsized":"2024-05-09T08:25:19.677013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:28:28.953434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.667316+0000","last_change":"2024-05-09T08:23:51.575389+0000","last_active":"2024-05-09T08:25:19.667316+0000","last_peered":"2024-05-09T08:25:19.667316+0000","last_clean":"2024-05-09T08:25:19.667316+0000","last_became_active":"2024-05-09T08:23:51.575071+0000","last_became_peered":"2024-05-09T08:23:51.575071+0000","last_unstale":"2024-05-09T08:25:19.667316+0000","last_undegraded":"2024-05-09T08:25:19.667316+0000","last_fullsized":"2024-05-09T08:25:19.667316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:34.844611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.676541+0000","last_change":"2024-05-09T08:23:51.572574+0000","last_active":"2024-05-09T08:25:19.676541+0000","last_peered":"2024-05-09T08:25:19.676541+0000","last_clean":"2024-05-09T08:25:19.676541+0000","last_became_active":"2024-05-09T08:23:51.572243+0000","last_became_peered":"2024-05-09T08:23:51.572243+0000","last_unstale":"2024-05-09T08:25:19.676541+0000","last_undegraded":"2024-05-09T08:25:19.676541+0000","last_fullsized":"2024-05-09T08:25:19.676541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:52.534365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.677778+0000","last_change":"2024-05-09T08:23:51.574963+0000","last_active":"2024-05-09T08:25:19.677778+0000","last_peered":"2024-05-09T08:25:19.677778+0000","last_clean":"2024-05-09T08:25:19.677778+0000","last_became_active":"2024-05-09T08:23:51.574597+0000","last_became_peered":"2024-05-09T08:23:51.574597+0000","last_unstale":"2024-05-09T08:25:19.677778+0000","last_undegraded":"2024-05-09T08:25:19.677778+0000","last_fullsized":"2024-05-09T08:25:19.677778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:33.348828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.680256+0000","last_change":"2024-05-09T08:23:51.577777+0000","last_active":"2024-05-09T08:25:19.680256+0000","last_peered":"2024-05-09T08:25:19.680256+0000","last_clean":"2024-05-09T08:25:19.680256+0000","last_became_active":"2024-05-09T08:23:51.577073+0000","last_became_peered":"2024-05-09T08:23:51.577073+0000","last_unstale":"2024-05-09T08:25:19.680256+0000","last_undegraded":"2024-05-09T08:25:19.680256+0000","last_fullsized":"2024-05-09T08:25:19.680256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:26.249488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.688063+0000","last_change":"2024-05-09T08:23:51.574476+0000","last_active":"2024-05-09T08:25:19.688063+0000","last_peered":"2024-05-09T08:25:19.688063+0000","last_clean":"2024-05-09T08:25:19.688063+0000","last_became_active":"2024-05-09T08:23:51.572748+0000","last_became_peered":"2024-05-09T08:23:51.572748+0000","last_unstale":"2024-05-09T08:25:19.688063+0000","last_undegraded":"2024-05-09T08:25:19.688063+0000","last_fullsized":"2024-05-09T08:25:19.688063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:43.515419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.679912+0000","last_change":"2024-05-09T08:23:51.580568+0000","last_active":"2024-05-09T08:25:19.679912+0000","last_peered":"2024-05-09T08:25:19.679912+0000","last_clean":"2024-05-09T08:25:19.679912+0000","last_became_active":"2024-05-09T08:23:51.579626+0000","last_became_peered":"2024-05-09T08:23:51.579626+0000","last_unstale":"2024-05-09T08:25:19.679912+0000","last_undegraded":"2024-05-09T08:25:19.679912+0000","last_fullsized":"2024-05-09T08:25:19.679912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:28.756079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.680128+0000","last_change":"2024-05-09T08:23:51.574363+0000","last_active":"2024-05-09T08:25:19.680128+0000","last_peered":"2024-05-09T08:25:19.680128+0000","last_clean":"2024-05-09T08:25:19.680128+0000","last_became_active":"2024-05-09T08:23:51.573858+0000","last_became_peered":"2024-05-09T08:23:51.573858+0000","last_unstale":"2024-05-09T08:25:19.680128+0000","last_undegraded":"2024-05-09T08:25:19.680128+0000","last_fullsized":"2024-05-09T08:25:19.680128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:30.752939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.665192+0000","last_change":"2024-05-09T08:23:51.579361+0000","last_active":"2024-05-09T08:25:19.665192+0000","last_peered":"2024-05-09T08:25:19.665192+0000","last_clean":"2024-05-09T08:25:19.665192+0000","last_became_active":"2024-05-09T08:23:51.579025+0000","last_became_peered":"2024-05-09T08:23:51.579025+0000","last_unstale":"2024-05-09T08:25:19.665192+0000","last_undegraded":"2024-05-09T08:25:19.665192+0000","last_fullsized":"2024-05-09T08:25:19.665192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:02.273999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478882+0000","last_change":"2024-05-09T08:23:55.670107+0000","last_active":"2024-05-09T08:24:51.478882+0000","last_peered":"2024-05-09T08:24:51.478882+0000","last_clean":"2024-05-09T08:24:51.478882+0000","last_became_active":"2024-05-09T08:23:55.669621+0000","last_became_peered":"2024-05-09T08:23:55.669621+0000","last_unstale":"2024-05-09T08:24:51.478882+0000","last_undegraded":"2024-05-09T08:24:51.478882+0000","last_fullsized":"2024-05-09T08:24:51.478882+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:00.547137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":133,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:29.539063+0000","last_change":"2024-05-09T08:24:51.498795+0000","last_active":"2024-05-09T08:25:29.539063+0000","last_peered":"2024-05-09T08:25:29.539063+0000","last_clean":"2024-05-09T08:25:29.539063+0000","last_became_active":"2024-05-09T08:24:51.487098+0000","last_became_peered":"2024-05-09T08:24:51.487098+0000","last_unstale":"2024-05-09T08:25:29.539063+0000","last_undegraded":"2024-05-09T08:25:29.539063+0000","last_fullsized":"2024-05-09T08:25:29.539063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:10.646049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.677654+0000","last_change":"2024-05-09T08:23:51.574336+0000","last_active":"2024-05-09T08:25:19.677654+0000","last_peered":"2024-05-09T08:25:19.677654+0000","last_clean":"2024-05-09T08:25:19.677654+0000","last_became_active":"2024-05-09T08:23:51.573801+0000","last_became_peered":"2024-05-09T08:23:51.573801+0000","last_unstale":"2024-05-09T08:25:19.677654+0000","last_undegraded":"2024-05-09T08:25:19.677654+0000","last_fullsized":"2024-05-09T08:25:19.677654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:47.389762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484058+0000","last_change":"2024-05-09T08:23:55.670194+0000","last_active":"2024-05-09T08:24:51.484058+0000","last_peered":"2024-05-09T08:24:51.484058+0000","last_clean":"2024-05-09T08:24:51.484058+0000","last_became_active":"2024-05-09T08:23:55.669762+0000","last_became_peered":"2024-05-09T08:23:55.669762+0000","last_unstale":"2024-05-09T08:24:51.484058+0000","last_undegraded":"2024-05-09T08:24:51.484058+0000","last_fullsized":"2024-05-09T08:24:51.484058+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:22.879635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484304+0000","last_change":"2024-05-09T08:23:50.574421+0000","last_active":"2024-05-09T08:24:51.484304+0000","last_peered":"2024-05-09T08:24:51.484304+0000","last_clean":"2024-05-09T08:24:51.484304+0000","last_became_active":"2024-05-09T08:23:50.574070+0000","last_became_peered":"2024-05-09T08:23:50.574070+0000","last_unstale":"2024-05-09T08:24:51.484304+0000","last_undegraded":"2024-05-09T08:24:51.484304+0000","last_fullsized":"2024-05-09T08:24:51.484304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:22.996034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.685731+0000","last_change":"2024-05-09T08:23:51.583275+0000","last_active":"2024-05-09T08:25:19.685731+0000","last_peered":"2024-05-09T08:25:19.685731+0000","last_clean":"2024-05-09T08:25:19.685731+0000","last_became_active":"2024-05-09T08:23:51.582847+0000","last_became_peered":"2024-05-09T08:23:51.582847+0000","last_unstale":"2024-05-09T08:25:19.685731+0000","last_undegraded":"2024-05-09T08:25:19.685731+0000","last_fullsized":"2024-05-09T08:25:19.685731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:03:03.512852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485323+0000","last_change":"2024-05-09T08:23:55.672256+0000","last_active":"2024-05-09T08:24:51.485323+0000","last_peered":"2024-05-09T08:24:51.485323+0000","last_clean":"2024-05-09T08:24:51.485323+0000","last_became_active":"2024-05-09T08:23:55.671855+0000","last_became_peered":"2024-05-09T08:23:55.671855+0000","last_unstale":"2024-05-09T08:24:51.485323+0000","last_undegraded":"2024-05-09T08:24:51.485323+0000","last_fullsized":"2024-05-09T08:24:51.485323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:45:12.199359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481664+0000","last_change":"2024-05-09T08:23:50.566686+0000","last_active":"2024-05-09T08:24:51.481664+0000","last_peered":"2024-05-09T08:24:51.481664+0000","last_clean":"2024-05-09T08:24:51.481664+0000","last_became_active":"2024-05-09T08:23:50.565910+0000","last_became_peered":"2024-05-09T08:23:50.565910+0000","last_unstale":"2024-05-09T08:24:51.481664+0000","last_undegraded":"2024-05-09T08:24:51.481664+0000","last_fullsized":"2024-05-09T08:24:51.481664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:02.211551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682187+0000","last_change":"2024-05-09T08:23:51.570972+0000","last_active":"2024-05-09T08:25:19.682187+0000","last_peered":"2024-05-09T08:25:19.682187+0000","last_clean":"2024-05-09T08:25:19.682187+0000","last_became_active":"2024-05-09T08:23:51.570638+0000","last_became_peered":"2024-05-09T08:23:51.570638+0000","last_unstale":"2024-05-09T08:25:19.682187+0000","last_undegraded":"2024-05-09T08:25:19.682187+0000","last_fullsized":"2024-05-09T08:25:19.682187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:38.738820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480699+0000","last_change":"2024-05-09T08:23:55.671929+0000","last_active":"2024-05-09T08:24:51.480699+0000","last_peered":"2024-05-09T08:24:51.480699+0000","last_clean":"2024-05-09T08:24:51.480699+0000","last_became_active":"2024-05-09T08:23:55.671483+0000","last_became_peered":"2024-05-09T08:23:55.671483+0000","last_unstale":"2024-05-09T08:24:51.480699+0000","last_undegraded":"2024-05-09T08:24:51.480699+0000","last_fullsized":"2024-05-09T08:24:51.480699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:21.527266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481776+0000","last_change":"2024-05-09T08:23:50.565644+0000","last_active":"2024-05-09T08:24:51.481776+0000","last_peered":"2024-05-09T08:24:51.481776+0000","last_clean":"2024-05-09T08:24:51.481776+0000","last_became_active":"2024-05-09T08:23:50.565265+0000","last_became_peered":"2024-05-09T08:23:50.565265+0000","last_unstale":"2024-05-09T08:24:51.481776+0000","last_undegraded":"2024-05-09T08:24:51.481776+0000","last_fullsized":"2024-05-09T08:24:51.481776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:51:57.972535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.671448+0000","last_change":"2024-05-09T08:23:51.571732+0000","last_active":"2024-05-09T08:25:19.671448+0000","last_peered":"2024-05-09T08:25:19.671448+0000","last_clean":"2024-05-09T08:25:19.671448+0000","last_became_active":"2024-05-09T08:23:51.570783+0000","last_became_peered":"2024-05-09T08:23:51.570783+0000","last_unstale":"2024-05-09T08:25:19.671448+0000","last_undegraded":"2024-05-09T08:25:19.671448+0000","last_fullsized":"2024-05-09T08:25:19.671448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:18.515905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479347+0000","last_change":"2024-05-09T08:23:55.666727+0000","last_active":"2024-05-09T08:24:51.479347+0000","last_peered":"2024-05-09T08:24:51.479347+0000","last_clean":"2024-05-09T08:24:51.479347+0000","last_became_active":"2024-05-09T08:23:55.666284+0000","last_became_peered":"2024-05-09T08:23:55.666284+0000","last_unstale":"2024-05-09T08:24:51.479347+0000","last_undegraded":"2024-05-09T08:24:51.479347+0000","last_fullsized":"2024-05-09T08:24:51.479347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:05.355404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:29.518466+0000","last_change":"2024-05-09T08:23:50.573308+0000","last_active":"2024-05-09T08:25:29.518466+0000","last_peered":"2024-05-09T08:25:29.518466+0000","last_clean":"2024-05-09T08:25:29.518466+0000","last_became_active":"2024-05-09T08:23:50.572946+0000","last_became_peered":"2024-05-09T08:23:50.572946+0000","last_unstale":"2024-05-09T08:25:29.518466+0000","last_undegraded":"2024-05-09T08:25:29.518466+0000","last_fullsized":"2024-05-09T08:25:29.518466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:44:35.816620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.671963+0000","last_change":"2024-05-09T08:23:51.571748+0000","last_active":"2024-05-09T08:25:19.671963+0000","last_peered":"2024-05-09T08:25:19.671963+0000","last_clean":"2024-05-09T08:25:19.671963+0000","last_became_active":"2024-05-09T08:23:51.571388+0000","last_became_peered":"2024-05-09T08:23:51.571388+0000","last_unstale":"2024-05-09T08:25:19.671963+0000","last_undegraded":"2024-05-09T08:25:19.671963+0000","last_fullsized":"2024-05-09T08:25:19.671963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:34.916491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479250+0000","last_change":"2024-05-09T08:23:55.674022+0000","last_active":"2024-05-09T08:24:51.479250+0000","last_peered":"2024-05-09T08:24:51.479250+0000","last_clean":"2024-05-09T08:24:51.479250+0000","last_became_active":"2024-05-09T08:23:55.673322+0000","last_became_peered":"2024-05-09T08:23:55.673322+0000","last_unstale":"2024-05-09T08:24:51.479250+0000","last_undegraded":"2024-05-09T08:24:51.479250+0000","last_fullsized":"2024-05-09T08:24:51.479250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:59:14.299886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481002+0000","last_change":"2024-05-09T08:23:50.571235+0000","last_active":"2024-05-09T08:24:51.481002+0000","last_peered":"2024-05-09T08:24:51.481002+0000","last_clean":"2024-05-09T08:24:51.481002+0000","last_became_active":"2024-05-09T08:23:50.570694+0000","last_became_peered":"2024-05-09T08:23:50.570694+0000","last_unstale":"2024-05-09T08:24:51.481002+0000","last_undegraded":"2024-05-09T08:24:51.481002+0000","last_fullsized":"2024-05-09T08:24:51.481002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:22.850248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681478+0000","last_change":"2024-05-09T08:23:51.572387+0000","last_active":"2024-05-09T08:25:19.681478+0000","last_peered":"2024-05-09T08:25:19.681478+0000","last_clean":"2024-05-09T08:25:19.681478+0000","last_became_active":"2024-05-09T08:23:51.572038+0000","last_became_peered":"2024-05-09T08:23:51.572038+0000","last_unstale":"2024-05-09T08:25:19.681478+0000","last_undegraded":"2024-05-09T08:25:19.681478+0000","last_fullsized":"2024-05-09T08:25:19.681478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:38.118125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.489278+0000","last_change":"2024-05-09T08:24:51.489278+0000","last_active":"2024-05-09T08:24:51.489278+0000","last_peered":"2024-05-09T08:24:51.489278+0000","last_clean":"2024-05-09T08:24:51.489278+0000","last_became_active":"2024-05-09T08:24:51.488875+0000","last_became_peered":"2024-05-09T08:24:51.488875+0000","last_unstale":"2024-05-09T08:24:51.489278+0000","last_undegraded":"2024-05-09T08:24:51.489278+0000","last_fullsized":"2024-05-09T08:24:51.489278+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:47:45.306525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.499072+0000","last_change":"2024-05-09T08:23:50.570040+0000","last_active":"2024-05-09T08:24:51.499072+0000","last_peered":"2024-05-09T08:24:51.499072+0000","last_clean":"2024-05-09T08:24:51.499072+0000","last_became_active":"2024-05-09T08:23:50.569665+0000","last_became_peered":"2024-05-09T08:23:50.569665+0000","last_unstale":"2024-05-09T08:24:51.499072+0000","last_undegraded":"2024-05-09T08:24:51.499072+0000","last_fullsized":"2024-05-09T08:24:51.499072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:22:29.639712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.683305+0000","last_change":"2024-05-09T08:23:51.568561+0000","last_active":"2024-05-09T08:25:19.683305+0000","last_peered":"2024-05-09T08:25:19.683305+0000","last_clean":"2024-05-09T08:25:19.683305+0000","last_became_active":"2024-05-09T08:23:51.568196+0000","last_became_peered":"2024-05-09T08:23:51.568196+0000","last_unstale":"2024-05-09T08:25:19.683305+0000","last_undegraded":"2024-05-09T08:25:19.683305+0000","last_fullsized":"2024-05-09T08:25:19.683305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:52.132837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.486358+0000","last_change":"2024-05-09T08:23:55.674960+0000","last_active":"2024-05-09T08:24:51.486358+0000","last_peered":"2024-05-09T08:24:51.486358+0000","last_clean":"2024-05-09T08:24:51.486358+0000","last_became_active":"2024-05-09T08:23:55.674509+0000","last_became_peered":"2024-05-09T08:23:55.674509+0000","last_unstale":"2024-05-09T08:24:51.486358+0000","last_undegraded":"2024-05-09T08:24:51.486358+0000","last_fullsized":"2024-05-09T08:24:51.486358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:22.479867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483898+0000","last_change":"2024-05-09T08:23:50.569896+0000","last_active":"2024-05-09T08:24:51.483898+0000","last_peered":"2024-05-09T08:24:51.483898+0000","last_clean":"2024-05-09T08:24:51.483898+0000","last_became_active":"2024-05-09T08:23:50.569562+0000","last_became_peered":"2024-05-09T08:23:50.569562+0000","last_unstale":"2024-05-09T08:24:51.483898+0000","last_undegraded":"2024-05-09T08:24:51.483898+0000","last_fullsized":"2024-05-09T08:24:51.483898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:25:20.278726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682617+0000","last_change":"2024-05-09T08:23:51.570260+0000","last_active":"2024-05-09T08:25:19.682617+0000","last_peered":"2024-05-09T08:25:19.682617+0000","last_clean":"2024-05-09T08:25:19.682617+0000","last_became_active":"2024-05-09T08:23:51.569837+0000","last_became_peered":"2024-05-09T08:23:51.569837+0000","last_unstale":"2024-05-09T08:25:19.682617+0000","last_undegraded":"2024-05-09T08:25:19.682617+0000","last_fullsized":"2024-05-09T08:25:19.682617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:01:44.805271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483875+0000","last_change":"2024-05-09T08:23:55.676639+0000","last_active":"2024-05-09T08:24:51.483875+0000","last_peered":"2024-05-09T08:24:51.483875+0000","last_clean":"2024-05-09T08:24:51.483875+0000","last_became_active":"2024-05-09T08:23:55.673643+0000","last_became_peered":"2024-05-09T08:23:55.673643+0000","last_unstale":"2024-05-09T08:24:51.483875+0000","last_undegraded":"2024-05-09T08:24:51.483875+0000","last_fullsized":"2024-05-09T08:24:51.483875+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:02.377184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:30.519587+0000","last_change":"2024-05-09T08:23:50.565745+0000","last_active":"2024-05-09T08:25:30.519587+0000","last_peered":"2024-05-09T08:25:30.519587+0000","last_clean":"2024-05-09T08:25:30.519587+0000","last_became_active":"2024-05-09T08:23:50.565385+0000","last_became_peered":"2024-05-09T08:23:50.565385+0000","last_unstale":"2024-05-09T08:25:30.519587+0000","last_undegraded":"2024-05-09T08:25:30.519587+0000","last_fullsized":"2024-05-09T08:25:30.519587+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:05:00.818398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681736+0000","last_change":"2024-05-09T08:23:51.572341+0000","last_active":"2024-05-09T08:25:19.681736+0000","last_peered":"2024-05-09T08:25:19.681736+0000","last_clean":"2024-05-09T08:25:19.681736+0000","last_became_active":"2024-05-09T08:23:51.571583+0000","last_became_peered":"2024-05-09T08:23:51.571583+0000","last_unstale":"2024-05-09T08:25:19.681736+0000","last_undegraded":"2024-05-09T08:25:19.681736+0000","last_fullsized":"2024-05-09T08:25:19.681736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:20.695995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478270+0000","last_change":"2024-05-09T08:23:55.677603+0000","last_active":"2024-05-09T08:24:51.478270+0000","last_peered":"2024-05-09T08:24:51.478270+0000","last_clean":"2024-05-09T08:24:51.478270+0000","last_became_active":"2024-05-09T08:23:55.677143+0000","last_became_peered":"2024-05-09T08:23:55.677143+0000","last_unstale":"2024-05-09T08:24:51.478270+0000","last_undegraded":"2024-05-09T08:24:51.478270+0000","last_fullsized":"2024-05-09T08:24:51.478270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:27.850997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485842+0000","last_change":"2024-05-09T08:23:50.565497+0000","last_active":"2024-05-09T08:24:51.485842+0000","last_peered":"2024-05-09T08:24:51.485842+0000","last_clean":"2024-05-09T08:24:51.485842+0000","last_became_active":"2024-05-09T08:23:50.565120+0000","last_became_peered":"2024-05-09T08:23:50.565120+0000","last_unstale":"2024-05-09T08:24:51.485842+0000","last_undegraded":"2024-05-09T08:24:51.485842+0000","last_fullsized":"2024-05-09T08:24:51.485842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:56.141997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.687454+0000","last_change":"2024-05-09T08:23:51.576489+0000","last_active":"2024-05-09T08:25:19.687454+0000","last_peered":"2024-05-09T08:25:19.687454+0000","last_clean":"2024-05-09T08:25:19.687454+0000","last_became_active":"2024-05-09T08:23:51.576145+0000","last_became_peered":"2024-05-09T08:23:51.576145+0000","last_unstale":"2024-05-09T08:25:19.687454+0000","last_undegraded":"2024-05-09T08:25:19.687454+0000","last_fullsized":"2024-05-09T08:25:19.687454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:37.650308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.490957+0000","last_change":"2024-05-09T08:24:51.490957+0000","last_active":"2024-05-09T08:24:51.490957+0000","last_peered":"2024-05-09T08:24:51.490957+0000","last_clean":"2024-05-09T08:24:51.490957+0000","last_became_active":"2024-05-09T08:24:51.490541+0000","last_became_peered":"2024-05-09T08:24:51.490541+0000","last_unstale":"2024-05-09T08:24:51.490957+0000","last_undegraded":"2024-05-09T08:24:51.490957+0000","last_fullsized":"2024-05-09T08:24:51.490957+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:03.168547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483916+0000","last_change":"2024-05-09T08:23:50.567182+0000","last_active":"2024-05-09T08:24:51.483916+0000","last_peered":"2024-05-09T08:24:51.483916+0000","last_clean":"2024-05-09T08:24:51.483916+0000","last_became_active":"2024-05-09T08:23:50.566851+0000","last_became_peered":"2024-05-09T08:23:50.566851+0000","last_unstale":"2024-05-09T08:24:51.483916+0000","last_undegraded":"2024-05-09T08:24:51.483916+0000","last_fullsized":"2024-05-09T08:24:51.483916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:26.540456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.683808+0000","last_change":"2024-05-09T08:23:51.574074+0000","last_active":"2024-05-09T08:25:19.683808+0000","last_peered":"2024-05-09T08:25:19.683808+0000","last_clean":"2024-05-09T08:25:19.683808+0000","last_became_active":"2024-05-09T08:23:51.572587+0000","last_became_peered":"2024-05-09T08:23:51.572587+0000","last_unstale":"2024-05-09T08:25:19.683808+0000","last_undegraded":"2024-05-09T08:25:19.683808+0000","last_fullsized":"2024-05-09T08:25:19.683808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:46.186358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478461+0000","last_change":"2024-05-09T08:23:55.667170+0000","last_active":"2024-05-09T08:24:51.478461+0000","last_peered":"2024-05-09T08:24:51.478461+0000","last_clean":"2024-05-09T08:24:51.478461+0000","last_became_active":"2024-05-09T08:23:55.666707+0000","last_became_peered":"2024-05-09T08:23:55.666707+0000","last_unstale":"2024-05-09T08:24:51.478461+0000","last_undegraded":"2024-05-09T08:24:51.478461+0000","last_fullsized":"2024-05-09T08:24:51.478461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:48.332207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":236,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:30.547637+0000","last_change":"2024-05-09T08:24:51.509495+0000","last_active":"2024-05-09T08:25:30.547637+0000","last_peered":"2024-05-09T08:25:30.547637+0000","last_clean":"2024-05-09T08:25:30.547637+0000","last_became_active":"2024-05-09T08:24:51.491187+0000","last_became_peered":"2024-05-09T08:24:51.491187+0000","last_unstale":"2024-05-09T08:25:30.547637+0000","last_undegraded":"2024-05-09T08:25:30.547637+0000","last_fullsized":"2024-05-09T08:25:30.547637+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:55.320000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":143,"num_read_kb":134,"num_write":40,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":63356,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.665403+0000","last_change":"2024-05-09T08:23:51.577595+0000","last_active":"2024-05-09T08:25:19.665403+0000","last_peered":"2024-05-09T08:25:19.665403+0000","last_clean":"2024-05-09T08:25:19.665403+0000","last_became_active":"2024-05-09T08:23:51.577271+0000","last_became_peered":"2024-05-09T08:23:51.577271+0000","last_unstale":"2024-05-09T08:25:19.665403+0000","last_undegraded":"2024-05-09T08:25:19.665403+0000","last_fullsized":"2024-05-09T08:25:19.665403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:34.679111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484410+0000","last_change":"2024-05-09T08:23:55.671128+0000","last_active":"2024-05-09T08:24:51.484410+0000","last_peered":"2024-05-09T08:24:51.484410+0000","last_clean":"2024-05-09T08:24:51.484410+0000","last_became_active":"2024-05-09T08:23:55.670414+0000","last_became_peered":"2024-05-09T08:23:55.670414+0000","last_unstale":"2024-05-09T08:24:51.484410+0000","last_undegraded":"2024-05-09T08:24:51.484410+0000","last_fullsized":"2024-05-09T08:24:51.484410+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:31:55.307412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479102+0000","last_change":"2024-05-09T08:23:50.568503+0000","last_active":"2024-05-09T08:24:51.479102+0000","last_peered":"2024-05-09T08:24:51.479102+0000","last_clean":"2024-05-09T08:24:51.479102+0000","last_became_active":"2024-05-09T08:23:50.568181+0000","last_became_peered":"2024-05-09T08:23:50.568181+0000","last_unstale":"2024-05-09T08:24:51.479102+0000","last_undegraded":"2024-05-09T08:24:51.479102+0000","last_fullsized":"2024-05-09T08:24:51.479102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:40.883792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681063+0000","last_change":"2024-05-09T08:23:51.573602+0000","last_active":"2024-05-09T08:25:19.681063+0000","last_peered":"2024-05-09T08:25:19.681063+0000","last_clean":"2024-05-09T08:25:19.681063+0000","last_became_active":"2024-05-09T08:23:51.572557+0000","last_became_peered":"2024-05-09T08:23:51.572557+0000","last_unstale":"2024-05-09T08:25:19.681063+0000","last_undegraded":"2024-05-09T08:25:19.681063+0000","last_fullsized":"2024-05-09T08:25:19.681063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.170962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481604+0000","last_change":"2024-05-09T08:23:55.673773+0000","last_active":"2024-05-09T08:24:51.481604+0000","last_peered":"2024-05-09T08:24:51.481604+0000","last_clean":"2024-05-09T08:24:51.481604+0000","last_became_active":"2024-05-09T08:23:55.673276+0000","last_became_peered":"2024-05-09T08:23:55.673276+0000","last_unstale":"2024-05-09T08:24:51.481604+0000","last_undegraded":"2024-05-09T08:24:51.481604+0000","last_fullsized":"2024-05-09T08:24:51.481604+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:59:00.629339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484874+0000","last_change":"2024-05-09T08:23:50.575894+0000","last_active":"2024-05-09T08:24:51.484874+0000","last_peered":"2024-05-09T08:24:51.484874+0000","last_clean":"2024-05-09T08:24:51.484874+0000","last_became_active":"2024-05-09T08:23:50.575555+0000","last_became_peered":"2024-05-09T08:23:50.575555+0000","last_unstale":"2024-05-09T08:24:51.484874+0000","last_undegraded":"2024-05-09T08:24:51.484874+0000","last_fullsized":"2024-05-09T08:24:51.484874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:19.070764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.680809+0000","last_change":"2024-05-09T08:23:51.574125+0000","last_active":"2024-05-09T08:25:19.680809+0000","last_peered":"2024-05-09T08:25:19.680809+0000","last_clean":"2024-05-09T08:25:19.680809+0000","last_became_active":"2024-05-09T08:23:51.573794+0000","last_became_peered":"2024-05-09T08:23:51.573794+0000","last_unstale":"2024-05-09T08:25:19.680809+0000","last_undegraded":"2024-05-09T08:25:19.680809+0000","last_fullsized":"2024-05-09T08:25:19.680809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:19:26.634208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479015+0000","last_change":"2024-05-09T08:23:55.677711+0000","last_active":"2024-05-09T08:24:51.479015+0000","last_peered":"2024-05-09T08:24:51.479015+0000","last_clean":"2024-05-09T08:24:51.479015+0000","last_became_active":"2024-05-09T08:23:55.677058+0000","last_became_peered":"2024-05-09T08:23:55.677058+0000","last_unstale":"2024-05-09T08:24:51.479015+0000","last_undegraded":"2024-05-09T08:24:51.479015+0000","last_fullsized":"2024-05-09T08:24:51.479015+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:26.842145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:31.531038+0000","last_change":"2024-05-09T08:23:50.566413+0000","last_active":"2024-05-09T08:25:31.531038+0000","last_peered":"2024-05-09T08:25:31.531038+0000","last_clean":"2024-05-09T08:25:31.531038+0000","last_became_active":"2024-05-09T08:23:50.566047+0000","last_became_peered":"2024-05-09T08:23:50.566047+0000","last_unstale":"2024-05-09T08:25:31.531038+0000","last_undegraded":"2024-05-09T08:25:31.531038+0000","last_fullsized":"2024-05-09T08:25:31.531038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:15.340947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.676256+0000","last_change":"2024-05-09T08:23:51.576779+0000","last_active":"2024-05-09T08:25:19.676256+0000","last_peered":"2024-05-09T08:25:19.676256+0000","last_clean":"2024-05-09T08:25:19.676256+0000","last_became_active":"2024-05-09T08:23:51.576119+0000","last_became_peered":"2024-05-09T08:23:51.576119+0000","last_unstale":"2024-05-09T08:25:19.676256+0000","last_undegraded":"2024-05-09T08:25:19.676256+0000","last_fullsized":"2024-05-09T08:25:19.676256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:02.250135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.499744+0000","last_change":"2024-05-09T08:23:55.678564+0000","last_active":"2024-05-09T08:24:51.499744+0000","last_peered":"2024-05-09T08:24:51.499744+0000","last_clean":"2024-05-09T08:24:51.499744+0000","last_became_active":"2024-05-09T08:23:55.678128+0000","last_became_peered":"2024-05-09T08:23:55.678128+0000","last_unstale":"2024-05-09T08:24:51.499744+0000","last_undegraded":"2024-05-09T08:24:51.499744+0000","last_fullsized":"2024-05-09T08:24:51.499744+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:13:21.868267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.501243+0000","last_change":"2024-05-09T08:24:51.501243+0000","last_active":"2024-05-09T08:24:51.501243+0000","last_peered":"2024-05-09T08:24:51.501243+0000","last_clean":"2024-05-09T08:24:51.501243+0000","last_became_active":"2024-05-09T08:24:51.490110+0000","last_became_peered":"2024-05-09T08:24:51.490110+0000","last_unstale":"2024-05-09T08:24:51.501243+0000","last_undegraded":"2024-05-09T08:24:51.501243+0000","last_fullsized":"2024-05-09T08:24:51.501243+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:03.086570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.666882+0000","last_change":"2024-05-09T08:23:51.574914+0000","last_active":"2024-05-09T08:25:19.666882+0000","last_peered":"2024-05-09T08:25:19.666882+0000","last_clean":"2024-05-09T08:25:19.666882+0000","last_became_active":"2024-05-09T08:23:51.574380+0000","last_became_peered":"2024-05-09T08:23:51.574380+0000","last_unstale":"2024-05-09T08:25:19.666882+0000","last_undegraded":"2024-05-09T08:25:19.666882+0000","last_fullsized":"2024-05-09T08:25:19.666882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:34.698328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484388+0000","last_change":"2024-05-09T08:23:55.670198+0000","last_active":"2024-05-09T08:24:51.484388+0000","last_peered":"2024-05-09T08:24:51.484388+0000","last_clean":"2024-05-09T08:24:51.484388+0000","last_became_active":"2024-05-09T08:23:55.669798+0000","last_became_peered":"2024-05-09T08:23:55.669798+0000","last_unstale":"2024-05-09T08:24:51.484388+0000","last_undegraded":"2024-05-09T08:24:51.484388+0000","last_fullsized":"2024-05-09T08:24:51.484388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:09.474264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.504286+0000","last_change":"2024-05-09T08:24:51.504286+0000","last_active":"2024-05-09T08:24:51.504286+0000","last_peered":"2024-05-09T08:24:51.504286+0000","last_clean":"2024-05-09T08:24:51.504286+0000","last_became_active":"2024-05-09T08:24:51.489111+0000","last_became_peered":"2024-05-09T08:24:51.489111+0000","last_unstale":"2024-05-09T08:24:51.504286+0000","last_undegraded":"2024-05-09T08:24:51.504286+0000","last_fullsized":"2024-05-09T08:24:51.504286+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:14.975959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.665938+0000","last_change":"2024-05-09T08:23:51.573434+0000","last_active":"2024-05-09T08:25:19.665938+0000","last_peered":"2024-05-09T08:25:19.665938+0000","last_clean":"2024-05-09T08:25:19.665938+0000","last_became_active":"2024-05-09T08:23:51.573081+0000","last_became_peered":"2024-05-09T08:23:51.573081+0000","last_unstale":"2024-05-09T08:25:19.665938+0000","last_undegraded":"2024-05-09T08:25:19.665938+0000","last_fullsized":"2024-05-09T08:25:19.665938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:49.270355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481580+0000","last_change":"2024-05-09T08:23:55.853114+0000","last_active":"2024-05-09T08:24:51.481580+0000","last_peered":"2024-05-09T08:24:51.481580+0000","last_clean":"2024-05-09T08:24:51.481580+0000","last_became_active":"2024-05-09T08:23:55.852640+0000","last_became_peered":"2024-05-09T08:23:55.852640+0000","last_unstale":"2024-05-09T08:24:51.481580+0000","last_undegraded":"2024-05-09T08:24:51.481580+0000","last_fullsized":"2024-05-09T08:24:51.481580+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:55:17.823585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478828+0000","last_change":"2024-05-09T08:23:50.567834+0000","last_active":"2024-05-09T08:24:51.478828+0000","last_peered":"2024-05-09T08:24:51.478828+0000","last_clean":"2024-05-09T08:24:51.478828+0000","last_became_active":"2024-05-09T08:23:50.567538+0000","last_became_peered":"2024-05-09T08:23:50.567538+0000","last_unstale":"2024-05-09T08:24:51.478828+0000","last_undegraded":"2024-05-09T08:24:51.478828+0000","last_fullsized":"2024-05-09T08:24:51.478828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.674352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682001+0000","last_change":"2024-05-09T08:23:51.580219+0000","last_active":"2024-05-09T08:25:19.682001+0000","last_peered":"2024-05-09T08:25:19.682001+0000","last_clean":"2024-05-09T08:25:19.682001+0000","last_became_active":"2024-05-09T08:23:51.579835+0000","last_became_peered":"2024-05-09T08:23:51.579835+0000","last_unstale":"2024-05-09T08:25:19.682001+0000","last_undegraded":"2024-05-09T08:25:19.682001+0000","last_fullsized":"2024-05-09T08:25:19.682001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:15.232318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.499482+0000","last_change":"2024-05-09T08:23:55.674324+0000","last_active":"2024-05-09T08:24:51.499482+0000","last_peered":"2024-05-09T08:24:51.499482+0000","last_clean":"2024-05-09T08:24:51.499482+0000","last_became_active":"2024-05-09T08:23:55.673254+0000","last_became_peered":"2024-05-09T08:23:55.673254+0000","last_unstale":"2024-05-09T08:24:51.499482+0000","last_undegraded":"2024-05-09T08:24:51.499482+0000","last_fullsized":"2024-05-09T08:24:51.499482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:05:23.303987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":107,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:31.542589+0000","last_change":"2024-05-09T08:23:50.571682+0000","last_active":"2024-05-09T08:25:31.542589+0000","last_peered":"2024-05-09T08:25:31.542589+0000","last_clean":"2024-05-09T08:25:31.542589+0000","last_became_active":"2024-05-09T08:23:50.571338+0000","last_became_peered":"2024-05-09T08:23:50.571338+0000","last_unstale":"2024-05-09T08:25:31.542589+0000","last_undegraded":"2024-05-09T08:25:31.542589+0000","last_fullsized":"2024-05-09T08:25:31.542589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:17.578071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.672368+0000","last_change":"2024-05-09T08:24:51.487208+0000","last_active":"2024-05-09T08:25:19.672368+0000","last_peered":"2024-05-09T08:25:19.672368+0000","last_clean":"2024-05-09T08:25:19.672368+0000","last_became_active":"2024-05-09T08:24:51.486665+0000","last_became_peered":"2024-05-09T08:24:51.486665+0000","last_unstale":"2024-05-09T08:25:19.672368+0000","last_undegraded":"2024-05-09T08:25:19.672368+0000","last_fullsized":"2024-05-09T08:25:19.672368+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:01:10.050846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.500000+0000","last_change":"2024-05-09T08:23:55.853086+0000","last_active":"2024-05-09T08:24:51.500000+0000","last_peered":"2024-05-09T08:24:51.500000+0000","last_clean":"2024-05-09T08:24:51.500000+0000","last_became_active":"2024-05-09T08:23:55.852302+0000","last_became_peered":"2024-05-09T08:23:55.852302+0000","last_unstale":"2024-05-09T08:24:51.500000+0000","last_undegraded":"2024-05-09T08:24:51.500000+0000","last_fullsized":"2024-05-09T08:24:51.500000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:42:18.547623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.666524+0000","last_change":"2024-05-09T08:23:51.575522+0000","last_active":"2024-05-09T08:25:19.666524+0000","last_peered":"2024-05-09T08:25:19.666524+0000","last_clean":"2024-05-09T08:25:19.666524+0000","last_became_active":"2024-05-09T08:23:51.575204+0000","last_became_peered":"2024-05-09T08:23:51.575204+0000","last_unstale":"2024-05-09T08:25:19.666524+0000","last_undegraded":"2024-05-09T08:25:19.666524+0000","last_fullsized":"2024-05-09T08:25:19.666524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:57.257435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:31.530466+0000","last_change":"2024-05-09T08:23:50.574996+0000","last_active":"2024-05-09T08:25:31.530466+0000","last_peered":"2024-05-09T08:25:31.530466+0000","last_clean":"2024-05-09T08:25:31.530466+0000","last_became_active":"2024-05-09T08:23:50.574619+0000","last_became_peered":"2024-05-09T08:23:50.574619+0000","last_unstale":"2024-05-09T08:25:31.530466+0000","last_undegraded":"2024-05-09T08:25:31.530466+0000","last_fullsized":"2024-05-09T08:25:31.530466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:44.681314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.493390+0000","last_change":"2024-05-09T08:24:51.493390+0000","last_active":"2024-05-09T08:24:51.493390+0000","last_peered":"2024-05-09T08:24:51.493390+0000","last_clean":"2024-05-09T08:24:51.493390+0000","last_became_active":"2024-05-09T08:24:51.487797+0000","last_became_peered":"2024-05-09T08:24:51.487797+0000","last_unstale":"2024-05-09T08:24:51.493390+0000","last_undegraded":"2024-05-09T08:24:51.493390+0000","last_fullsized":"2024-05-09T08:24:51.493390+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:46.646471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.678468+0000","last_change":"2024-05-09T08:23:51.573944+0000","last_active":"2024-05-09T08:25:19.678468+0000","last_peered":"2024-05-09T08:25:19.678468+0000","last_clean":"2024-05-09T08:25:19.678468+0000","last_became_active":"2024-05-09T08:23:51.573229+0000","last_became_peered":"2024-05-09T08:23:51.573229+0000","last_unstale":"2024-05-09T08:25:19.678468+0000","last_undegraded":"2024-05-09T08:25:19.678468+0000","last_fullsized":"2024-05-09T08:25:19.678468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:53.828582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485419+0000","last_change":"2024-05-09T08:23:50.566415+0000","last_active":"2024-05-09T08:24:51.485419+0000","last_peered":"2024-05-09T08:24:51.485419+0000","last_clean":"2024-05-09T08:24:51.485419+0000","last_became_active":"2024-05-09T08:23:50.565506+0000","last_became_peered":"2024-05-09T08:23:50.565506+0000","last_unstale":"2024-05-09T08:24:51.485419+0000","last_undegraded":"2024-05-09T08:24:51.485419+0000","last_fullsized":"2024-05-09T08:24:51.485419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:15.741565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485306+0000","last_change":"2024-05-09T08:22:38.366126+0000","last_active":"2024-05-09T08:24:51.485306+0000","last_peered":"2024-05-09T08:24:51.485306+0000","last_clean":"2024-05-09T08:24:51.485306+0000","last_became_active":"2024-05-09T08:22:38.365683+0000","last_became_peered":"2024-05-09T08:22:38.365683+0000","last_unstale":"2024-05-09T08:24:51.485306+0000","last_undegraded":"2024-05-09T08:24:51.485306+0000","last_fullsized":"2024-05-09T08:24:51.485306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_clean_scrub_stamp":"2024-05-09T08:19:12.435445+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:04.744060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.486182+0000","last_change":"2024-05-09T08:23:55.853248+0000","last_active":"2024-05-09T08:24:51.486182+0000","last_peered":"2024-05-09T08:24:51.486182+0000","last_clean":"2024-05-09T08:24:51.486182+0000","last_became_active":"2024-05-09T08:23:55.852646+0000","last_became_peered":"2024-05-09T08:23:55.852646+0000","last_unstale":"2024-05-09T08:24:51.486182+0000","last_undegraded":"2024-05-09T08:24:51.486182+0000","last_fullsized":"2024-05-09T08:24:51.486182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:19:54.378185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.678368+0000","last_change":"2024-05-09T08:23:51.574967+0000","last_active":"2024-05-09T08:25:19.678368+0000","last_peered":"2024-05-09T08:25:19.678368+0000","last_clean":"2024-05-09T08:25:19.678368+0000","last_became_active":"2024-05-09T08:23:51.574600+0000","last_became_peered":"2024-05-09T08:23:51.574600+0000","last_unstale":"2024-05-09T08:25:19.678368+0000","last_undegraded":"2024-05-09T08:25:19.678368+0000","last_fullsized":"2024-05-09T08:25:19.678368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:03.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485355+0000","last_change":"2024-05-09T08:23:50.569886+0000","last_active":"2024-05-09T08:24:51.485355+0000","last_peered":"2024-05-09T08:24:51.485355+0000","last_clean":"2024-05-09T08:24:51.485355+0000","last_became_active":"2024-05-09T08:23:50.569018+0000","last_became_peered":"2024-05-09T08:23:50.569018+0000","last_unstale":"2024-05-09T08:24:51.485355+0000","last_undegraded":"2024-05-09T08:24:51.485355+0000","last_fullsized":"2024-05-09T08:24:51.485355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:20.711485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480994+0000","last_change":"2024-05-09T08:23:55.676408+0000","last_active":"2024-05-09T08:24:51.480994+0000","last_peered":"2024-05-09T08:24:51.480994+0000","last_clean":"2024-05-09T08:24:51.480994+0000","last_became_active":"2024-05-09T08:23:55.675964+0000","last_became_peered":"2024-05-09T08:23:55.675964+0000","last_unstale":"2024-05-09T08:24:51.480994+0000","last_undegraded":"2024-05-09T08:24:51.480994+0000","last_fullsized":"2024-05-09T08:24:51.480994+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:29.210602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481491+0000","last_change":"2024-05-09T08:23:50.566946+0000","last_active":"2024-05-09T08:24:51.481491+0000","last_peered":"2024-05-09T08:24:51.481491+0000","last_clean":"2024-05-09T08:24:51.481491+0000","last_became_active":"2024-05-09T08:23:50.566366+0000","last_became_peered":"2024-05-09T08:23:50.566366+0000","last_unstale":"2024-05-09T08:24:51.481491+0000","last_undegraded":"2024-05-09T08:24:51.481491+0000","last_fullsized":"2024-05-09T08:24:51.481491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:37:30.720879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.673914+0000","last_change":"2024-05-09T08:23:51.571520+0000","last_active":"2024-05-09T08:25:19.673914+0000","last_peered":"2024-05-09T08:25:19.673914+0000","last_clean":"2024-05-09T08:25:19.673914+0000","last_became_active":"2024-05-09T08:23:51.571166+0000","last_became_peered":"2024-05-09T08:23:51.571166+0000","last_unstale":"2024-05-09T08:25:19.673914+0000","last_undegraded":"2024-05-09T08:25:19.673914+0000","last_fullsized":"2024-05-09T08:25:19.673914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:02:12.560238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483815+0000","last_change":"2024-05-09T08:23:55.852649+0000","last_active":"2024-05-09T08:24:51.483815+0000","last_peered":"2024-05-09T08:24:51.483815+0000","last_clean":"2024-05-09T08:24:51.483815+0000","last_became_active":"2024-05-09T08:23:55.852180+0000","last_became_peered":"2024-05-09T08:23:55.852180+0000","last_unstale":"2024-05-09T08:24:51.483815+0000","last_undegraded":"2024-05-09T08:24:51.483815+0000","last_fullsized":"2024-05-09T08:24:51.483815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:33.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479218+0000","last_change":"2024-05-09T08:23:50.561639+0000","last_active":"2024-05-09T08:24:51.479218+0000","last_peered":"2024-05-09T08:24:51.479218+0000","last_clean":"2024-05-09T08:24:51.479218+0000","last_became_active":"2024-05-09T08:23:50.561306+0000","last_became_peered":"2024-05-09T08:23:50.561306+0000","last_unstale":"2024-05-09T08:24:51.479218+0000","last_undegraded":"2024-05-09T08:24:51.479218+0000","last_fullsized":"2024-05-09T08:24:51.479218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:46.897965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.683808+0000","last_change":"2024-05-09T08:23:51.573765+0000","last_active":"2024-05-09T08:25:19.683808+0000","last_peered":"2024-05-09T08:25:19.683808+0000","last_clean":"2024-05-09T08:25:19.683808+0000","last_became_active":"2024-05-09T08:23:51.573134+0000","last_became_peered":"2024-05-09T08:23:51.573134+0000","last_unstale":"2024-05-09T08:25:19.683808+0000","last_undegraded":"2024-05-09T08:25:19.683808+0000","last_fullsized":"2024-05-09T08:25:19.683808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:25.334148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.478699+0000","last_change":"2024-05-09T08:23:55.663480+0000","last_active":"2024-05-09T08:24:51.478699+0000","last_peered":"2024-05-09T08:24:51.478699+0000","last_clean":"2024-05-09T08:24:51.478699+0000","last_became_active":"2024-05-09T08:23:55.662823+0000","last_became_peered":"2024-05-09T08:23:55.662823+0000","last_unstale":"2024-05-09T08:24:51.478699+0000","last_undegraded":"2024-05-09T08:24:51.478699+0000","last_fullsized":"2024-05-09T08:24:51.478699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:18.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481834+0000","last_change":"2024-05-09T08:23:50.574641+0000","last_active":"2024-05-09T08:24:51.481834+0000","last_peered":"2024-05-09T08:24:51.481834+0000","last_clean":"2024-05-09T08:24:51.481834+0000","last_became_active":"2024-05-09T08:23:50.574266+0000","last_became_peered":"2024-05-09T08:23:50.574266+0000","last_unstale":"2024-05-09T08:24:51.481834+0000","last_undegraded":"2024-05-09T08:24:51.481834+0000","last_fullsized":"2024-05-09T08:24:51.481834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.575211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.684985+0000","last_change":"2024-05-09T08:23:51.583698+0000","last_active":"2024-05-09T08:25:19.684985+0000","last_peered":"2024-05-09T08:25:19.684985+0000","last_clean":"2024-05-09T08:25:19.684985+0000","last_became_active":"2024-05-09T08:23:51.583342+0000","last_became_peered":"2024-05-09T08:23:51.583342+0000","last_unstale":"2024-05-09T08:25:19.684985+0000","last_undegraded":"2024-05-09T08:25:19.684985+0000","last_fullsized":"2024-05-09T08:25:19.684985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:37.444163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481329+0000","last_change":"2024-05-09T08:23:55.853759+0000","last_active":"2024-05-09T08:24:51.481329+0000","last_peered":"2024-05-09T08:24:51.481329+0000","last_clean":"2024-05-09T08:24:51.481329+0000","last_became_active":"2024-05-09T08:23:55.853045+0000","last_became_peered":"2024-05-09T08:23:55.853045+0000","last_unstale":"2024-05-09T08:24:51.481329+0000","last_undegraded":"2024-05-09T08:24:51.481329+0000","last_fullsized":"2024-05-09T08:24:51.481329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:53.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480004+0000","last_change":"2024-05-09T08:23:50.568373+0000","last_active":"2024-05-09T08:24:51.480004+0000","last_peered":"2024-05-09T08:24:51.480004+0000","last_clean":"2024-05-09T08:24:51.480004+0000","last_became_active":"2024-05-09T08:23:50.568015+0000","last_became_peered":"2024-05-09T08:23:50.568015+0000","last_unstale":"2024-05-09T08:24:51.480004+0000","last_undegraded":"2024-05-09T08:24:51.480004+0000","last_fullsized":"2024-05-09T08:24:51.480004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:05:16.391673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.630236+0000","last_change":"2024-05-09T08:23:51.576825+0000","last_active":"2024-05-09T08:25:19.630236+0000","last_peered":"2024-05-09T08:25:19.630236+0000","last_clean":"2024-05-09T08:25:19.630236+0000","last_became_active":"2024-05-09T08:23:51.573350+0000","last_became_peered":"2024-05-09T08:23:51.573350+0000","last_unstale":"2024-05-09T08:25:19.630236+0000","last_undegraded":"2024-05-09T08:25:19.630236+0000","last_fullsized":"2024-05-09T08:25:19.630236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:00.963281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.479110+0000","last_change":"2024-05-09T08:23:55.672145+0000","last_active":"2024-05-09T08:24:51.479110+0000","last_peered":"2024-05-09T08:24:51.479110+0000","last_clean":"2024-05-09T08:24:51.479110+0000","last_became_active":"2024-05-09T08:23:55.671743+0000","last_became_peered":"2024-05-09T08:23:55.671743+0000","last_unstale":"2024-05-09T08:24:51.479110+0000","last_undegraded":"2024-05-09T08:24:51.479110+0000","last_fullsized":"2024-05-09T08:24:51.479110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:09:23.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.483985+0000","last_change":"2024-05-09T08:23:50.568908+0000","last_active":"2024-05-09T08:24:51.483985+0000","last_peered":"2024-05-09T08:24:51.483985+0000","last_clean":"2024-05-09T08:24:51.483985+0000","last_became_active":"2024-05-09T08:23:50.567009+0000","last_became_peered":"2024-05-09T08:23:50.567009+0000","last_unstale":"2024-05-09T08:24:51.483985+0000","last_undegraded":"2024-05-09T08:24:51.483985+0000","last_fullsized":"2024-05-09T08:24:51.483985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:35.190156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.660457+0000","last_change":"2024-05-09T08:23:51.580121+0000","last_active":"2024-05-09T08:25:19.660457+0000","last_peered":"2024-05-09T08:25:19.660457+0000","last_clean":"2024-05-09T08:25:19.660457+0000","last_became_active":"2024-05-09T08:23:51.579402+0000","last_became_peered":"2024-05-09T08:23:51.579402+0000","last_unstale":"2024-05-09T08:25:19.660457+0000","last_undegraded":"2024-05-09T08:25:19.660457+0000","last_fullsized":"2024-05-09T08:25:19.660457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:11:26.144344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.481111+0000","last_change":"2024-05-09T08:23:55.664017+0000","last_active":"2024-05-09T08:24:51.481111+0000","last_peered":"2024-05-09T08:24:51.481111+0000","last_clean":"2024-05-09T08:24:51.481111+0000","last_became_active":"2024-05-09T08:23:55.663546+0000","last_became_peered":"2024-05-09T08:23:55.663546+0000","last_unstale":"2024-05-09T08:24:51.481111+0000","last_undegraded":"2024-05-09T08:24:51.481111+0000","last_fullsized":"2024-05-09T08:24:51.481111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:13:14.705971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.493220+0000","last_change":"2024-05-09T08:24:51.493220+0000","last_active":"2024-05-09T08:24:51.493220+0000","last_peered":"2024-05-09T08:24:51.493220+0000","last_clean":"2024-05-09T08:24:51.493220+0000","last_became_active":"2024-05-09T08:24:51.486332+0000","last_became_peered":"2024-05-09T08:24:51.486332+0000","last_unstale":"2024-05-09T08:24:51.493220+0000","last_undegraded":"2024-05-09T08:24:51.493220+0000","last_fullsized":"2024-05-09T08:24:51.493220+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:23.694545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.687747+0000","last_change":"2024-05-09T08:23:51.571414+0000","last_active":"2024-05-09T08:25:19.687747+0000","last_peered":"2024-05-09T08:25:19.687747+0000","last_clean":"2024-05-09T08:25:19.687747+0000","last_became_active":"2024-05-09T08:23:51.571085+0000","last_became_peered":"2024-05-09T08:23:51.571085+0000","last_unstale":"2024-05-09T08:25:19.687747+0000","last_undegraded":"2024-05-09T08:25:19.687747+0000","last_fullsized":"2024-05-09T08:25:19.687747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:37.513584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.485240+0000","last_change":"2024-05-09T08:23:55.852785+0000","last_active":"2024-05-09T08:24:51.485240+0000","last_peered":"2024-05-09T08:24:51.485240+0000","last_clean":"2024-05-09T08:24:51.485240+0000","last_became_active":"2024-05-09T08:23:55.852252+0000","last_became_peered":"2024-05-09T08:23:55.852252+0000","last_unstale":"2024-05-09T08:24:51.485240+0000","last_undegraded":"2024-05-09T08:24:51.485240+0000","last_fullsized":"2024-05-09T08:24:51.485240+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:44.503295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480830+0000","last_change":"2024-05-09T08:23:50.564787+0000","last_active":"2024-05-09T08:24:51.480830+0000","last_peered":"2024-05-09T08:24:51.480830+0000","last_clean":"2024-05-09T08:24:51.480830+0000","last_became_active":"2024-05-09T08:23:50.564409+0000","last_became_peered":"2024-05-09T08:23:50.564409+0000","last_unstale":"2024-05-09T08:24:51.480830+0000","last_undegraded":"2024-05-09T08:24:51.480830+0000","last_fullsized":"2024-05-09T08:24:51.480830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:23.051221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681007+0000","last_change":"2024-05-09T08:23:51.576830+0000","last_active":"2024-05-09T08:25:19.681007+0000","last_peered":"2024-05-09T08:25:19.681007+0000","last_clean":"2024-05-09T08:25:19.681007+0000","last_became_active":"2024-05-09T08:23:51.576311+0000","last_became_peered":"2024-05-09T08:23:51.576311+0000","last_unstale":"2024-05-09T08:25:19.681007+0000","last_undegraded":"2024-05-09T08:25:19.681007+0000","last_fullsized":"2024-05-09T08:25:19.681007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:26:08.651061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.480322+0000","last_change":"2024-05-09T08:23:55.672309+0000","last_active":"2024-05-09T08:24:51.480322+0000","last_peered":"2024-05-09T08:24:51.480322+0000","last_clean":"2024-05-09T08:24:51.480322+0000","last_became_active":"2024-05-09T08:23:55.671811+0000","last_became_peered":"2024-05-09T08:23:55.671811+0000","last_unstale":"2024-05-09T08:24:51.480322+0000","last_undegraded":"2024-05-09T08:24:51.480322+0000","last_fullsized":"2024-05-09T08:24:51.480322+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:19.400949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":107,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:30.530373+0000","last_change":"2024-05-09T08:23:50.575317+0000","last_active":"2024-05-09T08:25:30.530373+0000","last_peered":"2024-05-09T08:25:30.530373+0000","last_clean":"2024-05-09T08:25:30.530373+0000","last_became_active":"2024-05-09T08:23:50.574995+0000","last_became_peered":"2024-05-09T08:23:50.574995+0000","last_unstale":"2024-05-09T08:25:30.530373+0000","last_undegraded":"2024-05-09T08:25:30.530373+0000","last_fullsized":"2024-05-09T08:25:30.530373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:32:54.801799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.677140+0000","last_change":"2024-05-09T08:23:51.575904+0000","last_active":"2024-05-09T08:25:19.677140+0000","last_peered":"2024-05-09T08:25:19.677140+0000","last_clean":"2024-05-09T08:25:19.677140+0000","last_became_active":"2024-05-09T08:23:51.575599+0000","last_became_peered":"2024-05-09T08:23:51.575599+0000","last_unstale":"2024-05-09T08:25:19.677140+0000","last_undegraded":"2024-05-09T08:25:19.677140+0000","last_fullsized":"2024-05-09T08:25:19.677140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:20:13.087096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.486362+0000","last_change":"2024-05-09T08:23:55.680683+0000","last_active":"2024-05-09T08:24:51.486362+0000","last_peered":"2024-05-09T08:24:51.486362+0000","last_clean":"2024-05-09T08:24:51.486362+0000","last_became_active":"2024-05-09T08:23:55.680232+0000","last_became_peered":"2024-05-09T08:23:55.680232+0000","last_unstale":"2024-05-09T08:24:51.486362+0000","last_undegraded":"2024-05-09T08:24:51.486362+0000","last_fullsized":"2024-05-09T08:24:51.486362+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:33.583552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.484104+0000","last_change":"2024-05-09T08:23:50.575802+0000","last_active":"2024-05-09T08:24:51.484104+0000","last_peered":"2024-05-09T08:24:51.484104+0000","last_clean":"2024-05-09T08:24:51.484104+0000","last_became_active":"2024-05-09T08:23:50.575416+0000","last_became_peered":"2024-05-09T08:23:50.575416+0000","last_unstale":"2024-05-09T08:24:51.484104+0000","last_undegraded":"2024-05-09T08:24:51.484104+0000","last_fullsized":"2024-05-09T08:24:51.484104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:24:04.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682257+0000","last_change":"2024-05-09T08:23:51.578189+0000","last_active":"2024-05-09T08:25:19.682257+0000","last_peered":"2024-05-09T08:25:19.682257+0000","last_clean":"2024-05-09T08:25:19.682257+0000","last_became_active":"2024-05-09T08:23:51.577500+0000","last_became_peered":"2024-05-09T08:23:51.577500+0000","last_unstale":"2024-05-09T08:25:19.682257+0000","last_undegraded":"2024-05-09T08:25:19.682257+0000","last_fullsized":"2024-05-09T08:25:19.682257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:26:24.718791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.489679+0000","last_change":"2024-05-09T08:24:51.487659+0000","last_active":"2024-05-09T08:24:51.489679+0000","last_peered":"2024-05-09T08:24:51.489679+0000","last_clean":"2024-05-09T08:24:51.489679+0000","last_became_active":"2024-05-09T08:24:51.487212+0000","last_became_peered":"2024-05-09T08:24:51.487212+0000","last_unstale":"2024-05-09T08:24:51.489679+0000","last_undegraded":"2024-05-09T08:24:51.489679+0000","last_fullsized":"2024-05-09T08:24:51.489679+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:29:57.077744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:24:51.499956+0000","last_change":"2024-05-09T08:23:50.578438+0000","last_active":"2024-05-09T08:24:51.499956+0000","last_peered":"2024-05-09T08:24:51.499956+0000","last_clean":"2024-05-09T08:24:51.499956+0000","last_became_active":"2024-05-09T08:23:50.578039+0000","last_became_peered":"2024-05-09T08:23:50.578039+0000","last_unstale":"2024-05-09T08:24:51.499956+0000","last_undegraded":"2024-05-09T08:24:51.499956+0000","last_fullsized":"2024-05-09T08:24:51.499956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:59.420193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.670419+0000","last_change":"2024-05-09T08:23:51.568037+0000","last_active":"2024-05-09T08:25:19.670419+0000","last_peered":"2024-05-09T08:25:19.670419+0000","last_clean":"2024-05-09T08:25:19.670419+0000","last_became_active":"2024-05-09T08:23:51.567706+0000","last_became_peered":"2024-05-09T08:23:51.567706+0000","last_unstale":"2024-05-09T08:25:19.670419+0000","last_undegraded":"2024-05-09T08:25:19.670419+0000","last_fullsized":"2024-05-09T08:25:19.670419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:15:19.623805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.663888+0000","last_change":"2024-05-09T08:23:51.579885+0000","last_active":"2024-05-09T08:25:19.663888+0000","last_peered":"2024-05-09T08:25:19.663888+0000","last_clean":"2024-05-09T08:25:19.663888+0000","last_became_active":"2024-05-09T08:23:51.578953+0000","last_became_peered":"2024-05-09T08:23:51.578953+0000","last_unstale":"2024-05-09T08:25:19.663888+0000","last_undegraded":"2024-05-09T08:25:19.663888+0000","last_fullsized":"2024-05-09T08:25:19.663888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:57:23.793119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.669419+0000","last_change":"2024-05-09T08:23:51.583423+0000","last_active":"2024-05-09T08:25:19.669419+0000","last_peered":"2024-05-09T08:25:19.669419+0000","last_clean":"2024-05-09T08:25:19.669419+0000","last_became_active":"2024-05-09T08:23:51.583092+0000","last_became_peered":"2024-05-09T08:23:51.583092+0000","last_unstale":"2024-05-09T08:25:19.669419+0000","last_undegraded":"2024-05-09T08:25:19.669419+0000","last_fullsized":"2024-05-09T08:25:19.669419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:44:47.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.685762+0000","last_change":"2024-05-09T08:23:51.582116+0000","last_active":"2024-05-09T08:25:19.685762+0000","last_peered":"2024-05-09T08:25:19.685762+0000","last_clean":"2024-05-09T08:25:19.685762+0000","last_became_active":"2024-05-09T08:23:51.581715+0000","last_became_peered":"2024-05-09T08:23:51.581715+0000","last_unstale":"2024-05-09T08:25:19.685762+0000","last_undegraded":"2024-05-09T08:25:19.685762+0000","last_fullsized":"2024-05-09T08:25:19.685762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:17.528758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.684388+0000","last_change":"2024-05-09T08:23:51.575959+0000","last_active":"2024-05-09T08:25:19.684388+0000","last_peered":"2024-05-09T08:25:19.684388+0000","last_clean":"2024-05-09T08:25:19.684388+0000","last_became_active":"2024-05-09T08:23:51.575592+0000","last_became_peered":"2024-05-09T08:23:51.575592+0000","last_unstale":"2024-05-09T08:25:19.684388+0000","last_undegraded":"2024-05-09T08:25:19.684388+0000","last_fullsized":"2024-05-09T08:25:19.684388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:26.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.686361+0000","last_change":"2024-05-09T08:23:51.570756+0000","last_active":"2024-05-09T08:25:19.686361+0000","last_peered":"2024-05-09T08:25:19.686361+0000","last_clean":"2024-05-09T08:25:19.686361+0000","last_became_active":"2024-05-09T08:23:51.570412+0000","last_became_peered":"2024-05-09T08:23:51.570412+0000","last_unstale":"2024-05-09T08:25:19.686361+0000","last_undegraded":"2024-05-09T08:25:19.686361+0000","last_fullsized":"2024-05-09T08:25:19.686361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:11:53.959559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.663117+0000","last_change":"2024-05-09T08:23:51.572612+0000","last_active":"2024-05-09T08:25:19.663117+0000","last_peered":"2024-05-09T08:25:19.663117+0000","last_clean":"2024-05-09T08:25:19.663117+0000","last_became_active":"2024-05-09T08:23:51.572227+0000","last_became_peered":"2024-05-09T08:23:51.572227+0000","last_unstale":"2024-05-09T08:25:19.663117+0000","last_undegraded":"2024-05-09T08:25:19.663117+0000","last_fullsized":"2024-05-09T08:25:19.663117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:18.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.661992+0000","last_change":"2024-05-09T08:23:51.573021+0000","last_active":"2024-05-09T08:25:19.661992+0000","last_peered":"2024-05-09T08:25:19.661992+0000","last_clean":"2024-05-09T08:25:19.661992+0000","last_became_active":"2024-05-09T08:23:51.572716+0000","last_became_peered":"2024-05-09T08:23:51.572716+0000","last_unstale":"2024-05-09T08:25:19.661992+0000","last_undegraded":"2024-05-09T08:25:19.661992+0000","last_fullsized":"2024-05-09T08:25:19.661992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:31.494329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.683102+0000","last_change":"2024-05-09T08:23:51.572687+0000","last_active":"2024-05-09T08:25:19.683102+0000","last_peered":"2024-05-09T08:25:19.683102+0000","last_clean":"2024-05-09T08:25:19.683102+0000","last_became_active":"2024-05-09T08:23:51.572339+0000","last_became_peered":"2024-05-09T08:23:51.572339+0000","last_unstale":"2024-05-09T08:25:19.683102+0000","last_undegraded":"2024-05-09T08:25:19.683102+0000","last_fullsized":"2024-05-09T08:25:19.683102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:24.012360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.681218+0000","last_change":"2024-05-09T08:23:51.569404+0000","last_active":"2024-05-09T08:25:19.681218+0000","last_peered":"2024-05-09T08:25:19.681218+0000","last_clean":"2024-05-09T08:25:19.681218+0000","last_became_active":"2024-05-09T08:23:51.569044+0000","last_became_peered":"2024-05-09T08:23:51.569044+0000","last_unstale":"2024-05-09T08:25:19.681218+0000","last_undegraded":"2024-05-09T08:25:19.681218+0000","last_fullsized":"2024-05-09T08:25:19.681218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:26:01.343606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.680609+0000","last_change":"2024-05-09T08:23:51.577483+0000","last_active":"2024-05-09T08:25:19.680609+0000","last_peered":"2024-05-09T08:25:19.680609+0000","last_clean":"2024-05-09T08:25:19.680609+0000","last_became_active":"2024-05-09T08:23:51.577078+0000","last_became_peered":"2024-05-09T08:23:51.577078+0000","last_unstale":"2024-05-09T08:25:19.680609+0000","last_undegraded":"2024-05-09T08:25:19.680609+0000","last_fullsized":"2024-05-09T08:25:19.680609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:29.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.682945+0000","last_change":"2024-05-09T08:23:51.579736+0000","last_active":"2024-05-09T08:25:19.682945+0000","last_peered":"2024-05-09T08:25:19.682945+0000","last_clean":"2024-05-09T08:25:19.682945+0000","last_became_active":"2024-05-09T08:23:51.579409+0000","last_became_peered":"2024-05-09T08:23:51.579409+0000","last_unstale":"2024-05-09T08:25:19.682945+0000","last_undegraded":"2024-05-09T08:25:19.682945+0000","last_fullsized":"2024-05-09T08:25:19.682945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:05.697642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.686998+0000","last_change":"2024-05-09T08:23:51.569461+0000","last_active":"2024-05-09T08:25:19.686998+0000","last_peered":"2024-05-09T08:25:19.686998+0000","last_clean":"2024-05-09T08:25:19.686998+0000","last_became_active":"2024-05-09T08:23:51.569085+0000","last_became_peered":"2024-05-09T08:23:51.569085+0000","last_unstale":"2024-05-09T08:25:19.686998+0000","last_undegraded":"2024-05-09T08:25:19.686998+0000","last_fullsized":"2024-05-09T08:25:19.686998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:54.554949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:25:19.666790+0000","last_change":"2024-05-09T08:23:51.570593+0000","last_active":"2024-05-09T08:25:19.666790+0000","last_peered":"2024-05-09T08:25:19.666790+0000","last_clean":"2024-05-09T08:25:19.666790+0000","last_became_active":"2024-05-09T08:23:51.570195+0000","last_became_peered":"2024-05-09T08:23:51.570195+0000","last_unstale":"2024-05-09T08:25:19.666790+0000","last_undegraded":"2024-05-09T08:25:19.666790+0000","last_fullsized":"2024-05-09T08:25:19.666790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:18.376268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283814,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70559,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":752,"num_read_kb":450,"num_write":248,"num_write_kb":145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":64162,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":211677,"data_compressed":13616,"data_compressed_allocated":20480,"data_compressed_original":77824,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":501447,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32304,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712848,"statfs":{"total":95995035648,"available":95961956352,"internally_reserved":0,"allocated":1155072,"data_stored":1156474,"data_compressed":58336,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":501355,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":501039,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1098371,"data_compressed":56735,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496859,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":562166,"data_compressed":22245,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":498315,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":497997,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":496567,"data_compressed":20644,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":565182,"data_compressed":31058,"data_compressed_allocated":40960,"data_compressed_original":118784,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1085440,"data_stored":1093421,"data_compressed":56735,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34453,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66029,"data_compressed":1601,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":64949,"data_compressed":10414,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":65873,"data_compressed":1601,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:25:35.078 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-09T08:25:35.078 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-09T08:25:35.078 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-09T08:25:35.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph health --format=json 2024-05-09T08:25:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:35.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:35 smithi165 ceph-mon[34540]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-05-09T08:25:35.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:35 smithi165 ceph-mon[34540]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:25:35.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:35 smithi184 ceph-mon[34509]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-05-09T08:25:35.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:35 smithi184 ceph-mon[34509]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:25:35.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:35 smithi028 ceph-mon[30119]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-05-09T08:25:35.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:35 smithi028 ceph-mon[30119]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:25:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 -- 172.21.15.28:0/4145875882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc006aa50 msgr2=0x7fedc006aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 --2- 172.21.15.28:0/4145875882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc006aa50 0x7fedc006aeb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7feda8009990 tx=0x7feda8009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 -- 172.21.15.28:0/4145875882 shutdown_connections 2024-05-09T08:25:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 --2- 172.21.15.28:0/4145875882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedc006b3f0 0x7fedc010f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 --2- 172.21.15.28:0/4145875882 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedc01083a0 0x7fedc0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 --2- 172.21.15.28:0/4145875882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc006aa50 0x7fedc006aeb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 -- 172.21.15.28:0/4145875882 >> 172.21.15.28:0/4145875882 conn(0x7fedc00781d0 msgr2=0x7fedc00785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:37.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 -- 172.21.15.28:0/4145875882 shutdown_connections 2024-05-09T08:25:37.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 -- 172.21.15.28:0/4145875882 wait complete. 2024-05-09T08:25:37.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 Processor -- start 2024-05-09T08:25:37.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.096+0000 7fedc5b05700 1 -- start start 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedc5b05700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedc006aa50 0x7fedc019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedc5b05700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedc006b3f0 0x7fedc019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedc5b05700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc01083a0 0x7fedc01989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedc5b05700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedc0112660 con 0x7fedc01083a0 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedc5b05700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fedc01124e0 con 0x7fedc006b3f0 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedc5b05700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedc01127e0 con 0x7fedc006aa50 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedbffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc01083a0 0x7fedc01989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedbffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc01083a0 0x7fedc01989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40592/0 (socket says 172.21.15.28:40592) 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedbffff700 1 -- 172.21.15.28:0/1142383336 learned_addr learned my addr 172.21.15.28:0/1142383336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedbf7fe700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedc006aa50 0x7fedc019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedb6ffd700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedc006b3f0 0x7fedc019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedbffff700 1 -- 172.21.15.28:0/1142383336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedc006aa50 msgr2=0x7fedc019e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedbffff700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedc006aa50 0x7fedc019e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedbffff700 1 -- 172.21.15.28:0/1142383336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedc006b3f0 msgr2=0x7fedc019ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedbffff700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedc006b3f0 0x7fedc019ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.097+0000 7fedbffff700 1 -- 172.21.15.28:0/1142383336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc0199100 con 0x7fedc01083a0 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.098+0000 7fedbffff700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc01083a0 0x7fedc01989a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fedb800ada0 tx=0x7fedb800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.098+0000 7fedbd7fa700 1 -- 172.21.15.28:0/1142383336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8017020 con 0x7fedc01083a0 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.098+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc01993f0 con 0x7fedc01083a0 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.098+0000 7fedbd7fa700 1 -- 172.21.15.28:0/1142383336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb800cdb0 con 0x7fedc01083a0 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.098+0000 7fedbd7fa700 1 -- 172.21.15.28:0/1142383336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb80053e0 con 0x7fedc01083a0 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.098+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc01a56b0 con 0x7fedc01083a0 2024-05-09T08:25:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.099+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc0069010 con 0x7fedc01083a0 2024-05-09T08:25:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.100+0000 7fedbd7fa700 1 -- 172.21.15.28:0/1142383336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fedb800c8d0 con 0x7fedc01083a0 2024-05-09T08:25:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.101+0000 7fedbd7fa700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fedac0a2c20 0x7fedac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.101+0000 7fedbd7fa700 1 -- 172.21.15.28:0/1142383336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fedb80ceaa0 con 0x7fedc01083a0 2024-05-09T08:25:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.101+0000 7fedbf7fe700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fedac0a2c20 0x7fedac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.101+0000 7fedbf7fe700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fedac0a2c20 0x7fedac0a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fedc0199ae0 tx=0x7fedb0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.106+0000 7fedbd7fa700 1 -- 172.21.15.28:0/1142383336 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fedb8021020 con 0x7fedc01083a0 2024-05-09T08:25:37.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:37 smithi165 ceph-mon[34540]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:25:37.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:37 smithi184 ceph-mon[34509]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:25:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.358+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fedc002db80 con 0x7fedc01083a0 2024-05-09T08:25:37.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:37 smithi028 ceph-mon[30119]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:25:37.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.359+0000 7fedbd7fa700 1 -- 172.21.15.28:0/1142383336 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fedb809b730 con 0x7fedc01083a0 2024-05-09T08:25:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:25:37.360 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-09T08:25:37.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fedac0a2c20 msgr2=0x7fedac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:37.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fedac0a2c20 0x7fedac0a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fedc0199ae0 tx=0x7fedb0009190 comp rx=0 tx=0).stop 2024-05-09T08:25:37.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc01083a0 msgr2=0x7fedc01989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:37.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc01083a0 0x7fedc01989a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fedb800ada0 tx=0x7fedb800c330 comp rx=0 tx=0).stop 2024-05-09T08:25:37.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 shutdown_connections 2024-05-09T08:25:37.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fedac0a2c20 0x7fedac0a50e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedc006aa50 0x7fedc019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc01083a0 0x7fedc01989a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 --2- 172.21.15.28:0/1142383336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedc006b3f0 0x7fedc019ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:37.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 >> 172.21.15.28:0/1142383336 conn(0x7fedc00781d0 msgr2=0x7fedc01000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:37.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 shutdown_connections 2024-05-09T08:25:37.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:37.361+0000 7fedc5b05700 1 -- 172.21.15.28:0/1142383336 wait complete. 2024-05-09T08:25:38.229 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-09T08:25:38.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:25:38.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 -- 172.21.15.28:0/771503823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c107610 msgr2=0x7f268c109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 --2- 172.21.15.28:0/771503823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c107610 0x7f268c109a00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2684009ca0 tx=0x7f268402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 -- 172.21.15.28:0/771503823 shutdown_connections 2024-05-09T08:25:38.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 --2- 172.21.15.28:0/771503823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c107610 0x7f268c109a00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 --2- 172.21.15.28:0/771503823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f268c104ce0 0x7f268c1070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 --2- 172.21.15.28:0/771503823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f268c1023b0 0x7f268c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 -- 172.21.15.28:0/771503823 >> 172.21.15.28:0/771503823 conn(0x7f268c0fbd20 msgr2=0x7f268c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 -- 172.21.15.28:0/771503823 shutdown_connections 2024-05-09T08:25:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 -- 172.21.15.28:0/771503823 wait complete. 2024-05-09T08:25:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 Processor -- start 2024-05-09T08:25:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.342+0000 7f2692770640 1 -- start start 2024-05-09T08:25:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f2692770640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1023b0 0x7f268c101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f2692770640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f268c104ce0 0x7f268c100360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f2692770640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f268c107610 0x7f268c1008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f2692770640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10d510 con 0x7f268c1023b0 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f2692770640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10d390 con 0x7f268c107610 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f2692770640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10d690 con 0x7f268c104ce0 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1023b0 0x7f268c101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1023b0 0x7f268c101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40608/0 (socket says 172.21.15.28:40608) 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268bfff640 1 -- 172.21.15.28:0/3190728898 learned_addr learned my addr 172.21.15.28:0/3190728898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f2690d0d640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f268c107610 0x7f268c1008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268b7fe640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f268c104ce0 0x7f268c100360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268bfff640 1 -- 172.21.15.28:0/3190728898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f268c104ce0 msgr2=0x7f268c100360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268bfff640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f268c104ce0 0x7f268c100360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268bfff640 1 -- 172.21.15.28:0/3190728898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f268c107610 msgr2=0x7f268c1008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268bfff640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f268c107610 0x7f268c1008a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268bfff640 1 -- 172.21.15.28:0/3190728898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c101130 con 0x7f268c1023b0 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f2690d0d640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f268c107610 0x7f268c1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f268bfff640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1023b0 0x7f268c101d20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f267c00c890 tx=0x7f267c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f26897fa640 1 -- 172.21.15.28:0/3190728898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c004220 con 0x7f268c1023b0 2024-05-09T08:25:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c1a1c70 con 0x7f268c1023b0 2024-05-09T08:25:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f26897fa640 1 -- 172.21.15.28:0/3190728898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f267c0043c0 con 0x7f268c1023b0 2024-05-09T08:25:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.343+0000 7f26897fa640 1 -- 172.21.15.28:0/3190728898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c004a00 con 0x7f268c1023b0 2024-05-09T08:25:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.344+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c1a21c0 con 0x7f268c1023b0 2024-05-09T08:25:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.344+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f268c0fd810 con 0x7f268c1023b0 2024-05-09T08:25:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.346+0000 7f26897fa640 1 -- 172.21.15.28:0/3190728898 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f267c0079b0 con 0x7f268c1023b0 2024-05-09T08:25:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.346+0000 7f26897fa640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f26740a2b90 0x7f26740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.346+0000 7f268b7fe640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f26740a2b90 0x7f26740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.346+0000 7f26897fa640 1 -- 172.21.15.28:0/3190728898 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f267c0cf1a0 con 0x7f268c1023b0 2024-05-09T08:25:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.347+0000 7f268b7fe640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f26740a2b90 0x7f26740a5050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2680005fd0 tx=0x7f2680005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.351+0000 7f26897fa640 1 -- 172.21.15.28:0/3190728898 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f267c014030 con 0x7f268c1023b0 2024-05-09T08:25:38.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:38 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1142383336' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T08:25:38.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:38 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1142383336' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T08:25:38.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:38 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1142383336' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T08:25:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.540+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f268c062430 con 0x7f268c1023b0 2024-05-09T08:25:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.542+0000 7f26897fa640 1 -- 172.21.15.28:0/3190728898 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24666 (secure 0 0 0) 0x7f267c09bda0 con 0x7f268c1023b0 2024-05-09T08:25:38.543 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:25:38.543 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:24:51.475713+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:25:38.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f26740a2b90 msgr2=0x7f26740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f26740a2b90 0x7f26740a5050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2680005fd0 tx=0x7f2680005850 comp rx=0 tx=0).stop 2024-05-09T08:25:38.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1023b0 msgr2=0x7f268c101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1023b0 0x7f268c101d20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f267c00c890 tx=0x7f267c00cd50 comp rx=0 tx=0).stop 2024-05-09T08:25:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 shutdown_connections 2024-05-09T08:25:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f26740a2b90 0x7f26740a5050 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f268c107610 0x7f268c1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f268c104ce0 0x7f268c100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 --2- 172.21.15.28:0/3190728898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1023b0 0x7f268c101d20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 >> 172.21.15.28:0/3190728898 conn(0x7f268c0fbd20 msgr2=0x7f268c102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 shutdown_connections 2024-05-09T08:25:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.544+0000 7f2692770640 1 -- 172.21.15.28:0/3190728898 wait complete. 2024-05-09T08:25:38.556 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:25:38.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:25:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.665+0000 7f804939e640 1 -- 172.21.15.28:0/3255612006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8044104b20 msgr2=0x7f8044106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.665+0000 7f804939e640 1 --2- 172.21.15.28:0/3255612006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8044104b20 0x7f8044106f10 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8038009900 tx=0x7f803802a410 comp rx=0 tx=0).stop 2024-05-09T08:25:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.665+0000 7f804939e640 1 -- 172.21.15.28:0/3255612006 shutdown_connections 2024-05-09T08:25:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.665+0000 7f804939e640 1 --2- 172.21.15.28:0/3255612006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044107450 0x7f80441098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.665+0000 7f804939e640 1 --2- 172.21.15.28:0/3255612006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8044104b20 0x7f8044106f10 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.665+0000 7f804939e640 1 --2- 172.21.15.28:0/3255612006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80441021f0 0x7f80441045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.665+0000 7f804939e640 1 -- 172.21.15.28:0/3255612006 >> 172.21.15.28:0/3255612006 conn(0x7f80440fbbe0 msgr2=0x7f80440fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.665+0000 7f804939e640 1 -- 172.21.15.28:0/3255612006 shutdown_connections 2024-05-09T08:25:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.665+0000 7f804939e640 1 -- 172.21.15.28:0/3255612006 wait complete. 2024-05-09T08:25:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.666+0000 7f804939e640 1 Processor -- start 2024-05-09T08:25:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.666+0000 7f804939e640 1 -- start start 2024-05-09T08:25:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f804939e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80441021f0 0x7f8044197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f804939e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8044104b20 0x7f8044197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f804939e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044107450 0x7f804419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f804939e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f804410d3e0 con 0x7f80441021f0 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f804939e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f804410d260 con 0x7f8044107450 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f804939e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f804410d560 con 0x7f8044104b20 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8043fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80441021f0 0x7f8044197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8043fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80441021f0 0x7f8044197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40612/0 (socket says 172.21.15.28:40612) 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8043fff640 1 -- 172.21.15.28:0/1483494320 learned_addr learned my addr 172.21.15.28:0/1483494320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8048b9d640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044107450 0x7f804419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f80437fe640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8044104b20 0x7f8044197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8043fff640 1 -- 172.21.15.28:0/1483494320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8044104b20 msgr2=0x7f8044197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8043fff640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8044104b20 0x7f8044197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8043fff640 1 -- 172.21.15.28:0/1483494320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044107450 msgr2=0x7f804419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8043fff640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044107450 0x7f804419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8043fff640 1 -- 172.21.15.28:0/1483494320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804419f680 con 0x7f80441021f0 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8048b9d640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044107450 0x7f804419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f8043fff640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80441021f0 0x7f8044197a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f803400d950 tx=0x7f803400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f80417fa640 1 -- 172.21.15.28:0/1483494320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8034017030 con 0x7f80441021f0 2024-05-09T08:25:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f80417fa640 1 -- 172.21.15.28:0/1483494320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8034004020 con 0x7f80441021f0 2024-05-09T08:25:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f804419f970 con 0x7f80441021f0 2024-05-09T08:25:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f80417fa640 1 -- 172.21.15.28:0/1483494320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80340047a0 con 0x7f80441021f0 2024-05-09T08:25:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.667+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f804419fec0 con 0x7f80441021f0 2024-05-09T08:25:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.669+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80440fd6d0 con 0x7f80441021f0 2024-05-09T08:25:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.670+0000 7f80417fa640 1 -- 172.21.15.28:0/1483494320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f80340041d0 con 0x7f80441021f0 2024-05-09T08:25:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.670+0000 7f80417fa640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f802c0a2b90 0x7f802c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.670+0000 7f80417fa640 1 -- 172.21.15.28:0/1483494320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f80340ceeb0 con 0x7f80441021f0 2024-05-09T08:25:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.670+0000 7f80437fe640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f802c0a2b90 0x7f802c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.671+0000 7f80437fe640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f802c0a2b90 0x7f802c0a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8038006010 tx=0x7f8038000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.675+0000 7f80417fa640 1 -- 172.21.15.28:0/1483494320 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f803409bab0 con 0x7f80441021f0 2024-05-09T08:25:38.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.859+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f804402d1c0 con 0x7f80441021f0 2024-05-09T08:25:38.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.860+0000 7f80417fa640 1 -- 172.21.15.28:0/1483494320 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f804402d1c0 con 0x7f80441021f0 2024-05-09T08:25:38.861 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f802c0a2b90 msgr2=0x7f802c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f802c0a2b90 0x7f802c0a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8038006010 tx=0x7f8038000bc0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80441021f0 msgr2=0x7f8044197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80441021f0 0x7f8044197a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f803400d950 tx=0x7f803400de10 comp rx=0 tx=0).stop 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 shutdown_connections 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f802c0a2b90 0x7f802c0a5050 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044107450 0x7f804419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8044104b20 0x7f8044197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 --2- 172.21.15.28:0/1483494320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80441021f0 0x7f8044197a60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 >> 172.21.15.28:0/1483494320 conn(0x7f80440fbbe0 msgr2=0x7f8044102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:38.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 shutdown_connections 2024-05-09T08:25:38.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.862+0000 7f804939e640 1 -- 172.21.15.28:0/1483494320 wait complete. 2024-05-09T08:25:38.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:25:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.982+0000 7fd3f174f640 1 -- 172.21.15.28:0/2331892165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec104ca0 msgr2=0x7fd3ec107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.982+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2331892165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec104ca0 0x7fd3ec107090 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009900 tx=0x7fd3d402a3f0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.982+0000 7fd3f174f640 1 -- 172.21.15.28:0/2331892165 shutdown_connections 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.982+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2331892165 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec1075d0 0x7fd3ec1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.982+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2331892165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec104ca0 0x7fd3ec107090 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.982+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2331892165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3ec102370 0x7fd3ec104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.982+0000 7fd3f174f640 1 -- 172.21.15.28:0/2331892165 >> 172.21.15.28:0/2331892165 conn(0x7fd3ec0fbd20 msgr2=0x7fd3ec0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.982+0000 7fd3f174f640 1 -- 172.21.15.28:0/2331892165 shutdown_connections 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.982+0000 7fd3f174f640 1 -- 172.21.15.28:0/2331892165 wait complete. 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3f174f640 1 Processor -- start 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3f174f640 1 -- start start 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3f174f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec102370 0x7fd3ec197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3f174f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3ec104ca0 0x7fd3ec198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3f174f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec1075d0 0x7fd3ec19f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3f174f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec10d560 con 0x7fd3ec102370 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3f174f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec10d3e0 con 0x7fd3ec1075d0 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3f174f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec10d6e0 con 0x7fd3ec104ca0 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec102370 0x7fd3ec197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec102370 0x7fd3ec197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40620/0 (socket says 172.21.15.28:40620) 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eaffd640 1 -- 172.21.15.28:0/2893175967 learned_addr learned my addr 172.21.15.28:0/2893175967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3ea7fc640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3ec104ca0 0x7fd3ec198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eb7fe640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec1075d0 0x7fd3ec19f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eaffd640 1 -- 172.21.15.28:0/2893175967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3ec104ca0 msgr2=0x7fd3ec198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eaffd640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3ec104ca0 0x7fd3ec198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eaffd640 1 -- 172.21.15.28:0/2893175967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec1075d0 msgr2=0x7fd3ec19f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eaffd640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec1075d0 0x7fd3ec19f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eaffd640 1 -- 172.21.15.28:0/2893175967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ec19f830 con 0x7fd3ec102370 2024-05-09T08:25:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.983+0000 7fd3eb7fe640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec1075d0 0x7fd3ec19f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.984+0000 7fd3eaffd640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec102370 0x7fd3ec197c20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00ba50 tx=0x7fd3dc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.984+0000 7fd3e3fff640 1 -- 172.21.15.28:0/2893175967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc019070 con 0x7fd3ec102370 2024-05-09T08:25:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.984+0000 7fd3e3fff640 1 -- 172.21.15.28:0/2893175967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc00cbe0 con 0x7fd3ec102370 2024-05-09T08:25:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.984+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ec19fb20 con 0x7fd3ec102370 2024-05-09T08:25:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.984+0000 7fd3e3fff640 1 -- 172.21.15.28:0/2893175967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0076e0 con 0x7fd3ec102370 2024-05-09T08:25:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.984+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ec1a0070 con 0x7fd3ec102370 2024-05-09T08:25:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.985+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ec0fd810 con 0x7fd3ec102370 2024-05-09T08:25:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.986+0000 7fd3e3fff640 1 -- 172.21.15.28:0/2893175967 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd3dc01c070 con 0x7fd3ec102370 2024-05-09T08:25:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.987+0000 7fd3e3fff640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd3d80a2b90 0x7fd3d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.987+0000 7fd3ea7fc640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd3d80a2b90 0x7fd3d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.987+0000 7fd3e3fff640 1 -- 172.21.15.28:0/2893175967 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd3dc0cea40 con 0x7fd3ec102370 2024-05-09T08:25:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.987+0000 7fd3ea7fc640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd3d80a2b90 0x7fd3d80a5050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4005ab0 tx=0x7fd3d4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:38.991+0000 7fd3e3fff640 1 -- 172.21.15.28:0/2893175967 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd3dc09af90 con 0x7fd3ec102370 2024-05-09T08:25:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.177+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd3ec0624d0 con 0x7fd3ec102370 2024-05-09T08:25:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.177+0000 7fd3e3fff640 1 -- 172.21.15.28:0/2893175967 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fd3dc09f290 con 0x7fd3ec102370 2024-05-09T08:25:39.178 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:25:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.179+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd3d80a2b90 msgr2=0x7fd3d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd3d80a2b90 0x7fd3d80a5050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4005ab0 tx=0x7fd3d4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec102370 msgr2=0x7fd3ec197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec102370 0x7fd3ec197c20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00ba50 tx=0x7fd3dc00bf10 comp rx=0 tx=0).stop 2024-05-09T08:25:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 shutdown_connections 2024-05-09T08:25:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd3d80a2b90 0x7fd3d80a5050 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec1075d0 0x7fd3ec19f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3ec104ca0 0x7fd3ec198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 --2- 172.21.15.28:0/2893175967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3ec102370 0x7fd3ec197c20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 >> 172.21.15.28:0/2893175967 conn(0x7fd3ec0fbd20 msgr2=0x7fd3ec102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 shutdown_connections 2024-05-09T08:25:39.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.180+0000 7fd3f174f640 1 -- 172.21.15.28:0/2893175967 wait complete. 2024-05-09T08:25:39.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:39 smithi028 ceph-mon[30119]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:25:39.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:39 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3190728898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:25:39.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:39 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1483494320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:25:39.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:25:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.301+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/2705421739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 msgr2=0x7f7d38108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.301+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/2705421739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 0x7f7d38108bb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009900 tx=0x7f7d3002a410 comp rx=0 tx=0).stop 2024-05-09T08:25:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/2705421739 shutdown_connections 2024-05-09T08:25:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/2705421739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 0x7f7d38108bb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/2705421739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d38101960 0x7f7d38101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/2705421739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d38100770 0x7f7d38100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/2705421739 >> 172.21.15.28:0/2705421739 conn(0x7f7d380fbce0 msgr2=0x7f7d380fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/2705421739 shutdown_connections 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/2705421739 wait complete. 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 Processor -- start 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 -- start start 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d38100770 0x7f7d38197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d38101960 0x7f7d38198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 0x7f7d3819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.302+0000 7f7d3f8cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3810d4d0 con 0x7f7d38102320 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3f8cd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3810d350 con 0x7f7d38100770 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3f8cd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3810d650 con 0x7f7d38101960 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3de6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 0x7f7d3819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3de6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 0x7f7d3819f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40638/0 (socket says 172.21.15.28:40638) 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3de6a640 1 -- 172.21.15.28:0/556910045 learned_addr learned my addr 172.21.15.28:0/556910045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3ce68640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d38101960 0x7f7d38198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3d669640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d38100770 0x7f7d38197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3de6a640 1 -- 172.21.15.28:0/556910045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d38101960 msgr2=0x7f7d38198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3de6a640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d38101960 0x7f7d38198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3de6a640 1 -- 172.21.15.28:0/556910045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d38100770 msgr2=0x7f7d38197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3de6a640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d38100770 0x7f7d38197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3de6a640 1 -- 172.21.15.28:0/556910045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3819f760 con 0x7f7d38102320 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3d669640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d38100770 0x7f7d38197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3de6a640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 0x7f7d3819f0f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7d300314e0 tx=0x7f7d30031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d2e7fc640 1 -- 172.21.15.28:0/556910045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3003f030 con 0x7f7d38102320 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d2e7fc640 1 -- 172.21.15.28:0/556910045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d30003ad0 con 0x7f7d38102320 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3819f9f0 con 0x7f7d38102320 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d2e7fc640 1 -- 172.21.15.28:0/556910045 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d30035070 con 0x7f7d38102320 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.303+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3819ff30 con 0x7f7d38102320 2024-05-09T08:25:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.304+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d38107a90 con 0x7f7d38102320 2024-05-09T08:25:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.306+0000 7f7d2e7fc640 1 -- 172.21.15.28:0/556910045 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7d30042030 con 0x7f7d38102320 2024-05-09T08:25:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.306+0000 7f7d2e7fc640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7d240a2c10 0x7f7d240a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.306+0000 7f7d3d669640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7d240a2c10 0x7f7d240a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.306+0000 7f7d2e7fc640 1 -- 172.21.15.28:0/556910045 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7d300efd50 con 0x7f7d38102320 2024-05-09T08:25:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.307+0000 7f7d3d669640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7d240a2c10 0x7f7d240a50d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7d381017c0 tx=0x7f7d340095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:39.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.309+0000 7f7d2e7fc640 1 -- 172.21.15.28:0/556910045 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7d300bc950 con 0x7f7d38102320 2024-05-09T08:25:39.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:39 smithi165 ceph-mon[34540]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:25:39.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:39 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3190728898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:25:39.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:39 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1483494320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:25:39.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:39 smithi184 ceph-mon[34509]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:25:39.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:39 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3190728898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:25:39.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:39 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1483494320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:25:39.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.494+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7d3802d1c0 con 0x7f7d38102320 2024-05-09T08:25:39.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.495+0000 7f7d2e7fc640 1 -- 172.21.15.28:0/556910045 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f7d300bc250 con 0x7f7d38102320 2024-05-09T08:25:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:25:39.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7d240a2c10 msgr2=0x7f7d240a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7d240a2c10 0x7f7d240a50d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7d381017c0 tx=0x7f7d340095c0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 msgr2=0x7f7d3819f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 0x7f7d3819f0f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7d300314e0 tx=0x7f7d30031b60 comp rx=0 tx=0).stop 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 shutdown_connections 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7d240a2c10 0x7f7d240a50d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d38102320 0x7f7d3819f0f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d38101960 0x7f7d38198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 --2- 172.21.15.28:0/556910045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d38100770 0x7f7d38197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 >> 172.21.15.28:0/556910045 conn(0x7f7d380fbce0 msgr2=0x7f7d38104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 shutdown_connections 2024-05-09T08:25:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.497+0000 7f7d3f8cd640 1 -- 172.21.15.28:0/556910045 wait complete. 2024-05-09T08:25:39.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:25:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.617+0000 7f3dd0666640 1 -- 172.21.15.28:0/3355884844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc81009e0 msgr2=0x7f3dc8100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.617+0000 7f3dd0666640 1 --2- 172.21.15.28:0/3355884844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc81009e0 0x7f3dc8100e60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009900 tx=0x7f3db802a410 comp rx=0 tx=0).stop 2024-05-09T08:25:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.617+0000 7f3dd0666640 1 -- 172.21.15.28:0/3355884844 shutdown_connections 2024-05-09T08:25:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.617+0000 7f3dd0666640 1 --2- 172.21.15.28:0/3355884844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dc81013a0 0x7f3dc8108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.617+0000 7f3dd0666640 1 --2- 172.21.15.28:0/3355884844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc81009e0 0x7f3dc8100e60 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.617+0000 7f3dd0666640 1 --2- 172.21.15.28:0/3355884844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc8101d90 0x7f3dc81021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.617+0000 7f3dd0666640 1 -- 172.21.15.28:0/3355884844 >> 172.21.15.28:0/3355884844 conn(0x7f3dc80fb470 msgr2=0x7f3dc80fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 -- 172.21.15.28:0/3355884844 shutdown_connections 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 -- 172.21.15.28:0/3355884844 wait complete. 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 Processor -- start 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 -- start start 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dc81009e0 0x7f3dc8197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc81013a0 0x7f3dc8198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc8101d90 0x7f3dc819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc810d580 con 0x7f3dc8101d90 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc810d400 con 0x7f3dc81013a0 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dd0666640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc810d700 con 0x7f3dc81009e0 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.618+0000 7f3dcec03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc8101d90 0x7f3dc819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dcec03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc8101d90 0x7f3dc819f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40660/0 (socket says 172.21.15.28:40660) 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dcec03640 1 -- 172.21.15.28:0/2671460217 learned_addr learned my addr 172.21.15.28:0/2671460217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dcdc01640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc81013a0 0x7f3dc8198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dce402640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dc81009e0 0x7f3dc8197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dcec03640 1 -- 172.21.15.28:0/2671460217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dc81009e0 msgr2=0x7f3dc8197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dcec03640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dc81009e0 0x7f3dc8197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dcec03640 1 -- 172.21.15.28:0/2671460217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc81013a0 msgr2=0x7f3dc8198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dcec03640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc81013a0 0x7f3dc8198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dcec03640 1 -- 172.21.15.28:0/2671460217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc819f810 con 0x7f3dc8101d90 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dcec03640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc8101d90 0x7f3dc819f1a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0009fd0 tx=0x7f3dc000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dbf7fe640 1 -- 172.21.15.28:0/2671460217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0017070 con 0x7f3dc8101d90 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc819fb00 con 0x7f3dc8101d90 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dbf7fe640 1 -- 172.21.15.28:0/2671460217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dc0003e50 con 0x7f3dc8101d90 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dbf7fe640 1 -- 172.21.15.28:0/2671460217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc00106c0 con 0x7f3dc8101d90 2024-05-09T08:25:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.619+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc819ffd0 con 0x7f3dc8101d90 2024-05-09T08:25:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.620+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc8107bf0 con 0x7f3dc8101d90 2024-05-09T08:25:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.621+0000 7f3dbf7fe640 1 -- 172.21.15.28:0/2671460217 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3dc0003970 con 0x7f3dc8101d90 2024-05-09T08:25:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.622+0000 7f3dbf7fe640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3db40a2b90 0x7f3db40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.622+0000 7f3dbf7fe640 1 -- 172.21.15.28:0/2671460217 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3dc00cf2b0 con 0x7f3dc8101d90 2024-05-09T08:25:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.622+0000 7f3dce402640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3db40a2b90 0x7f3db40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.623+0000 7f3dce402640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3db40a2b90 0x7f3db40a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8069c30 tx=0x7f3dc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.626+0000 7f3dbf7fe640 1 -- 172.21.15.28:0/2671460217 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3dc009be50 con 0x7f3dc8101d90 2024-05-09T08:25:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.817+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3dc8062430 con 0x7f3dc8101d90 2024-05-09T08:25:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.818+0000 7f3dbf7fe640 1 -- 172.21.15.28:0/2671460217 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f3dc009b750 con 0x7f3dc8101d90 2024-05-09T08:25:39.818 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3db40a2b90 msgr2=0x7f3db40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3db40a2b90 0x7f3db40a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8069c30 tx=0x7f3dc4005f90 comp rx=0 tx=0).stop 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc8101d90 msgr2=0x7f3dc819f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc8101d90 0x7f3dc819f1a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0009fd0 tx=0x7f3dc000b260 comp rx=0 tx=0).stop 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 shutdown_connections 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3db40a2b90 0x7f3db40a5050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dc8101d90 0x7f3dc819f1a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc81013a0 0x7f3dc8198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 --2- 172.21.15.28:0/2671460217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3dc81009e0 0x7f3dc8197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 >> 172.21.15.28:0/2671460217 conn(0x7f3dc80fb470 msgr2=0x7f3dc8104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 shutdown_connections 2024-05-09T08:25:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.820+0000 7f3dd0666640 1 -- 172.21.15.28:0/2671460217 wait complete. 2024-05-09T08:25:39.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:25:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 -- 172.21.15.28:0/3586280063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 msgr2=0x7f1ac0108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 --2- 172.21.15.28:0/3586280063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 0x7f1ac0108bb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009ca0 tx=0x7f1ab802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 -- 172.21.15.28:0/3586280063 shutdown_connections 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 --2- 172.21.15.28:0/3586280063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 0x7f1ac0108bb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 --2- 172.21.15.28:0/3586280063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ac0101960 0x7f1ac0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 --2- 172.21.15.28:0/3586280063 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ac0100770 0x7f1ac0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 -- 172.21.15.28:0/3586280063 >> 172.21.15.28:0/3586280063 conn(0x7f1ac00fbce0 msgr2=0x7f1ac00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 -- 172.21.15.28:0/3586280063 shutdown_connections 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 -- 172.21.15.28:0/3586280063 wait complete. 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.937+0000 7f1ac8210640 1 Processor -- start 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac8210640 1 -- start start 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac8210640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ac0100770 0x7f1ac0069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac8210640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ac0101960 0x7f1ac019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac8210640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 0x7f1ac019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac8210640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac010c420 con 0x7f1ac0102320 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac8210640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac010c2a0 con 0x7f1ac0100770 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac8210640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac010c5a0 con 0x7f1ac0101960 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac67ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 0x7f1ac019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac5fac640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ac0100770 0x7f1ac0069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac67ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 0x7f1ac019ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40678/0 (socket says 172.21.15.28:40678) 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac5fac640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ac0100770 0x7f1ac0069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:40320/0 (socket says 172.21.15.28:40320) 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac67ad640 1 -- 172.21.15.28:0/467986281 learned_addr learned my addr 172.21.15.28:0/467986281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac57ab640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ac0101960 0x7f1ac019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac67ad640 1 -- 172.21.15.28:0/467986281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ac0101960 msgr2=0x7f1ac019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac67ad640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ac0101960 0x7f1ac019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac67ad640 1 -- 172.21.15.28:0/467986281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ac0100770 msgr2=0x7f1ac0069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac67ad640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ac0100770 0x7f1ac0069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac67ad640 1 -- 172.21.15.28:0/467986281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac019f780 con 0x7f1ac0102320 2024-05-09T08:25:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.938+0000 7f1ac67ad640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 0x7f1ac019ee70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80319f0 tx=0x7f1ab802f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.939+0000 7f1ab6ffd640 1 -- 172.21.15.28:0/467986281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8003c70 con 0x7f1ac0102320 2024-05-09T08:25:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.939+0000 7f1ab6ffd640 1 -- 172.21.15.28:0/467986281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ab8003e10 con 0x7f1ac0102320 2024-05-09T08:25:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.939+0000 7f1ab6ffd640 1 -- 172.21.15.28:0/467986281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab802fd30 con 0x7f1ac0102320 2024-05-09T08:25:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.939+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac019f9b0 con 0x7f1ac0102320 2024-05-09T08:25:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.939+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac019feb0 con 0x7f1ac0102320 2024-05-09T08:25:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.940+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ac0107a30 con 0x7f1ac0102320 2024-05-09T08:25:39.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.941+0000 7f1ab6ffd640 1 -- 172.21.15.28:0/467986281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1ab8004040 con 0x7f1ac0102320 2024-05-09T08:25:39.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.941+0000 7f1ab6ffd640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aac0a2c10 0x7f1aac0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:39.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.942+0000 7f1ab6ffd640 1 -- 172.21.15.28:0/467986281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1ab80f09f0 con 0x7f1ac0102320 2024-05-09T08:25:39.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.942+0000 7f1ac5fac640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aac0a2c10 0x7f1aac0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:39.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.942+0000 7f1ac5fac640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aac0a2c10 0x7f1aac0a50d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1ac01017c0 tx=0x7f1abc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:39.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:39.946+0000 7f1ab6ffd640 1 -- 172.21.15.28:0/467986281 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1ab80bd5f0 con 0x7f1ac0102320 2024-05-09T08:25:40.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.170+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ac00624d0 con 0x7f1ac0102320 2024-05-09T08:25:40.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.171+0000 7f1ab6ffd640 1 -- 172.21.15.28:0/467986281 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f1ab80bcef0 con 0x7f1ac0102320 2024-05-09T08:25:40.171 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:25:40.172 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:24:19.404649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24481,24499,24476,24512,24463]},"id":1}]} 2024-05-09T08:25:40.172 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-05-09T08:25:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aac0a2c10 msgr2=0x7f1aac0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aac0a2c10 0x7f1aac0a50d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1ac01017c0 tx=0x7f1abc00b390 comp rx=0 tx=0).stop 2024-05-09T08:25:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 msgr2=0x7f1ac019ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 0x7f1ac019ee70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80319f0 tx=0x7f1ab802f450 comp rx=0 tx=0).stop 2024-05-09T08:25:40.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 shutdown_connections 2024-05-09T08:25:40.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1aac0a2c10 0x7f1aac0a50d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac0102320 0x7f1ac019ee70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ac0101960 0x7f1ac019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 --2- 172.21.15.28:0/467986281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ac0100770 0x7f1ac0069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 >> 172.21.15.28:0/467986281 conn(0x7f1ac00fbce0 msgr2=0x7f1ac0104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:40.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 shutdown_connections 2024-05-09T08:25:40.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.173+0000 7f1ac8210640 1 -- 172.21.15.28:0/467986281 wait complete. 2024-05-09T08:25:40.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:25:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.291+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/4260730807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38102340 msgr2=0x7f7f38108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.291+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/4260730807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38102340 0x7f7f38108bd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009ca0 tx=0x7f7f3002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.291+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/4260730807 shutdown_connections 2024-05-09T08:25:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.291+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/4260730807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38102340 0x7f7f38108bd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.291+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/4260730807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f38101980 0x7f7f38101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.291+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/4260730807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f38100790 0x7f7f38100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.291+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/4260730807 >> 172.21.15.28:0/4260730807 conn(0x7f7f380fbd20 msgr2=0x7f7f380fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.291+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/4260730807 shutdown_connections 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.291+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/4260730807 wait complete. 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f3ddc1640 1 Processor -- start 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f3ddc1640 1 -- start start 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f3ddc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38100790 0x7f7f3806aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f3ddc1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f38101980 0x7f7f3819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f3ddc1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f38102340 0x7f7f3819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f3ddc1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3810c470 con 0x7f7f38100790 2024-05-09T08:25:40.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f3ddc1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3810c2f0 con 0x7f7f38101980 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f3ddc1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3810c5f0 con 0x7f7f38102340 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38100790 0x7f7f3806aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38100790 0x7f7f3806aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40692/0 (socket says 172.21.15.28:40692) 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f377fe640 1 -- 172.21.15.28:0/305533390 learned_addr learned my addr 172.21.15.28:0/305533390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.292+0000 7f7f36ffd640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f38101980 0x7f7f3819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f37fff640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f38102340 0x7f7f3819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f377fe640 1 -- 172.21.15.28:0/305533390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f38102340 msgr2=0x7f7f3819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f377fe640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f38102340 0x7f7f3819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f377fe640 1 -- 172.21.15.28:0/305533390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f38101980 msgr2=0x7f7f3819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f377fe640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f38101980 0x7f7f3819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f377fe640 1 -- 172.21.15.28:0/305533390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3819f850 con 0x7f7f38100790 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f37fff640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f38102340 0x7f7f3819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f377fe640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38100790 0x7f7f3806aa00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800d770 tx=0x7f7f2800db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f34ff9640 1 -- 172.21.15.28:0/305533390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28017070 con 0x7f7f38100790 2024-05-09T08:25:40.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3819fae0 con 0x7f7f38100790 2024-05-09T08:25:40.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f34ff9640 1 -- 172.21.15.28:0/305533390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f28009d00 con 0x7f7f38100790 2024-05-09T08:25:40.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f34ff9640 1 -- 172.21.15.28:0/305533390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28003d30 con 0x7f7f38100790 2024-05-09T08:25:40.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.293+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f381a0080 con 0x7f7f38100790 2024-05-09T08:25:40.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.294+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f38107a50 con 0x7f7f38100790 2024-05-09T08:25:40.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.295+0000 7f7f34ff9640 1 -- 172.21.15.28:0/305533390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7f28009820 con 0x7f7f38100790 2024-05-09T08:25:40.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.296+0000 7f7f34ff9640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f200a2b90 0x7f7f200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.296+0000 7f7f34ff9640 1 -- 172.21.15.28:0/305533390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7f280ced40 con 0x7f7f38100790 2024-05-09T08:25:40.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.296+0000 7f7f36ffd640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f200a2b90 0x7f7f200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.296+0000 7f7f36ffd640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f200a2b90 0x7f7f200a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7f3806a770 tx=0x7f7f2c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:40.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.300+0000 7f7f34ff9640 1 -- 172.21.15.28:0/305533390 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7f2809b940 con 0x7f7f38100790 2024-05-09T08:25:40.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:40 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2893175967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:25:40.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:40 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/556910045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:25:40.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:40 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2671460217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:25:40.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:40 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2893175967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:25:40.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:40 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/556910045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:25:40.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:40 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2671460217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:25:40.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:40 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2893175967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:25:40.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:40 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/556910045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:25:40.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:40 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2671460217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:25:40.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.490+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f380624d0 con 0x7f7f38100790 2024-05-09T08:25:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.491+0000 7f7f34ff9640 1 -- 172.21.15.28:0/305533390 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24666 (secure 0 0 0) 0x7f7f2809b240 con 0x7f7f38100790 2024-05-09T08:25:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:25:40.492 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:24:51.475713+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:25:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f200a2b90 msgr2=0x7f7f200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f200a2b90 0x7f7f200a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7f3806a770 tx=0x7f7f2c009110 comp rx=0 tx=0).stop 2024-05-09T08:25:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38100790 msgr2=0x7f7f3806aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38100790 0x7f7f3806aa00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800d770 tx=0x7f7f2800db70 comp rx=0 tx=0).stop 2024-05-09T08:25:40.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 shutdown_connections 2024-05-09T08:25:40.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f200a2b90 0x7f7f200a5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f38102340 0x7f7f3819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f38101980 0x7f7f3819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 --2- 172.21.15.28:0/305533390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38100790 0x7f7f3806aa00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 >> 172.21.15.28:0/305533390 conn(0x7f7f380fbd20 msgr2=0x7f7f38104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:40.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.493+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 shutdown_connections 2024-05-09T08:25:40.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.494+0000 7f7f3ddc1640 1 -- 172.21.15.28:0/305533390 wait complete. 2024-05-09T08:25:40.506 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:25:40.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:25:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.614+0000 7f2947948640 1 -- 172.21.15.28:0/743741115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 msgr2=0x7f29401046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.614+0000 7f2947948640 1 --2- 172.21.15.28:0/743741115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 0x7f29401046e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2930009ca0 tx=0x7f293002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 -- 172.21.15.28:0/743741115 shutdown_connections 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 --2- 172.21.15.28:0/743741115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940107550 0x7f29401099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 --2- 172.21.15.28:0/743741115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2940104c20 0x7f2940107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 --2- 172.21.15.28:0/743741115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 0x7f29401046e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 -- 172.21.15.28:0/743741115 >> 172.21.15.28:0/743741115 conn(0x7f29400fbce0 msgr2=0x7f29400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 -- 172.21.15.28:0/743741115 shutdown_connections 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 -- 172.21.15.28:0/743741115 wait complete. 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 Processor -- start 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 -- start start 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 0x7f2940101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940104c20 0x7f29401002a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2940107550 0x7f29401007e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f294010d560 con 0x7f29401022f0 2024-05-09T08:25:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f294010d3e0 con 0x7f2940107550 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f2947948640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f294010d6e0 con 0x7f2940104c20 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.615+0000 7f29456e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 0x7f2940101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f29456e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 0x7f2940101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40712/0 (socket says 172.21.15.28:40712) 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f29456e4640 1 -- 172.21.15.28:0/4277342964 learned_addr learned my addr 172.21.15.28:0/4277342964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f29456e4640 1 -- 172.21.15.28:0/4277342964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940104c20 msgr2=0x7f29401002a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f29456e4640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940104c20 0x7f29401002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f29456e4640 1 -- 172.21.15.28:0/4277342964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2940107550 msgr2=0x7f29401007e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f29456e4640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2940107550 0x7f29401007e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f29456e4640 1 -- 172.21.15.28:0/4277342964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2940101070 con 0x7f29401022f0 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f29456e4640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 0x7f2940101c60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2930031a40 tx=0x7f2930031ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f293e7fc640 1 -- 172.21.15.28:0/4277342964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293002faa0 con 0x7f29401022f0 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f293e7fc640 1 -- 172.21.15.28:0/4277342964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293002fc40 con 0x7f29401022f0 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29401a1d80 con 0x7f29401022f0 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f293e7fc640 1 -- 172.21.15.28:0/4277342964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293003a790 con 0x7f29401022f0 2024-05-09T08:25:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.616+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29401a22c0 con 0x7f29401022f0 2024-05-09T08:25:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.617+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29400fd7d0 con 0x7f29401022f0 2024-05-09T08:25:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.618+0000 7f293e7fc640 1 -- 172.21.15.28:0/4277342964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2930030b30 con 0x7f29401022f0 2024-05-09T08:25:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.619+0000 7f293e7fc640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f29280a2c10 0x7f29280a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.619+0000 7f293e7fc640 1 -- 172.21.15.28:0/4277342964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f29300f0930 con 0x7f29401022f0 2024-05-09T08:25:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.619+0000 7f2944ee3640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f29280a2c10 0x7f29280a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.619+0000 7f2944ee3640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f29280a2c10 0x7f29280a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2934005fd0 tx=0x7f2934005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.623+0000 7f293e7fc640 1 -- 172.21.15.28:0/4277342964 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f29300bd4d0 con 0x7f29401022f0 2024-05-09T08:25:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.812+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2940062500 con 0x7f29401022f0 2024-05-09T08:25:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.812+0000 7f293e7fc640 1 -- 172.21.15.28:0/4277342964 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f29300bcdd0 con 0x7f29401022f0 2024-05-09T08:25:40.813 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:25:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f29280a2c10 msgr2=0x7f29280a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f29280a2c10 0x7f29280a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2934005fd0 tx=0x7f2934005850 comp rx=0 tx=0).stop 2024-05-09T08:25:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 msgr2=0x7f2940101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 0x7f2940101c60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2930031a40 tx=0x7f2930031ec0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 shutdown_connections 2024-05-09T08:25:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f29280a2c10 0x7f29280a50d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2940107550 0x7f29401007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2940104c20 0x7f29401002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 --2- 172.21.15.28:0/4277342964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29401022f0 0x7f2940101c60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.814+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 >> 172.21.15.28:0/4277342964 conn(0x7f29400fbce0 msgr2=0x7f2940102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.815+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 shutdown_connections 2024-05-09T08:25:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.815+0000 7f2947948640 1 -- 172.21.15.28:0/4277342964 wait complete. 2024-05-09T08:25:40.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:25:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.933+0000 7fe8244da640 1 -- 172.21.15.28:0/4224787471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c104cc0 msgr2=0x7fe81c1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.933+0000 7fe8244da640 1 --2- 172.21.15.28:0/4224787471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c104cc0 0x7fe81c1070b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009900 tx=0x7fe80c02a410 comp rx=0 tx=0).stop 2024-05-09T08:25:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.933+0000 7fe8244da640 1 -- 172.21.15.28:0/4224787471 shutdown_connections 2024-05-09T08:25:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.933+0000 7fe8244da640 1 --2- 172.21.15.28:0/4224787471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe81c1075f0 0x7fe81c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.933+0000 7fe8244da640 1 --2- 172.21.15.28:0/4224787471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c104cc0 0x7fe81c1070b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.933+0000 7fe8244da640 1 --2- 172.21.15.28:0/4224787471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe81c102390 0x7fe81c104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 -- 172.21.15.28:0/4224787471 >> 172.21.15.28:0/4224787471 conn(0x7fe81c0fbd20 msgr2=0x7fe81c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 -- 172.21.15.28:0/4224787471 shutdown_connections 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 -- 172.21.15.28:0/4224787471 wait complete. 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 Processor -- start 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 -- start start 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe81c102390 0x7fe81c101d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe81c104cc0 0x7fe81c100340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c1075f0 0x7fe81c100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c10d560 con 0x7fe81c1075f0 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c10d3e0 con 0x7fe81c104cc0 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe8244da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c10d6e0 con 0x7fe81c102390 2024-05-09T08:25:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe822a77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c1075f0 0x7fe81c100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe822a77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c1075f0 0x7fe81c100880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40724/0 (socket says 172.21.15.28:40724) 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.934+0000 7fe822a77640 1 -- 172.21.15.28:0/1925036056 learned_addr learned my addr 172.21.15.28:0/1925036056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe821a75640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe81c104cc0 0x7fe81c100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe822276640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe81c102390 0x7fe81c101d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe822a77640 1 -- 172.21.15.28:0/1925036056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe81c102390 msgr2=0x7fe81c101d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe822a77640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe81c102390 0x7fe81c101d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe822a77640 1 -- 172.21.15.28:0/1925036056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe81c104cc0 msgr2=0x7fe81c100340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe822a77640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe81c104cc0 0x7fe81c100340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe822a77640 1 -- 172.21.15.28:0/1925036056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c101110 con 0x7fe81c1075f0 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe822276640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe81c102390 0x7fe81c101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe821a75640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe81c104cc0 0x7fe81c100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe822a77640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c1075f0 0x7fe81c100880 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe81400cad0 tx=0x7fe81400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe8137fe640 1 -- 172.21.15.28:0/1925036056 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81400ae20 con 0x7fe81c1075f0 2024-05-09T08:25:40.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c1a1d90 con 0x7fe81c1075f0 2024-05-09T08:25:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe8137fe640 1 -- 172.21.15.28:0/1925036056 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe814005020 con 0x7fe81c1075f0 2024-05-09T08:25:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe8137fe640 1 -- 172.21.15.28:0/1925036056 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81400f6f0 con 0x7fe81c1075f0 2024-05-09T08:25:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.935+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c1a2270 con 0x7fe81c1075f0 2024-05-09T08:25:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.936+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe81c0fd810 con 0x7fe81c1075f0 2024-05-09T08:25:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.937+0000 7fe8137fe640 1 -- 172.21.15.28:0/1925036056 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe8140051d0 con 0x7fe81c1075f0 2024-05-09T08:25:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.938+0000 7fe8137fe640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8080a2b90 0x7fe8080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.938+0000 7fe822276640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8080a2b90 0x7fe8080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.938+0000 7fe8137fe640 1 -- 172.21.15.28:0/1925036056 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe8140cf900 con 0x7fe81c1075f0 2024-05-09T08:25:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.938+0000 7fe822276640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8080a2b90 0x7fe8080a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe818006fd0 tx=0x7fe818008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:40.942+0000 7fe8137fe640 1 -- 172.21.15.28:0/1925036056 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe814014030 con 0x7fe81c1075f0 2024-05-09T08:25:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.127+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe81c02d1c0 con 0x7fe81c1075f0 2024-05-09T08:25:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.128+0000 7fe8137fe640 1 -- 172.21.15.28:0/1925036056 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fe81409c500 con 0x7fe81c1075f0 2024-05-09T08:25:41.128 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:25:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8080a2b90 msgr2=0x7fe8080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8080a2b90 0x7fe8080a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe818006fd0 tx=0x7fe818008040 comp rx=0 tx=0).stop 2024-05-09T08:25:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c1075f0 msgr2=0x7fe81c100880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c1075f0 0x7fe81c100880 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe81400cad0 tx=0x7fe81400cf90 comp rx=0 tx=0).stop 2024-05-09T08:25:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 shutdown_connections 2024-05-09T08:25:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe8080a2b90 0x7fe8080a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe81c1075f0 0x7fe81c100880 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe81c104cc0 0x7fe81c100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 --2- 172.21.15.28:0/1925036056 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe81c102390 0x7fe81c101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 >> 172.21.15.28:0/1925036056 conn(0x7fe81c0fbd20 msgr2=0x7fe81c102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 shutdown_connections 2024-05-09T08:25:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.130+0000 7fe8244da640 1 -- 172.21.15.28:0/1925036056 wait complete. 2024-05-09T08:25:41.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:25:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 -- 172.21.15.28:0/1304428109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 msgr2=0x7f6388101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 --2- 172.21.15.28:0/1304428109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 0x7f6388101de0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f637c009900 tx=0x7f637c02a410 comp rx=0 tx=0).stop 2024-05-09T08:25:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 -- 172.21.15.28:0/1304428109 shutdown_connections 2024-05-09T08:25:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 --2- 172.21.15.28:0/1304428109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6388102320 0x7f6388108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 --2- 172.21.15.28:0/1304428109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 0x7f6388101de0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 --2- 172.21.15.28:0/1304428109 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6388100770 0x7f6388100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 -- 172.21.15.28:0/1304428109 >> 172.21.15.28:0/1304428109 conn(0x7f63880fbce0 msgr2=0x7f63880fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 -- 172.21.15.28:0/1304428109 shutdown_connections 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 -- 172.21.15.28:0/1304428109 wait complete. 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 Processor -- start 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 -- start start 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6388100770 0x7f6388069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 0x7f638819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.250+0000 7f638f24c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6388102320 0x7f638819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f638f24c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f638810c3d0 con 0x7f6388101960 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f638f24c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f638810c250 con 0x7f6388102320 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f638f24c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f638810c550 con 0x7f6388100770 2024-05-09T08:25:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6387fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 0x7f638819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6387fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 0x7f638819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40736/0 (socket says 172.21.15.28:40736) 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6387fff640 1 -- 172.21.15.28:0/1676658656 learned_addr learned my addr 172.21.15.28:0/1676658656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f638cfe8640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6388100770 0x7f6388069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f638d7e9640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6388102320 0x7f638819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6387fff640 1 -- 172.21.15.28:0/1676658656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6388100770 msgr2=0x7f6388069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6387fff640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6388100770 0x7f6388069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6387fff640 1 -- 172.21.15.28:0/1676658656 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6388102320 msgr2=0x7f638819eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6387fff640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6388102320 0x7f638819eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6387fff640 1 -- 172.21.15.28:0/1676658656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638819f7f0 con 0x7f6388101960 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f638d7e9640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6388102320 0x7f638819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6387fff640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 0x7f638819c8c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f637c009230 tx=0x7f637c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6385ffb640 1 -- 172.21.15.28:0/1676658656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c03f030 con 0x7f6388101960 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6385ffb640 1 -- 172.21.15.28:0/1676658656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f637c003ce0 con 0x7f6388101960 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638819fa20 con 0x7f6388101960 2024-05-09T08:25:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.251+0000 7f6385ffb640 1 -- 172.21.15.28:0/1676658656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c035070 con 0x7f6388101960 2024-05-09T08:25:41.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.252+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f638819ff10 con 0x7f6388101960 2024-05-09T08:25:41.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.253+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6388107ab0 con 0x7f6388101960 2024-05-09T08:25:41.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.254+0000 7f6385ffb640 1 -- 172.21.15.28:0/1676658656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f637c042030 con 0x7f6388101960 2024-05-09T08:25:41.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.254+0000 7f6385ffb640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f63700a2b90 0x7f63700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.254+0000 7f638cfe8640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f63700a2b90 0x7f63700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.254+0000 7f6385ffb640 1 -- 172.21.15.28:0/1676658656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f637c0efb10 con 0x7f6388101960 2024-05-09T08:25:41.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.255+0000 7f638cfe8640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f63700a2b90 0x7f63700a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f63881017c0 tx=0x7f637800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:41.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.259+0000 7f6385ffb640 1 -- 172.21.15.28:0/1676658656 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f637c0bc710 con 0x7f6388101960 2024-05-09T08:25:41.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:41 smithi165 ceph-mon[34540]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:41.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:41 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/467986281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:25:41.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:41 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/305533390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:25:41.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:41 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4277342964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:25:41.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:41 smithi184 ceph-mon[34509]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:41.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:41 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/467986281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:25:41.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:41 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/305533390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:25:41.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:41 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4277342964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:25:41.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:41 smithi028 ceph-mon[30119]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:41.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:41 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/467986281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:25:41.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:41 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/305533390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:25:41.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:41 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4277342964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:25:41.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.444+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f638802d1c0 con 0x7f6388101960 2024-05-09T08:25:41.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.444+0000 7f6385ffb640 1 -- 172.21.15.28:0/1676658656 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f637c0bc010 con 0x7f6388101960 2024-05-09T08:25:41.445 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:25:41.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f63700a2b90 msgr2=0x7f63700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f63700a2b90 0x7f63700a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f63881017c0 tx=0x7f637800a2d0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 msgr2=0x7f638819c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 0x7f638819c8c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f637c009230 tx=0x7f637c004430 comp rx=0 tx=0).stop 2024-05-09T08:25:41.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 shutdown_connections 2024-05-09T08:25:41.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f63700a2b90 0x7f63700a5050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6388102320 0x7f638819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6388101960 0x7f638819c8c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 --2- 172.21.15.28:0/1676658656 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6388100770 0x7f6388069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 >> 172.21.15.28:0/1676658656 conn(0x7f63880fbce0 msgr2=0x7f6388104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:41.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 shutdown_connections 2024-05-09T08:25:41.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.446+0000 7f638f24c640 1 -- 172.21.15.28:0/1676658656 wait complete. 2024-05-09T08:25:41.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:25:41.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.566+0000 7ffb4be95640 1 -- 172.21.15.28:0/3993827227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44102a80 msgr2=0x7ffb44102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.566+0000 7ffb4be95640 1 --2- 172.21.15.28:0/3993827227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44102a80 0x7ffb44102f00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009900 tx=0x7ffb3402a410 comp rx=0 tx=0).stop 2024-05-09T08:25:41.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.566+0000 7ffb4be95640 1 -- 172.21.15.28:0/3993827227 shutdown_connections 2024-05-09T08:25:41.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.566+0000 7ffb4be95640 1 --2- 172.21.15.28:0/3993827227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb44103440 0x7ffb44109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.566+0000 7ffb4be95640 1 --2- 172.21.15.28:0/3993827227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44102a80 0x7ffb44102f00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.566+0000 7ffb4be95640 1 --2- 172.21.15.28:0/3993827227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb44101890 0x7ffb44101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.566+0000 7ffb4be95640 1 -- 172.21.15.28:0/3993827227 >> 172.21.15.28:0/3993827227 conn(0x7ffb440fce00 msgr2=0x7ffb440ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.566+0000 7ffb4be95640 1 -- 172.21.15.28:0/3993827227 shutdown_connections 2024-05-09T08:25:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.566+0000 7ffb4be95640 1 -- 172.21.15.28:0/3993827227 wait complete. 2024-05-09T08:25:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4be95640 1 Processor -- start 2024-05-09T08:25:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4be95640 1 -- start start 2024-05-09T08:25:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4be95640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb44101890 0x7ffb44197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4be95640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb44102a80 0x7ffb441981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4be95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44103440 0x7ffb4419f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4be95640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4410d600 con 0x7ffb44103440 2024-05-09T08:25:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4be95640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4410d480 con 0x7ffb44102a80 2024-05-09T08:25:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4be95640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4410d780 con 0x7ffb44101890 2024-05-09T08:25:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4a432640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44103440 0x7ffb4419f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4a432640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44103440 0x7ffb4419f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40746/0 (socket says 172.21.15.28:40746) 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4a432640 1 -- 172.21.15.28:0/1200327296 learned_addr learned my addr 172.21.15.28:0/1200327296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb49c31640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb44101890 0x7ffb44197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb49430640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb44102a80 0x7ffb441981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4a432640 1 -- 172.21.15.28:0/1200327296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb44101890 msgr2=0x7ffb44197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4a432640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb44101890 0x7ffb44197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4a432640 1 -- 172.21.15.28:0/1200327296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb44102a80 msgr2=0x7ffb441981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4a432640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb44102a80 0x7ffb441981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4a432640 1 -- 172.21.15.28:0/1200327296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb4419f870 con 0x7ffb44103440 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb4a432640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44103440 0x7ffb4419f1d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00eaa0 tx=0x7ffb3c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.567+0000 7ffb3affd640 1 -- 172.21.15.28:0/1200327296 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c00cdb0 con 0x7ffb44103440 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.568+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb4419fb60 con 0x7ffb44103440 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.568+0000 7ffb3affd640 1 -- 172.21.15.28:0/1200327296 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb3c005020 con 0x7ffb44103440 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.568+0000 7ffb3affd640 1 -- 172.21.15.28:0/1200327296 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c0116d0 con 0x7ffb44103440 2024-05-09T08:25:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.568+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb441a0100 con 0x7ffb44103440 2024-05-09T08:25:41.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.569+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb44108b50 con 0x7ffb44103440 2024-05-09T08:25:41.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.570+0000 7ffb3affd640 1 -- 172.21.15.28:0/1200327296 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ffb3c0054d0 con 0x7ffb44103440 2024-05-09T08:25:41.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.571+0000 7ffb3affd640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb300a2b90 0x7ffb300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.571+0000 7ffb49c31640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb300a2b90 0x7ffb300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.571+0000 7ffb3affd640 1 -- 172.21.15.28:0/1200327296 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ffb3c0cfad0 con 0x7ffb44103440 2024-05-09T08:25:41.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.571+0000 7ffb49c31640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb300a2b90 0x7ffb300a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffb44199130 tx=0x7ffb40008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:41.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.574+0000 7ffb3affd640 1 -- 172.21.15.28:0/1200327296 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffb3c016030 con 0x7ffb44103440 2024-05-09T08:25:41.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.759+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ffb4402d3c0 con 0x7ffb44103440 2024-05-09T08:25:41.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.759+0000 7ffb3affd640 1 -- 172.21.15.28:0/1200327296 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7ffb3c09c670 con 0x7ffb44103440 2024-05-09T08:25:41.760 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:25:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb300a2b90 msgr2=0x7ffb300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb300a2b90 0x7ffb300a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffb44199130 tx=0x7ffb40008040 comp rx=0 tx=0).stop 2024-05-09T08:25:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44103440 msgr2=0x7ffb4419f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44103440 0x7ffb4419f1d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00eaa0 tx=0x7ffb3c00ef60 comp rx=0 tx=0).stop 2024-05-09T08:25:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 shutdown_connections 2024-05-09T08:25:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ffb300a2b90 0x7ffb300a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb44103440 0x7ffb4419f1d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb44102a80 0x7ffb441981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 --2- 172.21.15.28:0/1200327296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb44101890 0x7ffb44197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 >> 172.21.15.28:0/1200327296 conn(0x7ffb440fce00 msgr2=0x7ffb44105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 shutdown_connections 2024-05-09T08:25:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.761+0000 7ffb4be95640 1 -- 172.21.15.28:0/1200327296 wait complete. 2024-05-09T08:25:41.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:25:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 -- 172.21.15.28:0/3414781286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58101960 msgr2=0x7f0c58101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 --2- 172.21.15.28:0/3414781286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58101960 0x7f0c58101de0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009900 tx=0x7f0c4c02a410 comp rx=0 tx=0).stop 2024-05-09T08:25:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 -- 172.21.15.28:0/3414781286 shutdown_connections 2024-05-09T08:25:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 --2- 172.21.15.28:0/3414781286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58102320 0x7f0c58108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 --2- 172.21.15.28:0/3414781286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58101960 0x7f0c58101de0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 --2- 172.21.15.28:0/3414781286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c58100770 0x7f0c58100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 -- 172.21.15.28:0/3414781286 >> 172.21.15.28:0/3414781286 conn(0x7f0c580fbce0 msgr2=0x7f0c580fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 -- 172.21.15.28:0/3414781286 shutdown_connections 2024-05-09T08:25:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 -- 172.21.15.28:0/3414781286 wait complete. 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.878+0000 7f0c602e1640 1 Processor -- start 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c602e1640 1 -- start start 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c602e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58100770 0x7f0c58069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c602e1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58101960 0x7f0c5819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c602e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c58102320 0x7f0c5819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c602e1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5810c3f0 con 0x7f0c58100770 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c602e1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5810c270 con 0x7f0c58101960 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c602e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5810c570 con 0x7f0c58102320 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e07d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58100770 0x7f0c58069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e07d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58100770 0x7f0c58069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40756/0 (socket says 172.21.15.28:40756) 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e07d640 1 -- 172.21.15.28:0/2962719696 learned_addr learned my addr 172.21.15.28:0/2962719696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5d87c640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58101960 0x7f0c5819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e07d640 1 -- 172.21.15.28:0/2962719696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c58102320 msgr2=0x7f0c5819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e87e640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c58102320 0x7f0c5819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e07d640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c58102320 0x7f0c5819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e07d640 1 -- 172.21.15.28:0/2962719696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58101960 msgr2=0x7f0c5819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e07d640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58101960 0x7f0c5819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e07d640 1 -- 172.21.15.28:0/2962719696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5819f810 con 0x7f0c58100770 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5d87c640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58101960 0x7f0c5819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c5e07d640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58100770 0x7f0c58069530 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800ccb0 tx=0x7f0c4800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c577fe640 1 -- 172.21.15.28:0/2962719696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48004b20 con 0x7f0c58100770 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c577fe640 1 -- 172.21.15.28:0/2962719696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c48004cc0 con 0x7f0c58100770 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5819faa0 con 0x7f0c58100770 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.879+0000 7f0c577fe640 1 -- 172.21.15.28:0/2962719696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48005020 con 0x7f0c58100770 2024-05-09T08:25:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.880+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c5819ffc0 con 0x7f0c58100770 2024-05-09T08:25:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.881+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c58107b10 con 0x7f0c58100770 2024-05-09T08:25:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.882+0000 7f0c577fe640 1 -- 172.21.15.28:0/2962719696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0c48004000 con 0x7f0c58100770 2024-05-09T08:25:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.882+0000 7f0c577fe640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c400a2c10 0x7f0c400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.883+0000 7f0c577fe640 1 -- 172.21.15.28:0/2962719696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0c480cf500 con 0x7f0c58100770 2024-05-09T08:25:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.883+0000 7f0c5d87c640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c400a2c10 0x7f0c400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.883+0000 7f0c5d87c640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c400a2c10 0x7f0c400a50d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c000c00 tx=0x7f0c4c0095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:41.887+0000 7f0c577fe640 1 -- 172.21.15.28:0/2962719696 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0c48014030 con 0x7f0c58100770 2024-05-09T08:25:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.112+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c580624d0 con 0x7f0c58100770 2024-05-09T08:25:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.112+0000 7f0c577fe640 1 -- 172.21.15.28:0/2962719696 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f0c4809c100 con 0x7f0c58100770 2024-05-09T08:25:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:25:42.114 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:24:19.404649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24481,24499,24476,24512,24463]},"id":1}]} 2024-05-09T08:25:42.114 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c400a2c10 msgr2=0x7f0c400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c400a2c10 0x7f0c400a50d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c000c00 tx=0x7f0c4c0095f0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58100770 msgr2=0x7f0c58069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58100770 0x7f0c58069530 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800ccb0 tx=0x7f0c4800a620 comp rx=0 tx=0).stop 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 shutdown_connections 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f0c400a2c10 0x7f0c400a50d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c58102320 0x7f0c5819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58101960 0x7f0c5819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 --2- 172.21.15.28:0/2962719696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c58100770 0x7f0c58069530 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 >> 172.21.15.28:0/2962719696 conn(0x7f0c580fbce0 msgr2=0x7f0c58104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 shutdown_connections 2024-05-09T08:25:42.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.115+0000 7f0c602e1640 1 -- 172.21.15.28:0/2962719696 wait complete. 2024-05-09T08:25:42.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:25:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.232+0000 7f01a41a9640 1 -- 172.21.15.28:0/3907715386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c107570 msgr2=0x7f019c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.232+0000 7f01a41a9640 1 --2- 172.21.15.28:0/3907715386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c107570 0x7f019c1099c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0194009ca0 tx=0x7f019402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.232+0000 7f01a41a9640 1 -- 172.21.15.28:0/3907715386 shutdown_connections 2024-05-09T08:25:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.232+0000 7f01a41a9640 1 --2- 172.21.15.28:0/3907715386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c107570 0x7f019c1099c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.232+0000 7f01a41a9640 1 --2- 172.21.15.28:0/3907715386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f019c104c40 0x7f019c107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.232+0000 7f01a41a9640 1 --2- 172.21.15.28:0/3907715386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f019c102310 0x7f019c104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.232+0000 7f01a41a9640 1 -- 172.21.15.28:0/3907715386 >> 172.21.15.28:0/3907715386 conn(0x7f019c0fbd20 msgr2=0x7f019c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.232+0000 7f01a41a9640 1 -- 172.21.15.28:0/3907715386 shutdown_connections 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.232+0000 7f01a41a9640 1 -- 172.21.15.28:0/3907715386 wait complete. 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a41a9640 1 Processor -- start 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a41a9640 1 -- start start 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a41a9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f019c102310 0x7f019c197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a41a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c104c40 0x7f019c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a41a9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f019c107570 0x7f019c19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a41a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f019c10d530 con 0x7f019c104c40 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a41a9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f019c10d3b0 con 0x7f019c107570 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a41a9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f019c10d6b0 con 0x7f019c102310 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a1744640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c104c40 0x7f019c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a1744640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c104c40 0x7f019c1980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40772/0 (socket says 172.21.15.28:40772) 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a1744640 1 -- 172.21.15.28:0/2138424242 learned_addr learned my addr 172.21.15.28:0/2138424242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a2746640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f019c107570 0x7f019c19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a1f45640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f019c102310 0x7f019c197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a1744640 1 -- 172.21.15.28:0/2138424242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f019c102310 msgr2=0x7f019c197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a1744640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f019c102310 0x7f019c197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a1744640 1 -- 172.21.15.28:0/2138424242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f019c107570 msgr2=0x7f019c19f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a1744640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f019c107570 0x7f019c19f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a1744640 1 -- 172.21.15.28:0/2138424242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019c19f780 con 0x7f019c104c40 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.233+0000 7f01a2746640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f019c107570 0x7f019c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.234+0000 7f01a1744640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c104c40 0x7f019c1980a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f019000d770 tx=0x7f019000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.234+0000 7f018effd640 1 -- 172.21.15.28:0/2138424242 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190017070 con 0x7f019c104c40 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.234+0000 7f018effd640 1 -- 172.21.15.28:0/2138424242 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0190009c30 con 0x7f019c104c40 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.234+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f019c19fa70 con 0x7f019c104c40 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.234+0000 7f018effd640 1 -- 172.21.15.28:0/2138424242 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019000f620 con 0x7f019c104c40 2024-05-09T08:25:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.234+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f019c19ffc0 con 0x7f019c104c40 2024-05-09T08:25:42.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.235+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f019c0fd810 con 0x7f019c104c40 2024-05-09T08:25:42.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.236+0000 7f018effd640 1 -- 172.21.15.28:0/2138424242 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f019001a070 con 0x7f019c104c40 2024-05-09T08:25:42.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.237+0000 7f018effd640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01880a2b90 0x7f01880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.237+0000 7f01a1f45640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01880a2b90 0x7f01880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.237+0000 7f018effd640 1 -- 172.21.15.28:0/2138424242 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f01900cefc0 con 0x7f019c104c40 2024-05-09T08:25:42.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.237+0000 7f01a1f45640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01880a2b90 0x7f01880a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0198005fd0 tx=0x7f0198005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.241+0000 7f018effd640 1 -- 172.21.15.28:0/2138424242 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0190009de0 con 0x7f019c104c40 2024-05-09T08:25:42.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:42 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1925036056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:25:42.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:42 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1676658656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:25:42.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:42 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1200327296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:25:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.427+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f019c062430 con 0x7f019c104c40 2024-05-09T08:25:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.429+0000 7f018effd640 1 -- 172.21.15.28:0/2138424242 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24666 (secure 0 0 0) 0x7f019009bb60 con 0x7f019c104c40 2024-05-09T08:25:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:25:42.430 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:24:51.475713+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01880a2b90 msgr2=0x7f01880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01880a2b90 0x7f01880a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0198005fd0 tx=0x7f0198005e20 comp rx=0 tx=0).stop 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c104c40 msgr2=0x7f019c1980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c104c40 0x7f019c1980a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f019000d770 tx=0x7f019000dc30 comp rx=0 tx=0).stop 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 shutdown_connections 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01880a2b90 0x7f01880a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f019c107570 0x7f019c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c104c40 0x7f019c1980a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 --2- 172.21.15.28:0/2138424242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f019c102310 0x7f019c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 >> 172.21.15.28:0/2138424242 conn(0x7f019c0fbd20 msgr2=0x7f019c102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 shutdown_connections 2024-05-09T08:25:42.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.431+0000 7f01a41a9640 1 -- 172.21.15.28:0/2138424242 wait complete. 2024-05-09T08:25:42.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:42 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1925036056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:25:42.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:42 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1676658656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:25:42.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:42 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1200327296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:25:42.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:25:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.548+0000 7fce47ebd640 1 -- 172.21.15.28:0/961257185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40100770 msgr2=0x7fce40100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.548+0000 7fce47ebd640 1 --2- 172.21.15.28:0/961257185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40100770 0x7fce40100b90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009900 tx=0x7fce3c02a410 comp rx=0 tx=0).stop 2024-05-09T08:25:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 -- 172.21.15.28:0/961257185 shutdown_connections 2024-05-09T08:25:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 --2- 172.21.15.28:0/961257185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce40102320 0x7fce40108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 --2- 172.21.15.28:0/961257185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce40101960 0x7fce40101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 --2- 172.21.15.28:0/961257185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40100770 0x7fce40100b90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 -- 172.21.15.28:0/961257185 >> 172.21.15.28:0/961257185 conn(0x7fce400fbce0 msgr2=0x7fce400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 -- 172.21.15.28:0/961257185 shutdown_connections 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 -- 172.21.15.28:0/961257185 wait complete. 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 Processor -- start 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 -- start start 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce40100770 0x7fce40197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce40101960 0x7fce40198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40102320 0x7fce4019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce4010c3e0 con 0x7fce40102320 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fce4010c260 con 0x7fce40100770 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.549+0000 7fce47ebd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce4010c560 con 0x7fce40101960 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce4645a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40102320 0x7fce4019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce4645a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40102320 0x7fce4019f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40776/0 (socket says 172.21.15.28:40776) 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce4645a640 1 -- 172.21.15.28:0/3018806135 learned_addr learned my addr 172.21.15.28:0/3018806135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce45c59640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce40100770 0x7fce40197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce4645a640 1 -- 172.21.15.28:0/3018806135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce40101960 msgr2=0x7fce40198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce4645a640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce40101960 0x7fce40198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce4645a640 1 -- 172.21.15.28:0/3018806135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce40100770 msgr2=0x7fce40197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce4645a640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce40100770 0x7fce40197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce4645a640 1 -- 172.21.15.28:0/3018806135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4019f7e0 con 0x7fce40102320 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce45c59640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce40100770 0x7fce40197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce4645a640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40102320 0x7fce4019f170 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fce3800eb10 tx=0x7fce3800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce36ffd640 1 -- 172.21.15.28:0/3018806135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce38004b20 con 0x7fce40102320 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4019fad0 con 0x7fce40102320 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce36ffd640 1 -- 172.21.15.28:0/3018806135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce38004cc0 con 0x7fce40102320 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce36ffd640 1 -- 172.21.15.28:0/3018806135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce38010760 con 0x7fce40102320 2024-05-09T08:25:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.550+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4019ffd0 con 0x7fce40102320 2024-05-09T08:25:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.551+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce401a0260 con 0x7fce40102320 2024-05-09T08:25:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.553+0000 7fce36ffd640 1 -- 172.21.15.28:0/3018806135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fce38010900 con 0x7fce40102320 2024-05-09T08:25:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.554+0000 7fce36ffd640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fce2c0a2b90 0x7fce2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.554+0000 7fce36ffd640 1 -- 172.21.15.28:0/3018806135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fce380cfce0 con 0x7fce40102320 2024-05-09T08:25:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.554+0000 7fce45c59640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fce2c0a2b90 0x7fce2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.554+0000 7fce45c59640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fce2c0a2b90 0x7fce2c0a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fce3c000c00 tx=0x7fce3c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.557+0000 7fce36ffd640 1 -- 172.21.15.28:0/3018806135 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fce38053740 con 0x7fce40102320 2024-05-09T08:25:42.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:42 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1925036056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:25:42.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:42 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1676658656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:25:42.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:42 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1200327296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:25:42.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.782+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce401a0430 con 0x7fce40102320 2024-05-09T08:25:42.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.783+0000 7fce36ffd640 1 -- 172.21.15.28:0/3018806135 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7fce3809c880 con 0x7fce40102320 2024-05-09T08:25:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:25:42.784 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:24:19.404649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24481,24499,24476,24512,24463]},"id":1}]} 2024-05-09T08:25:42.784 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-05-09T08:25:42.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.785+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fce2c0a2b90 msgr2=0x7fce2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.785+0000 7fce47ebd640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fce2c0a2b90 0x7fce2c0a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fce3c000c00 tx=0x7fce3c031400 comp rx=0 tx=0).stop 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40102320 msgr2=0x7fce4019f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40102320 0x7fce4019f170 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fce3800eb10 tx=0x7fce3800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 shutdown_connections 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fce2c0a2b90 0x7fce2c0a5050 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce40102320 0x7fce4019f170 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce40101960 0x7fce40198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 --2- 172.21.15.28:0/3018806135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce40100770 0x7fce40197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 >> 172.21.15.28:0/3018806135 conn(0x7fce400fbce0 msgr2=0x7fce400689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 shutdown_connections 2024-05-09T08:25:42.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.786+0000 7fce47ebd640 1 -- 172.21.15.28:0/3018806135 wait complete. 2024-05-09T08:25:42.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-09T08:23:50.918821+0000', 'modified': '2024-05-09T08:24:19.404649+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24512, 'mds_1': 24499, 'mds_2': 24481, 'mds_3': 24476, 'mds_4': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14613': {'gid': 14613, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/612632862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 612632862}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 612632862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14625': {'gid': 14625, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/9520094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 9520094}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 9520094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24410': {'gid': 24410, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.165:6833/3903056410', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6832', 'nonce': 3903056410}, {'type': 'v1', 'addr': '172.21.15.165:6833', 'nonce': 3903056410}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6833/1853687224', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6832', 'nonce': 1853687224}, {'type': 'v1', 'addr': '172.21.15.184:6833', 'nonce': 1853687224}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.165:6835/3736338591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6834', 'nonce': 3736338591}, {'type': 'v1', 'addr': '172.21.15.165:6835', 'nonce': 3736338591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.184:6835/4284085086', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6834', 'nonce': 4284085086}, {'type': 'v1', 'addr': '172.21.15.184:6835', 'nonce': 4284085086}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.165:6837/1615628763', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6836', 'nonce': 1615628763}, {'type': 'v1', 'addr': '172.21.15.165:6837', 'nonce': 1615628763}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.184:6837/3795789755', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6836', 'nonce': 3795789755}, {'type': 'v1', 'addr': '172.21.15.184:6837', 'nonce': 3795789755}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.184:6839/224925653', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6838', 'nonce': 224925653}, {'type': 'v1', 'addr': '172.21.15.184:6839', 'nonce': 224925653}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.165:6839/1723030158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6838', 'nonce': 1723030158}, {'type': 'v1', 'addr': '172.21.15.165:6839', 'nonce': 1723030158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24512, 'qdb_cluster': [24481, 24499, 24476, 24512, 24463]} 2024-05-09T08:25:42.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-09T08:25:42.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.904+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/249812945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 msgr2=0x7f6eac102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.904+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/249812945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 0x7f6eac102f00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009900 tx=0x7f6e9802a410 comp rx=0 tx=0).stop 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.904+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/249812945 shutdown_connections 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.904+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/249812945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac103440 0x7f6eac109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.904+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/249812945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 0x7f6eac102f00 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.904+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/249812945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6eac101890 0x7f6eac101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.904+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/249812945 >> 172.21.15.28:0/249812945 conn(0x7f6eac0fce00 msgr2=0x7f6eac0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.904+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/249812945 shutdown_connections 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.904+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/249812945 wait complete. 2024-05-09T08:25:42.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb2aa5640 1 Processor -- start 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb2aa5640 1 -- start start 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb2aa5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac101890 0x7f6eac197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb2aa5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 0x7f6eac1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb2aa5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6eac103440 0x7f6eac19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb2aa5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac10d4f0 con 0x7f6eac102a80 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb2aa5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac10d370 con 0x7f6eac103440 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb2aa5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac10d670 con 0x7f6eac101890 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 0x7f6eac1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 0x7f6eac1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40806/0 (socket says 172.21.15.28:40806) 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eabfff640 1 -- 172.21.15.28:0/465446910 learned_addr learned my addr 172.21.15.28:0/465446910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb0841640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac101890 0x7f6eac197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eb1042640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6eac103440 0x7f6eac19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eabfff640 1 -- 172.21.15.28:0/465446910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac101890 msgr2=0x7f6eac197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eabfff640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac101890 0x7f6eac197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eabfff640 1 -- 172.21.15.28:0/465446910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6eac103440 msgr2=0x7f6eac19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eabfff640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6eac103440 0x7f6eac19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eabfff640 1 -- 172.21.15.28:0/465446910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac19f790 con 0x7f6eac102a80 2024-05-09T08:25:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.905+0000 7f6eabfff640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 0x7f6eac1980b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009230 tx=0x7f6e98031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.906+0000 7f6ea9ffb640 1 -- 172.21.15.28:0/465446910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9803f030 con 0x7f6eac102a80 2024-05-09T08:25:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.906+0000 7f6ea9ffb640 1 -- 172.21.15.28:0/465446910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e98003ad0 con 0x7f6eac102a80 2024-05-09T08:25:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.906+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/465446910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eac19fa20 con 0x7f6eac102a80 2024-05-09T08:25:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.906+0000 7f6ea9ffb640 1 -- 172.21.15.28:0/465446910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98035070 con 0x7f6eac102a80 2024-05-09T08:25:42.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.906+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/465446910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eac19ff60 con 0x7f6eac102a80 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.908+0000 7f6ea9ffb640 1 -- 172.21.15.28:0/465446910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6e98042030 con 0x7f6eac102a80 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.908+0000 7f6eb57b5740 1 Processor -- start 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.908+0000 7f6eb57b5740 1 -- start start 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.908+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a28c0 0x55ef173a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.908+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173a7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.908+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a8420 0x55ef173aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.908+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef173064d0 con 0x55ef173a32b0 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.908+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ef172ae3e0 con 0x55ef173a8420 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.908+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ef17302490 con 0x55ef173a28c0 2024-05-09T08:25:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb0841640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173a7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb0841640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173a7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40822/0 (socket says 172.21.15.28:40822) 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb0841640 1 -- 172.21.15.28:0/2557572198 learned_addr learned my addr 172.21.15.28:0/2557572198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eabfff640 1 --2- 172.21.15.28:0/2557572198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a8420 0x55ef173aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb1042640 1 --2- 172.21.15.28:0/2557572198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a28c0 0x55ef173a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb0841640 1 -- 172.21.15.28:0/2557572198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a28c0 msgr2=0x55ef173a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb0841640 1 --2- 172.21.15.28:0/2557572198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a28c0 0x55ef173a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb0841640 1 -- 172.21.15.28:0/2557572198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a8420 msgr2=0x55ef173aa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb0841640 1 --2- 172.21.15.28:0/2557572198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a8420 0x55ef173aa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb0841640 1 -- 172.21.15.28:0/2557572198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef1701a2d0 con 0x55ef173a32b0 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6ea9ffb640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6e9c0a2c10 0x7f6e9c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb0841640 1 --2- 172.21.15.28:0/2557572198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173a7db0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009ca0 tx=0x7f6ea0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad33a6bf7f4d672 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6eb1042640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6e9c0a2c10 0x7f6e9c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6ea9ffb640 1 -- 172.21.15.28:0/465446910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6e980ef890 con 0x7f6eac102a80 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6e96ffd640 1 -- 172.21.15.28:0/2557572198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0004220 con 0x55ef173a32b0 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6e96ffd640 1 -- 172.21.15.28:0/2557572198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea00043c0 con 0x55ef173a32b0 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.909+0000 7f6e96ffd640 1 -- 172.21.15.28:0/2557572198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0019930 con 0x55ef173a32b0 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 -- 172.21.15.28:0/2557572198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 msgr2=0x55ef173a7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb1042640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6e9c0a2c10 0x7f6e9c0a50d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x55ef173a3b10 tx=0x7f6ea4009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/2557572198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173a7db0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009ca0 tx=0x7f6ea0011810 comp rx=0 tx=0).stop 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 -- 172.21.15.28:0/2557572198 shutdown_connections 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/2557572198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a8420 0x55ef173aa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/2557572198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173a7db0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/2557572198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a28c0 0x55ef173a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 -- 172.21.15.28:0/2557572198 >> 172.21.15.28:0/2557572198 conn(0x55ef1739df30 msgr2=0x55ef1739e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 -- 172.21.15.28:0/2557572198 shutdown_connections 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 -- 172.21.15.28:0/2557572198 wait complete. 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.910+0000 7f6eb57b5740 1 -- start start 2024-05-09T08:25:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb57b5740 1 Processor -- start 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173acce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 0x55ef173b4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef173ab3d0 con 0x55ef173a32b0 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ef172c18e0 con 0x55ef173a28c0 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ef173ab550 con 0x55ef173a8420 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb57b5740 10 client.0 _ll_register_callbacks cb 0x55ef1728e9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173ad220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40830/0 (socket says 172.21.15.28:40830) 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eabfff640 1 -- 172.21.15.28:0/1421648658 learned_addr learned my addr 172.21.15.28:0/1421648658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb1042640 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 0x55ef173b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb0841640 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eabfff640 1 -- 172.21.15.28:0/1421648658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 msgr2=0x55ef173b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eabfff640 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 0x55ef173b4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eabfff640 1 -- 172.21.15.28:0/1421648658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 msgr2=0x55ef173acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eabfff640 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eabfff640 1 -- 172.21.15.28:0/1421648658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef170e46b0 con 0x55ef173a32b0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb1042640 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 0x55ef173b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.911+0000 7f6eb0841640 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eabfff640 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173ad220 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6e980bbd70 tx=0x7f6e980bb740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff040cad49ac4a5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6e8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6e8bfff640 1 -- 172.21.15.28:0/1421648658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98049070 con 0x55ef173a32b0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6e8bfff640 1 -- 172.21.15.28:0/1421648658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6e9803b460 con 0x55ef173a32b0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eb57b5740 1 -- 172.21.15.28:0/1421648658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea00043c0 con 0x55ef173a32b0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6e8bfff640 1 -- 172.21.15.28:0/1421648658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6e980c49a0 con 0x55ef173a32b0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eb57b5740 10 client.15216 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eb57b5740 1 -- 172.21.15.28:0/1421648658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ef173b7230 con 0x55ef173a32b0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6e8bfff640 1 -- 172.21.15.28:0/1421648658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea00043c0 con 0x55ef173a32b0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6e8bfff640 1 -- 172.21.15.28:0/1421648658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6e980d2020 con 0x55ef173a32b0 2024-05-09T08:25:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eb57b5740 10 client.15216 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eb57b5740 10 client.15216 resolve_mds: resolved name 'f' to daemon mds.f 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eb57b5740 20 client.15216 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ef173b9900 0x55ef173bad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eb57b5740 4 client.15216 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.912+0000 7f6eb57b5740 1 -- 172.21.15.28:0/1421648658 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef17313c00 con 0x55ef173b9900 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.913+0000 7f6eb0841640 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ef173b9900 0x55ef173bad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.913+0000 7f6eb0841640 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ef173b9900 0x55ef173bad70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7c94a20511e16e20 server_cookie=718eb634da7fc8c in_seq=0 out_seq=0 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.913+0000 7f6e8bfff640 10 client.15216 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6e8bfff640 1 -- 172.21.15.28:0/1421648658 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef17313c00 con 0x55ef173b9900 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6e8bfff640 10 client.15216 handle_command_reply: tid=0 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 client.15216 shutdown 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 -- 172.21.15.28:0/1421648658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 msgr2=0x55ef173ad220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173ad220 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6e980bbd70 tx=0x7f6e980bb740 comp rx=0 tx=0).stop 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 -- 172.21.15.28:0/1421648658 shutdown_connections 2024-05-09T08:25:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ef173b9900 0x55ef173bad70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 0x55ef173b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6e8bfff640 0 client.15216 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:25:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173ad220 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/1421648658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 -- 172.21.15.28:0/1421648658 >> 172.21.15.28:0/1421648658 conn(0x55ef1739df30 msgr2=0x55ef1739d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 -- 172.21.15.28:0/1421648658 shutdown_connections 2024-05-09T08:25:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 1 -- 172.21.15.28:0/1421648658 wait complete. 2024-05-09T08:25:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.914+0000 7f6eb57b5740 20 client.15216 trim_cache size 0 max 16384 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.933+0000 7f6eb57b5740 1 Processor -- start 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.933+0000 7f6eb57b5740 1 -- start start 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.933+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a28c0 0x55ef173aa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.933+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a32b0 0x55ef173b4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.933+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a8420 0x55ef173acce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.933+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef173b90b0 con 0x55ef173a28c0 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.933+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ef173b9780 con 0x55ef173a8420 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.933+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ef173b9e50 con 0x55ef173a32b0 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.933+0000 7f6eb0841640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a28c0 0x55ef173aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6eb0841640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a28c0 0x55ef173aa810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40836/0 (socket says 172.21.15.28:40836) 2024-05-09T08:25:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6eb0841640 1 -- 172.21.15.28:0/3645894867 learned_addr learned my addr 172.21.15.28:0/3645894867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6eb0841640 1 -- 172.21.15.28:0/3645894867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a32b0 msgr2=0x55ef173b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6eb0841640 1 --2- 172.21.15.28:0/3645894867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a32b0 0x55ef173b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6eb0841640 1 -- 172.21.15.28:0/3645894867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a8420 msgr2=0x55ef173acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6eb0841640 1 --2- 172.21.15.28:0/3645894867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a8420 0x55ef173acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6eb0841640 1 -- 172.21.15.28:0/3645894867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef1739eca0 con 0x55ef173a28c0 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6eb0841640 1 --2- 172.21.15.28:0/3645894867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a28c0 0x55ef173aa810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009ca0 tx=0x7f6ea0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae5d46b4225888a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6e967fc640 1 -- 172.21.15.28:0/3645894867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea002cd90 con 0x55ef173a28c0 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6e967fc640 1 -- 172.21.15.28:0/3645894867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea0005520 con 0x55ef173a28c0 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.934+0000 7f6e967fc640 1 -- 172.21.15.28:0/3645894867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0003a20 con 0x55ef173a28c0 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 -- 172.21.15.28:0/3645894867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a28c0 msgr2=0x55ef173aa810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/3645894867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a28c0 0x55ef173aa810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009ca0 tx=0x7f6ea0009310 comp rx=0 tx=0).stop 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 -- 172.21.15.28:0/3645894867 shutdown_connections 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/3645894867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a8420 0x55ef173acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/3645894867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a32b0 0x55ef173b4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/3645894867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a28c0 0x55ef173aa810 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 -- 172.21.15.28:0/3645894867 >> 172.21.15.28:0/3645894867 conn(0x55ef1739df30 msgr2=0x55ef173b8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 -- 172.21.15.28:0/3645894867 shutdown_connections 2024-05-09T08:25:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 -- 172.21.15.28:0/3645894867 wait complete. 2024-05-09T08:25:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 -- start start 2024-05-09T08:25:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 Processor -- start 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.935+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173b0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb57b5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 0x55ef173b3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef173babf0 con 0x55ef173a32b0 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ef173ba520 con 0x55ef173a28c0 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb57b5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ef173bb2c0 con 0x55ef173a8420 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb57b5740 10 client.0 _ll_register_callbacks cb 0x7f6e98005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173b0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40848/0 (socket says 172.21.15.28:40848) 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eabfff640 1 -- 172.21.15.28:0/718704759 learned_addr learned my addr 172.21.15.28:0/718704759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb0841640 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eabfff640 1 -- 172.21.15.28:0/718704759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 msgr2=0x55ef173b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb1042640 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 0x55ef173b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eabfff640 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 0x55ef173b3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eabfff640 1 -- 172.21.15.28:0/718704759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 msgr2=0x55ef173b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eabfff640 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eabfff640 1 -- 172.21.15.28:0/718704759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef173b3d10 con 0x55ef173a32b0 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb0841640 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eabfff640 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173b0770 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6e98000c00 tx=0x7f6e98003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c602e4ab37dcf607 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6e8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6e8bfff640 1 -- 172.21.15.28:0/718704759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98049070 con 0x55ef173a32b0 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb57b5740 1 -- 172.21.15.28:0/718704759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea0005520 con 0x55ef173a32b0 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6e8bfff640 1 -- 172.21.15.28:0/718704759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6e980c4510 con 0x55ef173a32b0 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6e8bfff640 1 -- 172.21.15.28:0/718704759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6e980c4710 con 0x55ef173a32b0 2024-05-09T08:25:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb57b5740 10 client.15222 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.936+0000 7f6eb57b5740 1 -- 172.21.15.28:0/718704759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ef174201a0 con 0x55ef173a32b0 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6e8bfff640 1 -- 172.21.15.28:0/718704759 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea0005520 con 0x55ef173a32b0 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6e8bfff640 1 -- 172.21.15.28:0/718704759 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6e980d2060 con 0x55ef173a32b0 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6eb57b5740 10 client.15222 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6eb57b5740 10 client.15222 resolve_mds: resolved name 'f' to daemon mds.f 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6eb57b5740 20 client.15222 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ef17424b60 0x55ef17423870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6eb57b5740 4 client.15222 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6eb57b5740 1 -- 172.21.15.28:0/718704759 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "status"}) -- 0x55ef1730df50 con 0x55ef17424b60 2024-05-09T08:25:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6eb0841640 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ef17424b60 0x55ef17423870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6e8bfff640 10 client.15222 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.937+0000 7f6eb0841640 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ef17424b60 0x55ef17423870 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1cebd8d70c231277 server_cookie=5bc3c11532653571 in_seq=0 out_seq=0 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6e8bfff640 1 -- 172.21.15.28:0/718704759 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ef1730df50 con 0x55ef17424b60 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6e8bfff640 10 client.15222 handle_command_reply: tid=0 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 client.15222 shutdown 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 -- 172.21.15.28:0/718704759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 msgr2=0x55ef173b0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173b0770 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6e98000c00 tx=0x7f6e98003eb0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 -- 172.21.15.28:0/718704759 shutdown_connections 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ef17424b60 0x55ef17423870 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ef173a8420 0x55ef173b3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef173a32b0 0x55ef173b0770 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6e8bfff640 0 client.15222 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 --2- 172.21.15.28:0/718704759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ef173a28c0 0x55ef173b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 -- 172.21.15.28:0/718704759 >> 172.21.15.28:0/718704759 conn(0x55ef1739df30 msgr2=0x55ef173b8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 -- 172.21.15.28:0/718704759 shutdown_connections 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 1 -- 172.21.15.28:0/718704759 wait complete. 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.938+0000 7f6eb57b5740 20 client.15222 trim_cache size 0 max 16384 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24463, 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:25:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 84.541902578000006, 2024-05-09T08:25:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:25:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 91, 2024-05-09T08:25:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:25:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 134.91897560699999 2024-05-09T08:25:42.947 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:25:42.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6e9c0a2c10 msgr2=0x7f6e9c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6e9c0a2c10 0x7f6e9c0a50d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x55ef173a3b10 tx=0x7f6ea4009600 comp rx=0 tx=0).stop 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 msgr2=0x7f6eac1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 0x7f6eac1980b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009230 tx=0x7f6e98031b60 comp rx=0 tx=0).stop 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/465446910 shutdown_connections 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6e9c0a2c10 0x7f6e9c0a50d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6eac103440 0x7f6eac19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eac102a80 0x7f6eac1980b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 --2- 172.21.15.28:0/465446910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6eac101890 0x7f6eac197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/465446910 >> 172.21.15.28:0/465446910 conn(0x7f6eac0fce00 msgr2=0x7f6eac105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/465446910 shutdown_connections 2024-05-09T08:25:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:42.939+0000 7f6eb2aa5640 1 -- 172.21.15.28:0/465446910 wait complete. 2024-05-09T08:25:42.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.056+0000 7f69b4238640 1 -- 172.21.15.28:0/190935020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac100790 msgr2=0x7f69ac100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.056+0000 7f69b4238640 1 --2- 172.21.15.28:0/190935020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac100790 0x7f69ac100bb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009900 tx=0x7f69a802a410 comp rx=0 tx=0).stop 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.056+0000 7f69b4238640 1 -- 172.21.15.28:0/190935020 shutdown_connections 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.056+0000 7f69b4238640 1 --2- 172.21.15.28:0/190935020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69ac102340 0x7f69ac108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.056+0000 7f69b4238640 1 --2- 172.21.15.28:0/190935020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69ac101980 0x7f69ac101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.056+0000 7f69b4238640 1 --2- 172.21.15.28:0/190935020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac100790 0x7f69ac100bb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.056+0000 7f69b4238640 1 -- 172.21.15.28:0/190935020 >> 172.21.15.28:0/190935020 conn(0x7f69ac0fbd00 msgr2=0x7f69ac0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.056+0000 7f69b4238640 1 -- 172.21.15.28:0/190935020 shutdown_connections 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.056+0000 7f69b4238640 1 -- 172.21.15.28:0/190935020 wait complete. 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b4238640 1 Processor -- start 2024-05-09T08:25:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b4238640 1 -- start start 2024-05-09T08:25:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b4238640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69ac100790 0x7f69ac06a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b4238640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69ac101980 0x7f69ac19ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b4238640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac102340 0x7f69ac19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b4238640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac10c460 con 0x7f69ac102340 2024-05-09T08:25:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b4238640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac10c2e0 con 0x7f69ac100790 2024-05-09T08:25:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b4238640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac10c5e0 con 0x7f69ac101980 2024-05-09T08:25:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b27d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac102340 0x7f69ac19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b27d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac102340 0x7f69ac19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40864/0 (socket says 172.21.15.28:40864) 2024-05-09T08:25:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b27d5640 1 -- 172.21.15.28:0/3231892495 learned_addr learned my addr 172.21.15.28:0/3231892495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b27d5640 1 -- 172.21.15.28:0/3231892495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69ac101980 msgr2=0x7f69ac19ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b17d3640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69ac101980 0x7f69ac19ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69ac100790 0x7f69ac06a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b27d5640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69ac101980 0x7f69ac19ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b27d5640 1 -- 172.21.15.28:0/3231892495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69ac100790 msgr2=0x7f69ac06a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b27d5640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69ac100790 0x7f69ac06a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b27d5640 1 -- 172.21.15.28:0/3231892495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ac19f930 con 0x7f69ac102340 2024-05-09T08:25:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b17d3640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69ac101980 0x7f69ac19ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.057+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69ac100790 0x7f69ac06a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.058+0000 7f69b27d5640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac102340 0x7f69ac19f020 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f69a400cad0 tx=0x7f69a400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.058+0000 7f69a2ffd640 1 -- 172.21.15.28:0/3231892495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a400ae20 con 0x7f69ac102340 2024-05-09T08:25:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.058+0000 7f69a2ffd640 1 -- 172.21.15.28:0/3231892495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69a4005020 con 0x7f69ac102340 2024-05-09T08:25:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.058+0000 7f69b4238640 1 -- 172.21.15.28:0/3231892495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ac19fbc0 con 0x7f69ac102340 2024-05-09T08:25:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.058+0000 7f69a2ffd640 1 -- 172.21.15.28:0/3231892495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a400f6f0 con 0x7f69ac102340 2024-05-09T08:25:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.058+0000 7f69b4238640 1 -- 172.21.15.28:0/3231892495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ac1a0140 con 0x7f69ac102340 2024-05-09T08:25:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.060+0000 7f69a2ffd640 1 -- 172.21.15.28:0/3231892495 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f69a40051d0 con 0x7f69ac102340 2024-05-09T08:25:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.060+0000 7f69b7027740 1 Processor -- start 2024-05-09T08:25:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b7027740 1 -- start start 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b7027740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 0x559df037dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b7027740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 0x559df0382db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b7027740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03858b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b7027740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559df02e14d0 con 0x559df0383420 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b7027740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x559df02893e0 con 0x559df037e2b0 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b7027740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559df02dd490 con 0x559df037d8c0 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03858b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03858b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40866/0 (socket says 172.21.15.28:40866) 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 -- 172.21.15.28:0/2182059071 learned_addr learned my addr 172.21.15.28:0/2182059071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 0x559df037dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69a2ffd640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f69980a2c10 0x7f69980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 0x559df0382db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 -- 172.21.15.28:0/2182059071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 msgr2=0x559df037dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 0x559df037dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b27d5640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f69980a2c10 0x7f69980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 -- 172.21.15.28:0/2182059071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 msgr2=0x559df0382db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 0x559df0382db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 -- 172.21.15.28:0/2182059071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559defff52d0 con 0x559df0383420 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 0x559df0382db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69a2ffd640 1 -- 172.21.15.28:0/3231892495 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f69a40cf570 con 0x7f69ac102340 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.061+0000 7f69b17d3640 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03858b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f699c00cfd0 tx=0x7f699c00c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51dec8c2ab22ea4c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.062+0000 7f6997fff640 1 -- 172.21.15.28:0/2182059071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c01c070 con 0x559df0383420 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.062+0000 7f69b27d5640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f69980a2c10 0x7f69980a50d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f69a400a6e0 tx=0x7f69a4066a80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.062+0000 7f6997fff640 1 -- 172.21.15.28:0/2182059071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f699c017af0 con 0x559df0383420 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.062+0000 7f6997fff640 1 -- 172.21.15.28:0/2182059071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c00f040 con 0x559df0383420 2024-05-09T08:25:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.062+0000 7f69b7027740 1 -- 172.21.15.28:0/2182059071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 msgr2=0x559df03858b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.062+0000 7f69b7027740 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03858b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f699c00cfd0 tx=0x7f699c00c230 comp rx=0 tx=0).stop 2024-05-09T08:25:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.062+0000 7f69b7027740 1 -- 172.21.15.28:0/2182059071 shutdown_connections 2024-05-09T08:25:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.062+0000 7f69b7027740 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03858b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.062+0000 7f69b7027740 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 0x559df0382db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 --2- 172.21.15.28:0/2182059071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 0x559df037dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 -- 172.21.15.28:0/2182059071 >> 172.21.15.28:0/2182059071 conn(0x559df0378f30 msgr2=0x559df0379330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 -- 172.21.15.28:0/2182059071 shutdown_connections 2024-05-09T08:25:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 -- 172.21.15.28:0/2182059071 wait complete. 2024-05-09T08:25:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 -- start start 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 Processor -- start 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037d8c0 0x559df0387d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037e2b0 0x559df0388290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df0383420 0x559df038f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559df03863d0 con 0x559df037d8c0 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x559df029c8e0 con 0x559df0383420 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559df0386550 con 0x559df037e2b0 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b7027740 10 client.0 _ll_register_callbacks cb 0x559df02699c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b17d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037d8c0 0x559df0387d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.063+0000 7f69b17d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037d8c0 0x559df0387d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40876/0 (socket says 172.21.15.28:40876) 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b17d3640 1 -- 172.21.15.28:0/171565352 learned_addr learned my addr 172.21.15.28:0/171565352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b17d3640 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df0383420 0x559df038f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037e2b0 0x559df0388290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b17d3640 1 -- 172.21.15.28:0/171565352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037e2b0 msgr2=0x559df0388290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b17d3640 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037e2b0 0x559df0388290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b17d3640 1 -- 172.21.15.28:0/171565352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df0383420 msgr2=0x559df038f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b17d3640 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df0383420 0x559df038f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b17d3640 1 -- 172.21.15.28:0/171565352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559df00bf6b0 con 0x559df037d8c0 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b17d3640 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037d8c0 0x559df0387d50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f699c004810 tx=0x7f699c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0d7bee4c534b15 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69957fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69957fa640 1 -- 172.21.15.28:0/171565352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c01c070 con 0x559df037d8c0 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69957fa640 1 -- 172.21.15.28:0/171565352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f699c0176e0 con 0x559df037d8c0 2024-05-09T08:25:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b7027740 1 -- 172.21.15.28:0/171565352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f699c017af0 con 0x559df037d8c0 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69957fa640 1 -- 172.21.15.28:0/171565352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f699c031050 con 0x559df037d8c0 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b7027740 10 client.15234 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69b7027740 1 -- 172.21.15.28:0/171565352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559df0392230 con 0x559df037d8c0 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.064+0000 7f69957fa640 1 -- 172.21.15.28:0/171565352 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f699c017af0 con 0x559df037d8c0 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.065+0000 7f69957fa640 1 -- 172.21.15.28:0/171565352 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f699c023070 con 0x559df037d8c0 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.065+0000 7f69b7027740 10 client.15234 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.065+0000 7f69b7027740 10 client.15234 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.065+0000 7f69b7027740 20 client.15234 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.065+0000 7f69b7027740 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x559df0394460 0x559df03958d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.065+0000 7f69b7027740 4 client.15234 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:25:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.065+0000 7f69b7027740 1 -- 172.21.15.28:0/171565352 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559df02eec00 con 0x559df0394460 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.065+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x559df0394460 0x559df03958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.066+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x559df0394460 0x559df03958d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=65340212de19c0e6 server_cookie=f29e0699ff128184 in_seq=0 out_seq=0 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.066+0000 7f69957fa640 10 client.15234 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69957fa640 1 -- 172.21.15.28:0/171565352 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559df02eec00 con 0x559df0394460 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69957fa640 10 client.15234 handle_command_reply: tid=0 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 client.15234 shutdown 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 -- 172.21.15.28:0/171565352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037d8c0 msgr2=0x559df0387d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037d8c0 0x559df0387d50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f699c004810 tx=0x7f699c008040 comp rx=0 tx=0).stop 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 -- 172.21.15.28:0/171565352 shutdown_connections 2024-05-09T08:25:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x559df0394460 0x559df03958d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df0383420 0x559df038f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037e2b0 0x559df0388290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 --2- 172.21.15.28:0/171565352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037d8c0 0x559df0387d50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69957fa640 0 client.15234 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:25:43.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 -- 172.21.15.28:0/171565352 >> 172.21.15.28:0/171565352 conn(0x559df0378f30 msgr2=0x559df0378000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 -- 172.21.15.28:0/171565352 shutdown_connections 2024-05-09T08:25:43.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 1 -- 172.21.15.28:0/171565352 wait complete. 2024-05-09T08:25:43.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.067+0000 7f69b7027740 20 client.15234 trim_cache size 0 max 16384 2024-05-09T08:25:43.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.085+0000 7f69b7027740 1 Processor -- start 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b7027740 1 -- start start 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b7027740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037d8c0 0x559df037eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b7027740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037e2b0 0x559df0387960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b7027740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df0383420 0x559df0387ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b7027740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559df0393c10 con 0x559df037e2b0 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b7027740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x559df03942e0 con 0x559df037d8c0 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b7027740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559df03949b0 con 0x559df0383420 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b1fd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037e2b0 0x559df0387960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b1fd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037e2b0 0x559df0387960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40886/0 (socket says 172.21.15.28:40886) 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b1fd4640 1 -- 172.21.15.28:0/638742555 learned_addr learned my addr 172.21.15.28:0/638742555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b1fd4640 1 -- 172.21.15.28:0/638742555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df0383420 msgr2=0x559df0387ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/638742555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df0383420 0x559df0387ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b1fd4640 1 -- 172.21.15.28:0/638742555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037d8c0 msgr2=0x559df037eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/638742555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037d8c0 0x559df037eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b1fd4640 1 -- 172.21.15.28:0/638742555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559df037e690 con 0x559df037e2b0 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/638742555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037e2b0 0x559df0387960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f69a8032fd0 tx=0x7f69a8009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e4e46f0314077 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69977fe640 1 -- 172.21.15.28:0/638742555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a8046020 con 0x559df037e2b0 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69977fe640 1 -- 172.21.15.28:0/638742555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69a8042b50 con 0x559df037e2b0 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.086+0000 7f69977fe640 1 -- 172.21.15.28:0/638742555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a8035040 con 0x559df037e2b0 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 -- 172.21.15.28:0/638742555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037e2b0 msgr2=0x559df0387960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 --2- 172.21.15.28:0/638742555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037e2b0 0x559df0387960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f69a8032fd0 tx=0x7f69a8009680 comp rx=0 tx=0).stop 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 -- 172.21.15.28:0/638742555 shutdown_connections 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 --2- 172.21.15.28:0/638742555 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df0383420 0x559df0387ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 --2- 172.21.15.28:0/638742555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df037e2b0 0x559df0387960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 --2- 172.21.15.28:0/638742555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037d8c0 0x559df037eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 -- 172.21.15.28:0/638742555 >> 172.21.15.28:0/638742555 conn(0x559df0378f30 msgr2=0x559df0379310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 -- 172.21.15.28:0/638742555 shutdown_connections 2024-05-09T08:25:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 -- 172.21.15.28:0/638742555 wait complete. 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 -- start start 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.087+0000 7f69b7027740 1 Processor -- start 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b7027740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 0x559df03f8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b7027740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 0x559df03fa950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b7027740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03fcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b7027740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559df0395750 con 0x559df0383420 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b7027740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x559df0395080 con 0x559df037e2b0 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b7027740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559df0395e20 con 0x559df037d8c0 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b7027740 10 client.0 _ll_register_callbacks cb 0x7f699c01a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03fcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03fcf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40902/0 (socket says 172.21.15.28:40902) 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 -- 172.21.15.28:0/3825133263 learned_addr learned my addr 172.21.15.28:0/3825133263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 0x559df03fa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 0x559df03f8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 -- 172.21.15.28:0/3825133263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 msgr2=0x559df03f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 0x559df03f8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 -- 172.21.15.28:0/3825133263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 msgr2=0x559df03fa950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 0x559df03fa950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 -- 172.21.15.28:0/3825133263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559df03fd7c0 con 0x559df0383420 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69b17d3640 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03fcf40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f699c00cfd0 tx=0x7f699c0173a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a02aa74023da08 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.088+0000 7f69957fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69957fa640 1 -- 172.21.15.28:0/3825133263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c03b060 con 0x559df0383420 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69b7027740 1 -- 172.21.15.28:0/3825133263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69a8042b50 con 0x559df0383420 2024-05-09T08:25:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69957fa640 1 -- 172.21.15.28:0/3825133263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f699c00f040 con 0x559df0383420 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69957fa640 1 -- 172.21.15.28:0/3825133263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f699c042050 con 0x559df0383420 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69b7027740 10 client.15240 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69b7027740 1 -- 172.21.15.28:0/3825133263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559df03fea60 con 0x559df0383420 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69957fa640 1 -- 172.21.15.28:0/3825133263 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69a8042b50 con 0x559df0383420 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69957fa640 1 -- 172.21.15.28:0/3825133263 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f699c023050 con 0x559df0383420 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69b7027740 10 client.15240 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69b7027740 10 client.15240 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69b7027740 20 client.15240 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69b7027740 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x559df04034d0 0x559df04021e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69b7027740 4 client.15240 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:25:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.089+0000 7f69b7027740 1 -- 172.21.15.28:0/3825133263 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "status"}) -- 0x559df02e8f50 con 0x559df04034d0 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x559df04034d0 0x559df04021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b1fd4640 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x559df04034d0 0x559df04021e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6b7ea81e62a5ead3 server_cookie=57bdf1b70f99e47f in_seq=0 out_seq=0 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69957fa640 10 client.15240 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69957fa640 1 -- 172.21.15.28:0/3825133263 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559df02e8f50 con 0x559df04034d0 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69957fa640 10 client.15240 handle_command_reply: tid=0 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 client.15240 shutdown 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 -- 172.21.15.28:0/3825133263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 msgr2=0x559df03fcf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03fcf40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f699c00cfd0 tx=0x7f699c0173a0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 -- 172.21.15.28:0/3825133263 shutdown_connections 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x559df04034d0 0x559df04021e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559df0383420 0x559df03fcf40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559df037e2b0 0x559df03fa950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 --2- 172.21.15.28:0/3825133263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559df037d8c0 0x559df03f8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69957fa640 0 client.15240 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 -- 172.21.15.28:0/3825133263 >> 172.21.15.28:0/3825133263 conn(0x559df0378f30 msgr2=0x559df0379310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 -- 172.21.15.28:0/3825133263 shutdown_connections 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.090+0000 7f69b7027740 1 -- 172.21.15.28:0/3825133263 wait complete. 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b7027740 20 client.15240 trim_cache size 0 max 16384 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24476, 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:25:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:25:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:25:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 85.705965900999999, 2024-05-09T08:25:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:25:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 91, 2024-05-09T08:25:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:25:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 129.268827641 2024-05-09T08:25:43.100 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:25:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 -- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f69980a2c10 msgr2=0x7f69980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f69980a2c10 0x7f69980a50d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f69a400a6e0 tx=0x7f69a4066a80 comp rx=0 tx=0).stop 2024-05-09T08:25:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 -- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac102340 msgr2=0x7f69ac19f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac102340 0x7f69ac19f020 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f69a400cad0 tx=0x7f69a400cf90 comp rx=0 tx=0).stop 2024-05-09T08:25:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 -- 172.21.15.28:0/3231892495 shutdown_connections 2024-05-09T08:25:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f69980a2c10 0x7f69980a50d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69ac102340 0x7f69ac19f020 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69ac101980 0x7f69ac19ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 --2- 172.21.15.28:0/3231892495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69ac100790 0x7f69ac06a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 -- 172.21.15.28:0/3231892495 >> 172.21.15.28:0/3231892495 conn(0x7f69ac0fbd00 msgr2=0x7f69ac104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 -- 172.21.15.28:0/3231892495 shutdown_connections 2024-05-09T08:25:43.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.091+0000 7f69b4238640 1 -- 172.21.15.28:0/3231892495 wait complete. 2024-05-09T08:25:43.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T08:25:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.224+0000 7f01640ee640 1 -- 172.21.15.28:0/846264154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 msgr2=0x7f015c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.224+0000 7f01640ee640 1 --2- 172.21.15.28:0/846264154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 0x7f015c101de0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f014c009900 tx=0x7f014c02a410 comp rx=0 tx=0).stop 2024-05-09T08:25:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 -- 172.21.15.28:0/846264154 shutdown_connections 2024-05-09T08:25:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 --2- 172.21.15.28:0/846264154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c102320 0x7f015c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 --2- 172.21.15.28:0/846264154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 0x7f015c101de0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 --2- 172.21.15.28:0/846264154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c100770 0x7f015c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 -- 172.21.15.28:0/846264154 >> 172.21.15.28:0/846264154 conn(0x7f015c0fbce0 msgr2=0x7f015c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 -- 172.21.15.28:0/846264154 shutdown_connections 2024-05-09T08:25:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 -- 172.21.15.28:0/846264154 wait complete. 2024-05-09T08:25:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 Processor -- start 2024-05-09T08:25:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 -- start start 2024-05-09T08:25:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c100770 0x7f015c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 0x7f015c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c102320 0x7f015c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f015c10c3f0 con 0x7f015c101960 2024-05-09T08:25:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f015c10c270 con 0x7f015c102320 2024-05-09T08:25:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.225+0000 7f01640ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f015c10c570 con 0x7f015c100770 2024-05-09T08:25:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161689640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 0x7f015c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161689640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 0x7f015c19c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40916/0 (socket says 172.21.15.28:40916) 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161689640 1 -- 172.21.15.28:0/2967805363 learned_addr learned my addr 172.21.15.28:0/2967805363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161e8a640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c100770 0x7f015c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161689640 1 -- 172.21.15.28:0/2967805363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c100770 msgr2=0x7f015c069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161689640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c100770 0x7f015c069590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f016268b640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c102320 0x7f015c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161689640 1 -- 172.21.15.28:0/2967805363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c102320 msgr2=0x7f015c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161689640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c102320 0x7f015c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161689640 1 -- 172.21.15.28:0/2967805363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015c19f780 con 0x7f015c101960 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161e8a640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c100770 0x7f015c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f016268b640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c102320 0x7f015c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:43.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0161689640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 0x7f015c19c850 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f014c009230 tx=0x7f014c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0152ffd640 1 -- 172.21.15.28:0/2967805363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c03f030 con 0x7f015c101960 2024-05-09T08:25:43.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f01640ee640 1 -- 172.21.15.28:0/2967805363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f015c19fa10 con 0x7f015c101960 2024-05-09T08:25:43.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0152ffd640 1 -- 172.21.15.28:0/2967805363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f014c003ce0 con 0x7f015c101960 2024-05-09T08:25:43.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f0152ffd640 1 -- 172.21.15.28:0/2967805363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c035070 con 0x7f015c101960 2024-05-09T08:25:43.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.226+0000 7f01640ee640 1 -- 172.21.15.28:0/2967805363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f015c19ff00 con 0x7f015c101960 2024-05-09T08:25:43.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.229+0000 7f0166df8740 1 Processor -- start 2024-05-09T08:25:43.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0166df8740 1 -- start start 2024-05-09T08:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0152ffd640 1 -- 172.21.15.28:0/2967805363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f014c042030 con 0x7f015c101960 2024-05-09T08:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0166df8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201db8c0 0x55ed201dbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0166df8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201dc2b0 0x55ed201e0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0166df8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201e1420 0x55ed201e38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0166df8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed2013f4d0 con 0x55ed201e1420 2024-05-09T08:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0166df8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ed200e73e0 con 0x55ed201db8c0 2024-05-09T08:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0166df8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ed2013b490 con 0x55ed201dc2b0 2024-05-09T08:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161689640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201e1420 0x55ed201e38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161689640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201e1420 0x55ed201e38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40920/0 (socket says 172.21.15.28:40920) 2024-05-09T08:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161689640 1 -- 172.21.15.28:0/1253222784 learned_addr learned my addr 172.21.15.28:0/1253222784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f016268b640 1 --2- 172.21.15.28:0/1253222784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201db8c0 0x55ed201dbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161e8a640 1 --2- 172.21.15.28:0/1253222784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201dc2b0 0x55ed201e0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161689640 1 -- 172.21.15.28:0/1253222784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201dc2b0 msgr2=0x55ed201e0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161689640 1 --2- 172.21.15.28:0/1253222784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201dc2b0 0x55ed201e0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161689640 1 -- 172.21.15.28:0/1253222784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201db8c0 msgr2=0x55ed201dbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161689640 1 --2- 172.21.15.28:0/1253222784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201db8c0 0x55ed201dbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161689640 1 -- 172.21.15.28:0/1253222784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed1fe532d0 con 0x55ed201e1420 2024-05-09T08:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.230+0000 7f0161689640 1 --2- 172.21.15.28:0/1253222784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201e1420 0x55ed201e38b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f014c0f3e40 tx=0x7f014c0f3f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9e5fff8eec6dc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.231+0000 7f0152ffd640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01480a2b90 0x7f01480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.231+0000 7f016268b640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01480a2b90 0x7f01480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.231+0000 7f0147fff640 1 -- 172.21.15.28:0/1253222784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c0ff070 con 0x55ed201e1420 2024-05-09T08:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.231+0000 7f0147fff640 1 -- 172.21.15.28:0/1253222784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f014c0bc7c0 con 0x55ed201e1420 2024-05-09T08:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.231+0000 7f0147fff640 1 -- 172.21.15.28:0/1253222784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c03a3c0 con 0x55ed201e1420 2024-05-09T08:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.231+0000 7f0152ffd640 1 -- 172.21.15.28:0/2967805363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f014c0efbc0 con 0x7f015c101960 2024-05-09T08:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 -- 172.21.15.28:0/1253222784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201e1420 msgr2=0x55ed201e38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f016268b640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01480a2b90 0x7f01480a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55ed201dcb10 tx=0x7f0154008630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 --2- 172.21.15.28:0/1253222784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201e1420 0x55ed201e38b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f014c0f3e40 tx=0x7f014c0f3f30 comp rx=0 tx=0).stop 2024-05-09T08:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 -- 172.21.15.28:0/1253222784 shutdown_connections 2024-05-09T08:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 --2- 172.21.15.28:0/1253222784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201e1420 0x55ed201e38b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 --2- 172.21.15.28:0/1253222784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201dc2b0 0x55ed201e0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 --2- 172.21.15.28:0/1253222784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201db8c0 0x55ed201dbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 -- 172.21.15.28:0/1253222784 >> 172.21.15.28:0/1253222784 conn(0x55ed201d6f30 msgr2=0x55ed201d7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 -- 172.21.15.28:0/1253222784 shutdown_connections 2024-05-09T08:25:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 -- 172.21.15.28:0/1253222784 wait complete. 2024-05-09T08:25:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 -- start start 2024-05-09T08:25:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 Processor -- start 2024-05-09T08:25:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201e5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.232+0000 7f0166df8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201dc2b0 0x55ed201e5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0166df8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201e1420 0x55ed201ecf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0166df8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed201e4190 con 0x55ed201db8c0 2024-05-09T08:25:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0166df8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ed200fa8e0 con 0x55ed201dc2b0 2024-05-09T08:25:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0166df8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ed201e4310 con 0x55ed201e1420 2024-05-09T08:25:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0166df8740 10 client.0 _ll_register_callbacks cb 0x55ed200c79c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161e8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161e8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201e5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40932/0 (socket says 172.21.15.28:40932) 2024-05-09T08:25:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161e8a640 1 -- 172.21.15.28:0/1949919495 learned_addr learned my addr 172.21.15.28:0/1949919495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161689640 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201dc2b0 0x55ed201e5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f016268b640 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201e1420 0x55ed201ecf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161e8a640 1 -- 172.21.15.28:0/1949919495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201e1420 msgr2=0x55ed201ecf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161e8a640 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201e1420 0x55ed201ecf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161e8a640 1 -- 172.21.15.28:0/1949919495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201dc2b0 msgr2=0x55ed201e5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161e8a640 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201dc2b0 0x55ed201e5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161e8a640 1 -- 172.21.15.28:0/1949919495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed1ff1d6b0 con 0x55ed201db8c0 2024-05-09T08:25:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f016268b640 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201e1420 0x55ed201ecf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0161e8a640 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201e5990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f01580107f0 tx=0x7f0158010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d998f6ec2caba1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f01457fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f01457fa640 1 -- 172.21.15.28:0/1949919495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158004440 con 0x55ed201db8c0 2024-05-09T08:25:43.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f01457fa640 1 -- 172.21.15.28:0/1949919495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f015800e8b0 con 0x55ed201db8c0 2024-05-09T08:25:43.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0166df8740 1 -- 172.21.15.28:0/1949919495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f014c0bc7c0 con 0x55ed201db8c0 2024-05-09T08:25:43.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f01457fa640 1 -- 172.21.15.28:0/1949919495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f015800eab0 con 0x55ed201db8c0 2024-05-09T08:25:43.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0166df8740 10 client.15252 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:43.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.233+0000 7f0166df8740 1 -- 172.21.15.28:0/1949919495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ed201effc0 con 0x55ed201db8c0 2024-05-09T08:25:43.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.234+0000 7f01457fa640 1 -- 172.21.15.28:0/1949919495 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f014c0bc7c0 con 0x55ed201db8c0 2024-05-09T08:25:43.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.234+0000 7f01457fa640 1 -- 172.21.15.28:0/1949919495 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0158017070 con 0x55ed201db8c0 2024-05-09T08:25:43.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.234+0000 7f0166df8740 10 client.15252 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.234+0000 7f0166df8740 10 client.15252 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:25:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.234+0000 7f0166df8740 20 client.15252 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.234+0000 7f0166df8740 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55ed201f26c0 0x55ed201f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.234+0000 7f0166df8740 4 client.15252 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:25:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.234+0000 7f0166df8740 1 -- 172.21.15.28:0/1949919495 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed2014cc00 con 0x55ed201f26c0 2024-05-09T08:25:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.234+0000 7f0161689640 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55ed201f26c0 0x55ed201f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.235+0000 7f0161689640 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55ed201f26c0 0x55ed201f3b30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=851242be4f70d8f6 server_cookie=c149ac7912911cbb in_seq=0 out_seq=0 2024-05-09T08:25:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.235+0000 7f01457fa640 10 client.15252 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:25:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.235+0000 7f01457fa640 1 -- 172.21.15.28:0/1949919495 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ed2014cc00 con 0x55ed201f26c0 2024-05-09T08:25:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.235+0000 7f01457fa640 10 client.15252 handle_command_reply: tid=0 2024-05-09T08:25:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.235+0000 7f0166df8740 1 client.15252 shutdown 2024-05-09T08:25:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 -- 172.21.15.28:0/1949919495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 msgr2=0x55ed201e5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201e5990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f01580107f0 tx=0x7f0158010cb0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 -- 172.21.15.28:0/1949919495 shutdown_connections 2024-05-09T08:25:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55ed201f26c0 0x55ed201f3b30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201e1420 0x55ed201ecf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201dc2b0 0x55ed201e5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 --2- 172.21.15.28:0/1949919495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201e5990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f01457fa640 0 client.15252 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:25:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 -- 172.21.15.28:0/1949919495 >> 172.21.15.28:0/1949919495 conn(0x55ed201d6f30 msgr2=0x55ed201d6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 -- 172.21.15.28:0/1949919495 shutdown_connections 2024-05-09T08:25:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 1 -- 172.21.15.28:0/1949919495 wait complete. 2024-05-09T08:25:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.236+0000 7f0166df8740 20 client.15252 trim_cache size 0 max 16384 2024-05-09T08:25:43.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0166df8740 1 Processor -- start 2024-05-09T08:25:43.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0166df8740 1 -- start start 2024-05-09T08:25:43.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0166df8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201d7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0166df8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201dc2b0 0x55ed201ece30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0166df8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201e1420 0x55ed201d9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0166df8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed201f1e70 con 0x55ed201db8c0 2024-05-09T08:25:43.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0166df8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ed201f2540 con 0x55ed201e1420 2024-05-09T08:25:43.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0166df8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ed201f2c10 con 0x55ed201dc2b0 2024-05-09T08:25:43.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0161e8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201d7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0161e8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201d7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40942/0 (socket says 172.21.15.28:40942) 2024-05-09T08:25:43.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0161e8a640 1 -- 172.21.15.28:0/2470219840 learned_addr learned my addr 172.21.15.28:0/2470219840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0161e8a640 1 -- 172.21.15.28:0/2470219840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201dc2b0 msgr2=0x55ed201ece30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0161e8a640 1 --2- 172.21.15.28:0/2470219840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201dc2b0 0x55ed201ece30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0161e8a640 1 -- 172.21.15.28:0/2470219840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201e1420 msgr2=0x55ed201d9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0161e8a640 1 --2- 172.21.15.28:0/2470219840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201e1420 0x55ed201d9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.254+0000 7f0161e8a640 1 -- 172.21.15.28:0/2470219840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed201dc690 con 0x55ed201db8c0 2024-05-09T08:25:43.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0161e8a640 1 --2- 172.21.15.28:0/2470219840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201d7310 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0158000c00 tx=0x7f0158005c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e394c1beeb0cd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f01477fe640 1 -- 172.21.15.28:0/2470219840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015802b020 con 0x55ed201db8c0 2024-05-09T08:25:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f01477fe640 1 -- 172.21.15.28:0/2470219840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0158004cc0 con 0x55ed201db8c0 2024-05-09T08:25:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f01477fe640 1 -- 172.21.15.28:0/2470219840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01580044a0 con 0x55ed201db8c0 2024-05-09T08:25:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0166df8740 1 -- 172.21.15.28:0/2470219840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 msgr2=0x55ed201d7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0166df8740 1 --2- 172.21.15.28:0/2470219840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201d7310 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0158000c00 tx=0x7f0158005c40 comp rx=0 tx=0).stop 2024-05-09T08:25:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0166df8740 1 -- 172.21.15.28:0/2470219840 shutdown_connections 2024-05-09T08:25:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0166df8740 1 --2- 172.21.15.28:0/2470219840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201e1420 0x55ed201d9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0166df8740 1 --2- 172.21.15.28:0/2470219840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201dc2b0 0x55ed201ece30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0166df8740 1 --2- 172.21.15.28:0/2470219840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201db8c0 0x55ed201d7310 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0166df8740 1 -- 172.21.15.28:0/2470219840 >> 172.21.15.28:0/2470219840 conn(0x55ed201d6f30 msgr2=0x55ed201dcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0166df8740 1 -- 172.21.15.28:0/2470219840 shutdown_connections 2024-05-09T08:25:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.255+0000 7f0166df8740 1 -- 172.21.15.28:0/2470219840 wait complete. 2024-05-09T08:25:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0166df8740 1 -- start start 2024-05-09T08:25:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0166df8740 1 Processor -- start 2024-05-09T08:25:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0166df8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201db8c0 0x55ed201eb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0166df8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201dc2b0 0x55ed201eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0166df8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201e1420 0x55ed201ebb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0166df8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed201f39b0 con 0x55ed201dc2b0 2024-05-09T08:25:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0166df8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ed201f32e0 con 0x55ed201e1420 2024-05-09T08:25:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0166df8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ed201f4080 con 0x55ed201db8c0 2024-05-09T08:25:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0166df8740 10 client.0 _ll_register_callbacks cb 0x7f015800b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0161689640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201dc2b0 0x55ed201eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0161689640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201dc2b0 0x55ed201eb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40958/0 (socket says 172.21.15.28:40958) 2024-05-09T08:25:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.256+0000 7f0161689640 1 -- 172.21.15.28:0/2570700010 learned_addr learned my addr 172.21.15.28:0/2570700010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0161689640 1 -- 172.21.15.28:0/2570700010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201db8c0 msgr2=0x55ed201eb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0161689640 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201db8c0 0x55ed201eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0161689640 1 -- 172.21.15.28:0/2570700010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201e1420 msgr2=0x55ed201ebb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f016268b640 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201e1420 0x55ed201ebb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0161689640 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201e1420 0x55ed201ebb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0161689640 1 -- 172.21.15.28:0/2570700010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed201ec3a0 con 0x55ed201dc2b0 2024-05-09T08:25:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0161689640 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201dc2b0 0x55ed201eb5b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f014c0bbdc0 tx=0x7f014c005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac90d119fcf60ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f01457fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f01457fa640 1 -- 172.21.15.28:0/2570700010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c036d20 con 0x55ed201dc2b0 2024-05-09T08:25:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f01457fa640 1 -- 172.21.15.28:0/2570700010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f014c049de0 con 0x55ed201dc2b0 2024-05-09T08:25:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0166df8740 1 -- 172.21.15.28:0/2570700010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0158004cc0 con 0x55ed201dc2b0 2024-05-09T08:25:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f01457fa640 1 -- 172.21.15.28:0/2570700010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f014c049440 con 0x55ed201dc2b0 2024-05-09T08:25:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0166df8740 10 client.15258 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0166df8740 1 -- 172.21.15.28:0/2570700010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ed20258810 con 0x55ed201dc2b0 2024-05-09T08:25:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f01457fa640 1 -- 172.21.15.28:0/2570700010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0158004cc0 con 0x55ed201dc2b0 2024-05-09T08:25:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f01457fa640 1 -- 172.21.15.28:0/2570700010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f014c0b8730 con 0x55ed201dc2b0 2024-05-09T08:25:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0166df8740 10 client.15258 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:43.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0166df8740 10 client.15258 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:25:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0166df8740 20 client.15258 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0166df8740 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55ed2025d120 0x55ed2025be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0166df8740 4 client.15258 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:25:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.257+0000 7f0166df8740 1 -- 172.21.15.28:0/2570700010 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "status"}) -- 0x55ed20146f50 con 0x55ed2025d120 2024-05-09T08:25:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.258+0000 7f0161e8a640 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55ed2025d120 0x55ed2025be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.258+0000 7f0161e8a640 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55ed2025d120 0x55ed2025be30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=df166507f22fc1a2 server_cookie=77767c8129f804b8 in_seq=0 out_seq=0 2024-05-09T08:25:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.258+0000 7f01457fa640 10 client.15258 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:25:43.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.258+0000 7f01457fa640 1 -- 172.21.15.28:0/2570700010 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55ed20146f50 con 0x55ed2025d120 2024-05-09T08:25:43.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.258+0000 7f01457fa640 10 client.15258 handle_command_reply: tid=0 2024-05-09T08:25:43.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.258+0000 7f0166df8740 1 client.15258 shutdown 2024-05-09T08:25:43.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.258+0000 7f0166df8740 1 -- 172.21.15.28:0/2570700010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201dc2b0 msgr2=0x55ed201eb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.258+0000 7f0166df8740 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201dc2b0 0x55ed201eb5b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f014c0bbdc0 tx=0x7f014c005b70 comp rx=0 tx=0).stop 2024-05-09T08:25:43.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f0166df8740 1 -- 172.21.15.28:0/2570700010 shutdown_connections 2024-05-09T08:25:43.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f0166df8740 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55ed2025d120 0x55ed2025be30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f0166df8740 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed201e1420 0x55ed201ebb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f0166df8740 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed201dc2b0 0x55ed201eb5b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f0166df8740 1 --2- 172.21.15.28:0/2570700010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed201db8c0 0x55ed201eb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01457fa640 0 client.15258 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:25:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f0166df8740 1 -- 172.21.15.28:0/2570700010 >> 172.21.15.28:0/2570700010 conn(0x55ed201d6f30 msgr2=0x55ed201d7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f0166df8740 1 -- 172.21.15.28:0/2570700010 shutdown_connections 2024-05-09T08:25:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f0166df8740 1 -- 172.21.15.28:0/2570700010 wait complete. 2024-05-09T08:25:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f0166df8740 20 client.15258 trim_cache size 0 max 16384 2024-05-09T08:25:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:25:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:25:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-05-09T08:25:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24481, 2024-05-09T08:25:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:25:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:25:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:25:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 86.88063425, 2024-05-09T08:25:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:25:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 91, 2024-05-09T08:25:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:25:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 132.35457358299999 2024-05-09T08:25:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:25:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 -- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01480a2b90 msgr2=0x7f01480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01480a2b90 0x7f01480a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55ed201dcb10 tx=0x7f0154008630 comp rx=0 tx=0).stop 2024-05-09T08:25:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 -- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 msgr2=0x7f015c19c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 0x7f015c19c850 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f014c009230 tx=0x7f014c004430 comp rx=0 tx=0).stop 2024-05-09T08:25:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 -- 172.21.15.28:0/2967805363 shutdown_connections 2024-05-09T08:25:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f01480a2b90 0x7f01480a5050 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c102320 0x7f015c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101960 0x7f015c19c850 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 --2- 172.21.15.28:0/2967805363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c100770 0x7f015c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 -- 172.21.15.28:0/2967805363 >> 172.21.15.28:0/2967805363 conn(0x7f015c0fbce0 msgr2=0x7f015c104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 -- 172.21.15.28:0/2967805363 shutdown_connections 2024-05-09T08:25:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.259+0000 7f01640ee640 1 -- 172.21.15.28:0/2967805363 wait complete. 2024-05-09T08:25:43.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-09T08:25:43.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:43 smithi184 ceph-mon[34509]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-09T08:25:43.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:43 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2962719696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:25:43.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:43 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2138424242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:25:43.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:43 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3018806135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:25:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.388+0000 7f7f58c29640 1 -- 172.21.15.28:0/2256868445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54101980 msgr2=0x7f7f54101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.388+0000 7f7f58c29640 1 --2- 172.21.15.28:0/2256868445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54101980 0x7f7f54101e00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009900 tx=0x7f7f3c02a410 comp rx=0 tx=0).stop 2024-05-09T08:25:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.388+0000 7f7f58c29640 1 -- 172.21.15.28:0/2256868445 shutdown_connections 2024-05-09T08:25:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.388+0000 7f7f58c29640 1 --2- 172.21.15.28:0/2256868445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f54102340 0x7f7f54108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.388+0000 7f7f58c29640 1 --2- 172.21.15.28:0/2256868445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54101980 0x7f7f54101e00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.388+0000 7f7f58c29640 1 --2- 172.21.15.28:0/2256868445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f54100790 0x7f7f54100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.388+0000 7f7f58c29640 1 -- 172.21.15.28:0/2256868445 >> 172.21.15.28:0/2256868445 conn(0x7f7f540fbd20 msgr2=0x7f7f540fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.388+0000 7f7f58c29640 1 -- 172.21.15.28:0/2256868445 shutdown_connections 2024-05-09T08:25:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.388+0000 7f7f58c29640 1 -- 172.21.15.28:0/2256868445 wait complete. 2024-05-09T08:25:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f58c29640 1 Processor -- start 2024-05-09T08:25:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f58c29640 1 -- start start 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f58c29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54100790 0x7f7f54197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f58c29640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f54101980 0x7f7f541980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f58c29640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f54102340 0x7f7f5419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f58c29640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5410d590 con 0x7f7f54100790 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f58c29640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5410d410 con 0x7f7f54102340 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f58c29640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5410d710 con 0x7f7f54101980 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f5259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54100790 0x7f7f54197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f5259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54100790 0x7f7f54197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40970/0 (socket says 172.21.15.28:40970) 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f5259c640 1 -- 172.21.15.28:0/481831088 learned_addr learned my addr 172.21.15.28:0/481831088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.389+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f54102340 0x7f7f5419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f5259c640 1 -- 172.21.15.28:0/481831088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f54101980 msgr2=0x7f7f541980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f5259c640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f54101980 0x7f7f541980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f5259c640 1 -- 172.21.15.28:0/481831088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f54102340 msgr2=0x7f7f5419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f5259c640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f54102340 0x7f7f5419f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f5259c640 1 -- 172.21.15.28:0/481831088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5419f7c0 con 0x7f7f54100790 2024-05-09T08:25:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f54102340 0x7f7f5419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f5259c640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54100790 0x7f7f54197ba0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7f4400e9c0 tx=0x7f7f4400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f4b7fe640 1 -- 172.21.15.28:0/481831088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4400cd80 con 0x7f7f54100790 2024-05-09T08:25:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f58c29640 1 -- 172.21.15.28:0/481831088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5419fab0 con 0x7f7f54100790 2024-05-09T08:25:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f4b7fe640 1 -- 172.21.15.28:0/481831088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f440035f0 con 0x7f7f54100790 2024-05-09T08:25:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f4b7fe640 1 -- 172.21.15.28:0/481831088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f44010690 con 0x7f7f54100790 2024-05-09T08:25:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.390+0000 7f7f58c29640 1 -- 172.21.15.28:0/481831088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f541a0000 con 0x7f7f54100790 2024-05-09T08:25:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.393+0000 7f7f4b7fe640 1 -- 172.21.15.28:0/481831088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7f440037a0 con 0x7f7f54100790 2024-05-09T08:25:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.393+0000 7f7f5ba0c740 1 Processor -- start 2024-05-09T08:25:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.393+0000 7f7f5ba0c740 1 -- start start 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.393+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659e6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.393+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.393+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.393+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56436594a4d0 con 0x5643659ec420 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.393+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5643658f23e0 con 0x5643659e72b0 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.393+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564365946490 con 0x5643659e68c0 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659ee8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40982/0 (socket says 172.21.15.28:40982) 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4bfff640 1 -- 172.21.15.28:0/2923202508 learned_addr learned my addr 172.21.15.28:0/2923202508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659e6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f5259c640 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4bfff640 1 -- 172.21.15.28:0/2923202508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 msgr2=0x5643659e6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659e6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4b7fe640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f400a2b90 0x7f7f400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4bfff640 1 -- 172.21.15.28:0/2923202508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 msgr2=0x5643659ebdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659ebdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4bfff640 1 -- 172.21.15.28:0/2923202508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56436565e2d0 con 0x5643659ec420 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659e6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f400a2b90 0x7f7f400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4b7fe640 1 -- 172.21.15.28:0/481831088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7f440cf8b0 con 0x7f7f54100790 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659ee8b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0314e0 tx=0x7f7f3c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c760a99498bb5f6d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.394+0000 7f7f48ff9640 1 -- 172.21.15.28:0/2923202508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c03e070 con 0x5643659ec420 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f48ff9640 1 -- 172.21.15.28:0/2923202508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3c005530 con 0x5643659ec420 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f48ff9640 1 -- 172.21.15.28:0/2923202508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c035040 con 0x5643659ec420 2024-05-09T08:25:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/2923202508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 msgr2=0x5643659ee8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659ee8b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0314e0 tx=0x7f7f3c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f400a2b90 0x7f7f400a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0095f0 tx=0x7f7f4c009580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/2923202508 shutdown_connections 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659ee8b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659ebdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/2923202508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659e6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/2923202508 >> 172.21.15.28:0/2923202508 conn(0x5643659e1f30 msgr2=0x5643659e2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/2923202508 shutdown_connections 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.395+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/2923202508 wait complete. 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5ba0c740 1 -- start start 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5ba0c740 1 Processor -- start 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659e72b0 0x5643659f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659ec420 0x5643659f81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5643659ef370 con 0x5643659e72b0 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5643659058e0 con 0x5643659ec420 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5643659ef4f0 con 0x5643659e68c0 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5ba0c740 10 client.0 _ll_register_callbacks cb 0x5643658d29c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659e72b0 0x5643659f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659e72b0 0x5643659f11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40988/0 (socket says 172.21.15.28:40988) 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f4bfff640 1 -- 172.21.15.28:0/3359930072 learned_addr learned my addr 172.21.15.28:0/3359930072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659ec420 0x5643659f81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f5259c640 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659f0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f4bfff640 1 -- 172.21.15.28:0/3359930072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 msgr2=0x5643659f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659f0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f4bfff640 1 -- 172.21.15.28:0/3359930072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659ec420 msgr2=0x5643659f81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659ec420 0x5643659f81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f4bfff640 1 -- 172.21.15.28:0/3359930072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643657286b0 con 0x5643659e72b0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.396+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659ec420 0x5643659f81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659e72b0 0x5643659f11e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c006010 tx=0x7f7f3c004eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=687e0406b081c7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f39ffb640 1 -- 172.21.15.28:0/3359930072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c03e030 con 0x5643659e72b0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/3359930072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f3c005530 con 0x5643659e72b0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f39ffb640 1 -- 172.21.15.28:0/3359930072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7f3c0040c0 con 0x5643659e72b0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f39ffb640 1 -- 172.21.15.28:0/3359930072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7f3c055050 con 0x5643659e72b0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f5ba0c740 10 client.15270 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/3359930072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643659fb1e0 con 0x5643659e72b0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f39ffb640 1 -- 172.21.15.28:0/3359930072 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3c005530 con 0x5643659e72b0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f39ffb640 1 -- 172.21.15.28:0/3359930072 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7f3c0035f0 con 0x5643659e72b0 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f5ba0c740 10 client.15270 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f5ba0c740 10 client.15270 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f5ba0c740 20 client.15270 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5643659fd4a0 0x5643659fe910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f5ba0c740 4 client.15270 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.397+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/3359930072 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564365957c00 con 0x5643659fd4a0 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.398+0000 7f7f5259c640 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5643659fd4a0 0x5643659fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.398+0000 7f7f5259c640 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5643659fd4a0 0x5643659fe910 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=72df0861d5c45067 server_cookie=ec490a15ad4c3fc in_seq=0 out_seq=0 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.398+0000 7f7f39ffb640 10 client.15270 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f39ffb640 1 -- 172.21.15.28:0/3359930072 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564365957c00 con 0x5643659fd4a0 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f39ffb640 10 client.15270 handle_command_reply: tid=0 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 client.15270 shutdown 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/3359930072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659e72b0 msgr2=0x5643659f11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659e72b0 0x5643659f11e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c006010 tx=0x7f7f3c004eb0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/3359930072 shutdown_connections 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5643659fd4a0 0x5643659fe910 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659ec420 0x5643659f81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659e72b0 0x5643659f11e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/3359930072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659f0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f39ffb640 0 client.15270 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:25:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/3359930072 >> 172.21.15.28:0/3359930072 conn(0x5643659e1f30 msgr2=0x5643659e1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/3359930072 shutdown_connections 2024-05-09T08:25:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/3359930072 wait complete. 2024-05-09T08:25:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.399+0000 7f7f5ba0c740 20 client.15270 trim_cache size 0 max 16384 2024-05-09T08:25:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f5ba0c740 1 Processor -- start 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f5ba0c740 1 -- start start 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659ee810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659e2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5643659fcc50 con 0x5643659ec420 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5643659fd320 con 0x5643659e72b0 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5643659fd9f0 con 0x5643659e68c0 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f52d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f52d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659e2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41000/0 (socket says 172.21.15.28:41000) 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f52d9d640 1 -- 172.21.15.28:0/729124865 learned_addr learned my addr 172.21.15.28:0/729124865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f5259c640 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.418+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f52d9d640 1 -- 172.21.15.28:0/729124865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 msgr2=0x5643659ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f52d9d640 1 -- 172.21.15.28:0/729124865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 msgr2=0x5643659f80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f52d9d640 1 -- 172.21.15.28:0/729124865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643659e7b40 con 0x5643659ec420 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659e2310 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c01c4b0 tx=0x7f7f4c012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcef84deee3df893 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f3bfff640 1 -- 172.21.15.28:0/729124865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c004d20 con 0x5643659ec420 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f3bfff640 1 -- 172.21.15.28:0/729124865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f4c0109d0 con 0x5643659ec420 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f3bfff640 1 -- 172.21.15.28:0/729124865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c005630 con 0x5643659ec420 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/729124865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 msgr2=0x5643659e2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659e2310 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c01c4b0 tx=0x7f7f4c012f90 comp rx=0 tx=0).stop 2024-05-09T08:25:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/729124865 shutdown_connections 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.419+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659e2310 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/729124865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/729124865 >> 172.21.15.28:0/729124865 conn(0x5643659e1f30 msgr2=0x5643659fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/729124865 shutdown_connections 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/729124865 wait complete. 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 -- start start 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 Processor -- start 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5643659fe790 con 0x5643659ec420 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5643659fe0c0 con 0x5643659e72b0 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5643659fee60 con 0x5643659e68c0 2024-05-09T08:25:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5ba0c740 10 client.0 _ll_register_callbacks cb 0x7f7f3c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f52d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f52d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659f70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41012/0 (socket says 172.21.15.28:41012) 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f52d9d640 1 -- 172.21.15.28:0/59108335 learned_addr learned my addr 172.21.15.28:0/59108335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.420+0000 7f7f5259c640 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f52d9d640 1 -- 172.21.15.28:0/59108335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 msgr2=0x5643659f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f52d9d640 1 -- 172.21.15.28:0/59108335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 msgr2=0x5643659f40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f52d9d640 1 -- 172.21.15.28:0/59108335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643659f76e0 con 0x5643659ec420 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f4bfff640 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f52d9d640 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659f70f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c028040 tx=0x7f7f4c010df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb0506b1820afe5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f39ffb640 1 -- 172.21.15.28:0/59108335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c033020 con 0x5643659ec420 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/59108335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f4c0109d0 con 0x5643659ec420 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f39ffb640 1 -- 172.21.15.28:0/59108335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7f4c010700 con 0x5643659ec420 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f39ffb640 1 -- 172.21.15.28:0/59108335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7f4c03e050 con 0x5643659ec420 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f5ba0c740 10 client.15276 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/59108335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564365a63660 con 0x5643659ec420 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.421+0000 7f7f39ffb640 1 -- 172.21.15.28:0/59108335 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f4c0109d0 con 0x5643659ec420 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f39ffb640 1 -- 172.21.15.28:0/59108335 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7f4c00f040 con 0x5643659ec420 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f5ba0c740 10 client.15276 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f5ba0c740 10 client.15276 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f5ba0c740 20 client.15276 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564365a67f80 0x564365a66c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f5ba0c740 4 client.15276 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:25:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/59108335 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "status"}) -- 0x564365951f50 con 0x564365a67f80 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f5259c640 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564365a67f80 0x564365a66c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f39ffb640 10 client.15276 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.422+0000 7f7f5259c640 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564365a67f80 0x564365a66c90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74e6142286d139a0 server_cookie=821569810f2fd344 in_seq=0 out_seq=0 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f39ffb640 1 -- 172.21.15.28:0/59108335 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564365951f50 con 0x564365a67f80 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f39ffb640 10 client.15276 handle_command_reply: tid=0 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 client.15276 shutdown 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/59108335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 msgr2=0x5643659f70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659f70f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c028040 tx=0x7f7f4c010df0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/59108335 shutdown_connections 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564365a67f80 0x564365a66c90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643659ec420 0x5643659f70f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f39ffb640 0 client.15276 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5643659e72b0 0x5643659f40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 --2- 172.21.15.28:0/59108335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5643659e68c0 0x5643659f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/59108335 >> 172.21.15.28:0/59108335 conn(0x5643659e1f30 msgr2=0x5643659eb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/59108335 shutdown_connections 2024-05-09T08:25:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 1 -- 172.21.15.28:0/59108335 wait complete. 2024-05-09T08:25:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f5ba0c740 20 client.15276 trim_cache size 0 max 16384 2024-05-09T08:25:43.436 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:25:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:25:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-05-09T08:25:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24499, 2024-05-09T08:25:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:25:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:25:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:25:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 88.054935783000005, 2024-05-09T08:25:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:25:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 91, 2024-05-09T08:25:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:25:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 126.799597393 2024-05-09T08:25:43.437 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:25:43.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 -- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f400a2b90 msgr2=0x7f7f400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f400a2b90 0x7f7f400a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0095f0 tx=0x7f7f4c009580 comp rx=0 tx=0).stop 2024-05-09T08:25:43.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 -- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54100790 msgr2=0x7f7f54197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54100790 0x7f7f54197ba0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7f4400e9c0 tx=0x7f7f4400ee80 comp rx=0 tx=0).stop 2024-05-09T08:25:43.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 -- 172.21.15.28:0/481831088 shutdown_connections 2024-05-09T08:25:43.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7f400a2b90 0x7f7f400a5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f54102340 0x7f7f5419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f54101980 0x7f7f541980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 --2- 172.21.15.28:0/481831088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f54100790 0x7f7f54197ba0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 -- 172.21.15.28:0/481831088 >> 172.21.15.28:0/481831088 conn(0x7f7f540fbd20 msgr2=0x7f7f54104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.423+0000 7f7f58c29640 1 -- 172.21.15.28:0/481831088 shutdown_connections 2024-05-09T08:25:43.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.424+0000 7f7f58c29640 1 -- 172.21.15.28:0/481831088 wait complete. 2024-05-09T08:25:43.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:43 smithi028 ceph-mon[30119]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-09T08:25:43.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:43 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2962719696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:25:43.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:43 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2138424242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:25:43.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:43 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3018806135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:25:43.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T08:25:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 -- 172.21.15.28:0/4289346005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 msgr2=0x7f2eb8100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/4289346005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 0x7f2eb8100b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009900 tx=0x7f2ea802a410 comp rx=0 tx=0).stop 2024-05-09T08:25:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 -- 172.21.15.28:0/4289346005 shutdown_connections 2024-05-09T08:25:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/4289346005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb8102320 0x7f2eb8108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/4289346005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2eb8101960 0x7f2eb8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/4289346005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 0x7f2eb8100b90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 -- 172.21.15.28:0/4289346005 >> 172.21.15.28:0/4289346005 conn(0x7f2eb80fbce0 msgr2=0x7f2eb80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 -- 172.21.15.28:0/4289346005 shutdown_connections 2024-05-09T08:25:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 -- 172.21.15.28:0/4289346005 wait complete. 2024-05-09T08:25:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.552+0000 7f2ebe24d640 1 Processor -- start 2024-05-09T08:25:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2ebe24d640 1 -- start start 2024-05-09T08:25:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2ebe24d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 0x7f2eb8197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2ebe24d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2eb8101960 0x7f2eb8198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2ebe24d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb8102320 0x7f2eb819f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2ebe24d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb810c3e0 con 0x7f2eb8100770 2024-05-09T08:25:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2ebe24d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb810c260 con 0x7f2eb8102320 2024-05-09T08:25:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2ebe24d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb810c560 con 0x7f2eb8101960 2024-05-09T08:25:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 0x7f2eb8197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 0x7f2eb8197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41040/0 (socket says 172.21.15.28:41040) 2024-05-09T08:25:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb77fe640 1 -- 172.21.15.28:0/2296005822 learned_addr learned my addr 172.21.15.28:0/2296005822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2eb8101960 0x7f2eb8198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb8102320 0x7f2eb819f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb77fe640 1 -- 172.21.15.28:0/2296005822 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2eb8101960 msgr2=0x7f2eb8198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2eb8101960 0x7f2eb8198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb77fe640 1 -- 172.21.15.28:0/2296005822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb8102320 msgr2=0x7f2eb819f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb8102320 0x7f2eb819f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.553+0000 7f2eb77fe640 1 -- 172.21.15.28:0/2296005822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb819f7f0 con 0x7f2eb8100770 2024-05-09T08:25:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.554+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 0x7f2eb8197bd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8031610 tx=0x7f2ea8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.554+0000 7f2eb4ff9640 1 -- 172.21.15.28:0/2296005822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea803f030 con 0x7f2eb8100770 2024-05-09T08:25:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.554+0000 7f2ebe24d640 1 -- 172.21.15.28:0/2296005822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb819fa80 con 0x7f2eb8100770 2024-05-09T08:25:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.554+0000 7f2eb4ff9640 1 -- 172.21.15.28:0/2296005822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea8003ad0 con 0x7f2eb8100770 2024-05-09T08:25:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.554+0000 7f2eb4ff9640 1 -- 172.21.15.28:0/2296005822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8035070 con 0x7f2eb8100770 2024-05-09T08:25:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.554+0000 7f2ebe24d640 1 -- 172.21.15.28:0/2296005822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb819ff70 con 0x7f2eb8100770 2024-05-09T08:25:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.556+0000 7f2eb4ff9640 1 -- 172.21.15.28:0/2296005822 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2ea8042030 con 0x7f2eb8100770 2024-05-09T08:25:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.556+0000 7f2ec1036740 1 Processor -- start 2024-05-09T08:25:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.556+0000 7f2ec1036740 1 -- start start 2024-05-09T08:25:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc123cd0 0x5611bc1240f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc1246c0 0x5611bc1291c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc12bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2ec1036740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5611bc0834d0 con 0x5611bc1246c0 2024-05-09T08:25:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2ec1036740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5611bc02b3e0 con 0x5611bc129830 2024-05-09T08:25:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2ec1036740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5611bc07f490 con 0x5611bc123cd0 2024-05-09T08:25:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc1246c0 0x5611bc1291c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb6ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc12bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb6ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc12bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:40694/0 (socket says 172.21.15.28:40694) 2024-05-09T08:25:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb6ffd640 1 -- 172.21.15.28:0/4587342 learned_addr learned my addr 172.21.15.28:0/4587342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc1246c0 0x5611bc1291c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41048/0 (socket says 172.21.15.28:41048) 2024-05-09T08:25:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb4ff9640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2ea00a2b90 0x7f2ea00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/4587342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc123cd0 0x5611bc1240f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2ea00a2b90 0x7f2ea00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb6ffd640 1 -- 172.21.15.28:0/4587342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc123cd0 msgr2=0x5611bc1240f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/4587342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc123cd0 0x5611bc1240f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb6ffd640 1 -- 172.21.15.28:0/4587342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc1246c0 msgr2=0x5611bc1291c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/4587342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc1246c0 0x5611bc1291c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb6ffd640 1 -- 172.21.15.28:0/4587342 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611bbd972d0 con 0x5611bc129830 2024-05-09T08:25:43.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/4587342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc1246c0 0x5611bc1291c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:25:43.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.557+0000 7f2eb4ff9640 1 -- 172.21.15.28:0/2296005822 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2ea80ef5d0 con 0x7f2eb8100770 2024-05-09T08:25:43.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/4587342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc12bcc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009ca0 tx=0x7f2eac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=752ce7f359ae6753 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2e9dffb640 1 -- 172.21.15.28:0/4587342 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac01e070 con 0x5611bc129830 2024-05-09T08:25:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2ea00a2b90 0x7f2ea00a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0005c40 tx=0x7f2eb0004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2e9dffb640 1 -- 172.21.15.28:0/4587342 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eac019b10 con 0x5611bc129830 2024-05-09T08:25:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2e9dffb640 1 -- 172.21.15.28:0/4587342 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac00ea30 con 0x5611bc129830 2024-05-09T08:25:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2ec1036740 1 -- 172.21.15.28:0/4587342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 msgr2=0x5611bc12bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2ec1036740 1 --2- 172.21.15.28:0/4587342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc12bcc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009ca0 tx=0x7f2eac011810 comp rx=0 tx=0).stop 2024-05-09T08:25:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2ec1036740 1 -- 172.21.15.28:0/4587342 shutdown_connections 2024-05-09T08:25:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2ec1036740 1 --2- 172.21.15.28:0/4587342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc12bcc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2ec1036740 1 --2- 172.21.15.28:0/4587342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc1246c0 0x5611bc1291c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2ec1036740 1 --2- 172.21.15.28:0/4587342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc123cd0 0x5611bc1240f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.558+0000 7f2ec1036740 1 -- 172.21.15.28:0/4587342 >> 172.21.15.28:0/4587342 conn(0x5611bc11af30 msgr2=0x5611bc11b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 -- 172.21.15.28:0/4587342 shutdown_connections 2024-05-09T08:25:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 -- 172.21.15.28:0/4587342 wait complete. 2024-05-09T08:25:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 -- start start 2024-05-09T08:25:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 Processor -- start 2024-05-09T08:25:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc12e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc1246c0 0x5611bc12e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc1356b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5611bc12c7e0 con 0x5611bc123cd0 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5611bc03e8e0 con 0x5611bc129830 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5611bc12c960 con 0x5611bc1246c0 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc12e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2ec1036740 10 client.0 _ll_register_callbacks cb 0x5611bc00b9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc12e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41054/0 (socket says 172.21.15.28:41054) 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.559+0000 7f2eb77fe640 1 -- 172.21.15.28:0/1113938779 learned_addr learned my addr 172.21.15.28:0/1113938779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc1356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2eb77fe640 1 -- 172.21.15.28:0/1113938779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc1246c0 msgr2=0x5611bc12e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc1246c0 0x5611bc12e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2eb77fe640 1 -- 172.21.15.28:0/1113938779 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 msgr2=0x5611bc1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc1356b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2eb77fe640 1 -- 172.21.15.28:0/1113938779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611bbe616b0 con 0x5611bc123cd0 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc12e130 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8031440 tx=0x7f2ea8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5985d92160731c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2e96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2e96ffd640 1 -- 172.21.15.28:0/1113938779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8049040 con 0x5611bc123cd0 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2e96ffd640 1 -- 172.21.15.28:0/1113938779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2ea80ef330 con 0x5611bc123cd0 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2ec1036740 1 -- 172.21.15.28:0/1113938779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2eac019b10 con 0x5611bc123cd0 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2e96ffd640 1 -- 172.21.15.28:0/1113938779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2ea803be40 con 0x5611bc123cd0 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2ec1036740 10 client.15288 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2ec1036740 1 -- 172.21.15.28:0/1113938779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5611bc138690 con 0x5611bc123cd0 2024-05-09T08:25:43.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.560+0000 7f2e96ffd640 1 -- 172.21.15.28:0/1113938779 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eac019b10 con 0x5611bc123cd0 2024-05-09T08:25:43.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2e96ffd640 1 -- 172.21.15.28:0/1113938779 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2ea80b81f0 con 0x5611bc123cd0 2024-05-09T08:25:43.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2ec1036740 10 client.15288 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:43.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2ec1036740 10 client.15288 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:25:43.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2ec1036740 20 client.15288 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:43.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2ec1036740 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5611bc13ae10 0x5611bc13c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2ec1036740 4 client.15288 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2ec1036740 1 -- 172.21.15.28:0/1113938779 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611bc090c00 con 0x5611bc13ae10 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5611bc13ae10 0x5611bc13c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2e96ffd640 10 client.15288 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.561+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5611bc13ae10 0x5611bc13c280 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=684db9ea6c9c7085 server_cookie=7616f86863c30915 in_seq=0 out_seq=0 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2e96ffd640 1 -- 172.21.15.28:0/1113938779 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5611bc090c00 con 0x5611bc13ae10 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2e96ffd640 10 client.15288 handle_command_reply: tid=0 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 client.15288 shutdown 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 -- 172.21.15.28:0/1113938779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 msgr2=0x5611bc12e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc12e130 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8031440 tx=0x7f2ea8009560 comp rx=0 tx=0).stop 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 -- 172.21.15.28:0/1113938779 shutdown_connections 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5611bc13ae10 0x5611bc13c280 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc129830 0x5611bc1356b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc1246c0 0x5611bc12e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 --2- 172.21.15.28:0/1113938779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc12e130 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2e96ffd640 0 client.15288 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:25:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 -- 172.21.15.28:0/1113938779 >> 172.21.15.28:0/1113938779 conn(0x5611bc11af30 msgr2=0x5611bc11a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 -- 172.21.15.28:0/1113938779 shutdown_connections 2024-05-09T08:25:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 1 -- 172.21.15.28:0/1113938779 wait complete. 2024-05-09T08:25:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.562+0000 7f2ec1036740 20 client.15288 trim_cache size 0 max 16384 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2ec1036740 1 Processor -- start 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2ec1036740 1 -- start start 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc123cd0 0x5611bc12dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc1246c0 0x5611bc12e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc129830 0x5611bc119770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2ec1036740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5611bc13a5c0 con 0x5611bc129830 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2ec1036740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5611bc13ac90 con 0x5611bc123cd0 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2ec1036740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5611bc13b360 con 0x5611bc1246c0 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc129830 0x5611bc119770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc129830 0x5611bc119770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41056/0 (socket says 172.21.15.28:41056) 2024-05-09T08:25:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb7fff640 1 -- 172.21.15.28:0/2649587458 learned_addr learned my addr 172.21.15.28:0/2649587458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/2649587458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc1246c0 0x5611bc12e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/2649587458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc123cd0 0x5611bc12dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb7fff640 1 -- 172.21.15.28:0/2649587458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc1246c0 msgr2=0x5611bc12e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/2649587458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc1246c0 0x5611bc12e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb7fff640 1 -- 172.21.15.28:0/2649587458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc123cd0 msgr2=0x5611bc12dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/2649587458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc123cd0 0x5611bc12dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.584+0000 7f2eb7fff640 1 -- 172.21.15.28:0/2649587458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611bc124aa0 con 0x5611bc129830 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.585+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/2649587458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc129830 0x5611bc119770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2eb00136b0 tx=0x7f2eb0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407f9634eebf916e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.585+0000 7f2e9d7fa640 1 -- 172.21.15.28:0/2649587458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0019070 con 0x5611bc129830 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.585+0000 7f2e9d7fa640 1 -- 172.21.15.28:0/2649587458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eb00215c0 con 0x5611bc129830 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.585+0000 7f2e9d7fa640 1 -- 172.21.15.28:0/2649587458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0005310 con 0x5611bc129830 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 -- 172.21.15.28:0/2649587458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc129830 msgr2=0x5611bc119770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 --2- 172.21.15.28:0/2649587458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc129830 0x5611bc119770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2eb00136b0 tx=0x7f2eb0005bd0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 -- 172.21.15.28:0/2649587458 shutdown_connections 2024-05-09T08:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 --2- 172.21.15.28:0/2649587458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc129830 0x5611bc119770 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 --2- 172.21.15.28:0/2649587458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc1246c0 0x5611bc12e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 --2- 172.21.15.28:0/2649587458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc123cd0 0x5611bc12dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 -- 172.21.15.28:0/2649587458 >> 172.21.15.28:0/2649587458 conn(0x5611bc11af30 msgr2=0x5611bc1399e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 -- 172.21.15.28:0/2649587458 shutdown_connections 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 -- 172.21.15.28:0/2649587458 wait complete. 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 -- start start 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 Processor -- start 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc1346b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.586+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc1246c0 0x5611bc134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2ec1036740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc129830 0x5611bc131270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2ec1036740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5611bc13c100 con 0x5611bc123cd0 2024-05-09T08:25:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2ec1036740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5611bc13ba30 con 0x5611bc1246c0 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2ec1036740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5611bc13c7d0 con 0x5611bc129830 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2ec1036740 10 client.0 _ll_register_callbacks cb 0x7f2ea8031ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc1346b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc1346b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41062/0 (socket says 172.21.15.28:41062) 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb77fe640 1 -- 172.21.15.28:0/3049753911 learned_addr learned my addr 172.21.15.28:0/3049753911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc129830 0x5611bc131270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc1246c0 0x5611bc134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb77fe640 1 -- 172.21.15.28:0/3049753911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc129830 msgr2=0x5611bc131270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc129830 0x5611bc131270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb77fe640 1 -- 172.21.15.28:0/3049753911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc1246c0 msgr2=0x5611bc134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc1246c0 0x5611bc134bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb77fe640 1 -- 172.21.15.28:0/3049753911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611bc131aa0 con 0x5611bc123cd0 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb7fff640 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc129830 0x5611bc131270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2eb77fe640 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc1346b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8031b80 tx=0x7f2ea80058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8c34190b339cf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2e96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2e96ffd640 1 -- 172.21.15.28:0/3049753911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8049040 con 0x5611bc123cd0 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2e96ffd640 1 -- 172.21.15.28:0/3049753911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2ea80b81f0 con 0x5611bc123cd0 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2ec1036740 1 -- 172.21.15.28:0/3049753911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2eb00215c0 con 0x5611bc123cd0 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2e96ffd640 1 -- 172.21.15.28:0/3049753911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2ea80bb6c0 con 0x5611bc123cd0 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2ec1036740 10 client.15294 fetch_fsmap learned FSMap version 27 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.587+0000 7f2ec1036740 1 -- 172.21.15.28:0/3049753911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5611bc1a1c80 con 0x5611bc123cd0 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2e96ffd640 1 -- 172.21.15.28:0/3049753911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eb00215c0 con 0x5611bc123cd0 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2e96ffd640 1 -- 172.21.15.28:0/3049753911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2ea80c42b0 con 0x5611bc123cd0 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2ec1036740 10 client.15294 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2ec1036740 10 client.15294 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2ec1036740 20 client.15294 populate_metadata read hostname 'smithi028' 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2ec1036740 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5611bc1a6640 0x5611bc1a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2ec1036740 4 client.15294 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:25:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2ec1036740 1 -- 172.21.15.28:0/3049753911 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "status"}) -- 0x5611bc08af50 con 0x5611bc1a6640 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5611bc1a6640 0x5611bc1a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2e96ffd640 10 client.15294 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.588+0000 7f2eb6ffd640 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5611bc1a6640 0x5611bc1a2a80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b71a9bf78b85d3f server_cookie=161825513e13d660 in_seq=0 out_seq=0 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2e96ffd640 1 -- 172.21.15.28:0/3049753911 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5611bc08af50 con 0x5611bc1a6640 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2e96ffd640 10 client.15294 handle_command_reply: tid=0 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 client.15294 shutdown 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 -- 172.21.15.28:0/3049753911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 msgr2=0x5611bc1346b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc1346b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8031b80 tx=0x7f2ea80058d0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 -- 172.21.15.28:0/3049753911 shutdown_connections 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5611bc1a6640 0x5611bc1a2a80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5611bc129830 0x5611bc131270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5611bc1246c0 0x5611bc134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 --2- 172.21.15.28:0/3049753911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611bc123cd0 0x5611bc1346b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2e96ffd640 0 client.15294 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:25:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 -- 172.21.15.28:0/3049753911 >> 172.21.15.28:0/3049753911 conn(0x5611bc11af30 msgr2=0x5611bc12dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 -- 172.21.15.28:0/3049753911 shutdown_connections 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 1 -- 172.21.15.28:0/3049753911 wait complete. 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ec1036740 20 client.15294 trim_cache size 0 max 16384 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24512, 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:25:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 111.95777790699999, 2024-05-09T08:25:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:25:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 91, 2024-05-09T08:25:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:25:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 118.19082967999999 2024-05-09T08:25:43.599 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ebe24d640 1 -- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2ea00a2b90 msgr2=0x7f2ea00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2ea00a2b90 0x7f2ea00a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0005c40 tx=0x7f2eb0004ab0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ebe24d640 1 -- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 msgr2=0x7f2eb8197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.589+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 0x7f2eb8197bd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8031610 tx=0x7f2ea8004270 comp rx=0 tx=0).stop 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.590+0000 7f2ebe24d640 1 -- 172.21.15.28:0/2296005822 shutdown_connections 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.590+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2ea00a2b90 0x7f2ea00a5050 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.590+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2eb8102320 0x7f2eb819f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.590+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2eb8101960 0x7f2eb8198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.590+0000 7f2ebe24d640 1 --2- 172.21.15.28:0/2296005822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8100770 0x7f2eb8197bd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.590+0000 7f2ebe24d640 1 -- 172.21.15.28:0/2296005822 >> 172.21.15.28:0/2296005822 conn(0x7f2eb80fbce0 msgr2=0x7f2eb80fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.590+0000 7f2ebe24d640 1 -- 172.21.15.28:0/2296005822 shutdown_connections 2024-05-09T08:25:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:43.590+0000 7f2ebe24d640 1 -- 172.21.15.28:0/2296005822 wait complete. 2024-05-09T08:25:43.601 INFO:tasks.cephadm:Setup complete, yielding 2024-05-09T08:25:43.602 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-09T08:25:43.631 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:25:43.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch status' 2024-05-09T08:25:43.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:43 smithi165 ceph-mon[34540]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-09T08:25:43.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:43 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2962719696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:25:43.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:43 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2138424242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:25:43.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:43 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3018806135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:25:43.884 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:45.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:45 smithi184 ceph-mon[34509]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:45.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:45 smithi028 ceph-mon[30119]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:45.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:45 smithi165 ceph-mon[34540]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.820+0000 7f89e107a700 1 -- 172.21.15.28:0/2303819487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc102380 msgr2=0x7f89dc1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.820+0000 7f89e107a700 1 --2- 172.21.15.28:0/2303819487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc102380 0x7f89dc1027a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0099c0 tx=0x7f89cc009cd0 comp rx=0 tx=0).stop 2024-05-09T08:25:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.820+0000 7f89e107a700 1 -- 172.21.15.28:0/2303819487 shutdown_connections 2024-05-09T08:25:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.820+0000 7f89e107a700 1 --2- 172.21.15.28:0/2303819487 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89dc103f20 0x7f89dc10a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.820+0000 7f89e107a700 1 --2- 172.21.15.28:0/2303819487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc102380 0x7f89dc1027a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.820+0000 7f89e107a700 1 --2- 172.21.15.28:0/2303819487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89dc103580 0x7f89dc1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.820+0000 7f89e107a700 1 -- 172.21.15.28:0/2303819487 >> 172.21.15.28:0/2303819487 conn(0x7f89dc0fd920 msgr2=0x7f89dc0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.820+0000 7f89e107a700 1 -- 172.21.15.28:0/2303819487 shutdown_connections 2024-05-09T08:25:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.820+0000 7f89e107a700 1 -- 172.21.15.28:0/2303819487 wait complete. 2024-05-09T08:25:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89e107a700 1 Processor -- start 2024-05-09T08:25:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89e107a700 1 -- start start 2024-05-09T08:25:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89e107a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89dc102380 0x7f89dc19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89e107a700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89dc103580 0x7f89dc19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89e107a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103f20 0x7f89dc1a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89e107a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc10df90 con 0x7f89dc103f20 2024-05-09T08:25:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89e107a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc10de10 con 0x7f89dc102380 2024-05-09T08:25:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89e107a700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc10e110 con 0x7f89dc103580 2024-05-09T08:25:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89db59e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103f20 0x7f89dc1a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:45.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89db59e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103f20 0x7f89dc1a1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47410/0 (socket says 172.21.15.28:47410) 2024-05-09T08:25:45.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89db59e700 1 -- 172.21.15.28:0/2572208545 learned_addr learned my addr 172.21.15.28:0/2572208545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:45.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89dad9d700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89dc102380 0x7f89dc19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:45.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89da59c700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89dc103580 0x7f89dc19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:45.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.821+0000 7f89db59e700 1 -- 172.21.15.28:0/2572208545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89dc103580 msgr2=0x7f89dc19acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:45.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89db59e700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89dc103580 0x7f89dc19acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89db59e700 1 -- 172.21.15.28:0/2572208545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89dc102380 msgr2=0x7f89dc19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89db59e700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89dc102380 0x7f89dc19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89db59e700 1 -- 172.21.15.28:0/2572208545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc1a23e0 con 0x7f89dc103f20 2024-05-09T08:25:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89db59e700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103f20 0x7f89dc1a1d40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f89d4007f10 tx=0x7f89d400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89c3fff700 1 -- 172.21.15.28:0/2572208545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4004bb0 con 0x7f89dc103f20 2024-05-09T08:25:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc1a26d0 con 0x7f89dc103f20 2024-05-09T08:25:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89c3fff700 1 -- 172.21.15.28:0/2572208545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d4004d50 con 0x7f89dc103f20 2024-05-09T08:25:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89c3fff700 1 -- 172.21.15.28:0/2572208545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4005a60 con 0x7f89dc103f20 2024-05-09T08:25:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.822+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc1a2c10 con 0x7f89dc103f20 2024-05-09T08:25:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.823+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89dc069010 con 0x7f89dc103f20 2024-05-09T08:25:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.824+0000 7f89c3fff700 1 -- 172.21.15.28:0/2572208545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f89d401a020 con 0x7f89dc103f20 2024-05-09T08:25:45.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.825+0000 7f89c3fff700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89c40a2c20 0x7f89c40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.825+0000 7f89dad9d700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89c40a2c20 0x7f89c40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.825+0000 7f89c3fff700 1 -- 172.21.15.28:0/2572208545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f89d40cf1f0 con 0x7f89dc103f20 2024-05-09T08:25:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.825+0000 7f89dad9d700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89c40a2c20 0x7f89c40a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00b580 tx=0x7f89cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:45.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:45.828+0000 7f89c3fff700 1 -- 172.21.15.28:0/2572208545 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f89d4010070 con 0x7f89dc103f20 2024-05-09T08:25:46.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.053+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f89dc075c70 con 0x7f89c40a2c20 2024-05-09T08:25:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.055+0000 7f89c3fff700 1 -- 172.21.15.28:0/2572208545 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f89dc075c70 con 0x7f89c40a2c20 2024-05-09T08:25:46.055 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-05-09T08:25:46.055 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-05-09T08:25:46.056 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89c40a2c20 msgr2=0x7f89c40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89c40a2c20 0x7f89c40a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00b580 tx=0x7f89cc005fb0 comp rx=0 tx=0).stop 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103f20 msgr2=0x7f89dc1a1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103f20 0x7f89dc1a1d40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f89d4007f10 tx=0x7f89d400d3b0 comp rx=0 tx=0).stop 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 shutdown_connections 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f89c40a2c20 0x7f89c40a50e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103f20 0x7f89dc1a1d40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89dc102380 0x7f89dc19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 --2- 172.21.15.28:0/2572208545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89dc103580 0x7f89dc19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 >> 172.21.15.28:0/2572208545 conn(0x7f89dc0fd920 msgr2=0x7f89dc074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 shutdown_connections 2024-05-09T08:25:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:46.057+0000 7f89e107a700 1 -- 172.21.15.28:0/2572208545 wait complete. 2024-05-09T08:25:46.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-09T08:25:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:47.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:47 smithi028 ceph-mon[30119]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:25:47.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:47 smithi028 ceph-mon[30119]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:47.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:47 smithi165 ceph-mon[34540]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:25:47.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:47 smithi165 ceph-mon[34540]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:47.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:47 smithi184 ceph-mon[34509]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:25:47.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:47 smithi184 ceph-mon[34509]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:48.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.919+0000 7fe802ddf700 1 -- 172.21.15.28:0/1381758662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc102480 msgr2=0x7fe7fc1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.919+0000 7fe802ddf700 1 --2- 172.21.15.28:0/1381758662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc102480 0x7fe7fc1028e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80099c0 tx=0x7fe7e8009cd0 comp rx=0 tx=0).stop 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 -- 172.21.15.28:0/1381758662 shutdown_connections 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 --2- 172.21.15.28:0/1381758662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7fc102e20 0x7fe7fc1096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 --2- 172.21.15.28:0/1381758662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7fc101280 0x7fe7fc1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 --2- 172.21.15.28:0/1381758662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc102480 0x7fe7fc1028e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 -- 172.21.15.28:0/1381758662 >> 172.21.15.28:0/1381758662 conn(0x7fe7fc0fc820 msgr2=0x7fe7fc0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 -- 172.21.15.28:0/1381758662 shutdown_connections 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 -- 172.21.15.28:0/1381758662 wait complete. 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 Processor -- start 2024-05-09T08:25:48.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 -- start start 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc101280 0x7fe7fc06b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7fc102480 0x7fe7fc19d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7fc102e20 0x7fe7fc19f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc10cec0 con 0x7fe7fc101280 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc10cd40 con 0x7fe7fc102480 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe802ddf700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc10d040 con 0x7fe7fc102e20 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe800b7b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc101280 0x7fe7fc06b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe800b7b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc101280 0x7fe7fc06b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47420/0 (socket says 172.21.15.28:47420) 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.920+0000 7fe800b7b700 1 -- 172.21.15.28:0/3044731330 learned_addr learned my addr 172.21.15.28:0/3044731330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe80137c700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7fc102e20 0x7fe7fc19f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe7fbfff700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7fc102480 0x7fe7fc19d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe800b7b700 1 -- 172.21.15.28:0/3044731330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7fc102e20 msgr2=0x7fe7fc19f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe800b7b700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7fc102e20 0x7fe7fc19f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:48.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe800b7b700 1 -- 172.21.15.28:0/3044731330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7fc102480 msgr2=0x7fe7fc19d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe800b7b700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7fc102480 0x7fe7fc19d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe800b7b700 1 -- 172.21.15.28:0/3044731330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7fc1a0220 con 0x7fe7fc101280 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe80137c700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7fc102e20 0x7fe7fc19f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe800b7b700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc101280 0x7fe7fc06b5b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000c990 tx=0x7fe7f000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe7f9ffb700 1 -- 172.21.15.28:0/3044731330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0007b00 con 0x7fe7fc101280 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe7f9ffb700 1 -- 172.21.15.28:0/3044731330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f0007ca0 con 0x7fe7fc101280 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7fc1a0510 con 0x7fe7fc101280 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe7f9ffb700 1 -- 172.21.15.28:0/3044731330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0018780 con 0x7fe7fc101280 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.921+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7fc1a0b20 con 0x7fe7fc101280 2024-05-09T08:25:48.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.922+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7fc069010 con 0x7fe7fc101280 2024-05-09T08:25:48.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.924+0000 7fe7f9ffb700 1 -- 172.21.15.28:0/3044731330 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe7f000f450 con 0x7fe7fc101280 2024-05-09T08:25:48.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.924+0000 7fe7f9ffb700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe7ec0a2ca0 0x7fe7ec0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:48.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.925+0000 7fe7fbfff700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe7ec0a2ca0 0x7fe7ec0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:48.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.925+0000 7fe7f9ffb700 1 -- 172.21.15.28:0/3044731330 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe7f00cf320 con 0x7fe7fc101280 2024-05-09T08:25:48.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.925+0000 7fe7fbfff700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe7ec0a2ca0 0x7fe7ec0a5160 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80039c0 tx=0x7fe7e8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:48.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:48.929+0000 7fe7f9ffb700 1 -- 172.21.15.28:0/3044731330 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe7f0014030 con 0x7fe7fc101280 2024-05-09T08:25:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.148+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe7fc062760 con 0x7fe7ec0a2ca0 2024-05-09T08:25:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.161+0000 7fe7f9ffb700 1 -- 172.21.15.28:0/3044731330 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7fe7fc062760 con 0x7fe7ec0a2ca0 2024-05-09T08:25:49.163 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-09T08:25:49.163 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (2m) 28s ago 2m 16.2M - 19.0.0-2515-g96805dcf d7acac7d5657 9c62fc1c03ad 2024-05-09T08:25:49.163 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi165 running (2m) 57s ago 2m 28.7M - 19.0.0-2515-g96805dcf d7acac7d5657 55ee3e387e7a 2024-05-09T08:25:49.164 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi184 running (2m) 57s ago 2m 16.7M - 19.0.0-2515-g96805dcf d7acac7d5657 16b205826f39 2024-05-09T08:25:49.164 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (2m) 28s ago 2m 15.4M - 19.0.0-2515-g96805dcf d7acac7d5657 de5d67530b59 2024-05-09T08:25:49.164 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi165 running (2m) 57s ago 2m 18.3M - 19.0.0-2515-g96805dcf d7acac7d5657 5a02348fcbf9 2024-05-09T08:25:49.164 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi184 running (2m) 57s ago 2m 18.2M - 19.0.0-2515-g96805dcf d7acac7d5657 ac05c3efc91d 2024-05-09T08:25:49.164 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (2m) 28s ago 2m 16.2M - 19.0.0-2515-g96805dcf d7acac7d5657 a6350a7cbf4a 2024-05-09T08:25:49.165 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi165 running (2m) 57s ago 2m 17.0M - 19.0.0-2515-g96805dcf d7acac7d5657 65dfbf08fba2 2024-05-09T08:25:49.165 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi184 running (2m) 57s ago 2m 17.6M - 19.0.0-2515-g96805dcf d7acac7d5657 5514586d1cef 2024-05-09T08:25:49.165 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (2m) 28s ago 2m 14.0M - 19.0.0-2515-g96805dcf d7acac7d5657 aa97bd4f2d1f 2024-05-09T08:25:49.165 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi165 running (2m) 57s ago 2m 15.0M - 19.0.0-2515-g96805dcf d7acac7d5657 b45a185d7e34 2024-05-09T08:25:49.165 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi184 running (2m) 57s ago 2m 17.4M - 19.0.0-2515-g96805dcf d7acac7d5657 e8df3acc9d5d 2024-05-09T08:25:49.165 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (9m) 28s ago 9m 517M - 19.0.0-2515-g96805dcf d7acac7d5657 d33853de27af 2024-05-09T08:25:49.165 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi165 *:8443,8765 running (7m) 57s ago 7m 454M - 19.0.0-2515-g96805dcf d7acac7d5657 7f78607ac683 2024-05-09T08:25:49.166 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi184 *:8443,8765 running (7m) 57s ago 7m 454M - 19.0.0-2515-g96805dcf d7acac7d5657 4a3ae0841ccb 2024-05-09T08:25:49.166 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (9m) 28s ago 10m 53.1M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 ba577546d188 2024-05-09T08:25:49.166 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi165 running (8m) 57s ago 8m 39.8M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 4d34e1c042f3 2024-05-09T08:25:49.166 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi184 running (8m) 57s ago 8m 40.0M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 0923518f48c7 2024-05-09T08:25:49.166 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 28s ago 7m 56.1M 4273M 19.0.0-2515-g96805dcf d7acac7d5657 9f2df7f66d8b 2024-05-09T08:25:49.167 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi165 running (7m) 57s ago 7m 58.3M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 20f69da742ed 2024-05-09T08:25:49.167 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi184 running (6m) 57s ago 6m 48.4M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 9ee669cc621e 2024-05-09T08:25:49.167 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 28s ago 6m 53.2M 4273M 19.0.0-2515-g96805dcf d7acac7d5657 7ed65ca5eaa5 2024-05-09T08:25:49.167 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi165 running (5m) 57s ago 5m 42.0M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 888c840bf9cb 2024-05-09T08:25:49.167 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi184 running (5m) 57s ago 5m 56.0M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 f71598b9e4ee 2024-05-09T08:25:49.167 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (5m) 28s ago 5m 41.9M 4273M 19.0.0-2515-g96805dcf d7acac7d5657 37711b5ccc4d 2024-05-09T08:25:49.168 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi165 running (4m) 57s ago 4m 47.0M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 b656176f90c6 2024-05-09T08:25:49.168 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi184 running (4m) 57s ago 4m 45.6M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 03adce19dc4c 2024-05-09T08:25:49.168 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (3m) 28s ago 3m 52.3M 4273M 19.0.0-2515-g96805dcf d7acac7d5657 b619ad9c4073 2024-05-09T08:25:49.168 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi165 running (3m) 57s ago 3m 54.5M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 5ddfc3655ec8 2024-05-09T08:25:49.168 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi184 running (2m) 57s ago 2m 52.4M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 81f4d492a830 2024-05-09T08:25:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe7ec0a2ca0 msgr2=0x7fe7ec0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe7ec0a2ca0 0x7fe7ec0a5160 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80039c0 tx=0x7fe7e8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:25:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc101280 msgr2=0x7fe7fc06b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc101280 0x7fe7fc06b5b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000c990 tx=0x7fe7f000cd50 comp rx=0 tx=0).stop 2024-05-09T08:25:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 shutdown_connections 2024-05-09T08:25:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe7ec0a2ca0 0x7fe7ec0a5160 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7fc102e20 0x7fe7fc19f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7fc101280 0x7fe7fc06b5b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 --2- 172.21.15.28:0/3044731330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7fc102480 0x7fe7fc19d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 >> 172.21.15.28:0/3044731330 conn(0x7fe7fc0fc820 msgr2=0x7fe7fc105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 shutdown_connections 2024-05-09T08:25:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:49.164+0000 7fe802ddf700 1 -- 172.21.15.28:0/3044731330 wait complete. 2024-05-09T08:25:49.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:49 smithi028 ceph-mon[30119]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:25:49.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:49 smithi165 ceph-mon[34540]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:25:49.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:49 smithi184 ceph-mon[34509]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:25:49.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-09T08:25:50.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:50 smithi028 ceph-mon[30119]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:50.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:50 smithi165 ceph-mon[34540]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:50.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:50 smithi184 ceph-mon[34509]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:51 smithi028 ceph-mon[30119]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-05-09T08:25:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-05-09T08:25:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-05-09T08:25:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-05-09T08:25:51.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:51 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-05-09T08:25:51.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:51 smithi165 ceph-mon[34540]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-05-09T08:25:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-05-09T08:25:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-05-09T08:25:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-05-09T08:25:51.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:51 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-05-09T08:25:51.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:51 smithi184 ceph-mon[34509]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-09T08:25:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-05-09T08:25:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-05-09T08:25:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-05-09T08:25:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-05-09T08:25:51.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:51 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-05-09T08:25:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.993+0000 7ff0caa0b700 1 -- 172.21.15.28:0/3465038323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 msgr2=0x7ff0c41027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.993+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/3465038323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 0x7ff0c41027c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40099c0 tx=0x7ff0b4009cd0 comp rx=0 tx=0).stop 2024-05-09T08:25:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.993+0000 7ff0caa0b700 1 -- 172.21.15.28:0/3465038323 shutdown_connections 2024-05-09T08:25:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.993+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/3465038323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4103f40 0x7ff0c410a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.993+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/3465038323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 0x7ff0c41027c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.993+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/3465038323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0c41035a0 0x7ff0c4103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.993+0000 7ff0caa0b700 1 -- 172.21.15.28:0/3465038323 >> 172.21.15.28:0/3465038323 conn(0x7ff0c40fd940 msgr2=0x7ff0c40ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.993+0000 7ff0caa0b700 1 -- 172.21.15.28:0/3465038323 shutdown_connections 2024-05-09T08:25:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.993+0000 7ff0caa0b700 1 -- 172.21.15.28:0/3465038323 wait complete. 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0caa0b700 1 Processor -- start 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0caa0b700 1 -- start start 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0caa0b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 0x7ff0c4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0caa0b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0c41035a0 0x7ff0c4198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0caa0b700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4103f40 0x7ff0c419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0caa0b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c410df90 con 0x7ff0c41023a0 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0caa0b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c410de10 con 0x7ff0c41035a0 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0caa0b700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c410e110 con 0x7ff0c4103f40 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 0x7ff0c4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 0x7ff0c4198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47436/0 (socket says 172.21.15.28:47436) 2024-05-09T08:25:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0c3fff700 1 -- 172.21.15.28:0/2961095331 learned_addr learned my addr 172.21.15.28:0/2961095331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0c8fa8700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4103f40 0x7ff0c419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0c3fff700 1 -- 172.21.15.28:0/2961095331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4103f40 msgr2=0x7ff0c419fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0c3fff700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4103f40 0x7ff0c419fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.994+0000 7ff0bbfff700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0c41035a0 0x7ff0c4198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0c3fff700 1 -- 172.21.15.28:0/2961095331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0c41035a0 msgr2=0x7ff0c4198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0c3fff700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0c41035a0 0x7ff0c4198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0c3fff700 1 -- 172.21.15.28:0/2961095331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c41a0170 con 0x7ff0c41023a0 2024-05-09T08:25:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0c8fa8700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4103f40 0x7ff0c419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:25:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0c3fff700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 0x7ff0c4198510 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400fab0 tx=0x7ff0b400fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0c1ffb700 1 -- 172.21.15.28:0/2961095331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4027070 con 0x7ff0c41023a0 2024-05-09T08:25:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0c1ffb700 1 -- 172.21.15.28:0/2961095331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0b4022950 con 0x7ff0c41023a0 2024-05-09T08:25:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c41a0400 con 0x7ff0c41023a0 2024-05-09T08:25:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0c1ffb700 1 -- 172.21.15.28:0/2961095331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b400fe60 con 0x7ff0c41023a0 2024-05-09T08:25:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.995+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c41a0930 con 0x7ff0c41023a0 2024-05-09T08:25:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.996+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c4069010 con 0x7ff0c41023a0 2024-05-09T08:25:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.998+0000 7ff0c1ffb700 1 -- 172.21.15.28:0/2961095331 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff0b402a070 con 0x7ff0c41023a0 2024-05-09T08:25:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.999+0000 7ff0c1ffb700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff0b00a2ca0 0x7ff0b00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.999+0000 7ff0bbfff700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff0b00a2ca0 0x7ff0b00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.999+0000 7ff0c1ffb700 1 -- 172.21.15.28:0/2961095331 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9237+0+0 (secure 0 0 0) 0x7ff0b40d00e0 con 0x7ff0c41023a0 2024-05-09T08:25:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:51.999+0000 7ff0bbfff700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff0b00a2ca0 0x7ff0b00a5160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00ba60 tx=0x7ff0ac005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.002+0000 7ff0c1ffb700 1 -- 172.21.15.28:0/2961095331 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff0b4013070 con 0x7ff0c41023a0 2024-05-09T08:25:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.219+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ff0c4062760 con 0x7ff0b00a2ca0 2024-05-09T08:25:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.225+0000 7ff0c1ffb700 1 -- 172.21.15.28:0/2961095331 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ff0c4062760 con 0x7ff0b00a2ca0 2024-05-09T08:25:52.225 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-09T08:25:52.225 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 60s ago 2m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi165=b;smithi165=e;smithi165=h;smithi165=k;smithi184=c;smithi184=f;smithi184=i;smithi184=l;count:12 2024-05-09T08:25:52.225 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 60s ago 7m smithi028=x;smithi165=y;smithi184=z;count:3 2024-05-09T08:25:52.225 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 60s ago 8m smithi028:172.21.15.28=a;smithi165:172.21.15.165=b;smithi184:172.21.15.184=c;count:3 2024-05-09T08:25:52.225 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 60s ago - 2024-05-09T08:25:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff0b00a2ca0 msgr2=0x7ff0b00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff0b00a2ca0 0x7ff0b00a5160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00ba60 tx=0x7ff0ac005d50 comp rx=0 tx=0).stop 2024-05-09T08:25:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 msgr2=0x7ff0c4198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 0x7ff0c4198510 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400fab0 tx=0x7ff0b400fb90 comp rx=0 tx=0).stop 2024-05-09T08:25:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 shutdown_connections 2024-05-09T08:25:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff0b00a2ca0 0x7ff0b00a5160 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4103f40 0x7ff0c419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c41023a0 0x7ff0c4198510 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 --2- 172.21.15.28:0/2961095331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0c41035a0 0x7ff0c4198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.227+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 >> 172.21.15.28:0/2961095331 conn(0x7ff0c40fd940 msgr2=0x7ff0c4106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.228+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 shutdown_connections 2024-05-09T08:25:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:52.228+0000 7ff0caa0b700 1 -- 172.21.15.28:0/2961095331 wait complete. 2024-05-09T08:25:52.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:52 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-05-09T08:25:52.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:52 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-05-09T08:25:52.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:52 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-05-09T08:25:52.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:52 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-05-09T08:25:52.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:52 smithi028 ceph-mon[30119]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-05-09T08:25:52.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:52 smithi028 ceph-mon[30119]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T08:25:52.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:52 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-05-09T08:25:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:52 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-05-09T08:25:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:52 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-05-09T08:25:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:52 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-05-09T08:25:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:52 smithi165 ceph-mon[34540]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-05-09T08:25:52.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:52 smithi165 ceph-mon[34540]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T08:25:52.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:52 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-05-09T08:25:52.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:52 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-05-09T08:25:52.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:52 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-05-09T08:25:52.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:52 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-05-09T08:25:52.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:52 smithi184 ceph-mon[34509]: from='mgr.14152 172.21.15.28:0/4117824698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-05-09T08:25:52.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:52 smithi184 ceph-mon[34509]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T08:25:53.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-09T08:25:53.299 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:53.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:53 smithi028 ceph-mon[30119]: pgmap v353: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:25:53.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:53 smithi028 ceph-mon[30119]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:53.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:53 smithi028 ceph-mon[30119]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T08:25:53.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:53 smithi165 ceph-mon[34540]: pgmap v353: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:25:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:53 smithi165 ceph-mon[34540]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:53 smithi165 ceph-mon[34540]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T08:25:53.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:53 smithi184 ceph-mon[34509]: pgmap v353: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:25:53.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:53 smithi184 ceph-mon[34509]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:53.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:53 smithi184 ceph-mon[34509]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T08:25:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.013+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/3158693548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 msgr2=0x7f4cc81027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.013+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/3158693548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 0x7f4cc81027c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80099c0 tx=0x7f4cb8009cd0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.013+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/3158693548 shutdown_connections 2024-05-09T08:25:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.013+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/3158693548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc8103f40 0x7f4cc810a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.013+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/3158693548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 0x7f4cc81027c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.013+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/3158693548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc81035a0 0x7f4cc8103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.013+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/3158693548 >> 172.21.15.28:0/3158693548 conn(0x7f4cc80fd940 msgr2=0x7f4cc80ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.013+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/3158693548 shutdown_connections 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.013+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/3158693548 wait complete. 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4ccd2d6700 1 Processor -- start 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4ccd2d6700 1 -- start start 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4ccd2d6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 0x7f4cc8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4ccd2d6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc81035a0 0x7f4cc8198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4ccd2d6700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc8103f40 0x7f4cc819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4ccd2d6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc810e030 con 0x7f4cc81023a0 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4ccd2d6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc810deb0 con 0x7f4cc81035a0 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4ccd2d6700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc810e1b0 con 0x7f4cc8103f40 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4cc6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 0x7f4cc8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4cc6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 0x7f4cc8198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53288/0 (socket says 172.21.15.28:53288) 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.014+0000 7f4cc6ffd700 1 -- 172.21.15.28:0/2457020983 learned_addr learned my addr 172.21.15.28:0/2457020983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4cc67fc700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc81035a0 0x7f4cc8198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4cc77fe700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc8103f40 0x7f4cc819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4cc6ffd700 1 -- 172.21.15.28:0/2457020983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc8103f40 msgr2=0x7f4cc819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4cc6ffd700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc8103f40 0x7f4cc819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4cc6ffd700 1 -- 172.21.15.28:0/2457020983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc81035a0 msgr2=0x7f4cc8198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4cc6ffd700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc81035a0 0x7f4cc8198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4cc6ffd700 1 -- 172.21.15.28:0/2457020983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc81a0280 con 0x7f4cc81023a0 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4cc6ffd700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 0x7f4cc8198620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c010 tx=0x7f4cb80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4caffff700 1 -- 172.21.15.28:0/2457020983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8027070 con 0x7f4cc81023a0 2024-05-09T08:25:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc81a0510 con 0x7f4cc81023a0 2024-05-09T08:25:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4caffff700 1 -- 172.21.15.28:0/2457020983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb8022470 con 0x7f4cc81023a0 2024-05-09T08:25:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4caffff700 1 -- 172.21.15.28:0/2457020983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb800f6a0 con 0x7f4cc81023a0 2024-05-09T08:25:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.015+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc81a0a40 con 0x7f4cc81023a0 2024-05-09T08:25:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.016+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc8069010 con 0x7f4cc81023a0 2024-05-09T08:25:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.018+0000 7f4caffff700 1 -- 172.21.15.28:0/2457020983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4cb802a070 con 0x7f4cc81023a0 2024-05-09T08:25:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.018+0000 7f4caffff700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4cb00a2c20 0x7f4cb00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.019+0000 7f4cc67fc700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4cb00a2c20 0x7f4cb00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.019+0000 7f4caffff700 1 -- 172.21.15.28:0/2457020983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f4cb80d0840 con 0x7f4cc81023a0 2024-05-09T08:25:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.019+0000 7f4cc67fc700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4cb00a2c20 0x7f4cb00a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc005fd0 tx=0x7f4cbc00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.022+0000 7f4caffff700 1 -- 172.21.15.28:0/2457020983 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4cb8013070 con 0x7f4cc81023a0 2024-05-09T08:25:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.239+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f4cc8062760 con 0x7f4cb00a2c20 2024-05-09T08:25:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.240+0000 7f4caffff700 1 -- 172.21.15.28:0/2457020983 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f4cc8062760 con 0x7f4cb00a2c20 2024-05-09T08:25:55.240 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-05-09T08:25:55.240 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-05-09T08:25:55.240 INFO:teuthology.orchestra.run.smithi028.stdout:smithi165 172.21.15.165 2024-05-09T08:25:55.240 INFO:teuthology.orchestra.run.smithi028.stdout:smithi184 172.21.15.184 2024-05-09T08:25:55.240 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-05-09T08:25:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4cb00a2c20 msgr2=0x7f4cb00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4cb00a2c20 0x7f4cb00a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc005fd0 tx=0x7f4cbc00a560 comp rx=0 tx=0).stop 2024-05-09T08:25:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 msgr2=0x7f4cc8198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 0x7f4cc8198620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c010 tx=0x7f4cb80047b0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 shutdown_connections 2024-05-09T08:25:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4cb00a2c20 0x7f4cb00a50e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cc8103f40 0x7f4cc819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc81023a0 0x7f4cc8198620 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 --2- 172.21.15.28:0/2457020983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc81035a0 0x7f4cc8198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.242+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 >> 172.21.15.28:0/2457020983 conn(0x7f4cc80fd940 msgr2=0x7f4cc81068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.243+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 shutdown_connections 2024-05-09T08:25:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:55.243+0000 7f4ccd2d6700 1 -- 172.21.15.28:0/2457020983 wait complete. 2024-05-09T08:25:55.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:55 smithi028 ceph-mon[30119]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-09T08:25:55.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:55 smithi165 ceph-mon[34540]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-09T08:25:55.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:55 smithi184 ceph-mon[34509]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-09T08:25:55.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-09T08:25:56.183 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:56.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:56 smithi028 ceph-mon[30119]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:56.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:56 smithi165 ceph-mon[34540]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:56.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:56 smithi184 ceph-mon[34509]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:57.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:57 smithi028 ceph-mon[30119]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-05-09T08:25:57.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:57 smithi165 ceph-mon[34540]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-05-09T08:25:57.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:57 smithi184 ceph-mon[34509]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-05-09T08:25:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.917+0000 7fbf0f2de700 1 -- 172.21.15.28:0/4082220446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf08102eb0 msgr2=0x7fbf081052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.917+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/4082220446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf08102eb0 0x7fbf081052a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009990 tx=0x7fbf04009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.917+0000 7fbf0f2de700 1 -- 172.21.15.28:0/4082220446 shutdown_connections 2024-05-09T08:25:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.917+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/4082220446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf08108110 0x7fbf0810a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.917+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/4082220446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf08102eb0 0x7fbf081052a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.917+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/4082220446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf081057e0 0x7fbf08107bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.917+0000 7fbf0f2de700 1 -- 172.21.15.28:0/4082220446 >> 172.21.15.28:0/4082220446 conn(0x7fbf080fc820 msgr2=0x7fbf080fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.917+0000 7fbf0f2de700 1 -- 172.21.15.28:0/4082220446 shutdown_connections 2024-05-09T08:25:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.917+0000 7fbf0f2de700 1 -- 172.21.15.28:0/4082220446 wait complete. 2024-05-09T08:25:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0f2de700 1 Processor -- start 2024-05-09T08:25:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0f2de700 1 -- start start 2024-05-09T08:25:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0f2de700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf08102eb0 0x7fbf0819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0f2de700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf081057e0 0x7fbf0819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0f2de700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08108110 0x7fbf081a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0f2de700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0810df50 con 0x7fbf081057e0 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0f2de700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0810ddd0 con 0x7fbf08108110 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0f2de700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0810e0d0 con 0x7fbf08102eb0 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0c879700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf081057e0 0x7fbf0819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0c879700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf081057e0 0x7fbf0819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53300/0 (socket says 172.21.15.28:53300) 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0c879700 1 -- 172.21.15.28:0/127396679 learned_addr learned my addr 172.21.15.28:0/127396679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0d07a700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf08102eb0 0x7fbf0819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0d87b700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08108110 0x7fbf081a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0c879700 1 -- 172.21.15.28:0/127396679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf08102eb0 msgr2=0x7fbf0819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0c879700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf08102eb0 0x7fbf0819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0c879700 1 -- 172.21.15.28:0/127396679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08108110 msgr2=0x7fbf081a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.918+0000 7fbf0c879700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08108110 0x7fbf081a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.919+0000 7fbf0c879700 1 -- 172.21.15.28:0/127396679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf081a23b0 con 0x7fbf081057e0 2024-05-09T08:25:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.919+0000 7fbf0c879700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf081057e0 0x7fbf0819ac90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c9e0 tx=0x7fbef800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.919+0000 7fbefe7fc700 1 -- 172.21.15.28:0/127396679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8007ac0 con 0x7fbf081057e0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.919+0000 7fbefe7fc700 1 -- 172.21.15.28:0/127396679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef8007c60 con 0x7fbf081057e0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.919+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf081a26a0 con 0x7fbf081057e0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.919+0000 7fbefe7fc700 1 -- 172.21.15.28:0/127396679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8018730 con 0x7fbf081057e0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.919+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf081a2c30 con 0x7fbf081057e0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.920+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf08069010 con 0x7fbf081057e0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.921+0000 7fbefe7fc700 1 -- 172.21.15.28:0/127396679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbef80188d0 con 0x7fbf081057e0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.922+0000 7fbefe7fc700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbef40a2ca0 0x7fbef40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.922+0000 7fbf0d07a700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbef40a2ca0 0x7fbef40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.922+0000 7fbefe7fc700 1 -- 172.21.15.28:0/127396679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fbef80cf670 con 0x7fbf081057e0 2024-05-09T08:25:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.922+0000 7fbf0d07a700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbef40a2ca0 0x7fbef40a5160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009750 tx=0x7fbf04003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:57.925+0000 7fbefe7fc700 1 -- 172.21.15.28:0/127396679 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbef8014030 con 0x7fbf081057e0 2024-05-09T08:25:58.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.139+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 --> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fbf08062760 con 0x7fbef40a2ca0 2024-05-09T08:25:58.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.141+0000 7fbefe7fc700 1 -- 172.21.15.28:0/127396679 <== mgr.14152 v2:172.21.15.28:6800/953899001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fbf08062760 con 0x7fbef40a2ca0 2024-05-09T08:25:58.141 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-09T08:25:58.142 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-05-09T08:25:58.142 INFO:teuthology.orchestra.run.smithi028.stdout:smithi165 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CE400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-09T08:25:58.142 INFO:teuthology.orchestra.run.smithi028.stdout:smithi184 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT626500CE400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-05-09T08:25:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbef40a2ca0 msgr2=0x7fbef40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbef40a2ca0 0x7fbef40a5160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009750 tx=0x7fbf04003960 comp rx=0 tx=0).stop 2024-05-09T08:25:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf081057e0 msgr2=0x7fbf0819ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf081057e0 0x7fbf0819ac90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c9e0 tx=0x7fbef800ccf0 comp rx=0 tx=0).stop 2024-05-09T08:25:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 shutdown_connections 2024-05-09T08:25:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fbef40a2ca0 0x7fbef40a5160 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:58.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf08102eb0 0x7fbf0819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:58.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08108110 0x7fbf081a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:58.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 --2- 172.21.15.28:0/127396679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf081057e0 0x7fbf0819ac90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:58.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 >> 172.21.15.28:0/127396679 conn(0x7fbf080fc820 msgr2=0x7fbf081025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:58.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 shutdown_connections 2024-05-09T08:25:58.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:25:58.144+0000 7fbf0f2de700 1 -- 172.21.15.28:0/127396679 wait complete. 2024-05-09T08:25:58.806 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-09T08:25:58.820 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:25:58.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-09T08:25:59.075 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:59.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:59 smithi028 ceph-mon[30119]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-05-09T08:25:59.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:25:59 smithi028 ceph-mon[30119]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:59.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:59 smithi165 ceph-mon[34540]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-05-09T08:25:59.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:25:59 smithi165 ceph-mon[34540]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:59.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:59 smithi184 ceph-mon[34509]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-05-09T08:25:59.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:25:59 smithi184 ceph-mon[34509]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.806+0000 7f65ab2f9700 1 -- 172.21.15.28:0/628799792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4105740 msgr2=0x7f65a4107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.806+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/628799792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4105740 0x7f65a4107b30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6594009990 tx=0x7f6594009ca0 comp rx=0 tx=0).stop 2024-05-09T08:26:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.806+0000 7f65ab2f9700 1 -- 172.21.15.28:0/628799792 shutdown_connections 2024-05-09T08:26:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.806+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/628799792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65a4102e10 0x7f65a4105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.806+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/628799792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65a4108070 0x7f65a410a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.806+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/628799792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4105740 0x7f65a4107b30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.806+0000 7f65ab2f9700 1 -- 172.21.15.28:0/628799792 >> 172.21.15.28:0/628799792 conn(0x7f65a40fc820 msgr2=0x7f65a40fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.806+0000 7f65ab2f9700 1 -- 172.21.15.28:0/628799792 shutdown_connections 2024-05-09T08:26:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.806+0000 7f65ab2f9700 1 -- 172.21.15.28:0/628799792 wait complete. 2024-05-09T08:26:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65ab2f9700 1 Processor -- start 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65ab2f9700 1 -- start start 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65ab2f9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65a4102e10 0x7f65a4198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65ab2f9700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65a4105740 0x7f65a4198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65ab2f9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4108070 0x7f65a419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65ab2f9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65a410dfd0 con 0x7f65a4108070 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65ab2f9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f65a410de50 con 0x7f65a4102e10 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65ab2f9700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f65a410e150 con 0x7f65a4105740 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65a9896700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4108070 0x7f65a419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65a9896700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4108070 0x7f65a419fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53312/0 (socket says 172.21.15.28:53312) 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65a9896700 1 -- 172.21.15.28:0/1925644181 learned_addr learned my addr 172.21.15.28:0/1925644181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.807+0000 7f65a9095700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65a4102e10 0x7f65a4198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f65a8894700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65a4105740 0x7f65a4198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f65a9896700 1 -- 172.21.15.28:0/1925644181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65a4105740 msgr2=0x7f65a4198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f65a9896700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65a4105740 0x7f65a4198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f65a9896700 1 -- 172.21.15.28:0/1925644181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65a4102e10 msgr2=0x7f65a4198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f65a9896700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65a4102e10 0x7f65a4198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f65a9896700 1 -- 172.21.15.28:0/1925644181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a41a0250 con 0x7f65a4108070 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f65a9896700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4108070 0x7f65a419fbb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f659c007d70 tx=0x7f659c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f659a7fc700 1 -- 172.21.15.28:0/1925644181 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c016020 con 0x7f65a4108070 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a41a0540 con 0x7f65a4108070 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f659a7fc700 1 -- 172.21.15.28:0/1925644181 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f659c00ad00 con 0x7f65a4108070 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f659a7fc700 1 -- 172.21.15.28:0/1925644181 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c011600 con 0x7f65a4108070 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.808+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a41a0b00 con 0x7f65a4108070 2024-05-09T08:26:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.809+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a4069010 con 0x7f65a4108070 2024-05-09T08:26:00.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.809+0000 7f659a7fc700 1 -- 172.21.15.28:0/1925644181 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f659c019020 con 0x7f65a4108070 2024-05-09T08:26:00.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.810+0000 7f659a7fc700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f65900a2c20 0x7f65900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:00.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.810+0000 7f65a9095700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f65900a2c20 0x7f65900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:00.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.810+0000 7f659a7fc700 1 -- 172.21.15.28:0/1925644181 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f659c0cf7f0 con 0x7f65a4108070 2024-05-09T08:26:00.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.811+0000 7f65a9095700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f65900a2c20 0x7f65900a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009a10 tx=0x7f65a0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:00.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:00.814+0000 7f659a7fc700 1 -- 172.21.15.28:0/1925644181 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f659c020020 con 0x7f65a4108070 2024-05-09T08:26:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.068+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f65a402db80 con 0x7f65a4108070 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.069+0000 7f659a7fc700 1 -- 172.21.15.28:0/1925644181 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3092 (secure 0 0 0) 0x7f659c09c300 con 0x7f65a4108070 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout:e27 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 27 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-09T08:26:01.070 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-05-09T08:23:50.918821+0000 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-05-09T08:24:19.404649+0000 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 5 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:in 0,1,2,3,4 2024-05-09T08:26:01.071 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24512,1=24499,2=24481,3=24476,4=24463} 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3,4] 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24512 members: 24481,24499,24476,24512,24463 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{0:24512} state up:active seq 3 addr [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{0:24446} state up:standby-replay seq 1 addr [v2:172.21.15.165:6834/3736338591,v1:172.21.15.165:6835/3736338591] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{1:24499} state up:active seq 11 addr [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{1:24424} state up:standby-replay seq 1 addr [v2:172.21.15.184:6832/1853687224,v1:172.21.15.184:6833/1853687224] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{2:24481} state up:active seq 13 addr [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{2:24410} state up:standby-replay seq 1 addr [v2:172.21.15.165:6832/3903056410,v1:172.21.15.165:6833/3903056410] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{3:24476} state up:active seq 12 addr [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{3:14625} state up:standby-replay seq 1 addr [v2:172.21.15.28:6840/9520094,v1:172.21.15.28:6841/9520094] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{4:24463} state up:active seq 14 addr [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{4:14613} state up:standby-replay seq 1 addr [v2:172.21.15.28:6838/612632862,v1:172.21.15.28:6839/612632862] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:14538} state up:standby seq 1 addr [v2:172.21.15.28:6834/523365778,v1:172.21.15.28:6835/523365778] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.073 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:14550} state up:standby seq 1 addr [v2:172.21.15.28:6836/3978374239,v1:172.21.15.28:6837/3978374239] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:26:01.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f65900a2c20 msgr2=0x7f65900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:01.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f65900a2c20 0x7f65900a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009a10 tx=0x7f65a0009190 comp rx=0 tx=0).stop 2024-05-09T08:26:01.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4108070 msgr2=0x7f65a419fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:01.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4108070 0x7f65a419fbb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f659c007d70 tx=0x7f659c00a340 comp rx=0 tx=0).stop 2024-05-09T08:26:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 shutdown_connections 2024-05-09T08:26:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f65900a2c20 0x7f65900a50e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a4108070 0x7f65a419fbb0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65a4102e10 0x7f65a4198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 --2- 172.21.15.28:0/1925644181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65a4105740 0x7f65a4198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 >> 172.21.15.28:0/1925644181 conn(0x7f65a40fc820 msgr2=0x7f65a41036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 shutdown_connections 2024-05-09T08:26:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:01.071+0000 7f65ab2f9700 1 -- 172.21.15.28:0/1925644181 wait complete. 2024-05-09T08:26:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-05-09T08:26:01.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:01 smithi028 ceph-mon[30119]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-05-09T08:26:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:01 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1925644181' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T08:26:01.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-09T08:26:01.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:01 smithi165 ceph-mon[34540]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-05-09T08:26:01.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:01 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1925644181' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T08:26:01.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:01 smithi184 ceph-mon[34509]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-05-09T08:26:01.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:01 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1925644181' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T08:26:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:26:03.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:03 smithi028 ceph-mon[30119]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:26:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.797+0000 7fad39c57700 1 -- 172.21.15.28:0/802667678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102480 msgr2=0x7fad341028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.797+0000 7fad39c57700 1 --2- 172.21.15.28:0/802667678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102480 0x7fad341028e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fad280099c0 tx=0x7fad28009cd0 comp rx=0 tx=0).stop 2024-05-09T08:26:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.797+0000 7fad39c57700 1 -- 172.21.15.28:0/802667678 shutdown_connections 2024-05-09T08:26:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.797+0000 7fad39c57700 1 --2- 172.21.15.28:0/802667678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad34101280 0x7fad341016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.797+0000 7fad39c57700 1 --2- 172.21.15.28:0/802667678 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad34102e20 0x7fad341096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.797+0000 7fad39c57700 1 --2- 172.21.15.28:0/802667678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102480 0x7fad341028e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.797+0000 7fad39c57700 1 -- 172.21.15.28:0/802667678 >> 172.21.15.28:0/802667678 conn(0x7fad340fc820 msgr2=0x7fad340fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.797+0000 7fad39c57700 1 -- 172.21.15.28:0/802667678 shutdown_connections 2024-05-09T08:26:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.798+0000 7fad39c57700 1 -- 172.21.15.28:0/802667678 wait complete. 2024-05-09T08:26:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.798+0000 7fad39c57700 1 Processor -- start 2024-05-09T08:26:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.798+0000 7fad39c57700 1 -- start start 2024-05-09T08:26:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.798+0000 7fad39c57700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad34101280 0x7fad3406ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad39c57700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad34102480 0x7fad3419d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad39c57700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102e20 0x7fad3419f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad39c57700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad3410ce40 con 0x7fad34102e20 2024-05-09T08:26:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad39c57700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad3410ccc0 con 0x7fad34102480 2024-05-09T08:26:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad39c57700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad3410cfc0 con 0x7fad34101280 2024-05-09T08:26:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad33fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102e20 0x7fad3419f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad33fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102e20 0x7fad3419f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53326/0 (socket says 172.21.15.28:53326) 2024-05-09T08:26:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad33fff700 1 -- 172.21.15.28:0/1348319074 learned_addr learned my addr 172.21.15.28:0/1348319074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad32ffd700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad34102480 0x7fad3419d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad337fe700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad34101280 0x7fad3406ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad33fff700 1 -- 172.21.15.28:0/1348319074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad34101280 msgr2=0x7fad3406ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad33fff700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad34101280 0x7fad3406ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad33fff700 1 -- 172.21.15.28:0/1348319074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad34102480 msgr2=0x7fad3419d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad33fff700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad34102480 0x7fad3419d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad33fff700 1 -- 172.21.15.28:0/1348319074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad341a01f0 con 0x7fad34102e20 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad337fe700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad34101280 0x7fad3406ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.799+0000 7fad33fff700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102e20 0x7fad3419f920 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fad2c007ee0 tx=0x7fad2c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.800+0000 7fad30ff9700 1 -- 172.21.15.28:0/1348319074 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2c00dbd0 con 0x7fad34102e20 2024-05-09T08:26:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.800+0000 7fad30ff9700 1 -- 172.21.15.28:0/1348319074 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad2c00dd70 con 0x7fad34102e20 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.800+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad341a04e0 con 0x7fad34102e20 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.800+0000 7fad30ff9700 1 -- 172.21.15.28:0/1348319074 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2c005a60 con 0x7fad34102e20 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.800+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad341a0a20 con 0x7fad34102e20 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.801+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad34069010 con 0x7fad34102e20 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.802+0000 7fad30ff9700 1 -- 172.21.15.28:0/1348319074 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fad2c01a020 con 0x7fad34102e20 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.803+0000 7fad30ff9700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fad1c0a2ca0 0x7fad1c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.803+0000 7fad337fe700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fad1c0a2ca0 0x7fad1c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.803+0000 7fad30ff9700 1 -- 172.21.15.28:0/1348319074 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fad2c0ceb10 con 0x7fad34102e20 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.803+0000 7fad337fe700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fad1c0a2ca0 0x7fad1c0a5160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fad341022e0 tx=0x7fad24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:03.807+0000 7fad30ff9700 1 -- 172.21.15.28:0/1348319074 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fad2c010070 con 0x7fad34102e20 2024-05-09T08:26:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:03 smithi165 ceph-mon[34540]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:26:03.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:03 smithi184 ceph-mon[34509]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-09T08:26:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.013+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fad34062f30 con 0x7fad34102e20 2024-05-09T08:26:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.014+0000 7fad30ff9700 1 -- 172.21.15.28:0/1348319074 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5733 (secure 0 0 0) 0x7fad2c09b680 con 0x7fad34102e20 2024-05-09T08:26:04.015 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 93 2024-05-09T08:26:04.015 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:26:04.015 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-05-09T08:15:46.667230+0000 2024-05-09T08:26:04.015 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-05-09T08:25:52.396720+0000 2024-05-09T08:26:04.015 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-09T08:26:04.015 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 27 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-05-09T08:26:04.016 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/1756971731,v1:172.21.15.28:6803/1756971731] [v2:172.21.15.28:6804/1756971731,v1:172.21.15.28:6805/1756971731] exists,up bb0b1568-830d-4e49-afa7-96ea4d64ce19 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6800/3539015661,v1:172.21.15.165:6801/3539015661] [v2:172.21.15.165:6802/3539015661,v1:172.21.15.165:6803/3539015661] exists,up 1a57c6ac-92fa-4d2c-9ce8-ec3906cab422 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6800/3751267842,v1:172.21.15.184:6801/3751267842] [v2:172.21.15.184:6802/3751267842,v1:172.21.15.184:6803/3751267842] exists,up e9ab7ba3-b176-4560-a3d4-80a438a4efda 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/3526420147,v1:172.21.15.28:6811/3526420147] [v2:172.21.15.28:6812/3526420147,v1:172.21.15.28:6813/3526420147] exists,up 00f5fefc-7634-4f7a-b543-8b3d86be1a5d 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6808/3527398709,v1:172.21.15.165:6809/3527398709] [v2:172.21.15.165:6810/3527398709,v1:172.21.15.165:6811/3527398709] exists,up cac8fdba-d8a4-453d-9091-121d932d6904 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 38 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6808/2066099940,v1:172.21.15.184:6809/2066099940] [v2:172.21.15.184:6810/2066099940,v1:172.21.15.184:6811/2066099940] exists,up 072f3379-6c55-40cd-9898-639846b9369f 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/899794462,v1:172.21.15.28:6819/899794462] [v2:172.21.15.28:6820/899794462,v1:172.21.15.28:6821/899794462] exists,up d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6816/1526551047,v1:172.21.15.165:6817/1526551047] [v2:172.21.15.165:6818/1526551047,v1:172.21.15.165:6819/1526551047] exists,up d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6816/2287150908,v1:172.21.15.184:6817/2287150908] [v2:172.21.15.184:6818/2287150908,v1:172.21.15.184:6819/2287150908] exists,up 12434642-450b-4e7b-a86a-4f32e76d2286 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/712618237,v1:172.21.15.28:6827/712618237] [v2:172.21.15.28:6828/712618237,v1:172.21.15.28:6829/712618237] exists,up 00768b2e-17b7-4ba7-b857-f83ea092f65d 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6824/3106810724,v1:172.21.15.165:6825/3106810724] [v2:172.21.15.165:6826/3106810724,v1:172.21.15.165:6827/3106810724] exists,up 111970a1-1719-4b83-bedf-de211b2088c7 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.184:6824/1865031750,v1:172.21.15.184:6825/1865031750] [v2:172.21.15.184:6826/1865031750,v1:172.21.15.184:6827/1865031750] exists,up c95c2ce7-e296-4064-b913-311b7b9dcbd6 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.a [2,11] 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.f [8,11] 2024-05-09T08:26:04.017 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.10 [2,4] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.14 [2,4] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1f [8,1] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.2 [3,9] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.3 [4,9] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.6 [4,6] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.1b [4,7] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.1e [3,9] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.39 [5,8] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.5 [9,10] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.8 [5,8] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.13 [9,8] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.1f [9,8] 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/2931568341 expires 2024-05-10T08:16:49.552676+0000 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3510522803 expires 2024-05-10T08:16:49.552676+0000 2024-05-09T08:26:04.018 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/934935086 expires 2024-05-10T08:16:49.552676+0000 2024-05-09T08:26:04.019 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1194745694 expires 2024-05-10T08:16:22.859488+0000 2024-05-09T08:26:04.019 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3920156841 expires 2024-05-10T08:16:22.859488+0000 2024-05-09T08:26:04.019 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/1179097296 expires 2024-05-10T08:16:22.859488+0000 2024-05-09T08:26:04.019 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/1179097296 expires 2024-05-10T08:16:22.859488+0000 2024-05-09T08:26:04.019 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/2931568341 expires 2024-05-10T08:16:49.552676+0000 2024-05-09T08:26:04.019 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/466716681 expires 2024-05-10T08:16:49.552676+0000 2024-05-09T08:26:04.019 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2416478420 expires 2024-05-10T08:16:22.859488+0000 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fad1c0a2ca0 msgr2=0x7fad1c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fad1c0a2ca0 0x7fad1c0a5160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fad341022e0 tx=0x7fad24005f90 comp rx=0 tx=0).stop 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102e20 msgr2=0x7fad3419f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102e20 0x7fad3419f920 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fad2c007ee0 tx=0x7fad2c00d3b0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 shutdown_connections 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fad1c0a2ca0 0x7fad1c0a5160 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad34101280 0x7fad3406ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad34102e20 0x7fad3419f920 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 --2- 172.21.15.28:0/1348319074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad34102480 0x7fad3419d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.017+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 >> 172.21.15.28:0/1348319074 conn(0x7fad340fc820 msgr2=0x7fad34105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.018+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 shutdown_connections 2024-05-09T08:26:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.018+0000 7fad39c57700 1 -- 172.21.15.28:0/1348319074 wait complete. 2024-05-09T08:26:04.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1348319074' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T08:26:04.677 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-09T08:26:04.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:26:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.800+0000 7fcaf2b89640 1 -- 172.21.15.28:0/453715951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec100790 msgr2=0x7fcaec100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.800+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/453715951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec100790 0x7fcaec100bb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009ca0 tx=0x7fcae802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.800+0000 7fcaf2b89640 1 -- 172.21.15.28:0/453715951 shutdown_connections 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.800+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/453715951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaec102340 0x7fcaec108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/453715951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcaec101980 0x7fcaec101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/453715951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec100790 0x7fcaec100bb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 -- 172.21.15.28:0/453715951 >> 172.21.15.28:0/453715951 conn(0x7fcaec0fbd20 msgr2=0x7fcaec0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 -- 172.21.15.28:0/453715951 shutdown_connections 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 -- 172.21.15.28:0/453715951 wait complete. 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 Processor -- start 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 -- start start 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaec100790 0x7fcaec06aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec101980 0x7fcaec19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcaec102340 0x7fcaec1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec10c410 con 0x7fcaec101980 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec10c290 con 0x7fcaec102340 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcaf2b89640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec10c590 con 0x7fcaec100790 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcae3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec101980 0x7fcaec19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcae3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec101980 0x7fcaec19ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58472/0 (socket says 172.21.15.28:58472) 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.801+0000 7fcae3fff640 1 -- 172.21.15.28:0/1732480245 learned_addr learned my addr 172.21.15.28:0/1732480245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcaf1126640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcaec102340 0x7fcaec1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcaf0925640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaec100790 0x7fcaec06aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcae3fff640 1 -- 172.21.15.28:0/1732480245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaec100790 msgr2=0x7fcaec06aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcae3fff640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaec100790 0x7fcaec06aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcae3fff640 1 -- 172.21.15.28:0/1732480245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcaec102340 msgr2=0x7fcaec1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcae3fff640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcaec102340 0x7fcaec1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcae3fff640 1 -- 172.21.15.28:0/1732480245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaec1a1a50 con 0x7fcaec101980 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcae3fff640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec101980 0x7fcaec19ed10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00d950 tx=0x7fcadc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcae1ffb640 1 -- 172.21.15.28:0/1732480245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc017070 con 0x7fcaec101980 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcae1ffb640 1 -- 172.21.15.28:0/1732480245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcadc009c30 con 0x7fcaec101980 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaec1a1ce0 con 0x7fcaec101980 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcae1ffb640 1 -- 172.21.15.28:0/1732480245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc00f6b0 con 0x7fcaec101980 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.802+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaec1a21d0 con 0x7fcaec101980 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.803+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaec107ad0 con 0x7fcaec101980 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.804+0000 7fcae1ffb640 1 -- 172.21.15.28:0/1732480245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcadc0109d0 con 0x7fcaec101980 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.805+0000 7fcae1ffb640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcad40a2bc0 0x7fcad40a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.805+0000 7fcae1ffb640 1 -- 172.21.15.28:0/1732480245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fcadc0cf100 con 0x7fcaec101980 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.805+0000 7fcaf0925640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcad40a2bc0 0x7fcad40a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.806+0000 7fcaf0925640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcad40a2bc0 0x7fcad40a5080 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006010 tx=0x7fcae8009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.810+0000 7fcae1ffb640 1 -- 172.21.15.28:0/1732480245 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcadc009de0 con 0x7fcaec101980 2024-05-09T08:26:04.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1348319074' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T08:26:04.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1348319074' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T08:26:04.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.995+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcaec062430 con 0x7fcaec101980 2024-05-09T08:26:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.997+0000 7fcae1ffb640 1 -- 172.21.15.28:0/1732480245 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24899 (secure 0 0 0) 0x7fcadc0bb070 con 0x7fcaec101980 2024-05-09T08:26:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:04.998 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:25:52.396720+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcad40a2bc0 msgr2=0x7fcad40a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcad40a2bc0 0x7fcad40a5080 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006010 tx=0x7fcae8009360 comp rx=0 tx=0).stop 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec101980 msgr2=0x7fcaec19ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec101980 0x7fcaec19ed10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00d950 tx=0x7fcadc00de10 comp rx=0 tx=0).stop 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 shutdown_connections 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcad40a2bc0 0x7fcad40a5080 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcaec102340 0x7fcaec1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaec101980 0x7fcaec19ed10 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 --2- 172.21.15.28:0/1732480245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaec100790 0x7fcaec06aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 >> 172.21.15.28:0/1732480245 conn(0x7fcaec0fbd20 msgr2=0x7fcaec104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 shutdown_connections 2024-05-09T08:26:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:04.999+0000 7fcaf2b89640 1 -- 172.21.15.28:0/1732480245 wait complete. 2024-05-09T08:26:05.012 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:26:05.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:26:05.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.120+0000 7ff54da6d640 1 -- 172.21.15.28:0/609714903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548101980 msgr2=0x7ff548101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.120+0000 7ff54da6d640 1 --2- 172.21.15.28:0/609714903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548101980 0x7ff548101e00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff53c009900 tx=0x7ff53c02a410 comp rx=0 tx=0).stop 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.120+0000 7ff54da6d640 1 -- 172.21.15.28:0/609714903 shutdown_connections 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.120+0000 7ff54da6d640 1 --2- 172.21.15.28:0/609714903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff548102340 0x7ff548108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.120+0000 7ff54da6d640 1 --2- 172.21.15.28:0/609714903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548101980 0x7ff548101e00 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.120+0000 7ff54da6d640 1 --2- 172.21.15.28:0/609714903 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff548100790 0x7ff548100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.120+0000 7ff54da6d640 1 -- 172.21.15.28:0/609714903 >> 172.21.15.28:0/609714903 conn(0x7ff5480fbd20 msgr2=0x7ff5480fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.120+0000 7ff54da6d640 1 -- 172.21.15.28:0/609714903 shutdown_connections 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.120+0000 7ff54da6d640 1 -- 172.21.15.28:0/609714903 wait complete. 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.121+0000 7ff54da6d640 1 Processor -- start 2024-05-09T08:26:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.121+0000 7ff54da6d640 1 -- start start 2024-05-09T08:26:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.121+0000 7ff54da6d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff548100790 0x7ff54806aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.121+0000 7ff54da6d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff548101980 0x7ff54819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.121+0000 7ff54da6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548102340 0x7ff54819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.121+0000 7ff54da6d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff54810c410 con 0x7ff548102340 2024-05-09T08:26:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.121+0000 7ff54da6d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff54810c290 con 0x7ff548100790 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.121+0000 7ff54da6d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff54810c590 con 0x7ff548101980 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.121+0000 7ff547fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548102340 0x7ff54819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff547fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548102340 0x7ff54819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58496/0 (socket says 172.21.15.28:58496) 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff547fff640 1 -- 172.21.15.28:0/2040972381 learned_addr learned my addr 172.21.15.28:0/2040972381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff546ffd640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff548101980 0x7ff54819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff5477fe640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff548100790 0x7ff54806aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff547fff640 1 -- 172.21.15.28:0/2040972381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff548101980 msgr2=0x7ff54819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff547fff640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff548101980 0x7ff54819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff547fff640 1 -- 172.21.15.28:0/2040972381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff548100790 msgr2=0x7ff54806aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff547fff640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff548100790 0x7ff54806aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff547fff640 1 -- 172.21.15.28:0/2040972381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54819f820 con 0x7ff548102340 2024-05-09T08:26:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff5477fe640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff548100790 0x7ff54806aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff547fff640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548102340 0x7ff54819ef60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff54000eaa0 tx=0x7ff54000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff544ff9640 1 -- 172.21.15.28:0/2040972381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54000cdb0 con 0x7ff548102340 2024-05-09T08:26:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff544ff9640 1 -- 172.21.15.28:0/2040972381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff540005020 con 0x7ff548102340 2024-05-09T08:26:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff544ff9640 1 -- 172.21.15.28:0/2040972381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5400116b0 con 0x7ff548102340 2024-05-09T08:26:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff54819fb10 con 0x7ff548102340 2024-05-09T08:26:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.122+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5481a0030 con 0x7ff548102340 2024-05-09T08:26:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.123+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff548107b70 con 0x7ff548102340 2024-05-09T08:26:05.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.124+0000 7ff544ff9640 1 -- 172.21.15.28:0/2040972381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff540005560 con 0x7ff548102340 2024-05-09T08:26:05.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.125+0000 7ff544ff9640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff5300a2c10 0x7ff5300a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.125+0000 7ff544ff9640 1 -- 172.21.15.28:0/2040972381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7ff5400cf6d0 con 0x7ff548102340 2024-05-09T08:26:05.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.125+0000 7ff5477fe640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff5300a2c10 0x7ff5300a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.126+0000 7ff5477fe640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff5300a2c10 0x7ff5300a50d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff5481017e0 tx=0x7ff538005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:05.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.130+0000 7ff544ff9640 1 -- 172.21.15.28:0/2040972381 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff540016030 con 0x7ff548102340 2024-05-09T08:26:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.316+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff5480624d0 con 0x7ff548102340 2024-05-09T08:26:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.317+0000 7ff544ff9640 1 -- 172.21.15.28:0/2040972381 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7ff54009c240 con 0x7ff548102340 2024-05-09T08:26:05.317 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:26:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff5300a2c10 msgr2=0x7ff5300a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff5300a2c10 0x7ff5300a50d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff5481017e0 tx=0x7ff538005e70 comp rx=0 tx=0).stop 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548102340 msgr2=0x7ff54819ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548102340 0x7ff54819ef60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff54000eaa0 tx=0x7ff54000ef60 comp rx=0 tx=0).stop 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 shutdown_connections 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7ff5300a2c10 0x7ff5300a50d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff548102340 0x7ff54819ef60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff548101980 0x7ff54819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 --2- 172.21.15.28:0/2040972381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff548100790 0x7ff54806aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 >> 172.21.15.28:0/2040972381 conn(0x7ff5480fbd20 msgr2=0x7ff548104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 shutdown_connections 2024-05-09T08:26:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.319+0000 7ff54da6d640 1 -- 172.21.15.28:0/2040972381 wait complete. 2024-05-09T08:26:05.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4155515271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c104c80 msgr2=0x7f499c105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4155515271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c104c80 0x7f499c105100 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4984009900 tx=0x7f498402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4155515271 shutdown_connections 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4155515271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499c105640 0x7f499c10bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4155515271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c104c80 0x7f499c105100 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4155515271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f499c103a90 0x7f499c103eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4155515271 >> 172.21.15.28:0/4155515271 conn(0x7f499c0ff020 msgr2=0x7f499c101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4155515271 shutdown_connections 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4155515271 wait complete. 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 Processor -- start 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.439+0000 7f49a1fc8640 1 -- start start 2024-05-09T08:26:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49a1fc8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499c103a90 0x7f499c199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49a1fc8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f499c104c80 0x7f499c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49a1fc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c105640 0x7f499c1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49a1fc8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f499c10f710 con 0x7f499c105640 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49a1fc8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f499c10f590 con 0x7f499c103a90 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49a1fc8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f499c10f890 con 0x7f499c104c80 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c105640 0x7f499c1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c105640 0x7f499c1a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58512/0 (socket says 172.21.15.28:58512) 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499bfff640 1 -- 172.21.15.28:0/4216032584 learned_addr learned my addr 172.21.15.28:0/4216032584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f4992ffd640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f499c104c80 0x7f499c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499b7fe640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499c103a90 0x7f499c199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499bfff640 1 -- 172.21.15.28:0/4216032584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f499c104c80 msgr2=0x7f499c19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499bfff640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f499c104c80 0x7f499c19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499bfff640 1 -- 172.21.15.28:0/4216032584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499c103a90 msgr2=0x7f499c199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499bfff640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499c103a90 0x7f499c199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499bfff640 1 -- 172.21.15.28:0/4216032584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499c1a19b0 con 0x7f499c105640 2024-05-09T08:26:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499b7fe640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499c103a90 0x7f499c199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:05.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f499bfff640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c105640 0x7f499c1a1310 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f499400cce0 tx=0x7f499400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:05.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49997fa640 1 -- 172.21.15.28:0/4216032584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499400f450 con 0x7f499c105640 2024-05-09T08:26:05.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49997fa640 1 -- 172.21.15.28:0/4216032584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f499400fad0 con 0x7f499c105640 2024-05-09T08:26:05.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49997fa640 1 -- 172.21.15.28:0/4216032584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4994018c10 con 0x7f499c105640 2024-05-09T08:26:05.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.440+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499c1a1ca0 con 0x7f499c105640 2024-05-09T08:26:05.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.441+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f499c1a21a0 con 0x7f499c105640 2024-05-09T08:26:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.442+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f499c10adb0 con 0x7f499c105640 2024-05-09T08:26:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.443+0000 7f49997fa640 1 -- 172.21.15.28:0/4216032584 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f499400f5f0 con 0x7f499c105640 2024-05-09T08:26:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.443+0000 7f49997fa640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49880a2b90 0x7f49880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.443+0000 7f499b7fe640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49880a2b90 0x7f49880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.444+0000 7f49997fa640 1 -- 172.21.15.28:0/4216032584 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f49940cfc30 con 0x7f499c105640 2024-05-09T08:26:05.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.444+0000 7f499b7fe640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49880a2b90 0x7f49880a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f498c0058c0 tx=0x7f498c005d70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.448+0000 7f49997fa640 1 -- 172.21.15.28:0/4216032584 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4994014030 con 0x7f499c105640 2024-05-09T08:26:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.631+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f499c0624d0 con 0x7f499c105640 2024-05-09T08:26:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.631+0000 7f49997fa640 1 -- 172.21.15.28:0/4216032584 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f499409c7a0 con 0x7f499c105640 2024-05-09T08:26:05.632 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49880a2b90 msgr2=0x7f49880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49880a2b90 0x7f49880a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f498c0058c0 tx=0x7f498c005d70 comp rx=0 tx=0).stop 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c105640 msgr2=0x7f499c1a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c105640 0x7f499c1a1310 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f499400cce0 tx=0x7f499400a620 comp rx=0 tx=0).stop 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 shutdown_connections 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49880a2b90 0x7f49880a5050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499c105640 0x7f499c1a1310 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f499c104c80 0x7f499c19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 --2- 172.21.15.28:0/4216032584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499c103a90 0x7f499c199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 >> 172.21.15.28:0/4216032584 conn(0x7f499c0ff020 msgr2=0x7f499c107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 shutdown_connections 2024-05-09T08:26:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.633+0000 7f49a1fc8640 1 -- 172.21.15.28:0/4216032584 wait complete. 2024-05-09T08:26:05.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:05 smithi028 ceph-mon[30119]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-05-09T08:26:05.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:05 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1732480245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:05.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:05 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2040972381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:05.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:26:05.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.754+0000 7f4b4b948640 1 -- 172.21.15.28:0/3659171554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44101890 msgr2=0x7f4b44101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.754+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3659171554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44101890 0x7f4b44101cb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4b34009ca0 tx=0x7f4b3402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.754+0000 7f4b4b948640 1 -- 172.21.15.28:0/3659171554 shutdown_connections 2024-05-09T08:26:05.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.754+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3659171554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b44103440 0x7f4b44109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.754+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3659171554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b44102a80 0x7f4b44102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.754+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3659171554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44101890 0x7f4b44101cb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.754+0000 7f4b4b948640 1 -- 172.21.15.28:0/3659171554 >> 172.21.15.28:0/3659171554 conn(0x7f4b440fce00 msgr2=0x7f4b440ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.754+0000 7f4b4b948640 1 -- 172.21.15.28:0/3659171554 shutdown_connections 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.754+0000 7f4b4b948640 1 -- 172.21.15.28:0/3659171554 wait complete. 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b4b948640 1 Processor -- start 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b4b948640 1 -- start start 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b4b948640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b44101890 0x7f4b44197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b4b948640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b44102a80 0x7f4b441980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b4b948640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44103440 0x7f4b4419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b4b948640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4410d500 con 0x7f4b44103440 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b4b948640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4410d380 con 0x7f4b44101890 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b4b948640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4410d680 con 0x7f4b44102a80 2024-05-09T08:26:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b49ee5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44103440 0x7f4b4419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b49ee5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44103440 0x7f4b4419f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58538/0 (socket says 172.21.15.28:58538) 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b49ee5640 1 -- 172.21.15.28:0/3780117680 learned_addr learned my addr 172.21.15.28:0/3780117680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b496e4640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b44101890 0x7f4b44197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b48ee3640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b44102a80 0x7f4b441980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b49ee5640 1 -- 172.21.15.28:0/3780117680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b44102a80 msgr2=0x7f4b441980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b49ee5640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b44102a80 0x7f4b441980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b49ee5640 1 -- 172.21.15.28:0/3780117680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b44101890 msgr2=0x7f4b44197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b49ee5640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b44101890 0x7f4b44197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b49ee5640 1 -- 172.21.15.28:0/3780117680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4419f7d0 con 0x7f4b44103440 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.755+0000 7f4b48ee3640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b44102a80 0x7f4b441980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.756+0000 7f4b49ee5640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44103440 0x7f4b4419f160 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00e970 tx=0x7f4b3c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.756+0000 7f4b427fc640 1 -- 172.21.15.28:0/3780117680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c00ccc0 con 0x7f4b44103440 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.756+0000 7f4b427fc640 1 -- 172.21.15.28:0/3780117680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b3c00ce60 con 0x7f4b44103440 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.756+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4419fac0 con 0x7f4b44103440 2024-05-09T08:26:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.756+0000 7f4b427fc640 1 -- 172.21.15.28:0/3780117680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c010490 con 0x7f4b44103440 2024-05-09T08:26:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.756+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b441a0010 con 0x7f4b44103440 2024-05-09T08:26:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.757+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b44108c30 con 0x7f4b44103440 2024-05-09T08:26:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.758+0000 7f4b427fc640 1 -- 172.21.15.28:0/3780117680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4b3c005000 con 0x7f4b44103440 2024-05-09T08:26:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.759+0000 7f4b427fc640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b2c0a2c10 0x7f4b2c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.759+0000 7f4b496e4640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b2c0a2c10 0x7f4b2c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.759+0000 7f4b427fc640 1 -- 172.21.15.28:0/3780117680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f4b3c0cfae0 con 0x7f4b44103440 2024-05-09T08:26:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.759+0000 7f4b496e4640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b2c0a2c10 0x7f4b2c0a50d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4b34006010 tx=0x7f4b34000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.762+0000 7f4b427fc640 1 -- 172.21.15.28:0/3780117680 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4b3c053740 con 0x7f4b44103440 2024-05-09T08:26:05.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:05 smithi165 ceph-mon[34540]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-05-09T08:26:05.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:05 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1732480245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:05.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:05 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2040972381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:05.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:05 smithi184 ceph-mon[34509]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-05-09T08:26:05.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:05 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1732480245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:05.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:05 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2040972381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:05.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.948+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4b4402d410 con 0x7f4b44103440 2024-05-09T08:26:05.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.949+0000 7f4b427fc640 1 -- 172.21.15.28:0/3780117680 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f4b3c09c650 con 0x7f4b44103440 2024-05-09T08:26:05.950 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:26:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b2c0a2c10 msgr2=0x7f4b2c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b2c0a2c10 0x7f4b2c0a50d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4b34006010 tx=0x7f4b34000f40 comp rx=0 tx=0).stop 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44103440 msgr2=0x7f4b4419f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44103440 0x7f4b4419f160 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00e970 tx=0x7f4b3c00ed70 comp rx=0 tx=0).stop 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 shutdown_connections 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4b2c0a2c10 0x7f4b2c0a50d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44103440 0x7f4b4419f160 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b44102a80 0x7f4b441980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 --2- 172.21.15.28:0/3780117680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b44101890 0x7f4b44197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 >> 172.21.15.28:0/3780117680 conn(0x7f4b440fce00 msgr2=0x7f4b44105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 shutdown_connections 2024-05-09T08:26:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:05.951+0000 7f4b4b948640 1 -- 172.21.15.28:0/3780117680 wait complete. 2024-05-09T08:26:05.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:26:06.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.070+0000 7f6dc34e1640 1 -- 172.21.15.28:0/2047927628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 msgr2=0x7f6dbc104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.070+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/2047927628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 0x7f6dbc104700 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009900 tx=0x7f6db802a410 comp rx=0 tx=0).stop 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.070+0000 7f6dc34e1640 1 -- 172.21.15.28:0/2047927628 shutdown_connections 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.070+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/2047927628 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dbc107570 0x7f6dbc1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.070+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/2047927628 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dbc104c40 0x7f6dbc107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.070+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/2047927628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 0x7f6dbc104700 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.070+0000 7f6dc34e1640 1 -- 172.21.15.28:0/2047927628 >> 172.21.15.28:0/2047927628 conn(0x7f6dbc0fbd20 msgr2=0x7f6dbc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.070+0000 7f6dc34e1640 1 -- 172.21.15.28:0/2047927628 shutdown_connections 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.070+0000 7f6dc34e1640 1 -- 172.21.15.28:0/2047927628 wait complete. 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc34e1640 1 Processor -- start 2024-05-09T08:26:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc34e1640 1 -- start start 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc34e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 0x7f6dbc197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc34e1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dbc104c40 0x7f6dbc198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc34e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dbc107570 0x7f6dbc19f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc34e1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc10d540 con 0x7f6dbc102310 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc34e1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc10d3c0 con 0x7f6dbc104c40 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc34e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc10d6c0 con 0x7f6dbc107570 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc127d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 0x7f6dbc197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc127d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 0x7f6dbc197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58548/0 (socket says 172.21.15.28:58548) 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc127d640 1 -- 172.21.15.28:0/129377116 learned_addr learned my addr 172.21.15.28:0/129377116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.071+0000 7f6dc0a7c640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dbc104c40 0x7f6dbc198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc127d640 1 -- 172.21.15.28:0/129377116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dbc107570 msgr2=0x7f6dbc19f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc1a7e640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dbc107570 0x7f6dbc19f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc127d640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dbc107570 0x7f6dbc19f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc127d640 1 -- 172.21.15.28:0/129377116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dbc104c40 msgr2=0x7f6dbc198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc127d640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dbc104c40 0x7f6dbc198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc127d640 1 -- 172.21.15.28:0/129377116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dbc19f7e0 con 0x7f6dbc102310 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc0a7c640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dbc104c40 0x7f6dbc198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc1a7e640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dbc107570 0x7f6dbc19f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc127d640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 0x7f6dbc197bc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009230 tx=0x7f6db8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6db27fc640 1 -- 172.21.15.28:0/129377116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db803f030 con 0x7f6dbc102310 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6db27fc640 1 -- 172.21.15.28:0/129377116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6db8003ce0 con 0x7f6dbc102310 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6db27fc640 1 -- 172.21.15.28:0/129377116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db8035070 con 0x7f6dbc102310 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dbc19fa70 con 0x7f6dbc102310 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.072+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dbc19ff60 con 0x7f6dbc102310 2024-05-09T08:26:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.073+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dbc068ad0 con 0x7f6dbc102310 2024-05-09T08:26:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.074+0000 7f6db27fc640 1 -- 172.21.15.28:0/129377116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6db8042030 con 0x7f6dbc102310 2024-05-09T08:26:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.075+0000 7f6db27fc640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6da80a2b90 0x7f6da80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.075+0000 7f6dc0a7c640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6da80a2b90 0x7f6da80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.075+0000 7f6db27fc640 1 -- 172.21.15.28:0/129377116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f6db80efc40 con 0x7f6dbc102310 2024-05-09T08:26:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.075+0000 7f6dc0a7c640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6da80a2b90 0x7f6da80a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc1990d0 tx=0x7f6dac005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:06.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.079+0000 7f6db27fc640 1 -- 172.21.15.28:0/129377116 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6dbc068ad0 con 0x7f6dbc102310 2024-05-09T08:26:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.266+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6dbc062460 con 0x7f6dbc102310 2024-05-09T08:26:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.266+0000 7f6db27fc640 1 -- 172.21.15.28:0/129377116 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f6db80bc7b0 con 0x7f6dbc102310 2024-05-09T08:26:06.267 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6da80a2b90 msgr2=0x7f6da80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6da80a2b90 0x7f6da80a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc1990d0 tx=0x7f6dac005d30 comp rx=0 tx=0).stop 2024-05-09T08:26:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 msgr2=0x7f6dbc197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 0x7f6dbc197bc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009230 tx=0x7f6db8004430 comp rx=0 tx=0).stop 2024-05-09T08:26:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 shutdown_connections 2024-05-09T08:26:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f6da80a2b90 0x7f6da80a5050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dbc107570 0x7f6dbc19f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dbc104c40 0x7f6dbc198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 --2- 172.21.15.28:0/129377116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc102310 0x7f6dbc197bc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 >> 172.21.15.28:0/129377116 conn(0x7f6dbc0fbd20 msgr2=0x7f6dbc102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.268+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 shutdown_connections 2024-05-09T08:26:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.269+0000 7f6dc34e1640 1 -- 172.21.15.28:0/129377116 wait complete. 2024-05-09T08:26:06.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:26:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 -- 172.21.15.28:0/1762656341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 msgr2=0x7f9524074670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 --2- 172.21.15.28:0/1762656341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 0x7f9524074670 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9518009900 tx=0x7f951802a480 comp rx=0 tx=0).stop 2024-05-09T08:26:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 -- 172.21.15.28:0/1762656341 shutdown_connections 2024-05-09T08:26:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 --2- 172.21.15.28:0/1762656341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9524109750 0x7f952410bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 --2- 172.21.15.28:0/1762656341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 0x7f9524074670 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 --2- 172.21.15.28:0/1762656341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95240757f0 0x7f9524073c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 -- 172.21.15.28:0/1762656341 >> 172.21.15.28:0/1762656341 conn(0x7f95240fce30 msgr2=0x7f95240ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 -- 172.21.15.28:0/1762656341 shutdown_connections 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 -- 172.21.15.28:0/1762656341 wait complete. 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.385+0000 7f9529da2640 1 Processor -- start 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9529da2640 1 -- start start 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9529da2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 0x7f9524199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9529da2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95240757f0 0x7f952419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9529da2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9524109750 0x7f95241a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9529da2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f952410f720 con 0x7f9524074210 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9529da2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f952410f5a0 con 0x7f9524109750 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9529da2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f952410f8a0 con 0x7f95240757f0 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f95237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 0x7f9524199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f95237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 0x7f9524199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58564/0 (socket says 172.21.15.28:58564) 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f95237fe640 1 -- 172.21.15.28:0/2515103010 learned_addr learned my addr 172.21.15.28:0/2515103010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9522ffd640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95240757f0 0x7f952419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9523fff640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9524109750 0x7f95241a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f95237fe640 1 -- 172.21.15.28:0/2515103010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95240757f0 msgr2=0x7f952419a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f95237fe640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95240757f0 0x7f952419a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f95237fe640 1 -- 172.21.15.28:0/2515103010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9524109750 msgr2=0x7f95241a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f95237fe640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9524109750 0x7f95241a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f95237fe640 1 -- 172.21.15.28:0/2515103010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95241a19e0 con 0x7f9524074210 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9522ffd640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95240757f0 0x7f952419a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f9523fff640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9524109750 0x7f95241a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.386+0000 7f95237fe640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 0x7f9524199dc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f951400d770 tx=0x7f951400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.387+0000 7f9520ff9640 1 -- 172.21.15.28:0/2515103010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514017070 con 0x7f9524074210 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.387+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95241a1cd0 con 0x7f9524074210 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.387+0000 7f9520ff9640 1 -- 172.21.15.28:0/2515103010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9514005500 con 0x7f9524074210 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.387+0000 7f9520ff9640 1 -- 172.21.15.28:0/2515103010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514004e00 con 0x7f9524074210 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.387+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95241a21a0 con 0x7f9524074210 2024-05-09T08:26:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.388+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9524108510 con 0x7f9524074210 2024-05-09T08:26:06.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.389+0000 7f9520ff9640 1 -- 172.21.15.28:0/2515103010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9514004fa0 con 0x7f9524074210 2024-05-09T08:26:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.389+0000 7f9520ff9640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f950c0a2b90 0x7f950c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.390+0000 7f9522ffd640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f950c0a2b90 0x7f950c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.390+0000 7f9520ff9640 1 -- 172.21.15.28:0/2515103010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f95140cef60 con 0x7f9524074210 2024-05-09T08:26:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.390+0000 7f9522ffd640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f950c0a2b90 0x7f950c0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9518000c00 tx=0x7f95180095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:06.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.394+0000 7f9520ff9640 1 -- 172.21.15.28:0/2515103010 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f951409bad0 con 0x7f9524074210 2024-05-09T08:26:06.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.621+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9524062460 con 0x7f9524074210 2024-05-09T08:26:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.622+0000 7f9520ff9640 1 -- 172.21.15.28:0/2515103010 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f951409b3d0 con 0x7f9524074210 2024-05-09T08:26:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:24:19.404649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24481,24499,24476,24512,24463]},"id":1}]} 2024-05-09T08:26:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-05-09T08:26:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f950c0a2b90 msgr2=0x7f950c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f950c0a2b90 0x7f950c0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9518000c00 tx=0x7f95180095f0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 msgr2=0x7f9524199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 0x7f9524199dc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f951400d770 tx=0x7f951400dc30 comp rx=0 tx=0).stop 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 shutdown_connections 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f950c0a2b90 0x7f950c0a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9524109750 0x7f95241a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95240757f0 0x7f952419a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 --2- 172.21.15.28:0/2515103010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9524074210 0x7f9524199dc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 >> 172.21.15.28:0/2515103010 conn(0x7f95240fce30 msgr2=0x7f95241056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 shutdown_connections 2024-05-09T08:26:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.624+0000 7f9529da2640 1 -- 172.21.15.28:0/2515103010 wait complete. 2024-05-09T08:26:06.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4216032584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:06.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3780117680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:06.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/129377116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:06.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:26:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.742+0000 7f2fb889a640 1 -- 172.21.15.28:0/4293305949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb41018a0 msgr2=0x7f2fb4101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.742+0000 7f2fb889a640 1 --2- 172.21.15.28:0/4293305949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb41018a0 0x7f2fb4101cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4009900 tx=0x7f2fa402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.742+0000 7f2fb889a640 1 -- 172.21.15.28:0/4293305949 shutdown_connections 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.742+0000 7f2fb889a640 1 --2- 172.21.15.28:0/4293305949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fb4103450 0x7f2fb4109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.742+0000 7f2fb889a640 1 --2- 172.21.15.28:0/4293305949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb4102a90 0x7f2fb4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.742+0000 7f2fb889a640 1 --2- 172.21.15.28:0/4293305949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb41018a0 0x7f2fb4101cc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.742+0000 7f2fb889a640 1 -- 172.21.15.28:0/4293305949 >> 172.21.15.28:0/4293305949 conn(0x7f2fb40fce30 msgr2=0x7f2fb40ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.742+0000 7f2fb889a640 1 -- 172.21.15.28:0/4293305949 shutdown_connections 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.743+0000 7f2fb889a640 1 -- 172.21.15.28:0/4293305949 wait complete. 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.743+0000 7f2fb889a640 1 Processor -- start 2024-05-09T08:26:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.743+0000 7f2fb889a640 1 -- start start 2024-05-09T08:26:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.743+0000 7f2fb889a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb41018a0 0x7f2fb40755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.743+0000 7f2fb889a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fb4102a90 0x7f2fb4073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.743+0000 7f2fb889a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb4103450 0x7f2fb4074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb889a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb410d560 con 0x7f2fb4103450 2024-05-09T08:26:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb889a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb410d3e0 con 0x7f2fb4102a90 2024-05-09T08:26:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb889a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb410d6e0 con 0x7f2fb41018a0 2024-05-09T08:26:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb4103450 0x7f2fb4074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb4103450 0x7f2fb4074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58568/0 (socket says 172.21.15.28:58568) 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb2d9d640 1 -- 172.21.15.28:0/3407542526 learned_addr learned my addr 172.21.15.28:0/3407542526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb1d9b640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fb4102a90 0x7f2fb4073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb259c640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb41018a0 0x7f2fb40755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb2d9d640 1 -- 172.21.15.28:0/3407542526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb41018a0 msgr2=0x7f2fb40755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb2d9d640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb41018a0 0x7f2fb40755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb2d9d640 1 -- 172.21.15.28:0/3407542526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fb4102a90 msgr2=0x7f2fb4073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb2d9d640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fb4102a90 0x7f2fb4073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb2d9d640 1 -- 172.21.15.28:0/3407542526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb40749f0 con 0x7f2fb4103450 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb2d9d640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb4103450 0x7f2fb4074160 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2fac00cad0 tx=0x7f2fac00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fa37fe640 1 -- 172.21.15.28:0/3407542526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac00ae20 con 0x7f2fb4103450 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fa37fe640 1 -- 172.21.15.28:0/3407542526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fac005020 con 0x7f2fb4103450 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb41a1d30 con 0x7f2fb4103450 2024-05-09T08:26:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fa37fe640 1 -- 172.21.15.28:0/3407542526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac00f6f0 con 0x7f2fb4103450 2024-05-09T08:26:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.744+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb41a21d0 con 0x7f2fb4103450 2024-05-09T08:26:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.745+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb4108c40 con 0x7f2fb4103450 2024-05-09T08:26:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.747+0000 7f2fa37fe640 1 -- 172.21.15.28:0/3407542526 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2fac0051d0 con 0x7f2fb4103450 2024-05-09T08:26:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.747+0000 7f2fa37fe640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2f9c0a2b90 0x7f2f9c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.747+0000 7f2fb259c640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2f9c0a2b90 0x7f2f9c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:06.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.747+0000 7f2fa37fe640 1 -- 172.21.15.28:0/3407542526 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f2fac0cf3c0 con 0x7f2fb4103450 2024-05-09T08:26:06.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.748+0000 7f2fb259c640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2f9c0a2b90 0x7f2f9c0a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2fa402aa00 tx=0x7f2fa40315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.752+0000 7f2fa37fe640 1 -- 172.21.15.28:0/3407542526 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2fac014030 con 0x7f2fb4103450 2024-05-09T08:26:06.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:06 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4216032584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:06.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:06 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3780117680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:06.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:06 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/129377116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:06.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:06 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4216032584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:06.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:06 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3780117680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:06.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:06 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/129377116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.937+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2fb402d410 con 0x7f2fb4103450 2024-05-09T08:26:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.939+0000 7f2fa37fe640 1 -- 172.21.15.28:0/3407542526 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24899 (secure 0 0 0) 0x7f2fac09b980 con 0x7f2fb4103450 2024-05-09T08:26:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:06.940 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:25:52.396720+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:26:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2f9c0a2b90 msgr2=0x7f2f9c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2f9c0a2b90 0x7f2f9c0a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2fa402aa00 tx=0x7f2fa40315b0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb4103450 msgr2=0x7f2fb4074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb4103450 0x7f2fb4074160 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2fac00cad0 tx=0x7f2fac00cf90 comp rx=0 tx=0).stop 2024-05-09T08:26:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 shutdown_connections 2024-05-09T08:26:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2f9c0a2b90 0x7f2f9c0a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb4103450 0x7f2fb4074160 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fb4102a90 0x7f2fb4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 --2- 172.21.15.28:0/3407542526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fb41018a0 0x7f2fb40755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 >> 172.21.15.28:0/3407542526 conn(0x7f2fb40fce30 msgr2=0x7f2fb4105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 shutdown_connections 2024-05-09T08:26:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:06.941+0000 7f2fb889a640 1 -- 172.21.15.28:0/3407542526 wait complete. 2024-05-09T08:26:06.953 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:26:06.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:26:07.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.061+0000 7f5a7903f640 1 -- 172.21.15.28:0/259385716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74103440 msgr2=0x7f5a74109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.061+0000 7f5a7903f640 1 --2- 172.21.15.28:0/259385716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74103440 0x7f5a74109cd0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009ca0 tx=0x7f5a6c02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.061+0000 7f5a7903f640 1 -- 172.21.15.28:0/259385716 shutdown_connections 2024-05-09T08:26:07.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.061+0000 7f5a7903f640 1 --2- 172.21.15.28:0/259385716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74103440 0x7f5a74109cd0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.061+0000 7f5a7903f640 1 --2- 172.21.15.28:0/259385716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a74102a80 0x7f5a74102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.061+0000 7f5a7903f640 1 --2- 172.21.15.28:0/259385716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a74101890 0x7f5a74101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.061+0000 7f5a7903f640 1 -- 172.21.15.28:0/259385716 >> 172.21.15.28:0/259385716 conn(0x7f5a740fce00 msgr2=0x7f5a740ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.061+0000 7f5a7903f640 1 -- 172.21.15.28:0/259385716 shutdown_connections 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.061+0000 7f5a7903f640 1 -- 172.21.15.28:0/259385716 wait complete. 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7903f640 1 Processor -- start 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7903f640 1 -- start start 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7903f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a74101890 0x7f5a740755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7903f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74102a80 0x7f5a74073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7903f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a74103440 0x7f5a74074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7903f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7410d4f0 con 0x7f5a74102a80 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7903f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7410d370 con 0x7f5a74101890 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7903f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7410d670 con 0x7f5a74103440 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74102a80 0x7f5a74073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74102a80 0x7f5a74073c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58594/0 (socket says 172.21.15.28:58594) 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a7259c640 1 -- 172.21.15.28:0/2050666860 learned_addr learned my addr 172.21.15.28:0/2050666860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.062+0000 7f5a72d9d640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a74101890 0x7f5a740755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a7359e640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a74103440 0x7f5a74074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a7259c640 1 -- 172.21.15.28:0/2050666860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a74103440 msgr2=0x7f5a74074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a7259c640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a74103440 0x7f5a74074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a7259c640 1 -- 172.21.15.28:0/2050666860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a74101890 msgr2=0x7f5a740755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a7259c640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a74101890 0x7f5a740755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a7259c640 1 -- 172.21.15.28:0/2050666860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a740749f0 con 0x7f5a74102a80 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a7259c640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74102a80 0x7f5a74073c20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800d950 tx=0x7f5a6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a5bfff640 1 -- 172.21.15.28:0/2050666860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68017070 con 0x7f5a74102a80 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a5bfff640 1 -- 172.21.15.28:0/2050666860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a68009c30 con 0x7f5a74102a80 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a741a1c30 con 0x7f5a74102a80 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a5bfff640 1 -- 172.21.15.28:0/2050666860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6800f6b0 con 0x7f5a74102a80 2024-05-09T08:26:07.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.063+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a741a2180 con 0x7f5a74102a80 2024-05-09T08:26:07.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.064+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a74108b50 con 0x7f5a74102a80 2024-05-09T08:26:07.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.065+0000 7f5a5bfff640 1 -- 172.21.15.28:0/2050666860 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5a680109d0 con 0x7f5a74102a80 2024-05-09T08:26:07.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.066+0000 7f5a5bfff640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a5c0a2b90 0x7f5a5c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.066+0000 7f5a72d9d640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a5c0a2b90 0x7f5a5c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.066+0000 7f5a5bfff640 1 -- 172.21.15.28:0/2050666860 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f5a680cf0e0 con 0x7f5a74102a80 2024-05-09T08:26:07.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.066+0000 7f5a72d9d640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a5c0a2b90 0x7f5a5c0a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5a741028e0 tx=0x7f5a64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.070+0000 7f5a5bfff640 1 -- 172.21.15.28:0/2050666860 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5a68009de0 con 0x7f5a74102a80 2024-05-09T08:26:07.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.255+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5a740624d0 con 0x7f5a74102a80 2024-05-09T08:26:07.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.255+0000 7f5a5bfff640 1 -- 172.21.15.28:0/2050666860 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f5a6809bbf0 con 0x7f5a74102a80 2024-05-09T08:26:07.259 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:26:07.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a5c0a2b90 msgr2=0x7f5a5c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a5c0a2b90 0x7f5a5c0a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5a741028e0 tx=0x7f5a64005f90 comp rx=0 tx=0).stop 2024-05-09T08:26:07.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74102a80 msgr2=0x7f5a74073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74102a80 0x7f5a74073c20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800d950 tx=0x7f5a6800de10 comp rx=0 tx=0).stop 2024-05-09T08:26:07.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 shutdown_connections 2024-05-09T08:26:07.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f5a5c0a2b90 0x7f5a5c0a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a74103440 0x7f5a74074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74102a80 0x7f5a74073c20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 --2- 172.21.15.28:0/2050666860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a74101890 0x7f5a740755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 >> 172.21.15.28:0/2050666860 conn(0x7f5a740fce00 msgr2=0x7f5a74105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:07.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 shutdown_connections 2024-05-09T08:26:07.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.257+0000 7f5a7903f640 1 -- 172.21.15.28:0/2050666860 wait complete. 2024-05-09T08:26:07.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:26:07.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.376+0000 7fe2868bd640 1 -- 172.21.15.28:0/2791840616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 msgr2=0x7fe280107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.376+0000 7fe2868bd640 1 --2- 172.21.15.28:0/2791840616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 0x7fe280107030 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe274009900 tx=0x7fe27402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.377+0000 7fe2868bd640 1 -- 172.21.15.28:0/2791840616 shutdown_connections 2024-05-09T08:26:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.377+0000 7fe2868bd640 1 --2- 172.21.15.28:0/2791840616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe280107570 0x7fe2801099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.377+0000 7fe2868bd640 1 --2- 172.21.15.28:0/2791840616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 0x7fe280107030 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.377+0000 7fe2868bd640 1 --2- 172.21.15.28:0/2791840616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe280102310 0x7fe280104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.377+0000 7fe2868bd640 1 -- 172.21.15.28:0/2791840616 >> 172.21.15.28:0/2791840616 conn(0x7fe2800fbd20 msgr2=0x7fe2800fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.377+0000 7fe2868bd640 1 -- 172.21.15.28:0/2791840616 shutdown_connections 2024-05-09T08:26:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.377+0000 7fe2868bd640 1 -- 172.21.15.28:0/2791840616 wait complete. 2024-05-09T08:26:07.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.377+0000 7fe2868bd640 1 Processor -- start 2024-05-09T08:26:07.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.377+0000 7fe2868bd640 1 -- start start 2024-05-09T08:26:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe2868bd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe280102310 0x7fe280101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe2868bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 0x7fe2801002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe2868bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe280107570 0x7fe280100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe2868bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe28010d500 con 0x7fe280104c40 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe2868bd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe28010d380 con 0x7fe280102310 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe2868bd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe28010d680 con 0x7fe280107570 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 0x7fe2801002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 0x7fe2801002c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58612/0 (socket says 172.21.15.28:58612) 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27f7fe640 1 -- 172.21.15.28:0/904024374 learned_addr learned my addr 172.21.15.28:0/904024374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27ffff640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe280102310 0x7fe280101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe284e5a640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe280107570 0x7fe280100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27f7fe640 1 -- 172.21.15.28:0/904024374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe280107570 msgr2=0x7fe280100800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27f7fe640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe280107570 0x7fe280100800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27f7fe640 1 -- 172.21.15.28:0/904024374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe280102310 msgr2=0x7fe280101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27f7fe640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe280102310 0x7fe280101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27f7fe640 1 -- 172.21.15.28:0/904024374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe280101090 con 0x7fe280104c40 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27f7fe640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 0x7fe2801002c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe274031610 tx=0x7fe2740043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27d7fa640 1 -- 172.21.15.28:0/904024374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27403f030 con 0x7fe280104c40 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27d7fa640 1 -- 172.21.15.28:0/904024374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe274003ce0 con 0x7fe280104c40 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2801a1cc0 con 0x7fe280104c40 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.378+0000 7fe27d7fa640 1 -- 172.21.15.28:0/904024374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe274035070 con 0x7fe280104c40 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.379+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2801a21c0 con 0x7fe280104c40 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.380+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2800fd810 con 0x7fe280104c40 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.381+0000 7fe27d7fa640 1 -- 172.21.15.28:0/904024374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe274042030 con 0x7fe280104c40 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.381+0000 7fe27d7fa640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe2680a2b90 0x7fe2680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.382+0000 7fe27d7fa640 1 -- 172.21.15.28:0/904024374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fe2740efcc0 con 0x7fe280104c40 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.382+0000 7fe27ffff640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe2680a2b90 0x7fe2680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.382+0000 7fe27ffff640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe2680a2b90 0x7fe2680a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe270005fd0 tx=0x7fe270005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.385+0000 7fe27d7fa640 1 -- 172.21.15.28:0/904024374 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe2740bc830 con 0x7fe280104c40 2024-05-09T08:26:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.572+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe280062430 con 0x7fe280104c40 2024-05-09T08:26:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.572+0000 7fe27d7fa640 1 -- 172.21.15.28:0/904024374 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fe2740bc130 con 0x7fe280104c40 2024-05-09T08:26:07.573 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe2680a2b90 msgr2=0x7fe2680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe2680a2b90 0x7fe2680a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe270005fd0 tx=0x7fe270005850 comp rx=0 tx=0).stop 2024-05-09T08:26:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 msgr2=0x7fe2801002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 0x7fe2801002c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe274031610 tx=0x7fe2740043b0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 shutdown_connections 2024-05-09T08:26:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fe2680a2b90 0x7fe2680a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe280107570 0x7fe280100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe280104c40 0x7fe2801002c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 --2- 172.21.15.28:0/904024374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe280102310 0x7fe280101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 >> 172.21.15.28:0/904024374 conn(0x7fe2800fbd20 msgr2=0x7fe280102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:07.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 shutdown_connections 2024-05-09T08:26:07.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.575+0000 7fe2868bd640 1 -- 172.21.15.28:0/904024374 wait complete. 2024-05-09T08:26:07.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:07 smithi028 ceph-mon[30119]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:26:07.588 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2515103010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:07.588 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3407542526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:07.588 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2050666860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:07.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:26:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.697+0000 7fb9c38ee640 1 -- 172.21.15.28:0/2034863132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc102310 msgr2=0x7fb9bc104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.697+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/2034863132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc102310 0x7fb9bc104700 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009900 tx=0x7fb9b802a410 comp rx=0 tx=0).stop 2024-05-09T08:26:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.697+0000 7fb9c38ee640 1 -- 172.21.15.28:0/2034863132 shutdown_connections 2024-05-09T08:26:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.697+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/2034863132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9bc107570 0x7fb9bc1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.697+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/2034863132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9bc104c40 0x7fb9bc107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.697+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/2034863132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc102310 0x7fb9bc104700 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.697+0000 7fb9c38ee640 1 -- 172.21.15.28:0/2034863132 >> 172.21.15.28:0/2034863132 conn(0x7fb9bc0fbd20 msgr2=0x7fb9bc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.697+0000 7fb9c38ee640 1 -- 172.21.15.28:0/2034863132 shutdown_connections 2024-05-09T08:26:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.697+0000 7fb9c38ee640 1 -- 172.21.15.28:0/2034863132 wait complete. 2024-05-09T08:26:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c38ee640 1 Processor -- start 2024-05-09T08:26:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c38ee640 1 -- start start 2024-05-09T08:26:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c38ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9bc102310 0x7fb9bc197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c38ee640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9bc104c40 0x7fb9bc198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c38ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc107570 0x7fb9bc19f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c38ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc10d5d0 con 0x7fb9bc107570 2024-05-09T08:26:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c38ee640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc10d450 con 0x7fb9bc104c40 2024-05-09T08:26:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c38ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc10d750 con 0x7fb9bc102310 2024-05-09T08:26:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c1e8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc107570 0x7fb9bc19f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c1e8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc107570 0x7fb9bc19f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58630/0 (socket says 172.21.15.28:58630) 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c1e8b640 1 -- 172.21.15.28:0/4174756523 learned_addr learned my addr 172.21.15.28:0/4174756523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c168a640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9bc102310 0x7fb9bc197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.698+0000 7fb9c0e89640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9bc104c40 0x7fb9bc198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c1e8b640 1 -- 172.21.15.28:0/4174756523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9bc102310 msgr2=0x7fb9bc197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c1e8b640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9bc102310 0x7fb9bc197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c1e8b640 1 -- 172.21.15.28:0/4174756523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9bc104c40 msgr2=0x7fb9bc198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c1e8b640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9bc104c40 0x7fb9bc198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c1e8b640 1 -- 172.21.15.28:0/4174756523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc19f850 con 0x7fb9bc107570 2024-05-09T08:26:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c168a640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9bc102310 0x7fb9bc197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c0e89640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9bc104c40 0x7fb9bc198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c1e8b640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc107570 0x7fb9bc19f1b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400b970 tx=0x7fb9b400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9b27fc640 1 -- 172.21.15.28:0/4174756523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4019070 con 0x7fb9bc107570 2024-05-09T08:26:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9b27fc640 1 -- 172.21.15.28:0/4174756523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9b400cb40 con 0x7fb9bc107570 2024-05-09T08:26:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9bc19fb40 con 0x7fb9bc107570 2024-05-09T08:26:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9b27fc640 1 -- 172.21.15.28:0/4174756523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4007650 con 0x7fb9bc107570 2024-05-09T08:26:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.699+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9bc1a0040 con 0x7fb9bc107570 2024-05-09T08:26:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.700+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9bc0fd810 con 0x7fb9bc107570 2024-05-09T08:26:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.701+0000 7fb9b27fc640 1 -- 172.21.15.28:0/4174756523 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb9b401c070 con 0x7fb9bc107570 2024-05-09T08:26:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.702+0000 7fb9b27fc640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9a80a2b90 0x7fb9a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.702+0000 7fb9c168a640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9a80a2b90 0x7fb9a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.702+0000 7fb9b27fc640 1 -- 172.21.15.28:0/4174756523 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fb9b40ceb60 con 0x7fb9bc107570 2024-05-09T08:26:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.702+0000 7fb9c168a640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9a80a2b90 0x7fb9a80a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8006010 tx=0x7fb9b8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.706+0000 7fb9b27fc640 1 -- 172.21.15.28:0/4174756523 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb9b409b6d0 con 0x7fb9bc107570 2024-05-09T08:26:07.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:07 smithi165 ceph-mon[34540]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:26:07.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2515103010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:07.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3407542526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:07.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2050666860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:07.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:07 smithi184 ceph-mon[34509]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:26:07.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2515103010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:07.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3407542526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:07.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2050666860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.892+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb9bc02d1c0 con 0x7fb9bc107570 2024-05-09T08:26:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.893+0000 7fb9b27fc640 1 -- 172.21.15.28:0/4174756523 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fb9bc02d1c0 con 0x7fb9bc107570 2024-05-09T08:26:07.893 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:26:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9a80a2b90 msgr2=0x7fb9a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9a80a2b90 0x7fb9a80a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8006010 tx=0x7fb9b8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc107570 msgr2=0x7fb9bc19f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc107570 0x7fb9bc19f1b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400b970 tx=0x7fb9b400be30 comp rx=0 tx=0).stop 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 shutdown_connections 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb9a80a2b90 0x7fb9a80a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9bc107570 0x7fb9bc19f1b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9bc104c40 0x7fb9bc198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 --2- 172.21.15.28:0/4174756523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9bc102310 0x7fb9bc197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.894+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 >> 172.21.15.28:0/4174756523 conn(0x7fb9bc0fbd20 msgr2=0x7fb9bc102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.895+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 shutdown_connections 2024-05-09T08:26:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:07.895+0000 7fb9c38ee640 1 -- 172.21.15.28:0/4174756523 wait complete. 2024-05-09T08:26:07.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 -- 172.21.15.28:0/4164069855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610102340 msgr2=0x7f5610108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 --2- 172.21.15.28:0/4164069855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610102340 0x7f5610108bd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5608009ca0 tx=0x7f560802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 -- 172.21.15.28:0/4164069855 shutdown_connections 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 --2- 172.21.15.28:0/4164069855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610102340 0x7f5610108bd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 --2- 172.21.15.28:0/4164069855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5610101980 0x7f5610101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 --2- 172.21.15.28:0/4164069855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5610100790 0x7f5610100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 -- 172.21.15.28:0/4164069855 >> 172.21.15.28:0/4164069855 conn(0x7f56100fbd20 msgr2=0x7f56100fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 -- 172.21.15.28:0/4164069855 shutdown_connections 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 -- 172.21.15.28:0/4164069855 wait complete. 2024-05-09T08:26:08.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 Processor -- start 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.015+0000 7f5617fb7640 1 -- start start 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5617fb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610100790 0x7f5610197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5617fb7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5610101980 0x7f5610198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5617fb7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5610102340 0x7f561019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5617fb7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f561010d530 con 0x7f5610100790 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5617fb7640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f561010d3b0 con 0x7f5610102340 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5617fb7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f561010d6b0 con 0x7f5610101980 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615d53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610100790 0x7f5610197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615d53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610100790 0x7f5610197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58636/0 (socket says 172.21.15.28:58636) 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615d53640 1 -- 172.21.15.28:0/350220932 learned_addr learned my addr 172.21.15.28:0/350220932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5616554640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5610102340 0x7f561019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615552640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5610101980 0x7f5610198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615d53640 1 -- 172.21.15.28:0/350220932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5610101980 msgr2=0x7f5610198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615d53640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5610101980 0x7f5610198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615d53640 1 -- 172.21.15.28:0/350220932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5610102340 msgr2=0x7f561019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615d53640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5610102340 0x7f561019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615d53640 1 -- 172.21.15.28:0/350220932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561019f720 con 0x7f5610100790 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615552640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5610101980 0x7f5610198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5615d53640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610100790 0x7f5610197b00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f560c00d770 tx=0x7f560c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5606ffd640 1 -- 172.21.15.28:0/350220932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c017070 con 0x7f5610100790 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5606ffd640 1 -- 172.21.15.28:0/350220932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f560c009c30 con 0x7f5610100790 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561019fa10 con 0x7f5610100790 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5606ffd640 1 -- 172.21.15.28:0/350220932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c003dd0 con 0x7f5610100790 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.016+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561019ffb0 con 0x7f5610100790 2024-05-09T08:26:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.017+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5610068aa0 con 0x7f5610100790 2024-05-09T08:26:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.019+0000 7f5606ffd640 1 -- 172.21.15.28:0/350220932 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f560c003fb0 con 0x7f5610100790 2024-05-09T08:26:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.019+0000 7f5606ffd640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f55fc0a2c10 0x7f55fc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.019+0000 7f5615552640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f55fc0a2c10 0x7f55fc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.019+0000 7f5606ffd640 1 -- 172.21.15.28:0/350220932 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f560c0cf200 con 0x7f5610100790 2024-05-09T08:26:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.020+0000 7f5615552640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f55fc0a2c10 0x7f55fc0a50d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5610199010 tx=0x7f5600005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:08.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.022+0000 7f5606ffd640 1 -- 172.21.15.28:0/350220932 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f560c009de0 con 0x7f5610100790 2024-05-09T08:26:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.208+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f561002d2a0 con 0x7f5610100790 2024-05-09T08:26:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.208+0000 7f5606ffd640 1 -- 172.21.15.28:0/350220932 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f560c09bdf0 con 0x7f5610100790 2024-05-09T08:26:08.209 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f55fc0a2c10 msgr2=0x7f55fc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f55fc0a2c10 0x7f55fc0a50d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5610199010 tx=0x7f5600005e50 comp rx=0 tx=0).stop 2024-05-09T08:26:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610100790 msgr2=0x7f5610197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610100790 0x7f5610197b00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f560c00d770 tx=0x7f560c00dc30 comp rx=0 tx=0).stop 2024-05-09T08:26:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 shutdown_connections 2024-05-09T08:26:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f55fc0a2c10 0x7f55fc0a50d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5610102340 0x7f561019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5610101980 0x7f5610198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 --2- 172.21.15.28:0/350220932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5610100790 0x7f5610197b00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.210+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 >> 172.21.15.28:0/350220932 conn(0x7f56100fbd20 msgr2=0x7f5610104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.211+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 shutdown_connections 2024-05-09T08:26:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.211+0000 7f5617fb7640 1 -- 172.21.15.28:0/350220932 wait complete. 2024-05-09T08:26:08.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:26:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.328+0000 7f4a1b309640 1 -- 172.21.15.28:0/2356094028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 msgr2=0x7f4a141099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.328+0000 7f4a1b309640 1 --2- 172.21.15.28:0/2356094028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 0x7f4a141099a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009ca0 tx=0x7f4a0c02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.328+0000 7f4a1b309640 1 -- 172.21.15.28:0/2356094028 shutdown_connections 2024-05-09T08:26:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.328+0000 7f4a1b309640 1 --2- 172.21.15.28:0/2356094028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 0x7f4a141099a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.328+0000 7f4a1b309640 1 --2- 172.21.15.28:0/2356094028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14104c20 0x7f4a14107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.328+0000 7f4a1b309640 1 --2- 172.21.15.28:0/2356094028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a141022f0 0x7f4a141046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.328+0000 7f4a1b309640 1 -- 172.21.15.28:0/2356094028 >> 172.21.15.28:0/2356094028 conn(0x7f4a140fbce0 msgr2=0x7f4a140fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.328+0000 7f4a1b309640 1 -- 172.21.15.28:0/2356094028 shutdown_connections 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.328+0000 7f4a1b309640 1 -- 172.21.15.28:0/2356094028 wait complete. 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a1b309640 1 Processor -- start 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a1b309640 1 -- start start 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a1b309640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a141022f0 0x7f4a14199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a1b309640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14104c20 0x7f4a1419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a1b309640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 0x7f4a141a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a1b309640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1410d510 con 0x7f4a14107550 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a1b309640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1410d390 con 0x7f4a14104c20 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a1b309640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1410d690 con 0x7f4a141022f0 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a198a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 0x7f4a141a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a198a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 0x7f4a141a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58662/0 (socket says 172.21.15.28:58662) 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a198a6640 1 -- 172.21.15.28:0/1369070214 learned_addr learned my addr 172.21.15.28:0/1369070214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a190a5640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a141022f0 0x7f4a14199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a198a6640 1 -- 172.21.15.28:0/1369070214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a141022f0 msgr2=0x7f4a14199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a198a6640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a141022f0 0x7f4a14199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a198a6640 1 -- 172.21.15.28:0/1369070214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14104c20 msgr2=0x7f4a1419a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a188a4640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14104c20 0x7f4a1419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.329+0000 7f4a198a6640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14104c20 0x7f4a1419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.330+0000 7f4a198a6640 1 -- 172.21.15.28:0/1369070214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a141a1980 con 0x7f4a14107550 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.330+0000 7f4a198a6640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 0x7f4a141a1310 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c031ae0 tx=0x7f4a0c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.330+0000 7f4a067fc640 1 -- 172.21.15.28:0/1369070214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c003c30 con 0x7f4a14107550 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.330+0000 7f4a067fc640 1 -- 172.21.15.28:0/1369070214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a0c003dd0 con 0x7f4a14107550 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.330+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a141a1c10 con 0x7f4a14107550 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.330+0000 7f4a067fc640 1 -- 172.21.15.28:0/1369070214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c02fcb0 con 0x7f4a14107550 2024-05-09T08:26:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.330+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a141a21b0 con 0x7f4a14107550 2024-05-09T08:26:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.331+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a14109040 con 0x7f4a14107550 2024-05-09T08:26:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.332+0000 7f4a067fc640 1 -- 172.21.15.28:0/1369070214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4a0c004050 con 0x7f4a14107550 2024-05-09T08:26:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.333+0000 7f4a067fc640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4a000a2c10 0x7f4a000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.333+0000 7f4a190a5640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4a000a2c10 0x7f4a000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.333+0000 7f4a067fc640 1 -- 172.21.15.28:0/1369070214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f4a0c0f0970 con 0x7f4a14107550 2024-05-09T08:26:08.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.333+0000 7f4a190a5640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4a000a2c10 0x7f4a000a50d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009b60 tx=0x7f4a10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.337+0000 7f4a067fc640 1 -- 172.21.15.28:0/1369070214 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a0c0bd480 con 0x7f4a14107550 2024-05-09T08:26:08.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.563+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a140624d0 con 0x7f4a14107550 2024-05-09T08:26:08.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.564+0000 7f4a067fc640 1 -- 172.21.15.28:0/1369070214 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f4a0c0bcd80 con 0x7f4a14107550 2024-05-09T08:26:08.565 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:08.565 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:24:19.404649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24481,24499,24476,24512,24463]},"id":1}]} 2024-05-09T08:26:08.565 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4a000a2c10 msgr2=0x7f4a000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4a000a2c10 0x7f4a000a50d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009b60 tx=0x7f4a10005f90 comp rx=0 tx=0).stop 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 msgr2=0x7f4a141a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 0x7f4a141a1310 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c031ae0 tx=0x7f4a0c031ee0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 shutdown_connections 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4a000a2c10 0x7f4a000a50d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a14107550 0x7f4a141a1310 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a14104c20 0x7f4a1419a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 --2- 172.21.15.28:0/1369070214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a141022f0 0x7f4a14199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 >> 172.21.15.28:0/1369070214 conn(0x7f4a140fbce0 msgr2=0x7f4a14101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 shutdown_connections 2024-05-09T08:26:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.566+0000 7f4a1b309640 1 -- 172.21.15.28:0/1369070214 wait complete. 2024-05-09T08:26:08.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:26:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.683+0000 7f883049a640 1 -- 172.21.15.28:0/951732224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828102310 msgr2=0x7f8828104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.683+0000 7f883049a640 1 --2- 172.21.15.28:0/951732224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828102310 0x7f8828104700 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8824009900 tx=0x7f882402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.683+0000 7f883049a640 1 -- 172.21.15.28:0/951732224 shutdown_connections 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.683+0000 7f883049a640 1 --2- 172.21.15.28:0/951732224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8828107570 0x7f88281099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.683+0000 7f883049a640 1 --2- 172.21.15.28:0/951732224 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8828104c40 0x7f8828107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.683+0000 7f883049a640 1 --2- 172.21.15.28:0/951732224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828102310 0x7f8828104700 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.683+0000 7f883049a640 1 -- 172.21.15.28:0/951732224 >> 172.21.15.28:0/951732224 conn(0x7f88280fbd20 msgr2=0x7f88280fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 -- 172.21.15.28:0/951732224 shutdown_connections 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 -- 172.21.15.28:0/951732224 wait complete. 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 Processor -- start 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 -- start start 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8828102310 0x7f8828101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8828104c40 0x7f88281002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828107570 0x7f8828100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f882810d600 con 0x7f8828107570 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f882810d480 con 0x7f8828102310 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f883049a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f882810d780 con 0x7f8828104c40 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f882ea37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828107570 0x7f8828100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f882ea37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828107570 0x7f8828100800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58688/0 (socket says 172.21.15.28:58688) 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.684+0000 7f882ea37640 1 -- 172.21.15.28:0/1704324191 learned_addr learned my addr 172.21.15.28:0/1704324191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f882e236640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8828102310 0x7f8828101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f882da35640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8828104c40 0x7f88281002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f882ea37640 1 -- 172.21.15.28:0/1704324191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8828104c40 msgr2=0x7f88281002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f882ea37640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8828104c40 0x7f88281002c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f882ea37640 1 -- 172.21.15.28:0/1704324191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8828102310 msgr2=0x7f8828101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f882ea37640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8828102310 0x7f8828101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f882ea37640 1 -- 172.21.15.28:0/1704324191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8828101090 con 0x7f8828107570 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f882ea37640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828107570 0x7f8828100800 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f882000b970 tx=0x7f882000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f881f7fe640 1 -- 172.21.15.28:0/1704324191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8820019070 con 0x7f8828107570 2024-05-09T08:26:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f881f7fe640 1 -- 172.21.15.28:0/1704324191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f882000cb40 con 0x7f8828107570 2024-05-09T08:26:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88281a1e50 con 0x7f8828107570 2024-05-09T08:26:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f881f7fe640 1 -- 172.21.15.28:0/1704324191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8820007650 con 0x7f8828107570 2024-05-09T08:26:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.685+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88281a22e0 con 0x7f8828107570 2024-05-09T08:26:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.686+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88280fd810 con 0x7f8828107570 2024-05-09T08:26:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.688+0000 7f881f7fe640 1 -- 172.21.15.28:0/1704324191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f882001c070 con 0x7f8828107570 2024-05-09T08:26:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.689+0000 7f881f7fe640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f88140a2b90 0x7f88140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.689+0000 7f882e236640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f88140a2b90 0x7f88140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.689+0000 7f881f7fe640 1 -- 172.21.15.28:0/1704324191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f88200cedd0 con 0x7f8828107570 2024-05-09T08:26:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.689+0000 7f882e236640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f88140a2b90 0x7f88140a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f88240314e0 tx=0x7f8824000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.692+0000 7f881f7fe640 1 -- 172.21.15.28:0/1704324191 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f882009b2f0 con 0x7f8828107570 2024-05-09T08:26:08.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:08 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/904024374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:08.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:08 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4174756523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:08.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:08 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/350220932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:08.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:08 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/904024374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:08.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:08 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4174756523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:08.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:08 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/350220932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.877+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8828062430 con 0x7f8828107570 2024-05-09T08:26:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.878+0000 7f881f7fe640 1 -- 172.21.15.28:0/1704324191 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24899 (secure 0 0 0) 0x7f882009f5f0 con 0x7f8828107570 2024-05-09T08:26:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:08.879 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:25:52.396720+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:26:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f88140a2b90 msgr2=0x7f88140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f88140a2b90 0x7f88140a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f88240314e0 tx=0x7f8824000bc0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828107570 msgr2=0x7f8828100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828107570 0x7f8828100800 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f882000b970 tx=0x7f882000be30 comp rx=0 tx=0).stop 2024-05-09T08:26:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 shutdown_connections 2024-05-09T08:26:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f88140a2b90 0x7f88140a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8828107570 0x7f8828100800 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8828104c40 0x7f88281002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 --2- 172.21.15.28:0/1704324191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8828102310 0x7f8828101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 >> 172.21.15.28:0/1704324191 conn(0x7f88280fbd20 msgr2=0x7f8828102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 shutdown_connections 2024-05-09T08:26:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:08.880+0000 7f883049a640 1 -- 172.21.15.28:0/1704324191 wait complete. 2024-05-09T08:26:08.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:08 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/904024374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:08.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:08 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4174756523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:08.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:08 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/350220932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:08.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-09T08:23:50.918821+0000', 'modified': '2024-05-09T08:24:19.404649+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24512, 'mds_1': 24499, 'mds_2': 24481, 'mds_3': 24476, 'mds_4': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14613': {'gid': 14613, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/612632862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 612632862}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 612632862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14625': {'gid': 14625, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/9520094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 9520094}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 9520094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24410': {'gid': 24410, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.165:6833/3903056410', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6832', 'nonce': 3903056410}, {'type': 'v1', 'addr': '172.21.15.165:6833', 'nonce': 3903056410}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.184:6833/1853687224', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6832', 'nonce': 1853687224}, {'type': 'v1', 'addr': '172.21.15.184:6833', 'nonce': 1853687224}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.165:6835/3736338591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6834', 'nonce': 3736338591}, {'type': 'v1', 'addr': '172.21.15.165:6835', 'nonce': 3736338591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.184:6835/4284085086', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6834', 'nonce': 4284085086}, {'type': 'v1', 'addr': '172.21.15.184:6835', 'nonce': 4284085086}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.165:6837/1615628763', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6836', 'nonce': 1615628763}, {'type': 'v1', 'addr': '172.21.15.165:6837', 'nonce': 1615628763}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.184:6837/3795789755', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6836', 'nonce': 3795789755}, {'type': 'v1', 'addr': '172.21.15.184:6837', 'nonce': 3795789755}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.184:6839/224925653', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6838', 'nonce': 224925653}, {'type': 'v1', 'addr': '172.21.15.184:6839', 'nonce': 224925653}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.165:6839/1723030158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6838', 'nonce': 1723030158}, {'type': 'v1', 'addr': '172.21.15.165:6839', 'nonce': 1723030158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24512, 'qdb_cluster': [24481, 24499, 24476, 24512, 24463]} 2024-05-09T08:26:08.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-09T08:26:08.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-05-09T08:26:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.002+0000 7f1332134640 1 -- 172.21.15.28:0/3750859538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 msgr2=0x7f132c100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.002+0000 7f1332134640 1 --2- 172.21.15.28:0/3750859538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 0x7f132c100bb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f131c009900 tx=0x7f131c02a410 comp rx=0 tx=0).stop 2024-05-09T08:26:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.002+0000 7f1332134640 1 -- 172.21.15.28:0/3750859538 shutdown_connections 2024-05-09T08:26:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.002+0000 7f1332134640 1 --2- 172.21.15.28:0/3750859538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f132c102340 0x7f132c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.002+0000 7f1332134640 1 --2- 172.21.15.28:0/3750859538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f132c101980 0x7f132c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.002+0000 7f1332134640 1 --2- 172.21.15.28:0/3750859538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 0x7f132c100bb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.002+0000 7f1332134640 1 -- 172.21.15.28:0/3750859538 >> 172.21.15.28:0/3750859538 conn(0x7f132c0fbd20 msgr2=0x7f132c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.002+0000 7f1332134640 1 -- 172.21.15.28:0/3750859538 shutdown_connections 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.002+0000 7f1332134640 1 -- 172.21.15.28:0/3750859538 wait complete. 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f1332134640 1 Processor -- start 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f1332134640 1 -- start start 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f1332134640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 0x7f132c197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f1332134640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f132c101980 0x7f132c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f1332134640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f132c102340 0x7f132c19f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f1332134640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f132c10c4e0 con 0x7f132c100790 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f1332134640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f132c10c360 con 0x7f132c102340 2024-05-09T08:26:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f1332134640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f132c10c660 con 0x7f132c101980 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 0x7f132c197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 0x7f132c197d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58708/0 (socket says 172.21.15.28:58708) 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.003+0000 7f132b7fe640 1 -- 172.21.15.28:0/1447320024 learned_addr learned my addr 172.21.15.28:0/1447320024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f132affd640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f132c101980 0x7f132c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f132b7fe640 1 -- 172.21.15.28:0/1447320024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f132c101980 msgr2=0x7f132c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f132b7fe640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f132c101980 0x7f132c1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f132b7fe640 1 -- 172.21.15.28:0/1447320024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f132c102340 msgr2=0x7f132c19f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f132bfff640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f132c102340 0x7f132c19f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f132b7fe640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f132c102340 0x7f132c19f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f132b7fe640 1 -- 172.21.15.28:0/1447320024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c19f990 con 0x7f132c100790 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f132affd640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f132c101980 0x7f132c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f132b7fe640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 0x7f132c197d70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f131c031610 tx=0x7f131c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f1328ff9640 1 -- 172.21.15.28:0/1447320024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c03f030 con 0x7f132c100790 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f1328ff9640 1 -- 172.21.15.28:0/1447320024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f131c003ce0 con 0x7f132c100790 2024-05-09T08:26:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f1332134640 1 -- 172.21.15.28:0/1447320024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c19fc20 con 0x7f132c100790 2024-05-09T08:26:09.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f1328ff9640 1 -- 172.21.15.28:0/1447320024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c035070 con 0x7f132c100790 2024-05-09T08:26:09.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.004+0000 7f1332134640 1 -- 172.21.15.28:0/1447320024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c1a0110 con 0x7f132c100790 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.006+0000 7f1328ff9640 1 -- 172.21.15.28:0/1447320024 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f131c042030 con 0x7f132c100790 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f1334e40740 1 Processor -- start 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f1334e40740 1 -- start start 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f1334e40740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 0x55ce61688ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f1334e40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce6168ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f1334e40740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce6168e420 0x55ce616908b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f1334e40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce615ec4d0 con 0x55ce616892b0 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f1334e40740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ce615943e0 con 0x55ce616888c0 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f1334e40740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ce615e8490 con 0x55ce6168e420 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce6168ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce6168ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58722/0 (socket says 172.21.15.28:58722) 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132b7fe640 1 -- 172.21.15.28:0/503981259 learned_addr learned my addr 172.21.15.28:0/503981259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132bfff640 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 0x55ce61688ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132affd640 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce6168e420 0x55ce616908b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f1328ff9640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f13180a2b90 0x7f13180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132b7fe640 1 -- 172.21.15.28:0/503981259 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce6168e420 msgr2=0x55ce616908b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132b7fe640 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce6168e420 0x55ce616908b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132b7fe640 1 -- 172.21.15.28:0/503981259 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 msgr2=0x55ce61688ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132b7fe640 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 0x55ce61688ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132b7fe640 1 -- 172.21.15.28:0/503981259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce613002d0 con 0x55ce616892b0 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.007+0000 7f132affd640 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce6168e420 0x55ce616908b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.008+0000 7f132bfff640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f13180a2b90 0x7f13180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.008+0000 7f1328ff9640 1 -- 172.21.15.28:0/1447320024 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f131c0efb90 con 0x7f132c100790 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.008+0000 7f132b7fe640 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce6168ddb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f131c0d60a0 tx=0x7f131c0b80d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e130f243243d79e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.008+0000 7f1321ffb640 1 -- 172.21.15.28:0/503981259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c049070 con 0x55ce616892b0 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.008+0000 7f1321ffb640 1 -- 172.21.15.28:0/503981259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f131c0dc0e0 con 0x55ce616892b0 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.008+0000 7f1321ffb640 1 -- 172.21.15.28:0/503981259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c0f4040 con 0x55ce616892b0 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 -- 172.21.15.28:0/503981259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 msgr2=0x55ce6168ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f132bfff640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f13180a2b90 0x7f13180a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55ce61689b10 tx=0x7f1324004b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce6168ddb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f131c0d60a0 tx=0x7f131c0b80d0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 -- 172.21.15.28:0/503981259 shutdown_connections 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce6168e420 0x55ce616908b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce6168ddb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 --2- 172.21.15.28:0/503981259 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 0x55ce61688ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 -- 172.21.15.28:0/503981259 >> 172.21.15.28:0/503981259 conn(0x55ce61683f30 msgr2=0x55ce61684330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 -- 172.21.15.28:0/503981259 shutdown_connections 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 -- 172.21.15.28:0/503981259 wait complete. 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 -- start start 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 Processor -- start 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616888c0 0x55ce61692920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce61692e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 0x55ce61699ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.009+0000 7f1334e40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce615ec4d0 con 0x55ce616892b0 2024-05-09T08:26:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f1334e40740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ce615a78e0 con 0x55ce6168e420 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f1334e40740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ce61691190 con 0x55ce616888c0 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce61692e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f1334e40740 10 client.0 _ll_register_callbacks cb 0x55ce615749c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce61692e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58732/0 (socket says 172.21.15.28:58732) 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132affd640 1 -- 172.21.15.28:0/3836491542 learned_addr learned my addr 172.21.15.28:0/3836491542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132bfff640 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 0x55ce61699ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132b7fe640 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616888c0 0x55ce61692920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132affd640 1 -- 172.21.15.28:0/3836491542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616888c0 msgr2=0x55ce61692920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132affd640 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616888c0 0x55ce61692920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132affd640 1 -- 172.21.15.28:0/3836491542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 msgr2=0x55ce61699ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132affd640 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 0x55ce61699ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132affd640 1 -- 172.21.15.28:0/3836491542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce613ca6b0 con 0x55ce616892b0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f132affd640 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce61692e60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f131400e280 tx=0x7f131400e740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=825e2c015fbf8c8e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f130affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f130affd640 1 -- 172.21.15.28:0/3836491542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1314018020 con 0x55ce616892b0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f130affd640 1 -- 172.21.15.28:0/3836491542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f1314013660 con 0x55ce616892b0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f1334e40740 1 -- 172.21.15.28:0/3836491542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ce615f9c00 con 0x55ce616892b0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f130affd640 1 -- 172.21.15.28:0/3836491542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1314023050 con 0x55ce616892b0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f1334e40740 10 client.15426 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.010+0000 7f1334e40740 1 -- 172.21.15.28:0/3836491542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce6169cf80 con 0x55ce616892b0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f130affd640 1 -- 172.21.15.28:0/3836491542 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ce615f9c00 con 0x55ce616892b0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f130affd640 1 -- 172.21.15.28:0/3836491542 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f1314014960 con 0x55ce616892b0 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f1334e40740 10 client.15426 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f1334e40740 10 client.15426 resolve_mds: resolved name 'f' to daemon mds.f 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f1334e40740 20 client.15426 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f1334e40740 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ce6169f6e0 0x55ce616a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f1334e40740 4 client.15426 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f1334e40740 1 -- 172.21.15.28:0/3836491542 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce615f3f50 con 0x55ce6169f6e0 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f132b7fe640 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ce6169f6e0 0x55ce616a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f132b7fe640 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ce6169f6e0 0x55ce616a0b50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c07a40e5b28f88c3 server_cookie=ebcd757f4c87b7b8 in_seq=0 out_seq=0 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.011+0000 7f130affd640 10 client.15426 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.012+0000 7f130affd640 1 -- 172.21.15.28:0/3836491542 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce615f3f50 con 0x55ce6169f6e0 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.012+0000 7f130affd640 10 client.15426 handle_command_reply: tid=0 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.012+0000 7f1334e40740 1 client.15426 shutdown 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.012+0000 7f1334e40740 1 -- 172.21.15.28:0/3836491542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 msgr2=0x55ce61692e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.012+0000 7f1334e40740 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce61692e60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f131400e280 tx=0x7f131400e740 comp rx=0 tx=0).stop 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f1334e40740 1 -- 172.21.15.28:0/3836491542 shutdown_connections 2024-05-09T08:26:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f1334e40740 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 0x55ce61699ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f1334e40740 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616892b0 0x55ce61692e60 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f1334e40740 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ce6169f6e0 0x55ce616a0b50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f1334e40740 1 --2- 172.21.15.28:0/3836491542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616888c0 0x55ce61692920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f130affd640 0 client.15426 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:26:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f1334e40740 1 -- 172.21.15.28:0/3836491542 >> 172.21.15.28:0/3836491542 conn(0x55ce61683f30 msgr2=0x55ce61683000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f1334e40740 1 -- 172.21.15.28:0/3836491542 shutdown_connections 2024-05-09T08:26:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f1334e40740 1 -- 172.21.15.28:0/3836491542 wait complete. 2024-05-09T08:26:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.013+0000 7f1334e40740 20 client.15426 trim_cache size 0 max 16384 2024-05-09T08:26:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f1334e40740 1 Processor -- start 2024-05-09T08:26:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f1334e40740 1 -- start start 2024-05-09T08:26:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f1334e40740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 0x55ce61691bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f1334e40740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 0x55ce61699e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f1334e40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce6168e420 0x55ce61692130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f1334e40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce6169ee90 con 0x55ce6168e420 2024-05-09T08:26:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f1334e40740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ce6169f560 con 0x55ce616888c0 2024-05-09T08:26:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f1334e40740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ce6169fc30 con 0x55ce616892b0 2024-05-09T08:26:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce6168e420 0x55ce61692130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce6168e420 0x55ce61692130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58740/0 (socket says 172.21.15.28:58740) 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132bfff640 1 -- 172.21.15.28:0/1263131569 learned_addr learned my addr 172.21.15.28:0/1263131569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132b7fe640 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 0x55ce61691bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132affd640 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 0x55ce61699e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132bfff640 1 -- 172.21.15.28:0/1263131569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 msgr2=0x55ce61699e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132bfff640 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 0x55ce61699e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132bfff640 1 -- 172.21.15.28:0/1263131569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 msgr2=0x55ce61691bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132bfff640 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 0x55ce61691bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132bfff640 1 -- 172.21.15.28:0/1263131569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce61689b40 con 0x55ce6168e420 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.031+0000 7f132affd640 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 0x55ce61699e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f132bfff640 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce6168e420 0x55ce61692130 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1324013470 tx=0x7f1324012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17458c67514e2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f13217fa640 1 -- 172.21.15.28:0/1263131569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324024070 con 0x55ce6168e420 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f13217fa640 1 -- 172.21.15.28:0/1263131569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f132400c040 con 0x55ce6168e420 2024-05-09T08:26:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f13217fa640 1 -- 172.21.15.28:0/1263131569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132401f8a0 con 0x55ce6168e420 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f1334e40740 1 -- 172.21.15.28:0/1263131569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce6168e420 msgr2=0x55ce61692130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f1334e40740 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce6168e420 0x55ce61692130 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1324013470 tx=0x7f1324012000 comp rx=0 tx=0).stop 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f1334e40740 1 -- 172.21.15.28:0/1263131569 shutdown_connections 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f1334e40740 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce6168e420 0x55ce61692130 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f1334e40740 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 0x55ce61699e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f1334e40740 1 --2- 172.21.15.28:0/1263131569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce616888c0 0x55ce61691bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f1334e40740 1 -- 172.21.15.28:0/1263131569 >> 172.21.15.28:0/1263131569 conn(0x55ce61683f30 msgr2=0x55ce6168a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f1334e40740 1 -- 172.21.15.28:0/1263131569 shutdown_connections 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.032+0000 7f1334e40740 1 -- 172.21.15.28:0/1263131569 wait complete. 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f1334e40740 1 -- start start 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f1334e40740 1 Processor -- start 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f1334e40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616888c0 0x55ce61696090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f1334e40740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 0x55ce616965d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f1334e40740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 0x55ce61698ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f1334e40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce616a09d0 con 0x55ce616888c0 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f1334e40740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ce616a0300 con 0x55ce6168e420 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f1334e40740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ce616a10a0 con 0x55ce616892b0 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f1334e40740 10 client.0 _ll_register_callbacks cb 0x7f131400c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616888c0 0x55ce61696090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616888c0 0x55ce61696090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58752/0 (socket says 172.21.15.28:58752) 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132b7fe640 1 -- 172.21.15.28:0/839529177 learned_addr learned my addr 172.21.15.28:0/839529177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132bfff640 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 0x55ce61698ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132affd640 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 0x55ce616965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132b7fe640 1 -- 172.21.15.28:0/839529177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 msgr2=0x55ce616965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132b7fe640 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 0x55ce616965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132b7fe640 1 -- 172.21.15.28:0/839529177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 msgr2=0x55ce61698ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132b7fe640 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 0x55ce61698ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132b7fe640 1 -- 172.21.15.28:0/839529177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce61699530 con 0x55ce616888c0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.033+0000 7f132bfff640 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 0x55ce61698ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f132b7fe640 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616888c0 0x55ce61696090 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f131c031440 tx=0x7f131c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2103b4877073e92f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f130affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f130affd640 1 -- 172.21.15.28:0/839529177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c049040 con 0x55ce616888c0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f130affd640 1 -- 172.21.15.28:0/839529177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f131c0f3920 con 0x55ce616888c0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f1334e40740 1 -- 172.21.15.28:0/839529177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f132400c040 con 0x55ce616888c0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f130affd640 1 -- 172.21.15.28:0/839529177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f131c0bbb40 con 0x55ce616888c0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f1334e40740 10 client.15432 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f1334e40740 1 -- 172.21.15.28:0/839529177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce617059b0 con 0x55ce616888c0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f130affd640 1 -- 172.21.15.28:0/839529177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f132400c040 con 0x55ce616888c0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f130affd640 1 -- 172.21.15.28:0/839529177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f131c0b8670 con 0x55ce616888c0 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f1334e40740 10 client.15432 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f1334e40740 10 client.15432 resolve_mds: resolved name 'f' to daemon mds.f 2024-05-09T08:26:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f1334e40740 20 client.15432 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f1334e40740 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ce6170a430 0x55ce61709140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f1334e40740 4 client.15432 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.034+0000 7f1334e40740 1 -- 172.21.15.28:0/839529177 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "status"}) -- 0x55ce61571b80 con 0x55ce6170a430 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f132affd640 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ce6170a430 0x55ce61709140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f130affd640 10 client.15432 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f132affd640 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ce6170a430 0x55ce61709140 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e99abc18b4feaa56 server_cookie=d00199987212d56f in_seq=0 out_seq=0 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f130affd640 1 -- 172.21.15.28:0/839529177 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ce61571b80 con 0x55ce6170a430 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f130affd640 10 client.15432 handle_command_reply: tid=0 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f1334e40740 1 client.15432 shutdown 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f1334e40740 1 -- 172.21.15.28:0/839529177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616888c0 msgr2=0x55ce61696090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f1334e40740 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616888c0 0x55ce61696090 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f131c031440 tx=0x7f131c005b50 comp rx=0 tx=0).stop 2024-05-09T08:26:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f1334e40740 1 -- 172.21.15.28:0/839529177 shutdown_connections 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f1334e40740 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ce6168e420 0x55ce61698ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f1334e40740 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55ce6170a430 0x55ce61709140 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f1334e40740 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ce616892b0 0x55ce616965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f1334e40740 1 --2- 172.21.15.28:0/839529177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce616888c0 0x55ce61696090 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f130affd640 0 client.15432 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.035+0000 7f1334e40740 1 -- 172.21.15.28:0/839529177 >> 172.21.15.28:0/839529177 conn(0x55ce61683f30 msgr2=0x55ce6168a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1334e40740 1 -- 172.21.15.28:0/839529177 shutdown_connections 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1334e40740 1 -- 172.21.15.28:0/839529177 wait complete. 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1334e40740 20 client.15432 trim_cache size 0 max 16384 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24463, 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 110.639423167, 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 161.01649578600001 2024-05-09T08:26:09.044 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 -- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f13180a2b90 msgr2=0x7f13180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f13180a2b90 0x7f13180a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55ce61689b10 tx=0x7f1324004b50 comp rx=0 tx=0).stop 2024-05-09T08:26:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 -- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 msgr2=0x7f132c197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 0x7f132c197d70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f131c031610 tx=0x7f131c0043b0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 -- 172.21.15.28:0/1447320024 shutdown_connections 2024-05-09T08:26:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f13180a2b90 0x7f13180a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f132c102340 0x7f132c19f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f132c101980 0x7f132c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 --2- 172.21.15.28:0/1447320024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100790 0x7f132c197d70 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 -- 172.21.15.28:0/1447320024 >> 172.21.15.28:0/1447320024 conn(0x7f132c0fbd20 msgr2=0x7f132c069f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 -- 172.21.15.28:0/1447320024 shutdown_connections 2024-05-09T08:26:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.036+0000 7f1332134640 1 -- 172.21.15.28:0/1447320024 wait complete. 2024-05-09T08:26:09.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-05-09T08:26:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 -- 172.21.15.28:0/2792400651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4100770 msgr2=0x7f49e4100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/2792400651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4100770 0x7f49e4100b90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009900 tx=0x7f49e002a410 comp rx=0 tx=0).stop 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 -- 172.21.15.28:0/2792400651 shutdown_connections 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/2792400651 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49e4102320 0x7f49e4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/2792400651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e4101960 0x7f49e4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/2792400651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4100770 0x7f49e4100b90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 -- 172.21.15.28:0/2792400651 >> 172.21.15.28:0/2792400651 conn(0x7f49e40fbce0 msgr2=0x7f49e40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 -- 172.21.15.28:0/2792400651 shutdown_connections 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 -- 172.21.15.28:0/2792400651 wait complete. 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 Processor -- start 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.154+0000 7f49ebb1b640 1 -- start start 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ebb1b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49e4100770 0x7f49e4197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ebb1b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e4101960 0x7f49e41981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ebb1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4102320 0x7f49e419f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ebb1b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49e410d510 con 0x7f49e4102320 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ebb1b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f49e410d390 con 0x7f49e4100770 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ebb1b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49e410d690 con 0x7f49e4101960 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ea0b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4102320 0x7f49e419f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ea0b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4102320 0x7f49e419f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58774/0 (socket says 172.21.15.28:58774) 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ea0b8640 1 -- 172.21.15.28:0/183104959 learned_addr learned my addr 172.21.15.28:0/183104959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49e90b6640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e4101960 0x7f49e41981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49e98b7640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49e4100770 0x7f49e4197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ea0b8640 1 -- 172.21.15.28:0/183104959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e4101960 msgr2=0x7f49e41981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ea0b8640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e4101960 0x7f49e41981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ea0b8640 1 -- 172.21.15.28:0/183104959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49e4100770 msgr2=0x7f49e4197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ea0b8640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49e4100770 0x7f49e4197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ea0b8640 1 -- 172.21.15.28:0/183104959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e419f870 con 0x7f49e4102320 2024-05-09T08:26:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49e98b7640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49e4100770 0x7f49e4197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ea0b8640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4102320 0x7f49e419f1d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00cad0 tx=0x7f49dc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49daffd640 1 -- 172.21.15.28:0/183104959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc00ae20 con 0x7f49e4102320 2024-05-09T08:26:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49daffd640 1 -- 172.21.15.28:0/183104959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49dc005020 con 0x7f49e4102320 2024-05-09T08:26:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49daffd640 1 -- 172.21.15.28:0/183104959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc00f6f0 con 0x7f49e4102320 2024-05-09T08:26:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ebb1b640 1 -- 172.21.15.28:0/183104959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e419fb60 con 0x7f49e4102320 2024-05-09T08:26:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.155+0000 7f49ebb1b640 1 -- 172.21.15.28:0/183104959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e41a00b0 con 0x7f49e4102320 2024-05-09T08:26:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.158+0000 7f49daffd640 1 -- 172.21.15.28:0/183104959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f49dc0051d0 con 0x7f49e4102320 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.158+0000 7f49ee82d740 1 Processor -- start 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.158+0000 7f49ee82d740 1 -- start start 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.158+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d46ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.158+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d4bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.158+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d4e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.158+0000 7f49ee82d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a514caa4d0 con 0x55a514d468c0 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.158+0000 7f49ee82d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a514c523e0 con 0x55a514d4c420 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49ee82d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a514ca6490 con 0x55a514d472b0 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d46ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d46ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58776/0 (socket says 172.21.15.28:58776) 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 -- 172.21.15.28:0/2026794329 learned_addr learned my addr 172.21.15.28:0/2026794329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49daffd640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49d00a2c10 0x7f49d00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e98b7640 1 --2- 172.21.15.28:0/2026794329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d4bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 --2- 172.21.15.28:0/2026794329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d4e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49ea0b8640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49d00a2c10 0x7f49d00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 -- 172.21.15.28:0/2026794329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 msgr2=0x55a514d4bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 --2- 172.21.15.28:0/2026794329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d4bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 -- 172.21.15.28:0/2026794329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 msgr2=0x55a514d4e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 --2- 172.21.15.28:0/2026794329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d4e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 -- 172.21.15.28:0/2026794329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5149be2d0 con 0x55a514d468c0 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49daffd640 1 -- 172.21.15.28:0/183104959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f49dc0cf580 con 0x7f49e4102320 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49e90b6640 1 --2- 172.21.15.28:0/2026794329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d46ce0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f49d40124f0 tx=0x7f49d400c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5ee4c0367db67c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49cffff640 1 -- 172.21.15.28:0/2026794329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4018030 con 0x55a514d468c0 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49ea0b8640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49d00a2c10 0x7f49d00a50d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f49dc09b980 tx=0x7f49dc086040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49cffff640 1 -- 172.21.15.28:0/2026794329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d4005500 con 0x55a514d468c0 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.159+0000 7f49cffff640 1 -- 172.21.15.28:0/2026794329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d40108a0 con 0x55a514d468c0 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.160+0000 7f49ee82d740 1 -- 172.21.15.28:0/2026794329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 msgr2=0x55a514d46ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.160+0000 7f49ee82d740 1 --2- 172.21.15.28:0/2026794329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d46ce0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f49d40124f0 tx=0x7f49d400c5e0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.160+0000 7f49ee82d740 1 -- 172.21.15.28:0/2026794329 shutdown_connections 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.160+0000 7f49ee82d740 1 --2- 172.21.15.28:0/2026794329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d4e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.160+0000 7f49ee82d740 1 --2- 172.21.15.28:0/2026794329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d4bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.160+0000 7f49ee82d740 1 --2- 172.21.15.28:0/2026794329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d46ce0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.160+0000 7f49ee82d740 1 -- 172.21.15.28:0/2026794329 >> 172.21.15.28:0/2026794329 conn(0x55a514d41f30 msgr2=0x55a514d42330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.160+0000 7f49ee82d740 1 -- 172.21.15.28:0/2026794329 shutdown_connections 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.160+0000 7f49ee82d740 1 -- 172.21.15.28:0/2026794329 wait complete. 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49ee82d740 1 -- start start 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49ee82d740 1 Processor -- start 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d468c0 0x55a514d50cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d472b0 0x55a514d511f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49ee82d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a514d4f3d0 con 0x55a514d472b0 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49ee82d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a514c658e0 con 0x55a514d4c420 2024-05-09T08:26:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49ee82d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a514d4f550 con 0x55a514d468c0 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49ee82d740 10 client.0 _ll_register_callbacks cb 0x55a514c329c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49e98b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d472b0 0x55a514d511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49e98b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d472b0 0x55a514d511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58790/0 (socket says 172.21.15.28:58790) 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49e98b7640 1 -- 172.21.15.28:0/3725047800 learned_addr learned my addr 172.21.15.28:0/3725047800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.161+0000 7f49e90b6640 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d468c0 0x55a514d50cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49e90b6640 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49e98b7640 1 -- 172.21.15.28:0/3725047800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d468c0 msgr2=0x55a514d50cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49e98b7640 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d468c0 0x55a514d50cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49e98b7640 1 -- 172.21.15.28:0/3725047800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 msgr2=0x55a514d581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49e98b7640 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49e98b7640 1 -- 172.21.15.28:0/3725047800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a514a886b0 con 0x55a514d472b0 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49e90b6640 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49e98b7640 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d472b0 0x55a514d511f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f49e0031440 tx=0x7f49e0031aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bffd781033b7f865 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49cd7fa640 1 -- 172.21.15.28:0/3725047800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e003f070 con 0x55a514d472b0 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49cd7fa640 1 -- 172.21.15.28:0/3725047800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f49e0005030 con 0x55a514d472b0 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49ee82d740 1 -- 172.21.15.28:0/3725047800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d4005500 con 0x55a514d472b0 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49cd7fa640 1 -- 172.21.15.28:0/3725047800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49e004c050 con 0x55a514d472b0 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49ee82d740 10 client.15444 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49ee82d740 1 -- 172.21.15.28:0/3725047800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a514d5b260 con 0x55a514d472b0 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49cd7fa640 1 -- 172.21.15.28:0/3725047800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d4005500 con 0x55a514d472b0 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49cd7fa640 1 -- 172.21.15.28:0/3725047800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f49e00035f0 con 0x55a514d472b0 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.162+0000 7f49ee82d740 10 client.15444 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.163+0000 7f49ee82d740 10 client.15444 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.163+0000 7f49ee82d740 20 client.15444 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.163+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x55a514d5d830 0x55a514d5eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.163+0000 7f49ee82d740 4 client.15444 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.163+0000 7f49ee82d740 1 -- 172.21.15.28:0/3725047800 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a514cb7c00 con 0x55a514d5d830 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.163+0000 7f49e90b6640 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x55a514d5d830 0x55a514d5eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.163+0000 7f49e90b6640 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x55a514d5d830 0x55a514d5eca0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f207e17bfead6c3e server_cookie=6bf58f245d3cc509 in_seq=0 out_seq=0 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.163+0000 7f49cd7fa640 10 client.15444 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49cd7fa640 1 -- 172.21.15.28:0/3725047800 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a514cb7c00 con 0x55a514d5d830 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49cd7fa640 10 client.15444 handle_command_reply: tid=0 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 client.15444 shutdown 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 -- 172.21.15.28:0/3725047800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d472b0 msgr2=0x55a514d511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d472b0 0x55a514d511f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f49e0031440 tx=0x7f49e0031aa0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 -- 172.21.15.28:0/3725047800 shutdown_connections 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x55a514d5d830 0x55a514d5eca0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d472b0 0x55a514d511f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49cd7fa640 0 client.15444 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3725047800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d468c0 0x55a514d50cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 -- 172.21.15.28:0/3725047800 >> 172.21.15.28:0/3725047800 conn(0x55a514d41f30 msgr2=0x55a514d41000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 -- 172.21.15.28:0/3725047800 shutdown_connections 2024-05-09T08:26:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 1 -- 172.21.15.28:0/3725047800 wait complete. 2024-05-09T08:26:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.164+0000 7f49ee82d740 20 client.15444 trim_cache size 0 max 16384 2024-05-09T08:26:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49ee82d740 1 Processor -- start 2024-05-09T08:26:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49ee82d740 1 -- start start 2024-05-09T08:26:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d4e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d42310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49ee82d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a514d5cfe0 con 0x55a514d468c0 2024-05-09T08:26:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49ee82d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a514d5d6b0 con 0x55a514d4c420 2024-05-09T08:26:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49ee82d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a514d5dd80 con 0x55a514d472b0 2024-05-09T08:26:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e90b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d4e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e90b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d4e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58800/0 (socket says 172.21.15.28:58800) 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e90b6640 1 -- 172.21.15.28:0/3339007379 learned_addr learned my addr 172.21.15.28:0/3339007379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e98b7640 1 --2- 172.21.15.28:0/3339007379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e90b6640 1 -- 172.21.15.28:0/3339007379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 msgr2=0x55a514d580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e90b6640 1 --2- 172.21.15.28:0/3339007379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e90b6640 1 -- 172.21.15.28:0/3339007379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 msgr2=0x55a514d42310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e90b6640 1 --2- 172.21.15.28:0/3339007379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d42310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e90b6640 1 -- 172.21.15.28:0/3339007379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a514d51e10 con 0x55a514d468c0 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e98b7640 1 --2- 172.21.15.28:0/3339007379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.183+0000 7f49e90b6640 1 --2- 172.21.15.28:0/3339007379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d4e810 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x55a514d47b10 tx=0x7f49d400c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4382389526c684c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49cf7fe640 1 -- 172.21.15.28:0/3339007379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4018040 con 0x55a514d468c0 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49cf7fe640 1 -- 172.21.15.28:0/3339007379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d4019040 con 0x55a514d468c0 2024-05-09T08:26:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49cf7fe640 1 -- 172.21.15.28:0/3339007379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d401d460 con 0x55a514d468c0 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49ee82d740 1 -- 172.21.15.28:0/3339007379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 msgr2=0x55a514d4e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3339007379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d4e810 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x55a514d47b10 tx=0x7f49d400c5e0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49ee82d740 1 -- 172.21.15.28:0/3339007379 shutdown_connections 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3339007379 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d42310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3339007379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49ee82d740 1 --2- 172.21.15.28:0/3339007379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d4e810 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49ee82d740 1 -- 172.21.15.28:0/3339007379 >> 172.21.15.28:0/3339007379 conn(0x55a514d41f30 msgr2=0x55a514d47b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49ee82d740 1 -- 172.21.15.28:0/3339007379 shutdown_connections 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.184+0000 7f49ee82d740 1 -- 172.21.15.28:0/3339007379 wait complete. 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49ee82d740 1 -- start start 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49ee82d740 1 Processor -- start 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d53a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d53f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49ee82d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d56f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49ee82d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a514d5eb20 con 0x55a514d468c0 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49ee82d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a514d5e450 con 0x55a514d4c420 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49ee82d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a514d5f1f0 con 0x55a514d472b0 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49ee82d740 10 client.0 _ll_register_callbacks cb 0x7f49e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e90b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d53a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e90b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d53a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58808/0 (socket says 172.21.15.28:58808) 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e90b6640 1 -- 172.21.15.28:0/282718735 learned_addr learned my addr 172.21.15.28:0/282718735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e98b7640 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d53f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e90b6640 1 -- 172.21.15.28:0/282718735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 msgr2=0x55a514d53f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e90b6640 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d53f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e90b6640 1 -- 172.21.15.28:0/282718735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 msgr2=0x55a514d56f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e90b6640 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d56f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e90b6640 1 -- 172.21.15.28:0/282718735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a514d57550 con 0x55a514d468c0 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.185+0000 7f49e98b7640 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d53f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49e90b6640 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d53a50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f49d4005ed0 tx=0x7f49d4005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e933f7b90412d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49cd7fa640 1 -- 172.21.15.28:0/282718735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4037070 con 0x55a514d468c0 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49cd7fa640 1 -- 172.21.15.28:0/282718735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f49d402a470 con 0x55a514d468c0 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49ee82d740 1 -- 172.21.15.28:0/282718735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d4019040 con 0x55a514d468c0 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49cd7fa640 1 -- 172.21.15.28:0/282718735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49d4043050 con 0x55a514d468c0 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49ee82d740 10 client.15450 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49ee82d740 1 -- 172.21.15.28:0/282718735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a514dc37e0 con 0x55a514d468c0 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49cd7fa640 1 -- 172.21.15.28:0/282718735 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d4019040 con 0x55a514d468c0 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49cd7fa640 1 -- 172.21.15.28:0/282718735 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f49d4033420 con 0x55a514d468c0 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49ee82d740 10 client.15450 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49ee82d740 10 client.15450 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49ee82d740 20 client.15450 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49ee82d740 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x55a514dc7c50 0x55a514dc6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49ee82d740 4 client.15450 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:26:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.186+0000 7f49ee82d740 1 -- 172.21.15.28:0/282718735 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "status"}) -- 0x55a514cb1f50 con 0x55a514dc7c50 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.187+0000 7f49e98b7640 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x55a514dc7c50 0x55a514dc6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.187+0000 7f49cd7fa640 10 client.15450 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.187+0000 7f49e98b7640 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x55a514dc7c50 0x55a514dc6960 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e241a90d8080809f server_cookie=c5c8a242570a0871 in_seq=0 out_seq=0 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.187+0000 7f49cd7fa640 1 -- 172.21.15.28:0/282718735 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55a514cb1f50 con 0x55a514dc7c50 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.187+0000 7f49cd7fa640 10 client.15450 handle_command_reply: tid=0 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.187+0000 7f49ee82d740 1 client.15450 shutdown 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.187+0000 7f49ee82d740 1 -- 172.21.15.28:0/282718735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 msgr2=0x55a514d53a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.187+0000 7f49ee82d740 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d53a50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f49d4005ed0 tx=0x7f49d4005f00 comp rx=0 tx=0).stop 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ee82d740 1 -- 172.21.15.28:0/282718735 shutdown_connections 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ee82d740 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x55a514dc7c50 0x55a514dc6960 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ee82d740 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a514d4c420 0x55a514d56f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49cd7fa640 0 client.15450 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:26:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ee82d740 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a514d472b0 0x55a514d53f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ee82d740 1 --2- 172.21.15.28:0/282718735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a514d468c0 0x55a514d53a50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ee82d740 1 -- 172.21.15.28:0/282718735 >> 172.21.15.28:0/282718735 conn(0x55a514d41f30 msgr2=0x55a514d42310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ee82d740 1 -- 172.21.15.28:0/282718735 shutdown_connections 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ee82d740 1 -- 172.21.15.28:0/282718735 wait complete. 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ee82d740 20 client.15450 trim_cache size 0 max 16384 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24476, 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:26:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:26:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:26:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 111.80312269, 2024-05-09T08:26:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:26:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-05-09T08:26:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:26:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 155.36598418400001 2024-05-09T08:26:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 -- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49d00a2c10 msgr2=0x7f49d00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49d00a2c10 0x7f49d00a50d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f49dc09b980 tx=0x7f49dc086040 comp rx=0 tx=0).stop 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 -- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4102320 msgr2=0x7f49e419f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4102320 0x7f49e419f1d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00cad0 tx=0x7f49dc00cf90 comp rx=0 tx=0).stop 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 -- 172.21.15.28:0/183104959 shutdown_connections 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f49d00a2c10 0x7f49d00a50d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e4102320 0x7f49e419f1d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49e4101960 0x7f49e41981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 --2- 172.21.15.28:0/183104959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49e4100770 0x7f49e4197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 -- 172.21.15.28:0/183104959 >> 172.21.15.28:0/183104959 conn(0x7f49e40fbce0 msgr2=0x7f49e4104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 -- 172.21.15.28:0/183104959 shutdown_connections 2024-05-09T08:26:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.188+0000 7f49ebb1b640 1 -- 172.21.15.28:0/183104959 wait complete. 2024-05-09T08:26:09.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T08:26:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.305+0000 7f1bd9615640 1 -- 172.21.15.28:0/2678782284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 msgr2=0x7f1bd4102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.305+0000 7f1bd9615640 1 --2- 172.21.15.28:0/2678782284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 0x7f1bd4102f00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009900 tx=0x7f1bc802a410 comp rx=0 tx=0).stop 2024-05-09T08:26:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.305+0000 7f1bd9615640 1 -- 172.21.15.28:0/2678782284 shutdown_connections 2024-05-09T08:26:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.305+0000 7f1bd9615640 1 --2- 172.21.15.28:0/2678782284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bd4103440 0x7f1bd4109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.305+0000 7f1bd9615640 1 --2- 172.21.15.28:0/2678782284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 0x7f1bd4102f00 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.305+0000 7f1bd9615640 1 --2- 172.21.15.28:0/2678782284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1bd4101890 0x7f1bd4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.305+0000 7f1bd9615640 1 -- 172.21.15.28:0/2678782284 >> 172.21.15.28:0/2678782284 conn(0x7f1bd40fce00 msgr2=0x7f1bd40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.305+0000 7f1bd9615640 1 -- 172.21.15.28:0/2678782284 shutdown_connections 2024-05-09T08:26:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.305+0000 7f1bd9615640 1 -- 172.21.15.28:0/2678782284 wait complete. 2024-05-09T08:26:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.306+0000 7f1bd9615640 1 Processor -- start 2024-05-09T08:26:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.306+0000 7f1bd9615640 1 -- start start 2024-05-09T08:26:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.306+0000 7f1bd9615640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bd4101890 0x7f1bd4197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.306+0000 7f1bd9615640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 0x7f1bd41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.306+0000 7f1bd9615640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1bd4103440 0x7f1bd419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.306+0000 7f1bd9615640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd410d510 con 0x7f1bd4102a80 2024-05-09T08:26:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.306+0000 7f1bd9615640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd410d390 con 0x7f1bd4103440 2024-05-09T08:26:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.306+0000 7f1bd9615640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd410d690 con 0x7f1bd4101890 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 0x7f1bd41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 0x7f1bd41980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58832/0 (socket says 172.21.15.28:58832) 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd27fc640 1 -- 172.21.15.28:0/1508568293 learned_addr learned my addr 172.21.15.28:0/1508568293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd2ffd640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bd4101890 0x7f1bd4197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd27fc640 1 -- 172.21.15.28:0/1508568293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bd4101890 msgr2=0x7f1bd4197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1bd4103440 0x7f1bd419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bd4101890 0x7f1bd4197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd27fc640 1 -- 172.21.15.28:0/1508568293 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1bd4103440 msgr2=0x7f1bd419f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1bd4103440 0x7f1bd419f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd27fc640 1 -- 172.21.15.28:0/1508568293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd419f7a0 con 0x7f1bd4102a80 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1bd4103440 0x7f1bd419f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 0x7f1bd41980c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009230 tx=0x7f1bc8004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bbbfff640 1 -- 172.21.15.28:0/1508568293 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc803f030 con 0x7f1bd4102a80 2024-05-09T08:26:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd9615640 1 -- 172.21.15.28:0/1508568293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd419fa30 con 0x7f1bd4102a80 2024-05-09T08:26:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bbbfff640 1 -- 172.21.15.28:0/1508568293 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bc8003ce0 con 0x7f1bd4102a80 2024-05-09T08:26:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bbbfff640 1 -- 172.21.15.28:0/1508568293 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8035070 con 0x7f1bd4102a80 2024-05-09T08:26:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.307+0000 7f1bd9615640 1 -- 172.21.15.28:0/1508568293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd419ff20 con 0x7f1bd4102a80 2024-05-09T08:26:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bbbfff640 1 -- 172.21.15.28:0/1508568293 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1bc8042030 con 0x7f1bd4102a80 2024-05-09T08:26:09.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bdc403740 1 Processor -- start 2024-05-09T08:26:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bdc403740 1 -- start start 2024-05-09T08:26:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3acce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bdc403740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56094d3104d0 con 0x56094d3b2420 2024-05-09T08:26:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bdc403740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56094d2b83e0 con 0x56094d3ac8c0 2024-05-09T08:26:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bdc403740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56094d30c490 con 0x56094d3ad2b0 2024-05-09T08:26:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bd27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bd27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58834/0 (socket says 172.21.15.28:58834) 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bd27fc640 1 -- 172.21.15.28:0/3745515790 learned_addr learned my addr 172.21.15.28:0/3745515790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bbbfff640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1bbc0a2b90 0x7f1bbc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.310+0000 7f1bd2ffd640 1 --2- 172.21.15.28:0/3745515790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/3745515790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bbbfff640 1 -- 172.21.15.28:0/1508568293 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f1bc80efed0 con 0x7f1bd4102a80 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bd27fc640 1 -- 172.21.15.28:0/3745515790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 msgr2=0x56094d3b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1bbc0a2b90 0x7f1bbc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/3745515790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3b1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bd27fc640 1 -- 172.21.15.28:0/3745515790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 msgr2=0x56094d3acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/3745515790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bd27fc640 1 -- 172.21.15.28:0/3745515790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56094d0242d0 con 0x56094d3b2420 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/3745515790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3b48b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1bc80ed5e0 tx=0x7f1bc8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c5e1dfcdcaabcd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bb97fa640 1 -- 172.21.15.28:0/3745515790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8049070 con 0x56094d3b2420 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1bbc0a2b90 0x7f1bbc0a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc009270 tx=0x7f1bcc005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bb97fa640 1 -- 172.21.15.28:0/3745515790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bc80f35c0 con 0x56094d3b2420 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.311+0000 7f1bb97fa640 1 -- 172.21.15.28:0/3745515790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc80ba640 con 0x56094d3b2420 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 -- 172.21.15.28:0/3745515790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 msgr2=0x56094d3b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 --2- 172.21.15.28:0/3745515790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3b48b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1bc80ed5e0 tx=0x7f1bc8009260 comp rx=0 tx=0).stop 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 -- 172.21.15.28:0/3745515790 shutdown_connections 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 --2- 172.21.15.28:0/3745515790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3b48b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 --2- 172.21.15.28:0/3745515790 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 --2- 172.21.15.28:0/3745515790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 -- 172.21.15.28:0/3745515790 >> 172.21.15.28:0/3745515790 conn(0x56094d3a7f30 msgr2=0x56094d3a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 -- 172.21.15.28:0/3745515790 shutdown_connections 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 -- 172.21.15.28:0/3745515790 wait complete. 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 -- start start 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 Processor -- start 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ad2b0 0x56094d3b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3b2420 0x56094d3bdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56094d3b5190 con 0x56094d3ad2b0 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56094d2cb8e0 con 0x56094d3ac8c0 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.312+0000 7f1bdc403740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56094d3b5310 con 0x56094d3b2420 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bdc403740 10 client.0 _ll_register_callbacks cb 0x56094d2989c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ad2b0 0x56094d3b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ad2b0 0x56094d3b6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58844/0 (socket says 172.21.15.28:58844) 2024-05-09T08:26:09.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd27fc640 1 -- 172.21.15.28:0/1671437572 learned_addr learned my addr 172.21.15.28:0/1671437572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3b2420 0x56094d3bdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd2ffd640 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd27fc640 1 -- 172.21.15.28:0/1671437572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3b2420 msgr2=0x56094d3bdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3b2420 0x56094d3bdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd27fc640 1 -- 172.21.15.28:0/1671437572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 msgr2=0x56094d3b69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3b69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd27fc640 1 -- 172.21.15.28:0/1671437572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56094d0ee6b0 con 0x56094d3ad2b0 2024-05-09T08:26:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3b2420 0x56094d3bdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ad2b0 0x56094d3b6f10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009a20 tx=0x7f1bc80095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d746335928b892 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bb6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1671437572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8049070 con 0x56094d3ad2b0 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1671437572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f1bc803b460 con 0x56094d3ad2b0 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bdc403740 1 -- 172.21.15.28:0/1671437572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56094d31dc00 con 0x56094d3ad2b0 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1671437572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1bc80bca40 con 0x56094d3ad2b0 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bdc403740 10 client.15462 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bdc403740 1 -- 172.21.15.28:0/1671437572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56094d3b5190 con 0x56094d3ad2b0 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.313+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1671437572 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56094d31dc00 con 0x56094d3ad2b0 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1671437572 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f1bc80c4b20 con 0x56094d3ad2b0 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bdc403740 10 client.15462 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bdc403740 10 client.15462 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bdc403740 20 client.15462 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x56094d3c34d0 0x56094d3c4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bdc403740 4 client.15462 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bdc403740 1 -- 172.21.15.28:0/1671437572 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56094d317f50 con 0x56094d3c34d0 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bd2ffd640 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x56094d3c34d0 0x56094d3c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bb6ff5640 10 client.15462 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.314+0000 7f1bd2ffd640 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x56094d3c34d0 0x56094d3c4940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=56c2f3bb5714c64b server_cookie=4216efccd7584100 in_seq=0 out_seq=0 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1671437572 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56094d317f50 con 0x56094d3c34d0 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bb6ff5640 10 client.15462 handle_command_reply: tid=0 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 client.15462 shutdown 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 -- 172.21.15.28:0/1671437572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ad2b0 msgr2=0x56094d3b6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ad2b0 0x56094d3b6f10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009a20 tx=0x7f1bc80095d0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 -- 172.21.15.28:0/1671437572 shutdown_connections 2024-05-09T08:26:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x56094d3c34d0 0x56094d3c4940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3b2420 0x56094d3bdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ad2b0 0x56094d3b6f10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bb6ff5640 0 client.15462 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:26:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1671437572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3b69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 -- 172.21.15.28:0/1671437572 >> 172.21.15.28:0/1671437572 conn(0x56094d3a7f30 msgr2=0x56094d3a7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 -- 172.21.15.28:0/1671437572 shutdown_connections 2024-05-09T08:26:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 1 -- 172.21.15.28:0/1671437572 wait complete. 2024-05-09T08:26:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.315+0000 7f1bdc403740 20 client.15462 trim_cache size 0 max 16384 2024-05-09T08:26:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bdc403740 1 Processor -- start 2024-05-09T08:26:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bdc403740 1 -- start start 2024-05-09T08:26:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3ab730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bdc403740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56094d3c2c80 con 0x56094d3b2420 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bdc403740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56094d3c3350 con 0x56094d3ac8c0 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bdc403740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56094d3c3a20 con 0x56094d3ad2b0 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3ab730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58856/0 (socket says 172.21.15.28:58856) 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd37fe640 1 -- 172.21.15.28:0/2833596407 learned_addr learned my addr 172.21.15.28:0/2833596407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/2833596407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd37fe640 1 -- 172.21.15.28:0/2833596407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 msgr2=0x56094d3ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/2833596407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3ab1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd2ffd640 1 --2- 172.21.15.28:0/2833596407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd37fe640 1 -- 172.21.15.28:0/2833596407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 msgr2=0x56094d3aacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/2833596407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3aacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.334+0000 7f1bd37fe640 1 -- 172.21.15.28:0/2833596407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56094d3a8980 con 0x56094d3b2420 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/2833596407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3ab730 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00f830 tx=0x7f1bcc0096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9851d02be55b75b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bb8ff9640 1 -- 172.21.15.28:0/2833596407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc018040 con 0x56094d3b2420 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bb8ff9640 1 -- 172.21.15.28:0/2833596407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bcc011e50 con 0x56094d3b2420 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bb8ff9640 1 -- 172.21.15.28:0/2833596407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc0203e0 con 0x56094d3b2420 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bdc403740 1 -- 172.21.15.28:0/2833596407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 msgr2=0x56094d3ab730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bdc403740 1 --2- 172.21.15.28:0/2833596407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3ab730 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00f830 tx=0x7f1bcc0096f0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bdc403740 1 -- 172.21.15.28:0/2833596407 shutdown_connections 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bdc403740 1 --2- 172.21.15.28:0/2833596407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3b2420 0x56094d3ab730 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bdc403740 1 --2- 172.21.15.28:0/2833596407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bdc403740 1 --2- 172.21.15.28:0/2833596407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3ac8c0 0x56094d3aacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bdc403740 1 -- 172.21.15.28:0/2833596407 >> 172.21.15.28:0/2833596407 conn(0x56094d3a7f30 msgr2=0x56094d3c20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bdc403740 1 -- 172.21.15.28:0/2833596407 shutdown_connections 2024-05-09T08:26:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.335+0000 7f1bdc403740 1 -- 172.21.15.28:0/2833596407 wait complete. 2024-05-09T08:26:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bdc403740 1 -- start start 2024-05-09T08:26:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bdc403740 1 Processor -- start 2024-05-09T08:26:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ac8c0 0x56094d3b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3b9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bdc403740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3b2420 0x56094d3bc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bdc403740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56094d3c47c0 con 0x56094d3ac8c0 2024-05-09T08:26:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bdc403740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56094d3c40f0 con 0x56094d3b2420 2024-05-09T08:26:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bdc403740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56094d3c4e90 con 0x56094d3ad2b0 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bdc403740 10 client.0 _ll_register_callbacks cb 0x7f1bc40092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ac8c0 0x56094d3b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ac8c0 0x56094d3b8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58866/0 (socket says 172.21.15.28:58866) 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd2ffd640 1 -- 172.21.15.28:0/1702280774 learned_addr learned my addr 172.21.15.28:0/1702280774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3b2420 0x56094d3bc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd2ffd640 1 -- 172.21.15.28:0/1702280774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 msgr2=0x56094d3b9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd2ffd640 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3b9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd2ffd640 1 -- 172.21.15.28:0/1702280774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3b2420 msgr2=0x56094d3bc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd2ffd640 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3b2420 0x56094d3bc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd2ffd640 1 -- 172.21.15.28:0/1702280774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56094d3bc940 con 0x56094d3ac8c0 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd37fe640 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3b2420 0x56094d3bc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bd2ffd640 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ac8c0 0x56094d3b8df0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x56094d3ad6b0 tx=0x7f1bc4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c693ad38f86e2ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.336+0000 7f1bb6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1702280774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4005020 con 0x56094d3ac8c0 2024-05-09T08:26:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1702280774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f1bc40051c0 con 0x56094d3ac8c0 2024-05-09T08:26:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bdc403740 1 -- 172.21.15.28:0/1702280774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bcc011e50 con 0x56094d3ac8c0 2024-05-09T08:26:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1702280774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1bc402b050 con 0x56094d3ac8c0 2024-05-09T08:26:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bdc403740 10 client.15468 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bdc403740 1 -- 172.21.15.28:0/1702280774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56094d4297a0 con 0x56094d3ac8c0 2024-05-09T08:26:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1702280774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bcc011e50 con 0x56094d3ac8c0 2024-05-09T08:26:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1702280774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f1bc400f040 con 0x56094d3ac8c0 2024-05-09T08:26:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bdc403740 10 client.15468 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bdc403740 10 client.15468 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:26:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bdc403740 20 client.15468 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x56094d42e430 0x56094d42a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bdc403740 4 client.15468 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:26:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.337+0000 7f1bdc403740 1 -- 172.21.15.28:0/1702280774 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "status"}) -- 0x56094d295b80 con 0x56094d42e430 2024-05-09T08:26:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x56094d42e430 0x56094d42a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bb6ff5640 10 client.15468 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:26:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bd27fc640 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x56094d42e430 0x56094d42a870 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dcbe25579c16465b server_cookie=7d1d8debca2c92cd in_seq=0 out_seq=0 2024-05-09T08:26:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bb6ff5640 1 -- 172.21.15.28:0/1702280774 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56094d295b80 con 0x56094d42e430 2024-05-09T08:26:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bb6ff5640 10 client.15468 handle_command_reply: tid=0 2024-05-09T08:26:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bdc403740 1 client.15468 shutdown 2024-05-09T08:26:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bdc403740 1 -- 172.21.15.28:0/1702280774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ac8c0 msgr2=0x56094d3b8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ac8c0 0x56094d3b8df0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x56094d3ad6b0 tx=0x7f1bc4012820 comp rx=0 tx=0).stop 2024-05-09T08:26:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bdc403740 1 -- 172.21.15.28:0/1702280774 shutdown_connections 2024-05-09T08:26:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x56094d42e430 0x56094d42a870 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56094d3b2420 0x56094d3bc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56094d3ad2b0 0x56094d3b9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bb6ff5640 0 client.15468 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:26:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bdc403740 1 --2- 172.21.15.28:0/1702280774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56094d3ac8c0 0x56094d3b8df0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.338+0000 7f1bdc403740 1 -- 172.21.15.28:0/1702280774 >> 172.21.15.28:0/1702280774 conn(0x56094d3a7f30 msgr2=0x56094d3a6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bdc403740 1 -- 172.21.15.28:0/1702280774 shutdown_connections 2024-05-09T08:26:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bdc403740 1 -- 172.21.15.28:0/1702280774 wait complete. 2024-05-09T08:26:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bdc403740 20 client.15468 trim_cache size 0 max 16384 2024-05-09T08:26:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:26:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-05-09T08:26:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24481, 2024-05-09T08:26:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:26:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:26:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:26:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 112.96050912299999, 2024-05-09T08:26:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:26:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-05-09T08:26:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:26:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 158.43444665199999 2024-05-09T08:26:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 -- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1bbc0a2b90 msgr2=0x7f1bbc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1bbc0a2b90 0x7f1bbc0a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc009270 tx=0x7f1bcc005bd0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 -- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 msgr2=0x7f1bd41980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 0x7f1bd41980c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009230 tx=0x7f1bc8004360 comp rx=0 tx=0).stop 2024-05-09T08:26:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 -- 172.21.15.28:0/1508568293 shutdown_connections 2024-05-09T08:26:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f1bbc0a2b90 0x7f1bbc0a5050 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1bd4103440 0x7f1bd419f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd4102a80 0x7f1bd41980c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 --2- 172.21.15.28:0/1508568293 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bd4101890 0x7f1bd4197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 -- 172.21.15.28:0/1508568293 >> 172.21.15.28:0/1508568293 conn(0x7f1bd40fce00 msgr2=0x7f1bd4105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 -- 172.21.15.28:0/1508568293 shutdown_connections 2024-05-09T08:26:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.339+0000 7f1bd9615640 1 -- 172.21.15.28:0/1508568293 wait complete. 2024-05-09T08:26:09.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-09T08:26:09.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.462+0000 7f4f6a97e640 1 -- 172.21.15.28:0/3419600491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 msgr2=0x7f4f64104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.462+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/3419600491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 0x7f4f64104780 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4f54009900 tx=0x7f4f5402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:09.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.462+0000 7f4f6a97e640 1 -- 172.21.15.28:0/3419600491 shutdown_connections 2024-05-09T08:26:09.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.462+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/3419600491 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f641075f0 0x7f4f641099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.462+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/3419600491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f64104cc0 0x7f4f641070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.462+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/3419600491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 0x7f4f64104780 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.462+0000 7f4f6a97e640 1 -- 172.21.15.28:0/3419600491 >> 172.21.15.28:0/3419600491 conn(0x7f4f640fbce0 msgr2=0x7f4f640fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.462+0000 7f4f6a97e640 1 -- 172.21.15.28:0/3419600491 shutdown_connections 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.463+0000 7f4f6a97e640 1 -- 172.21.15.28:0/3419600491 wait complete. 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.463+0000 7f4f6a97e640 1 Processor -- start 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.463+0000 7f4f6a97e640 1 -- start start 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.463+0000 7f4f6a97e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 0x7f4f64101d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.463+0000 7f4f6a97e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f64104cc0 0x7f4f64100340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.463+0000 7f4f6a97e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f641075f0 0x7f4f64100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.463+0000 7f4f6a97e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6410d510 con 0x7f4f64102390 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.463+0000 7f4f6a97e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6410d390 con 0x7f4f64104cc0 2024-05-09T08:26:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.463+0000 7f4f6a97e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6410d690 con 0x7f4f641075f0 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 0x7f4f64101d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 0x7f4f64101d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58870/0 (socket says 172.21.15.28:58870) 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f63fff640 1 -- 172.21.15.28:0/505769245 learned_addr learned my addr 172.21.15.28:0/505769245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f641075f0 0x7f4f64100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f637fe640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f64104cc0 0x7f4f64100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f63fff640 1 -- 172.21.15.28:0/505769245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f641075f0 msgr2=0x7f4f64100880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f63fff640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f641075f0 0x7f4f64100880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f63fff640 1 -- 172.21.15.28:0/505769245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f64104cc0 msgr2=0x7f4f64100340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f63fff640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f64104cc0 0x7f4f64100340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f63fff640 1 -- 172.21.15.28:0/505769245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f64101110 con 0x7f4f64102390 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f641075f0 0x7f4f64100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f63fff640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 0x7f4f64101d00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4f54009230 tx=0x7f4f54004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f617fa640 1 -- 172.21.15.28:0/505769245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5403f030 con 0x7f4f64102390 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f617fa640 1 -- 172.21.15.28:0/505769245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f54003ce0 con 0x7f4f64102390 2024-05-09T08:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f6a97e640 1 -- 172.21.15.28:0/505769245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f641a1cd0 con 0x7f4f64102390 2024-05-09T08:26:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f617fa640 1 -- 172.21.15.28:0/505769245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f54035070 con 0x7f4f64102390 2024-05-09T08:26:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.464+0000 7f4f6a97e640 1 -- 172.21.15.28:0/505769245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f641a21c0 con 0x7f4f64102390 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f617fa640 1 -- 172.21.15.28:0/505769245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4f54042030 con 0x7f4f64102390 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f6d76f740 1 Processor -- start 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f6d76f740 1 -- start start 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e58c0 0x560cce4e5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4eadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4eb420 0x560cce4ed8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cce4494d0 con 0x560cce4e62b0 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560cce3f13e0 con 0x560cce4e58c0 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560cce445490 con 0x560cce4eb420 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.467+0000 7f4f63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4eadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4eadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58872/0 (socket says 172.21.15.28:58872) 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f63fff640 1 -- 172.21.15.28:0/2574953235 learned_addr learned my addr 172.21.15.28:0/2574953235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e58c0 0x560cce4e5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f637fe640 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4eb420 0x560cce4ed8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f63fff640 1 -- 172.21.15.28:0/2574953235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4eb420 msgr2=0x560cce4ed8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f63fff640 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4eb420 0x560cce4ed8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f63fff640 1 -- 172.21.15.28:0/2574953235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e58c0 msgr2=0x560cce4e5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f63fff640 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e58c0 0x560cce4e5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f63fff640 1 -- 172.21.15.28:0/2574953235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cce15d2d0 con 0x560cce4e62b0 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f617fa640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4f4c0a2c10 0x7f4f4c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e58c0 0x560cce4e5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4f4c0a2c10 0x7f4f4c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f617fa640 1 -- 172.21.15.28:0/505769245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f4f540efb20 con 0x7f4f64102390 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f63fff640 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4eadb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4f540f3c50 tx=0x7f4f540b85a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda6f9b1440cd189 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f4a7fc640 1 -- 172.21.15.28:0/2574953235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f54049070 con 0x560cce4e62b0 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f4a7fc640 1 -- 172.21.15.28:0/2574953235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f540ba030 con 0x560cce4e62b0 2024-05-09T08:26:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.468+0000 7f4f4a7fc640 1 -- 172.21.15.28:0/2574953235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f540bbd00 con 0x560cce4e62b0 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 -- 172.21.15.28:0/2574953235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 msgr2=0x560cce4eadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4f4c0a2c10 0x7f4f4c0a50d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009690 tx=0x7f4f5c009620 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4eadb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4f540f3c50 tx=0x7f4f540b85a0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 -- 172.21.15.28:0/2574953235 shutdown_connections 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4eb420 0x560cce4ed8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4eadb0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/2574953235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e58c0 0x560cce4e5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 -- 172.21.15.28:0/2574953235 >> 172.21.15.28:0/2574953235 conn(0x560cce4e0f30 msgr2=0x560cce4e1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 -- 172.21.15.28:0/2574953235 shutdown_connections 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 -- 172.21.15.28:0/2574953235 wait complete. 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.469+0000 7f4f6d76f740 1 -- start start 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f6d76f740 1 Processor -- start 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e62b0 0x560cce4eff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4eb420 0x560cce4f6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cce4ee190 con 0x560cce4eb420 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560cce4048e0 con 0x560cce4e62b0 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560cce4ee310 con 0x560cce4e58c0 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f6d76f740 10 client.0 _ll_register_callbacks cb 0x560cce3d19c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f68f1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4eb420 0x560cce4f6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f68f1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4eb420 0x560cce4f6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58888/0 (socket says 172.21.15.28:58888) 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f68f1b640 1 -- 172.21.15.28:0/1079115136 learned_addr learned my addr 172.21.15.28:0/1079115136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f63fff640 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f637fe640 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e62b0 0x560cce4eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f68f1b640 1 -- 172.21.15.28:0/1079115136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 msgr2=0x560cce4ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4ef9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f68f1b640 1 -- 172.21.15.28:0/1079115136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e62b0 msgr2=0x560cce4eff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e62b0 0x560cce4eff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.470+0000 7f4f68f1b640 1 -- 172.21.15.28:0/1079115136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cce2276b0 con 0x560cce4eb420 2024-05-09T08:26:09.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4eb420 0x560cce4f6f50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c012010 tx=0x7f4f5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6ed1900148b846 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f437fe640 1 -- 172.21.15.28:0/1079115136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c00fe60 con 0x560cce4eb420 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f437fe640 1 -- 172.21.15.28:0/1079115136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f4f5c003970 con 0x560cce4eb420 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f6d76f740 1 -- 172.21.15.28:0/1079115136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f540ba030 con 0x560cce4eb420 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f437fe640 1 -- 172.21.15.28:0/1079115136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4f5c02f050 con 0x560cce4eb420 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f6d76f740 10 client.15480 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f6d76f740 1 -- 172.21.15.28:0/1079115136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cce4fa000 con 0x560cce4eb420 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f437fe640 1 -- 172.21.15.28:0/1079115136 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f540ba030 con 0x560cce4eb420 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f437fe640 1 -- 172.21.15.28:0/1079115136 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4f5c01f440 con 0x560cce4eb420 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f6d76f740 10 client.15480 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f6d76f740 10 client.15480 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f6d76f740 20 client.15480 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x560cce4fc730 0x560cce4fdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f6d76f740 4 client.15480 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.471+0000 7f4f6d76f740 1 -- 172.21.15.28:0/1079115136 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cce456c00 con 0x560cce4fc730 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.472+0000 7f4f63fff640 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x560cce4fc730 0x560cce4fdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.472+0000 7f4f63fff640 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x560cce4fc730 0x560cce4fdba0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=525b7cf31f63d251 server_cookie=a70db0b5cb64ad82 in_seq=0 out_seq=0 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.472+0000 7f4f437fe640 10 client.15480 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f437fe640 1 -- 172.21.15.28:0/1079115136 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560cce456c00 con 0x560cce4fc730 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f437fe640 10 client.15480 handle_command_reply: tid=0 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 client.15480 shutdown 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 -- 172.21.15.28:0/1079115136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4eb420 msgr2=0x560cce4f6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4eb420 0x560cce4f6f50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c012010 tx=0x7f4f5c004a40 comp rx=0 tx=0).stop 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 -- 172.21.15.28:0/1079115136 shutdown_connections 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x560cce4fc730 0x560cce4fdba0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4eb420 0x560cce4f6f50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4e62b0 0x560cce4eff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f437fe640 0 client.15480 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:26:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/1079115136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 -- 172.21.15.28:0/1079115136 >> 172.21.15.28:0/1079115136 conn(0x560cce4e0f30 msgr2=0x560cce4e0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 -- 172.21.15.28:0/1079115136 shutdown_connections 2024-05-09T08:26:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 1 -- 172.21.15.28:0/1079115136 wait complete. 2024-05-09T08:26:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.473+0000 7f4f6d76f740 20 client.15480 trim_cache size 0 max 16384 2024-05-09T08:26:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f6d76f740 1 Processor -- start 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f6d76f740 1 -- start start 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4ee600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 0x560cce4eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cce4fbee0 con 0x560cce4e62b0 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560cce4fc5b0 con 0x560cce4eb420 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560cce4fcc80 con 0x560cce4e58c0 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58894/0 (socket says 172.21.15.28:58894) 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f637fe640 1 -- 172.21.15.28:0/3246150155 learned_addr learned my addr 172.21.15.28:0/3246150155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f63fff640 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f637fe640 1 -- 172.21.15.28:0/3246150155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 msgr2=0x560cce4ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 0x560cce4eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f637fe640 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f637fe640 1 -- 172.21.15.28:0/3246150155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 msgr2=0x560cce4eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f637fe640 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 0x560cce4eeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f637fe640 1 -- 172.21.15.28:0/3246150155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cce4e1ca0 con 0x560cce4e62b0 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f63fff640 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f68f1b640 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 0x560cce4eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.492+0000 7f4f637fe640 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f6f10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4f58011ab0 tx=0x7f4f58011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3ebdb7af3a7c70 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f49ffb640 1 -- 172.21.15.28:0/3246150155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58020070 con 0x560cce4e62b0 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f49ffb640 1 -- 172.21.15.28:0/3246150155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f5801bad0 con 0x560cce4e62b0 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f49ffb640 1 -- 172.21.15.28:0/3246150155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5800fcf0 con 0x560cce4e62b0 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3246150155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 msgr2=0x560cce4f6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f6f10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4f58011ab0 tx=0x7f4f58011f70 comp rx=0 tx=0).stop 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3246150155 shutdown_connections 2024-05-09T08:26:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 0x560cce4eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f6f10 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3246150155 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3246150155 >> 172.21.15.28:0/3246150155 conn(0x560cce4e0f30 msgr2=0x560cce4ef9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3246150155 shutdown_connections 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.493+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3246150155 wait complete. 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f6d76f740 1 -- start start 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f6d76f740 1 Processor -- start 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f6d76f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 0x560cce4f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cce4fda20 con 0x560cce4e62b0 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560cce4fd350 con 0x560cce4eb420 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f6d76f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560cce4fe0f0 con 0x560cce4e58c0 2024-05-09T08:26:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f6d76f740 10 client.0 _ll_register_callbacks cb 0x7f4f5c012570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58902/0 (socket says 172.21.15.28:58902) 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f637fe640 1 -- 172.21.15.28:0/3639916469 learned_addr learned my addr 172.21.15.28:0/3639916469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f63fff640 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f637fe640 1 -- 172.21.15.28:0/3639916469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 msgr2=0x560cce4f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f637fe640 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4f30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f637fe640 1 -- 172.21.15.28:0/3639916469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 msgr2=0x560cce4f61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f637fe640 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 0x560cce4f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f637fe640 1 -- 172.21.15.28:0/3639916469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cce4f67b0 con 0x560cce4e62b0 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.494+0000 7f4f63fff640 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f637fe640 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f35e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800f830 tx=0x7f4f5801beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84417ec53ecfc726 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f437fe640 1 -- 172.21.15.28:0/3639916469 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5802b020 con 0x560cce4e62b0 2024-05-09T08:26:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f437fe640 1 -- 172.21.15.28:0/3639916469 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f4f5800fc30 con 0x560cce4e62b0 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3639916469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f5801bad0 con 0x560cce4e62b0 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f437fe640 1 -- 172.21.15.28:0/3639916469 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4f58036050 con 0x560cce4e62b0 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f6d76f740 10 client.15486 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3639916469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cce562c20 con 0x560cce4e62b0 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f437fe640 1 -- 172.21.15.28:0/3639916469 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f5801bad0 con 0x560cce4e62b0 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f437fe640 1 -- 172.21.15.28:0/3639916469 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4f5801a030 con 0x560cce4e62b0 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f6d76f740 10 client.15486 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f6d76f740 10 client.15486 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f6d76f740 20 client.15486 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x560cce4fc730 0x560cce566300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f6d76f740 4 client.15486 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.495+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3639916469 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "status"}) -- 0x560cce450f50 con 0x560cce4fc730 2024-05-09T08:26:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.496+0000 7f4f63fff640 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x560cce4fc730 0x560cce566300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.496+0000 7f4f63fff640 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x560cce4fc730 0x560cce566300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=78df5364b83c8e6d server_cookie=302beaf065ebafbb in_seq=0 out_seq=0 2024-05-09T08:26:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.496+0000 7f4f437fe640 10 client.15486 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:26:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.496+0000 7f4f437fe640 1 -- 172.21.15.28:0/3639916469 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560cce450f50 con 0x560cce4fc730 2024-05-09T08:26:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.496+0000 7f4f437fe640 10 client.15486 handle_command_reply: tid=0 2024-05-09T08:26:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.496+0000 7f4f6d76f740 1 client.15486 shutdown 2024-05-09T08:26:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3639916469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 msgr2=0x560cce4f35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f35e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800f830 tx=0x7f4f5801beb0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3639916469 shutdown_connections 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x560cce4fc730 0x560cce566300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560cce4eb420 0x560cce4f61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cce4e62b0 0x560cce4f35e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f437fe640 0 client.15486 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 --2- 172.21.15.28:0/3639916469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560cce4e58c0 0x560cce4f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3639916469 >> 172.21.15.28:0/3639916469 conn(0x560cce4e0f30 msgr2=0x560cce4e3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3639916469 shutdown_connections 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 1 -- 172.21.15.28:0/3639916469 wait complete. 2024-05-09T08:26:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6d76f740 20 client.15486 trim_cache size 0 max 16384 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24499, 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 114.12884877499999, 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 152.87351408999999 2024-05-09T08:26:09.508 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:09.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6a97e640 1 -- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4f4c0a2c10 msgr2=0x7f4f4c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4f4c0a2c10 0x7f4f4c0a50d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009690 tx=0x7f4f5c009620 comp rx=0 tx=0).stop 2024-05-09T08:26:09.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6a97e640 1 -- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 msgr2=0x7f4f64101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 0x7f4f64101d00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4f54009230 tx=0x7f4f54004360 comp rx=0 tx=0).stop 2024-05-09T08:26:09.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6a97e640 1 -- 172.21.15.28:0/505769245 shutdown_connections 2024-05-09T08:26:09.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f4f4c0a2c10 0x7f4f4c0a50d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f641075f0 0x7f4f64100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f64104cc0 0x7f4f64100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.497+0000 7f4f6a97e640 1 --2- 172.21.15.28:0/505769245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f64102390 0x7f4f64101d00 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.498+0000 7f4f6a97e640 1 -- 172.21.15.28:0/505769245 >> 172.21.15.28:0/505769245 conn(0x7f4f640fbce0 msgr2=0x7f4f64102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.498+0000 7f4f6a97e640 1 -- 172.21.15.28:0/505769245 shutdown_connections 2024-05-09T08:26:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.498+0000 7f4f6a97e640 1 -- 172.21.15.28:0/505769245 wait complete. 2024-05-09T08:26:09.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:09 smithi028 ceph-mon[30119]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:26:09.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:09 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1369070214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:09.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:09 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1704324191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:09.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T08:26:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.616+0000 7f56d3f63640 1 -- 172.21.15.28:0/1478587909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 msgr2=0x7f56cc1099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.616+0000 7f56d3f63640 1 --2- 172.21.15.28:0/1478587909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 0x7f56cc1099a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009900 tx=0x7f56c402a3f0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.616+0000 7f56d3f63640 1 -- 172.21.15.28:0/1478587909 shutdown_connections 2024-05-09T08:26:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.616+0000 7f56d3f63640 1 --2- 172.21.15.28:0/1478587909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 0x7f56cc1099a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.616+0000 7f56d3f63640 1 --2- 172.21.15.28:0/1478587909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56cc104c20 0x7f56cc107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.616+0000 7f56d3f63640 1 --2- 172.21.15.28:0/1478587909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc1022f0 0x7f56cc1046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.616+0000 7f56d3f63640 1 -- 172.21.15.28:0/1478587909 >> 172.21.15.28:0/1478587909 conn(0x7f56cc0fbce0 msgr2=0x7f56cc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 -- 172.21.15.28:0/1478587909 shutdown_connections 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 -- 172.21.15.28:0/1478587909 wait complete. 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 Processor -- start 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 -- start start 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56cc1022f0 0x7f56cc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc104c20 0x7f56cc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 0x7f56cc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc10d4f0 con 0x7f56cc107550 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc10d370 con 0x7f56cc1022f0 2024-05-09T08:26:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d3f63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc10d670 con 0x7f56cc104c20 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.617+0000 7f56d2500640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 0x7f56cc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d2500640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 0x7f56cc19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58918/0 (socket says 172.21.15.28:58918) 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d2500640 1 -- 172.21.15.28:0/133072972 learned_addr learned my addr 172.21.15.28:0/133072972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d1cff640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56cc1022f0 0x7f56cc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d14fe640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc104c20 0x7f56cc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d2500640 1 -- 172.21.15.28:0/133072972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc104c20 msgr2=0x7f56cc197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d2500640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc104c20 0x7f56cc197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d2500640 1 -- 172.21.15.28:0/133072972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56cc1022f0 msgr2=0x7f56cc197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d2500640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56cc1022f0 0x7f56cc197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d2500640 1 -- 172.21.15.28:0/133072972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56cc19f670 con 0x7f56cc107550 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d2500640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 0x7f56cc19f000 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f56c40314e0 tx=0x7f56c40043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56beffd640 1 -- 172.21.15.28:0/133072972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c403f030 con 0x7f56cc107550 2024-05-09T08:26:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56beffd640 1 -- 172.21.15.28:0/133072972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56c4003ce0 con 0x7f56cc107550 2024-05-09T08:26:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d3f63640 1 -- 172.21.15.28:0/133072972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56cc19f900 con 0x7f56cc107550 2024-05-09T08:26:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56beffd640 1 -- 172.21.15.28:0/133072972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c4035070 con 0x7f56cc107550 2024-05-09T08:26:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.618+0000 7f56d3f63640 1 -- 172.21.15.28:0/133072972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56cc19fdf0 con 0x7f56cc107550 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.620+0000 7f56beffd640 1 -- 172.21.15.28:0/133072972 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f56c4042030 con 0x7f56cc107550 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d6c75740 1 Processor -- start 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d6c75740 1 -- start start 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582303cd0 0x5575823040f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 0x5575823091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582309830 0x55758230bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d6c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5575822634d0 con 0x557582303cd0 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d6c75740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55758220b3e0 con 0x557582309830 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d6c75740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55758225f490 con 0x5575823046c0 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d14fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582303cd0 0x5575823040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d14fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582303cd0 0x5575823040f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58924/0 (socket says 172.21.15.28:58924) 2024-05-09T08:26:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56d14fe640 1 -- 172.21.15.28:0/968121949 learned_addr learned my addr 172.21.15.28:0/968121949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.621+0000 7f56beffd640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56b80a2b90 0x7f56b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d1cff640 1 --2- 172.21.15.28:0/968121949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 0x5575823091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d14fe640 1 --2- 172.21.15.28:0/968121949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582309830 0x55758230bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d2500640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56b80a2b90 0x7f56b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d14fe640 1 -- 172.21.15.28:0/968121949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 msgr2=0x5575823091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d14fe640 1 --2- 172.21.15.28:0/968121949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 0x5575823091c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d14fe640 1 -- 172.21.15.28:0/968121949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582309830 msgr2=0x55758230bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d14fe640 1 --2- 172.21.15.28:0/968121949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582309830 0x55758230bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d14fe640 1 -- 172.21.15.28:0/968121949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557581f772d0 con 0x557582303cd0 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56beffd640 1 -- 172.21.15.28:0/133072972 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f56c40efd30 con 0x7f56cc107550 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d14fe640 1 --2- 172.21.15.28:0/968121949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582303cd0 0x5575823040f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f56c80144f0 tx=0x7f56c800b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b913d78aa71c271 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56b7fff640 1 -- 172.21.15.28:0/968121949 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c8005020 con 0x557582303cd0 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56d2500640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56b80a2b90 0x7f56b80a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f56c40bc1f0 tx=0x7f56c408f3e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56b7fff640 1 -- 172.21.15.28:0/968121949 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56c80047a0 con 0x557582303cd0 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.622+0000 7f56b7fff640 1 -- 172.21.15.28:0/968121949 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c8012770 con 0x557582303cd0 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 -- 172.21.15.28:0/968121949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582303cd0 msgr2=0x5575823040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 --2- 172.21.15.28:0/968121949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582303cd0 0x5575823040f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f56c80144f0 tx=0x7f56c800b4e0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 -- 172.21.15.28:0/968121949 shutdown_connections 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 --2- 172.21.15.28:0/968121949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582309830 0x55758230bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 --2- 172.21.15.28:0/968121949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 0x5575823091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 --2- 172.21.15.28:0/968121949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582303cd0 0x5575823040f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 -- 172.21.15.28:0/968121949 >> 172.21.15.28:0/968121949 conn(0x5575822faf30 msgr2=0x5575822fb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 -- 172.21.15.28:0/968121949 shutdown_connections 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 -- 172.21.15.28:0/968121949 wait complete. 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 -- start start 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.623+0000 7f56d6c75740 1 Processor -- start 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582303cd0 0x55758230e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 0x55758230e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582315620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d6c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55758230c780 con 0x557582309830 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d6c75740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55758221e8e0 con 0x557582303cd0 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d6c75740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55758230c900 con 0x5575823046c0 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d6c75740 10 client.0 _ll_register_callbacks cb 0x5575821eb9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582315620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582315620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58926/0 (socket says 172.21.15.28:58926) 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 -- 172.21.15.28:0/3574693977 learned_addr learned my addr 172.21.15.28:0/3574693977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582303cd0 0x55758230e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d1cff640 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 0x55758230e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 -- 172.21.15.28:0/3574693977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 msgr2=0x55758230e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 0x55758230e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 -- 172.21.15.28:0/3574693977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582303cd0 msgr2=0x55758230e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582303cd0 0x55758230e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 -- 172.21.15.28:0/3574693977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575820416b0 con 0x557582309830 2024-05-09T08:26:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d1cff640 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 0x55758230e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56d14fe640 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582315620 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f56c8014fd0 tx=0x7f56c8005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3d2a18543c2a0f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.624+0000 7f56b57fa640 1 -- 172.21.15.28:0/3574693977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c8005370 con 0x557582309830 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56b57fa640 1 -- 172.21.15.28:0/3574693977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f56c8012510 con 0x557582309830 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56d6c75740 1 -- 172.21.15.28:0/3574693977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56c80047a0 con 0x557582309830 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56b57fa640 1 -- 172.21.15.28:0/3574693977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f56c8012710 con 0x557582309830 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56d6c75740 10 client.15498 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56d6c75740 1 -- 172.21.15.28:0/3574693977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5575823185e0 con 0x557582309830 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56b57fa640 1 -- 172.21.15.28:0/3574693977 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56c80047a0 con 0x557582309830 2024-05-09T08:26:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56b57fa640 1 -- 172.21.15.28:0/3574693977 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f56c800e750 con 0x557582309830 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56d6c75740 10 client.15498 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56d6c75740 10 client.15498 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56d6c75740 20 client.15498 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55758231a9e0 0x55758231be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56d6c75740 4 client.15498 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.625+0000 7f56d6c75740 1 -- 172.21.15.28:0/3574693977 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557582270c00 con 0x55758231a9e0 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.626+0000 7f56d1cff640 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55758231a9e0 0x55758231be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.626+0000 7f56d1cff640 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55758231a9e0 0x55758231be50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c70fab358f273aa4 server_cookie=b1da8ad53e3a25d7 in_seq=0 out_seq=0 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.626+0000 7f56b57fa640 10 client.15498 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56b57fa640 1 -- 172.21.15.28:0/3574693977 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557582270c00 con 0x55758231a9e0 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56b57fa640 10 client.15498 handle_command_reply: tid=0 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 client.15498 shutdown 2024-05-09T08:26:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 -- 172.21.15.28:0/3574693977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 msgr2=0x557582315620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582315620 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f56c8014fd0 tx=0x7f56c8005f00 comp rx=0 tx=0).stop 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 -- 172.21.15.28:0/3574693977 shutdown_connections 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55758231a9e0 0x55758231be50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582315620 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5575823046c0 0x55758230e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3574693977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557582303cd0 0x55758230e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56b57fa640 0 client.15498 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 -- 172.21.15.28:0/3574693977 >> 172.21.15.28:0/3574693977 conn(0x5575822faf30 msgr2=0x5575822fa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 -- 172.21.15.28:0/3574693977 shutdown_connections 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 1 -- 172.21.15.28:0/3574693977 wait complete. 2024-05-09T08:26:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.627+0000 7f56d6c75740 20 client.15498 trim_cache size 0 max 16384 2024-05-09T08:26:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d6c75740 1 Processor -- start 2024-05-09T08:26:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d6c75740 1 -- start start 2024-05-09T08:26:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557582303cd0 0x55758230bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5575823046c0 0x557582315540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582319aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d6c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55758231a190 con 0x557582309830 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d6c75740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55758231a860 con 0x5575823046c0 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d6c75740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55758231af30 con 0x557582303cd0 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d14fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582319aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d14fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582319aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58942/0 (socket says 172.21.15.28:58942) 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.645+0000 7f56d14fe640 1 -- 172.21.15.28:0/3278200043 learned_addr learned my addr 172.21.15.28:0/3278200043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56d14fe640 1 -- 172.21.15.28:0/3278200043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557582303cd0 msgr2=0x55758230bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56d14fe640 1 --2- 172.21.15.28:0/3278200043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557582303cd0 0x55758230bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56d14fe640 1 -- 172.21.15.28:0/3278200043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5575823046c0 msgr2=0x557582315540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56d14fe640 1 --2- 172.21.15.28:0/3278200043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5575823046c0 0x557582315540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56d14fe640 1 -- 172.21.15.28:0/3278200043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557582308d20 con 0x557582309830 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56d14fe640 1 --2- 172.21.15.28:0/3278200043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582319aa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f56c80143a0 tx=0x7f56c800ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5385baa99367c0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56b77fe640 1 -- 172.21.15.28:0/3278200043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c800be50 con 0x557582309830 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56b77fe640 1 -- 172.21.15.28:0/3278200043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56c8025070 con 0x557582309830 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56b77fe640 1 -- 172.21.15.28:0/3278200043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c800e6d0 con 0x557582309830 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56d6c75740 1 -- 172.21.15.28:0/3278200043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 msgr2=0x557582319aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.646+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3278200043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582319aa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f56c80143a0 tx=0x7f56c800ba00 comp rx=0 tx=0).stop 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 -- 172.21.15.28:0/3278200043 shutdown_connections 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3278200043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x557582319aa0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3278200043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5575823046c0 0x557582315540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 --2- 172.21.15.28:0/3278200043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557582303cd0 0x55758230bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 -- 172.21.15.28:0/3278200043 >> 172.21.15.28:0/3278200043 conn(0x5575822faf30 msgr2=0x5575822fb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 -- 172.21.15.28:0/3278200043 shutdown_connections 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 -- 172.21.15.28:0/3278200043 wait complete. 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 -- start start 2024-05-09T08:26:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 Processor -- start 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557582303cd0 0x557582311390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5575823046c0 0x5575823118d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x5575823148f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55758231bcd0 con 0x557582309830 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55758231b600 con 0x5575823046c0 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55758231c3a0 con 0x557582303cd0 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d6c75740 10 client.0 _ll_register_callbacks cb 0x7f56c0008510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.647+0000 7f56d14fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x5575823148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d14fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x5575823148f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58954/0 (socket says 172.21.15.28:58954) 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d14fe640 1 -- 172.21.15.28:0/2137310241 learned_addr learned my addr 172.21.15.28:0/2137310241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d1cff640 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5575823046c0 0x5575823118d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d14fe640 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557582303cd0 0x557582311390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d14fe640 1 -- 172.21.15.28:0/2137310241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557582303cd0 msgr2=0x557582311390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d14fe640 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557582303cd0 0x557582311390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d14fe640 1 -- 172.21.15.28:0/2137310241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5575823046c0 msgr2=0x5575823118d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d14fe640 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5575823046c0 0x5575823118d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d14fe640 1 -- 172.21.15.28:0/2137310241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557582314ee0 con 0x557582309830 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d14fe640 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x5575823148f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f56c800ed60 tx=0x7f56c8014ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9302eef0ff498d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56b57fa640 1 -- 172.21.15.28:0/2137310241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c8004e80 con 0x557582309830 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56b57fa640 1 -- 172.21.15.28:0/2137310241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f56c800b370 con 0x557582309830 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d6c75740 1 -- 172.21.15.28:0/2137310241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56c8025070 con 0x557582309830 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56b57fa640 1 -- 172.21.15.28:0/2137310241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f56c8005580 con 0x557582309830 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d6c75740 10 client.15504 fetch_fsmap learned FSMap version 27 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.648+0000 7f56d6c75740 1 -- 172.21.15.28:0/2137310241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5575823810f0 con 0x557582309830 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56b57fa640 1 -- 172.21.15.28:0/2137310241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56c8025070 con 0x557582309830 2024-05-09T08:26:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56b57fa640 1 -- 172.21.15.28:0/2137310241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f56c801f730 con 0x557582309830 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56d6c75740 10 client.15504 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56d6c75740 10 client.15504 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56d6c75740 20 client.15504 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56d6c75740 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557582385840 0x557582382350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56d6c75740 4 client.15504 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56d6c75740 1 -- 172.21.15.28:0/2137310241 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "status"}) -- 0x55758226af50 con 0x557582385840 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56d1cff640 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557582385840 0x557582382350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56b57fa640 10 client.15504 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.649+0000 7f56d1cff640 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557582385840 0x557582382350 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3102ad21895ee5c1 server_cookie=c29eab55738d5db0 in_seq=0 out_seq=0 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56b57fa640 1 -- 172.21.15.28:0/2137310241 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55758226af50 con 0x557582385840 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56b57fa640 10 client.15504 handle_command_reply: tid=0 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 client.15504 shutdown 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 -- 172.21.15.28:0/2137310241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 msgr2=0x5575823148f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x5575823148f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f56c800ed60 tx=0x7f56c8014ec0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 -- 172.21.15.28:0/2137310241 shutdown_connections 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557582385840 0x557582382350 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557582309830 0x5575823148f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5575823046c0 0x5575823118d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56b57fa640 0 client.15504 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 --2- 172.21.15.28:0/2137310241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557582303cd0 0x557582311390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 -- 172.21.15.28:0/2137310241 >> 172.21.15.28:0/2137310241 conn(0x5575822faf30 msgr2=0x5575822fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 -- 172.21.15.28:0/2137310241 shutdown_connections 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 1 -- 172.21.15.28:0/2137310241 wait complete. 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.650+0000 7f56d6c75740 20 client.15504 trim_cache size 0 max 16384 2024-05-09T08:26:09.657 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "3891b212-0ddc-11ef-bc98-c7b262605968", 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24512, 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 138.01881163900001, 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 144.251863219 2024-05-09T08:26:09.658 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 -- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56b80a2b90 msgr2=0x7f56b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56b80a2b90 0x7f56b80a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f56c40bc1f0 tx=0x7f56c408f3e0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 -- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 msgr2=0x7f56cc19f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 0x7f56cc19f000 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f56c40314e0 tx=0x7f56c40043c0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 -- 172.21.15.28:0/133072972 shutdown_connections 2024-05-09T08:26:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f56b80a2b90 0x7f56b80a5050 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56cc107550 0x7f56cc19f000 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc104c20 0x7f56cc197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 --2- 172.21.15.28:0/133072972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56cc1022f0 0x7f56cc197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 -- 172.21.15.28:0/133072972 >> 172.21.15.28:0/133072972 conn(0x7f56cc0fbce0 msgr2=0x7f56cc102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 -- 172.21.15.28:0/133072972 shutdown_connections 2024-05-09T08:26:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.651+0000 7f56d3f63640 1 -- 172.21.15.28:0/133072972 wait complete. 2024-05-09T08:26:09.662 INFO:teuthology.run_tasks:Running task sequential... 2024-05-09T08:26:09.673 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-09T08:26:09.673 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-09T08:26:09.673 INFO:teuthology.task.print:Enabling mgr modules 2024-05-09T08:26:09.674 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-09T08:26:09.674 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:26:09.674 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:26:09.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-09T08:26:09.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.778+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/2483833563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 msgr2=0x7fc9b4107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.778+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/2483833563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 0x7fc9b4107030 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009900 tx=0x7fc9a802a410 comp rx=0 tx=0).stop 2024-05-09T08:26:09.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.778+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/2483833563 shutdown_connections 2024-05-09T08:26:09.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.778+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/2483833563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9b4107570 0x7fc9b41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.778+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/2483833563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 0x7fc9b4107030 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.778+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/2483833563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9b4102310 0x7fc9b4104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.778+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/2483833563 >> 172.21.15.28:0/2483833563 conn(0x7fc9b40fbd20 msgr2=0x7fc9b40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:09.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.778+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/2483833563 shutdown_connections 2024-05-09T08:26:09.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.778+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/2483833563 wait complete. 2024-05-09T08:26:09.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b8fdc640 1 Processor -- start 2024-05-09T08:26:09.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b8fdc640 1 -- start start 2024-05-09T08:26:09.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b8fdc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9b4102310 0x7fc9b4199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b8fdc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 0x7fc9b419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b8fdc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9b4107570 0x7fc9b41a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b8fdc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b410d560 con 0x7fc9b4104c40 2024-05-09T08:26:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b8fdc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b410d3e0 con 0x7fc9b4107570 2024-05-09T08:26:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b8fdc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b410d6e0 con 0x7fc9b4102310 2024-05-09T08:26:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 0x7fc9b419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 0x7fc9b419a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58980/0 (socket says 172.21.15.28:58980) 2024-05-09T08:26:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b1d9b640 1 -- 172.21.15.28:0/725406709 learned_addr learned my addr 172.21.15.28:0/725406709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b2d9d640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9b4107570 0x7fc9b41a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b259c640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9b4102310 0x7fc9b4199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.779+0000 7fc9b1d9b640 1 -- 172.21.15.28:0/725406709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9b4102310 msgr2=0x7fc9b4199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9b1d9b640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9b4102310 0x7fc9b4199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9b1d9b640 1 -- 172.21.15.28:0/725406709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9b4107570 msgr2=0x7fc9b41a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9b1d9b640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9b4107570 0x7fc9b41a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9b1d9b640 1 -- 172.21.15.28:0/725406709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b41a1960 con 0x7fc9b4104c40 2024-05-09T08:26:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9b2d9d640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9b4107570 0x7fc9b41a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9b1d9b640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 0x7fc9b419a290 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8031610 tx=0x7fc9a80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9a37fe640 1 -- 172.21.15.28:0/725406709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a803f030 con 0x7fc9b4104c40 2024-05-09T08:26:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9a37fe640 1 -- 172.21.15.28:0/725406709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9a8003ce0 con 0x7fc9b4104c40 2024-05-09T08:26:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b41a1bf0 con 0x7fc9b4104c40 2024-05-09T08:26:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9a37fe640 1 -- 172.21.15.28:0/725406709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8035070 con 0x7fc9b4104c40 2024-05-09T08:26:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.780+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b41a20e0 con 0x7fc9b4104c40 2024-05-09T08:26:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.781+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b4109060 con 0x7fc9b4104c40 2024-05-09T08:26:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.783+0000 7fc9a37fe640 1 -- 172.21.15.28:0/725406709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc9a8042030 con 0x7fc9b4104c40 2024-05-09T08:26:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.784+0000 7fc9a37fe640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc99c0a2c30 0x7fc99c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.784+0000 7fc9b259c640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc99c0a2c30 0x7fc99c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.784+0000 7fc9a37fe640 1 -- 172.21.15.28:0/725406709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fc9a80efb10 con 0x7fc9b4104c40 2024-05-09T08:26:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.784+0000 7fc9b259c640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc99c0a2c30 0x7fc99c0a50f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4005fd0 tx=0x7fc9a4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:09.788+0000 7fc9a37fe640 1 -- 172.21.15.28:0/725406709 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc9a80bc680 con 0x7fc9b4104c40 2024-05-09T08:26:09.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:09 smithi165 ceph-mon[34540]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:26:09.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:09 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1369070214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:09.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:09 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1704324191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:09.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:09 smithi184 ceph-mon[34509]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-09T08:26:09.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:09 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1369070214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:09.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:09 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1704324191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:10.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.026+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fc9b40624d0 con 0x7fc9b4104c40 2024-05-09T08:26:10.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.445+0000 7fc9a37fe640 1 -- 172.21.15.28:0/725406709 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fc9a80bbf80 con 0x7fc9b4104c40 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc99c0a2c30 msgr2=0x7fc99c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc99c0a2c30 0x7fc99c0a50f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4005fd0 tx=0x7fc9a4005850 comp rx=0 tx=0).stop 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 msgr2=0x7fc9b419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 0x7fc9b419a290 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8031610 tx=0x7fc9a80043b0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 shutdown_connections 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fc99c0a2c30 0x7fc99c0a50f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9b4107570 0x7fc9b41a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4104c40 0x7fc9b419a290 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 --2- 172.21.15.28:0/725406709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9b4102310 0x7fc9b4199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 >> 172.21.15.28:0/725406709 conn(0x7fc9b40fbd20 msgr2=0x7fc9b41018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 shutdown_connections 2024-05-09T08:26:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.448+0000 7fc9b8fdc640 1 -- 172.21.15.28:0/725406709 wait complete. 2024-05-09T08:26:10.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-09T08:26:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.583+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/1133003143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec100790 msgr2=0x7f2bec100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.583+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/1133003143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec100790 0x7f2bec100bb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009ca0 tx=0x7f2bdc02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.583+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/1133003143 shutdown_connections 2024-05-09T08:26:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.583+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/1133003143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bec102340 0x7f2bec108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.583+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/1133003143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bec101980 0x7f2bec101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.583+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/1133003143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec100790 0x7f2bec100bb0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.583+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/1133003143 >> 172.21.15.28:0/1133003143 conn(0x7f2bec0fbd00 msgr2=0x7f2bec0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.583+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/1133003143 shutdown_connections 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.583+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/1133003143 wait complete. 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf3d1d640 1 Processor -- start 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf3d1d640 1 -- start start 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf3d1d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bec100790 0x7f2bec197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf3d1d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bec101980 0x7f2bec198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf3d1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec102340 0x7f2bec19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf3d1d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec10c460 con 0x7f2bec102340 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf3d1d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec10c2e0 con 0x7f2bec101980 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf3d1d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec10c5e0 con 0x7f2bec100790 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf22ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec102340 0x7f2bec19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf22ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec102340 0x7f2bec19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59024/0 (socket says 172.21.15.28:59024) 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf22ba640 1 -- 172.21.15.28:0/2488708028 learned_addr learned my addr 172.21.15.28:0/2488708028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf12b8640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bec101980 0x7f2bec198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.584+0000 7f2bf1ab9640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bec100790 0x7f2bec197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2bf22ba640 1 -- 172.21.15.28:0/2488708028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bec100790 msgr2=0x7f2bec197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2bf22ba640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bec100790 0x7f2bec197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2bf22ba640 1 -- 172.21.15.28:0/2488708028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bec101980 msgr2=0x7f2bec198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2bf22ba640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bec101980 0x7f2bec198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2bf22ba640 1 -- 172.21.15.28:0/2488708028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bec19f760 con 0x7f2bec102340 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2bf22ba640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec102340 0x7f2bec19f0f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2be400d750 tx=0x7f2be400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2beaffd640 1 -- 172.21.15.28:0/2488708028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4017070 con 0x7f2bec102340 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2beaffd640 1 -- 172.21.15.28:0/2488708028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be4005500 con 0x7f2bec102340 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bec19fa50 con 0x7f2bec102340 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2beaffd640 1 -- 172.21.15.28:0/2488708028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4004d90 con 0x7f2bec102340 2024-05-09T08:26:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.585+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bec19fff0 con 0x7f2bec102340 2024-05-09T08:26:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.586+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bec1a0280 con 0x7f2bec102340 2024-05-09T08:26:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.587+0000 7f2beaffd640 1 -- 172.21.15.28:0/2488708028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f2be4004f30 con 0x7f2bec102340 2024-05-09T08:26:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.588+0000 7f2beaffd640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2bd40a2d00 0x7f2bd40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.588+0000 7f2bf1ab9640 1 -- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2bd40a2d00 msgr2=0x7f2bd40a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.589+0000 7f2beaffd640 1 -- 172.21.15.28:0/2488708028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f2be40ced00 con 0x7f2bec102340 2024-05-09T08:26:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.589+0000 7f2bf1ab9640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2bd40a2d00 0x7f2bd40a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:26:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.593+0000 7f2beaffd640 1 -- 172.21.15.28:0/2488708028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2be409b220 con 0x7f2bec102340 2024-05-09T08:26:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.781+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f2bec1a0450 con 0x7f2bec102340 2024-05-09T08:26:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.785+0000 7f2beaffd640 1 -- 172.21.15.28:0/2488708028 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7f2be409f520 con 0x7f2bec102340 2024-05-09T08:26:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2bd40a2d00 msgr2=0x7f2bd40a51c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:26:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2bd40a2d00 0x7f2bd40a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec102340 msgr2=0x7f2bec19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec102340 0x7f2bec19f0f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2be400d750 tx=0x7f2be400dc10 comp rx=0 tx=0).stop 2024-05-09T08:26:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 shutdown_connections 2024-05-09T08:26:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f2bd40a2d00 0x7f2bd40a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bec102340 0x7f2bec19f0f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bec101980 0x7f2bec198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 --2- 172.21.15.28:0/2488708028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bec100790 0x7f2bec197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 >> 172.21.15.28:0/2488708028 conn(0x7f2bec0fbd00 msgr2=0x7f2bec0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 shutdown_connections 2024-05-09T08:26:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.787+0000 7f2bf3d1d640 1 -- 172.21.15.28:0/2488708028 wait complete. 2024-05-09T08:26:10.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:10 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/725406709' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-09T08:26:10.799 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: ignoring --setuser ceph since I am not root 2024-05-09T08:26:10.799 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: ignoring --setgroup ceph since I am not root 2024-05-09T08:26:10.799 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:10.646+0000 7fc80319c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:26:10.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-09T08:26:10.810 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:10 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/725406709' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-09T08:26:10.810 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: ignoring --setuser ceph since I am not root 2024-05-09T08:26:10.810 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: ignoring --setgroup ceph since I am not root 2024-05-09T08:26:10.810 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:10 smithi184 ceph-mgr[36247]: -- 172.21.15.184:0/2796228480 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561b490b05a0 con 0x561b49085000 2024-05-09T08:26:10.810 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:10.626+0000 7fca9db38200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:26:10.810 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:10.809+0000 7fca9db38200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:26:10.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:10 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/725406709' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-09T08:26:10.816 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: ignoring --setuser ceph since I am not root 2024-05-09T08:26:10.816 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: ignoring --setgroup ceph since I am not root 2024-05-09T08:26:10.816 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:10 smithi165 ceph-mgr[35928]: -- 172.21.15.165:0/4180304994 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56521f5664e0 con 0x56521f53c800 2024-05-09T08:26:10.816 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:10 smithi165 ceph-mgr[35928]: -- 172.21.15.165:0/4180304994 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56521f5685a0 con 0x56521f53c800 2024-05-09T08:26:10.816 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:10.631+0000 7ff0df322200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:26:10.816 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:10.816+0000 7ff0df322200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:26:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.908+0000 7f7e3d8da640 1 -- 172.21.15.28:0/3389033923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38102a80 msgr2=0x7f7e38102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.908+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/3389033923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38102a80 0x7f7e38102f00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009900 tx=0x7f7e2002a410 comp rx=0 tx=0).stop 2024-05-09T08:26:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.908+0000 7f7e3d8da640 1 -- 172.21.15.28:0/3389033923 shutdown_connections 2024-05-09T08:26:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.908+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/3389033923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e38103440 0x7f7e38109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.908+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/3389033923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38102a80 0x7f7e38102f00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.908+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/3389033923 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e38101890 0x7f7e38101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.908+0000 7f7e3d8da640 1 -- 172.21.15.28:0/3389033923 >> 172.21.15.28:0/3389033923 conn(0x7f7e380fce20 msgr2=0x7f7e380ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.908+0000 7f7e3d8da640 1 -- 172.21.15.28:0/3389033923 shutdown_connections 2024-05-09T08:26:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.908+0000 7f7e3d8da640 1 -- 172.21.15.28:0/3389033923 wait complete. 2024-05-09T08:26:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.909+0000 7f7e3d8da640 1 Processor -- start 2024-05-09T08:26:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.909+0000 7f7e3d8da640 1 -- start start 2024-05-09T08:26:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3d8da640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e38101890 0x7f7e38197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3d8da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e38102a80 0x7f7e38197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3d8da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38103440 0x7f7e3819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3d8da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3810d510 con 0x7f7e38103440 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3d8da640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3810d390 con 0x7f7e38101890 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3d8da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3810d690 con 0x7f7e38102a80 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38103440 0x7f7e3819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38103440 0x7f7e3819f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59038/0 (socket says 172.21.15.28:59038) 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e377fe640 1 -- 172.21.15.28:0/2209922117 learned_addr learned my addr 172.21.15.28:0/2209922117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e36ffd640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e38101890 0x7f7e38197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e2ffff640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e38102a80 0x7f7e38197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e377fe640 1 -- 172.21.15.28:0/2209922117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e38102a80 msgr2=0x7f7e38197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e377fe640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e38102a80 0x7f7e38197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e377fe640 1 -- 172.21.15.28:0/2209922117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e38101890 msgr2=0x7f7e38197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e377fe640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e38101890 0x7f7e38197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e377fe640 1 -- 172.21.15.28:0/2209922117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3819f6c0 con 0x7f7e38103440 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e36ffd640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e38101890 0x7f7e38197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e377fe640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38103440 0x7f7e3819f050 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000eb10 tx=0x7f7e3000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3c8d8640 1 -- 172.21.15.28:0/2209922117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e30004b20 con 0x7f7e38103440 2024-05-09T08:26:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3c8d8640 1 -- 172.21.15.28:0/2209922117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e30004cc0 con 0x7f7e38103440 2024-05-09T08:26:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3819f9b0 con 0x7f7e38103440 2024-05-09T08:26:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.910+0000 7f7e3c8d8640 1 -- 172.21.15.28:0/2209922117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e30010760 con 0x7f7e38103440 2024-05-09T08:26:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.911+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e3819fed0 con 0x7f7e38103440 2024-05-09T08:26:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.912+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e38108ae0 con 0x7f7e38103440 2024-05-09T08:26:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.913+0000 7f7e3c8d8640 1 -- 172.21.15.28:0/2209922117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f7e30005000 con 0x7f7e38103440 2024-05-09T08:26:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.913+0000 7f7e3c8d8640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7e240a2c80 0x7f7e240a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.914+0000 7f7e36ffd640 1 -- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7e240a2c80 msgr2=0x7f7e240a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.914+0000 7f7e3c8d8640 1 -- 172.21.15.28:0/2209922117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f7e300cfeb0 con 0x7f7e38103440 2024-05-09T08:26:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.914+0000 7f7e36ffd640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7e240a2c80 0x7f7e240a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:26:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:10.917+0000 7f7e3c8d8640 1 -- 172.21.15.28:0/2209922117 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7e3009ca20 con 0x7f7e38103440 2024-05-09T08:26:11.101 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:10.918+0000 7ff0df322200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:26:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.105+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f7e3802d470 con 0x7f7e38103440 2024-05-09T08:26:11.107 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:10.910+0000 7fca9db38200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:26:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.110+0000 7f7e3c8d8640 1 -- 172.21.15.28:0/2209922117 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f7e3009c320 con 0x7f7e38103440 2024-05-09T08:26:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.112+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7e240a2c80 msgr2=0x7f7e240a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.112+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7e240a2c80 0x7f7e240a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.112+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38103440 msgr2=0x7f7e3819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.112+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38103440 0x7f7e3819f050 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000eb10 tx=0x7f7e3000c2d0 comp rx=0 tx=0).stop 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.113+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 shutdown_connections 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.113+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f7e240a2c80 0x7f7e240a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.113+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e38103440 0x7f7e3819f050 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.113+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e38102a80 0x7f7e38197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.113+0000 7f7e3d8da640 1 --2- 172.21.15.28:0/2209922117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e38101890 0x7f7e38197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.113+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 >> 172.21.15.28:0/2209922117 conn(0x7f7e380fce20 msgr2=0x7f7e38105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.113+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 shutdown_connections 2024-05-09T08:26:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:11.113+0000 7f7e3d8da640 1 -- 172.21.15.28:0/2209922117 wait complete. 2024-05-09T08:26:11.124 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:10.844+0000 7fc80319c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:26:11.124 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:10.957+0000 7fc80319c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:26:11.125 INFO:teuthology.run_tasks:Running task kclient... 2024-05-09T08:26:11.139 INFO:tasks.kclient:Mounting kernel clients... 2024-05-09T08:26:11.140 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-09T08:26:11.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:11.140 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-05-09T08:26:11.157 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-09T08:26:11.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:11.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:26:11.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:11.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-05-09T08:26:11.339 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-05-09T08:26:11.339 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T08:26:11.339 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T08:26:11.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-09T08:26:11.394 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:11.297+0000 7fc80319c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:26:11.602 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:11.225+0000 7ff0df322200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:26:11.602 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:11.310+0000 7ff0df322200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:26:11.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:11 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/725406709' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-09T08:26:11.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:11 smithi165 ceph-mon[34540]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-05-09T08:26:11.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:11 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2488708028' entity='client.admin' 2024-05-09T08:26:11.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:11 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2209922117' entity='client.admin' 2024-05-09T08:26:11.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:11 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/725406709' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-09T08:26:11.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:11 smithi184 ceph-mon[34509]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-05-09T08:26:11.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:11 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2488708028' entity='client.admin' 2024-05-09T08:26:11.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:11 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2209922117' entity='client.admin' 2024-05-09T08:26:11.607 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:11.218+0000 7fca9db38200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:26:11.607 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:11.301+0000 7fca9db38200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:26:11.611 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:26:11.682 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:11.392+0000 7fc80319c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:26:11.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:11 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/725406709' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-09T08:26:11.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:11 smithi028 ceph-mon[30119]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-05-09T08:26:11.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:11 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2488708028' entity='client.admin' 2024-05-09T08:26:11.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:11 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2209922117' entity='client.admin' 2024-05-09T08:26:12.830 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:12.487+0000 7fca9db38200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:26:12.830 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:12.659+0000 7fca9db38200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:26:12.830 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:12.830+0000 7fca9db38200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:26:12.830 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:12.490+0000 7ff0df322200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:26:12.831 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:12.659+0000 7ff0df322200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:26:12.831 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:12.830+0000 7ff0df322200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:26:12.831 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:12 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:12.699+0000 7fc80319c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:26:13.135 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:12 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:12.882+0000 7fc80319c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:26:13.136 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:13.064+0000 7fc80319c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:26:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 -- 172.21.15.28:0/1333758904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27406a030 msgr2=0x7fd27406a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 --2- 172.21.15.28:0/1333758904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27406a030 0x7fd27406a410 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd270009990 tx=0x7fd270009ca0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 -- 172.21.15.28:0/1333758904 shutdown_connections 2024-05-09T08:26:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 --2- 172.21.15.28:0/1333758904 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd27410edd0 0x7fd2741111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 --2- 172.21.15.28:0/1333758904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27406a030 0x7fd27406a410 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 --2- 172.21.15.28:0/1333758904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27406a950 0x7fd27410e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 -- 172.21.15.28:0/1333758904 >> 172.21.15.28:0/1333758904 conn(0x7fd2740781d0 msgr2=0x7fd2740785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 -- 172.21.15.28:0/1333758904 shutdown_connections 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 -- 172.21.15.28:0/1333758904 wait complete. 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 Processor -- start 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.368+0000 7fd27c5df700 1 -- start start 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27c5df700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27406a030 0x7fd27419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27c5df700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd27406a950 0x7fd27419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27c5df700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27410edd0 0x7fd27419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27c5df700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd274114840 con 0x7fd27410edd0 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27c5df700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd2741146c0 con 0x7fd27406a030 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27c5df700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd2741149c0 con 0x7fd27406a950 2024-05-09T08:26:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27ab7c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27410edd0 0x7fd27419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27a37b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27406a030 0x7fd27419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27ab7c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27410edd0 0x7fd27419f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59058/0 (socket says 172.21.15.28:59058) 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27ab7c700 1 -- 172.21.15.28:0/3932818170 learned_addr learned my addr 172.21.15.28:0/3932818170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27a37b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27406a030 0x7fd27419add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:58092/0 (socket says 172.21.15.28:58092) 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd279b7a700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd27406a950 0x7fd27419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27ab7c700 1 -- 172.21.15.28:0/3932818170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd27406a950 msgr2=0x7fd27419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27ab7c700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd27406a950 0x7fd27419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27ab7c700 1 -- 172.21.15.28:0/3932818170 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27406a030 msgr2=0x7fd27419add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27ab7c700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27406a030 0x7fd27419add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27ab7c700 1 -- 172.21.15.28:0/3932818170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27419fde0 con 0x7fd27410edd0 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd279b7a700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd27406a950 0x7fd27419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.369+0000 7fd27ab7c700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27410edd0 0x7fd27419f6a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00ef90 tx=0x7fd26c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.370+0000 7fd2677fe700 1 -- 172.21.15.28:0/3932818170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0092e0 con 0x7fd27410edd0 2024-05-09T08:26:13.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.370+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2741a0070 con 0x7fd27410edd0 2024-05-09T08:26:13.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.370+0000 7fd2677fe700 1 -- 172.21.15.28:0/3932818170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd26c004bb0 con 0x7fd27410edd0 2024-05-09T08:26:13.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.370+0000 7fd2677fe700 1 -- 172.21.15.28:0/3932818170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c004280 con 0x7fd27410edd0 2024-05-09T08:26:13.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.370+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2741a7950 con 0x7fd27410edd0 2024-05-09T08:26:13.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.371+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd274069010 con 0x7fd27410edd0 2024-05-09T08:26:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.373+0000 7fd2677fe700 1 -- 172.21.15.28:0/3932818170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fd26c01b030 con 0x7fd27410edd0 2024-05-09T08:26:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.374+0000 7fd2677fe700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd2600a2c70 0x7fd2600a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.374+0000 7fd27a37b700 1 -- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd2600a2c70 msgr2=0x7fd2600a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.374+0000 7fd2677fe700 1 -- 172.21.15.28:0/3932818170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fd26c0cf960 con 0x7fd27410edd0 2024-05-09T08:26:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.374+0000 7fd27a37b700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd2600a2c70 0x7fd2600a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:26:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.376+0000 7fd2677fe700 1 -- 172.21.15.28:0/3932818170 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd26c022020 con 0x7fd27410edd0 2024-05-09T08:26:13.411 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:13.116+0000 7ff0df322200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:26:13.411 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:13.207+0000 7ff0df322200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:26:13.412 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:13.412+0000 7ff0df322200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:26:13.415 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:13.118+0000 7fca9db38200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:26:13.415 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:13.211+0000 7fca9db38200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:26:13.415 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:13.415+0000 7fca9db38200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:26:13.432 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:13.383+0000 7fc80319c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:26:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.575+0000 7fd27a37b700 1 -- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd2600a2c70 msgr2=0x7fd2600a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.575+0000 7fd27a37b700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd2600a2c70 0x7fd2600a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:26:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.635+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd27402db80 con 0x7fd27410edd0 2024-05-09T08:26:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.635+0000 7fd2677fe700 1 -- 172.21.15.28:0/3932818170 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7fd26c010040 con 0x7fd27410edd0 2024-05-09T08:26:13.636 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T08:26:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd2600a2c70 msgr2=0x7fd2600a5130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:26:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd2600a2c70 0x7fd2600a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27410edd0 msgr2=0x7fd27419f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27410edd0 0x7fd27419f6a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00ef90 tx=0x7fd26c00c3a0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 shutdown_connections 2024-05-09T08:26:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fd2600a2c70 0x7fd2600a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd27410edd0 0x7fd27419f6a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27406a030 0x7fd27419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 --2- 172.21.15.28:0/3932818170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd27406a950 0x7fd27419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.637+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 >> 172.21.15.28:0/3932818170 conn(0x7fd2740781d0 msgr2=0x7fd274110320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.638+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 shutdown_connections 2024-05-09T08:26:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:13.638+0000 7fd27c5df700 1 -- 172.21.15.28:0/3932818170 wait complete. 2024-05-09T08:26:13.688 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:13.609+0000 7fca9db38200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:26:13.688 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:13.604+0000 7ff0df322200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:26:13.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:13 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3932818170' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:26:13.913 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:13.483+0000 7fc80319c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:26:13.913 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:13.705+0000 7fc80319c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:26:14.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:13 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3932818170' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:26:14.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:13 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3932818170' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:26:14.182 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:13.912+0000 7fc80319c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:26:14.343 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T08:26:14.343 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T08:26:14.343 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-05-09T08:26:14.344 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T08:26:14.344 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T08:26:14.344 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T08:26:14.344 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T08:26:14.344 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T08:26:14.344 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T08:26:14.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:14.344 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-05-09T08:26:14.361 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T08:26:14.361 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T08:26:14.361 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T08:26:14.361 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-05-09T08:26:14.362 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T08:26:14.363 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42101sec preferred_lft 42101sec 2024-05-09T08:26:14.363 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-05-09T08:26:14.363 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:26:14.363 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:26:14.363 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-05-09T08:26:14.363 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-05-09T08:26:14.363 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T08:26:14.363 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:26:14.363 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:26:14.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-05-09T08:26:14.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-05-09T08:26:14.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-05-09T08:26:14.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T08:26:14.364 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:26:14.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:26:14.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:26:14.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:14.518 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T08:26:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-05-09T08:26:14.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:14.546 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-05-09T08:26:14.601 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:14 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:14.207+0000 7ff0df322200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:26:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-05-09T08:26:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T08:26:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T08:26:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T08:26:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T08:26:14.605 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:26:14.605 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:26:14.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T08:26:14.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T08:26:14.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T08:26:14.606 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:26:14.607 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:14 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:14.212+0000 7fca9db38200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:26:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:26:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:26:14.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:14.739 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-05-09T08:26:14.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:14.755 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-05-09T08:26:14.812 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:26:14.812 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:26:14.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T08:26:14.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-09T08:26:14.812 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:26:14.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:26:14.912 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:14 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:14.580+0000 7fc80319c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:26:14.923 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:14 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:14.628+0000 7ff0df322200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:26:14.923 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:14 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:14.724+0000 7ff0df322200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:26:14.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:26:14.935 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-09T08:26:14.935 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:26:14.935 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:26:14.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-09T08:26:14.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T08:26:14.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-09T08:26:14.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-09T08:26:14.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-09T08:26:14.936 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:26:14.938 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:14 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:14.640+0000 7fca9db38200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:26:14.938 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:14 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:14.737+0000 7fca9db38200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:26:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:26:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:26:15.058 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:26:15.058 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-09T08:26:15.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-05-09T08:26:15.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T08:26:15.058 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-09T08:26:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:26:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:26:15.143 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:26:15.143 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T08:26:15.159 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T08:26:15.159 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:26:15.159 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T08:26:15.167 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:15.053+0000 7fc80319c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:26:15.182 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-09T08:26:15.182 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-09T08:26:15.182 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00 2024-05-09T08:26:15.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:15.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-05-09T08:26:15.267 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-05-09T08:26:15.267 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-05-09T08:26:15.351 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:14 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:14.923+0000 7ff0df322200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:26:15.392 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-05-09T08:26:15.392 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-05-09T08:26:15.392 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.165:6789,172.21.15.184:6789" 2024-05-09T08:26:15.392 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@3891b212-0ddc-11ef-bc98-c7b262605968.cephfs=/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00 2024-05-09T08:26:15.393 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync,name=0,key=0,mon_addr=172.21.15.28:6789/172.21.15.165:6789/172.21.15.184:6789" will pass to kernel 2024-05-09T08:26:15.393 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T08:26:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-05-09T08:26:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T08:26:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T08:26:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T08:26:15.395 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T08:26:15.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:15.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T08:26:15.418 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:15.166+0000 7fc80319c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:26:15.418 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:15.391+0000 7fc80319c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:26:15.425 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T08:26:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T08:26:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:26:15.525 INFO:teuthology.orchestra.run.smithi028.stdout:client15549 2024-05-09T08:26:15.526 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:26:15.526 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:26:15.544 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T08:26:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T08:26:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:26:15.645 INFO:teuthology.orchestra.run.smithi028.stdout:client15549 2024-05-09T08:26:15.646 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T08:26:15.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T08:26:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:26:15.706 INFO:teuthology.orchestra.run.smithi028.stdout:client15549 2024-05-09T08:26:15.706 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/3891b212-0ddc-11ef-bc98-c7b262605968.client15549/status') 2024-05-09T08:26:15.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:26:15.768 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.15549 (3)192.168.0.1:0/19799291 2024-05-09T08:26:15.768 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-05-09T08:26:15.768 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-05-09T08:26:15.768 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-05-09T08:26:15.768 INFO:teuthology.orchestra.run.smithi028.stderr:68 bytes copied, 4.2062e-05 s, 1.6 MB/s 2024-05-09T08:26:15.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:26:15.776 INFO:teuthology.run_tasks:Running task exec... 2024-05-09T08:26:15.785 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:26:15.785 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:26:15.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-05-09T08:26:15.789 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:15.450+0000 7ff0df322200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:26:15.789 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:15.543+0000 7ff0df322200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:26:15.801 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:14 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:14.938+0000 7fca9db38200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:26:15.802 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:15.465+0000 7fca9db38200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:26:15.802 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:15.557+0000 7fca9db38200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:26:15.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 -- 172.21.15.28:0/275271295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4102340 msgr2=0x7efcd4108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:15.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 --2- 172.21.15.28:0/275271295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4102340 0x7efcd4108bd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7efccc009ca0 tx=0x7efccc02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 -- 172.21.15.28:0/275271295 shutdown_connections 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 --2- 172.21.15.28:0/275271295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4102340 0x7efcd4108bd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 --2- 172.21.15.28:0/275271295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcd4101980 0x7efcd4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 --2- 172.21.15.28:0/275271295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd4100790 0x7efcd4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 -- 172.21.15.28:0/275271295 >> 172.21.15.28:0/275271295 conn(0x7efcd40fbd00 msgr2=0x7efcd40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 -- 172.21.15.28:0/275271295 shutdown_connections 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 -- 172.21.15.28:0/275271295 wait complete. 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 Processor -- start 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 -- start start 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcd4100790 0x7efcd4069a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.892+0000 7efcdc56a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4101980 0x7efcd419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcdc56a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd4102340 0x7efcd419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcdc56a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410c460 con 0x7efcd4101980 2024-05-09T08:26:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcdc56a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410c2e0 con 0x7efcd4102340 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcdc56a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410c5e0 con 0x7efcd4100790 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcd9b05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4101980 0x7efcd419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcd9b05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4101980 0x7efcd419c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38008/0 (socket says 172.21.15.28:38008) 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcd9b05640 1 -- 172.21.15.28:0/1969841540 learned_addr learned my addr 172.21.15.28:0/1969841540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcda306640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcd4100790 0x7efcd4069a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcdab07640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd4102340 0x7efcd419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcd9b05640 1 -- 172.21.15.28:0/1969841540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcd4100790 msgr2=0x7efcd4069a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcd9b05640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcd4100790 0x7efcd4069a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcd9b05640 1 -- 172.21.15.28:0/1969841540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd4102340 msgr2=0x7efcd419ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcd9b05640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd4102340 0x7efcd419ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcd9b05640 1 -- 172.21.15.28:0/1969841540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd419f730 con 0x7efcd4101980 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcda306640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcd4100790 0x7efcd4069a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcd9b05640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4101980 0x7efcd419c800 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efcc400e9f0 tx=0x7efcc400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efccb7fe640 1 -- 172.21.15.28:0/1969841540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc400cd80 con 0x7efcd4101980 2024-05-09T08:26:15.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efccb7fe640 1 -- 172.21.15.28:0/1969841540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcc4005500 con 0x7efcd4101980 2024-05-09T08:26:15.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd419f9c0 con 0x7efcd4101980 2024-05-09T08:26:15.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.893+0000 7efccb7fe640 1 -- 172.21.15.28:0/1969841540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc4010690 con 0x7efcd4101980 2024-05-09T08:26:15.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.894+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd419fec0 con 0x7efcd4101980 2024-05-09T08:26:15.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.894+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcd41079e0 con 0x7efcd4101980 2024-05-09T08:26:15.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.897+0000 7efccb7fe640 1 -- 172.21.15.28:0/1969841540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7efcc40046c0 con 0x7efcd4101980 2024-05-09T08:26:15.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.897+0000 7efccb7fe640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efcc00a2c80 0x7efcc00a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:15.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.897+0000 7efcda306640 1 -- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efcc00a2c80 msgr2=0x7efcc00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:15.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.897+0000 7efccb7fe640 1 -- 172.21.15.28:0/1969841540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7efcc40cf870 con 0x7efcd4101980 2024-05-09T08:26:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.897+0000 7efcda306640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efcc00a2c80 0x7efcc00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:26:15.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:15.902+0000 7efccb7fe640 1 -- 172.21.15.28:0/1969841540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efcc409c3e0 con 0x7efcd4101980 2024-05-09T08:26:16.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:15.966+0000 7fc80319c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:26:16.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:16.069+0000 7fc80319c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:26:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.098+0000 7efcda306640 1 -- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efcc00a2c80 msgr2=0x7efcc00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.098+0000 7efcda306640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efcc00a2c80 0x7efcc00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:26:16.101 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:15.789+0000 7ff0df322200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:26:16.101 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:15.923+0000 7ff0df322200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:26:16.101 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:16.018+0000 7ff0df322200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:26:16.107 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:15.802+0000 7fca9db38200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:26:16.108 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:15.934+0000 7fca9db38200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:26:16.108 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:16.028+0000 7fca9db38200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:26:16.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.128+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7efcd4062430 con 0x7efcd4101980 2024-05-09T08:26:16.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.180+0000 7efccb7fe640 1 -- 172.21.15.28:0/1969841540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v28) ==== 117+0+0 (secure 0 0 0) 0x7efcc409bce0 con 0x7efcd4101980 2024-05-09T08:26:16.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efcc00a2c80 msgr2=0x7efcc00a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efcc00a2c80 0x7efcc00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4101980 msgr2=0x7efcd419c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4101980 0x7efcd419c800 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efcc400e9f0 tx=0x7efcc400edf0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 shutdown_connections 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7efcc00a2c80 0x7efcc00a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcd4102340 0x7efcd419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcd4101980 0x7efcd419c800 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 --2- 172.21.15.28:0/1969841540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcd4100790 0x7efcd4069a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 >> 172.21.15.28:0/1969841540 conn(0x7efcd40fbd00 msgr2=0x7efcd4104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 shutdown_connections 2024-05-09T08:26:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.183+0000 7efcdc56a640 1 -- 172.21.15.28:0/1969841540 wait complete. 2024-05-09T08:26:16.197 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-09T08:26:16.208 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-09T08:26:16.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:26:16.336 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:16 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1969841540' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-09T08:26:16.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 -- 172.21.15.28:0/1863697770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 msgr2=0x7f3af41099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 --2- 172.21.15.28:0/1863697770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 0x7f3af41099a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009ca0 tx=0x7f3aec02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 -- 172.21.15.28:0/1863697770 shutdown_connections 2024-05-09T08:26:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 --2- 172.21.15.28:0/1863697770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 0x7f3af41099a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 --2- 172.21.15.28:0/1863697770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4104c80 0x7f3af4107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 --2- 172.21.15.28:0/1863697770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af4102350 0x7f3af4104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 -- 172.21.15.28:0/1863697770 >> 172.21.15.28:0/1863697770 conn(0x7f3af40fbce0 msgr2=0x7f3af40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:16.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 -- 172.21.15.28:0/1863697770 shutdown_connections 2024-05-09T08:26:16.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 -- 172.21.15.28:0/1863697770 wait complete. 2024-05-09T08:26:16.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 Processor -- start 2024-05-09T08:26:16.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.359+0000 7f3afab28640 1 -- start start 2024-05-09T08:26:16.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3afab28640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af4102350 0x7f3af4101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:16.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3afab28640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4104c80 0x7f3af4100300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:16.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3afab28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 0x7f3af4100840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:16.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3afab28640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3af410d4f0 con 0x7f3af41075b0 2024-05-09T08:26:16.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3afab28640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3af410d370 con 0x7f3af4102350 2024-05-09T08:26:16.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3afab28640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3af410d670 con 0x7f3af4104c80 2024-05-09T08:26:16.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af90c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 0x7f3af4100840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:16.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af90c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 0x7f3af4100840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38022/0 (socket says 172.21.15.28:38022) 2024-05-09T08:26:16.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af90c5640 1 -- 172.21.15.28:0/947862782 learned_addr learned my addr 172.21.15.28:0/947862782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:16.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af3fff640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4104c80 0x7f3af4100300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:16.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af88c4640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af4102350 0x7f3af4101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:16.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af90c5640 1 -- 172.21.15.28:0/947862782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4104c80 msgr2=0x7f3af4100300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:16.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af90c5640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4104c80 0x7f3af4100300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af90c5640 1 -- 172.21.15.28:0/947862782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af4102350 msgr2=0x7f3af4101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:16.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af90c5640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af4102350 0x7f3af4101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af90c5640 1 -- 172.21.15.28:0/947862782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af41010d0 con 0x7f3af41075b0 2024-05-09T08:26:16.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af90c5640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 0x7f3af4100840 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3aec031ae0 tx=0x7f3aec031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:16.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.360+0000 7f3af1ffb640 1 -- 172.21.15.28:0/947862782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec003c30 con 0x7f3af41075b0 2024-05-09T08:26:16.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.361+0000 7f3af1ffb640 1 -- 172.21.15.28:0/947862782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3aec003dd0 con 0x7f3af41075b0 2024-05-09T08:26:16.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.361+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af41a1c80 con 0x7f3af41075b0 2024-05-09T08:26:16.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.361+0000 7f3af1ffb640 1 -- 172.21.15.28:0/947862782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec02fcb0 con 0x7f3af41075b0 2024-05-09T08:26:16.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.361+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af41a21d0 con 0x7f3af41075b0 2024-05-09T08:26:16.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.362+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3af40fd7d0 con 0x7f3af41075b0 2024-05-09T08:26:16.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.363+0000 7f3af1ffb640 1 -- 172.21.15.28:0/947862782 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f3aec004050 con 0x7f3af41075b0 2024-05-09T08:26:16.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.364+0000 7f3af1ffb640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3ae40a2be0 0x7f3ae40a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:16.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.364+0000 7f3af88c4640 1 -- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3ae40a2be0 msgr2=0x7f3ae40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:16.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.364+0000 7f3af1ffb640 1 -- 172.21.15.28:0/947862782 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f3aec0f0710 con 0x7f3af41075b0 2024-05-09T08:26:16.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.364+0000 7f3af88c4640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3ae40a2be0 0x7f3ae40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:26:16.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.368+0000 7f3af1ffb640 1 -- 172.21.15.28:0/947862782 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3aec0bd220 con 0x7f3af41075b0 2024-05-09T08:26:16.438 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:16 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1969841540' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-09T08:26:16.438 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:16.133+0000 7ff0df322200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:26:16.447 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:16 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1969841540' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-09T08:26:16.447 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:16.143+0000 7fca9db38200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:26:16.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.557+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3af4062430 con 0x7f3af41075b0 2024-05-09T08:26:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.559+0000 7f3af1ffb640 1 -- 172.21.15.28:0/947862782 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24899 (secure 0 0 0) 0x7f3aec0bcb20 con 0x7f3af41075b0 2024-05-09T08:26:16.560 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:16.560 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:25:52.396720+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:26:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3ae40a2be0 msgr2=0x7f3ae40a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:26:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3ae40a2be0 0x7f3ae40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 msgr2=0x7f3af4100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 0x7f3af4100840 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3aec031ae0 tx=0x7f3aec031ee0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 shutdown_connections 2024-05-09T08:26:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f3ae40a2be0 0x7f3ae40a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af41075b0 0x7f3af4100840 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af4104c80 0x7f3af4100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 --2- 172.21.15.28:0/947862782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af4102350 0x7f3af4101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.561+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 >> 172.21.15.28:0/947862782 conn(0x7f3af40fbce0 msgr2=0x7f3af4102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.562+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 shutdown_connections 2024-05-09T08:26:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.562+0000 7f3afab28640 1 -- 172.21.15.28:0/947862782 wait complete. 2024-05-09T08:26:16.577 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:26:16.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:26:16.597 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:16.336+0000 7fc80319c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:26:16.597 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:16.485+0000 7fc80319c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:26:16.597 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:16.597+0000 7fc80319c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:26:16.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.688+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1651212894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4100770 msgr2=0x7fb4b4100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:16.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.688+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1651212894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4100770 0x7fb4b4100b90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009900 tx=0x7fb4a402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:16.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.688+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1651212894 shutdown_connections 2024-05-09T08:26:16.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.688+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1651212894 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b4102320 0x7fb4b4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.688+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1651212894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4b4101960 0x7fb4b4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.688+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1651212894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4100770 0x7fb4b4100b90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.688+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1651212894 >> 172.21.15.28:0/1651212894 conn(0x7fb4b40fbce0 msgr2=0x7fb4b40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:16.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.688+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1651212894 shutdown_connections 2024-05-09T08:26:16.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.688+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1651212894 wait complete. 2024-05-09T08:26:16.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.689+0000 7fb4b8c9e640 1 Processor -- start 2024-05-09T08:26:16.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.689+0000 7fb4b8c9e640 1 -- start start 2024-05-09T08:26:16.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.689+0000 7fb4b8c9e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b4100770 0x7fb4b4197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.689+0000 7fb4b8c9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4101960 0x7fb4b4198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.689+0000 7fb4b8c9e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4b4102320 0x7fb4b419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.689+0000 7fb4b8c9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b410c3c0 con 0x7fb4b4101960 2024-05-09T08:26:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.689+0000 7fb4b8c9e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b410c240 con 0x7fb4b4102320 2024-05-09T08:26:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.689+0000 7fb4b8c9e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b410c540 con 0x7fb4b4100770 2024-05-09T08:26:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4101960 0x7fb4b4198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4101960 0x7fb4b4198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38036/0 (socket says 172.21.15.28:38036) 2024-05-09T08:26:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4abfff640 1 -- 172.21.15.28:0/1013192895 learned_addr learned my addr 172.21.15.28:0/1013192895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4b2d9d640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4b4102320 0x7fb4b419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4b259c640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b4100770 0x7fb4b4197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4abfff640 1 -- 172.21.15.28:0/1013192895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b4100770 msgr2=0x7fb4b4197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4abfff640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b4100770 0x7fb4b4197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4abfff640 1 -- 172.21.15.28:0/1013192895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4b4102320 msgr2=0x7fb4b419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4abfff640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4b4102320 0x7fb4b419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4abfff640 1 -- 172.21.15.28:0/1013192895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b419f7e0 con 0x7fb4b4101960 2024-05-09T08:26:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4abfff640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4101960 0x7fb4b4198100 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00e9f0 tx=0x7fb49c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4ab7fe640 1 -- 172.21.15.28:0/1013192895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00cd80 con 0x7fb4b4101960 2024-05-09T08:26:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b419fad0 con 0x7fb4b4101960 2024-05-09T08:26:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4ab7fe640 1 -- 172.21.15.28:0/1013192895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb49c010910 con 0x7fb4b4101960 2024-05-09T08:26:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4ab7fe640 1 -- 172.21.15.28:0/1013192895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c0188f0 con 0x7fb4b4101960 2024-05-09T08:26:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.690+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b41a0020 con 0x7fb4b4101960 2024-05-09T08:26:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.691+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4b41a01f0 con 0x7fb4b4101960 2024-05-09T08:26:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.693+0000 7fb4ab7fe640 1 -- 172.21.15.28:0/1013192895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fb49c010ac0 con 0x7fb4b4101960 2024-05-09T08:26:16.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.693+0000 7fb4ab7fe640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb4a00a2be0 0x7fb4a00a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:16.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.693+0000 7fb4b259c640 1 -- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb4a00a2be0 msgr2=0x7fb4a00a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:16.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.694+0000 7fb4ab7fe640 1 -- 172.21.15.28:0/1013192895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fb49c0cfba0 con 0x7fb4b4101960 2024-05-09T08:26:16.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.694+0000 7fb4b259c640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb4a00a2be0 0x7fb4a00a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:26:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.701+0000 7fb4ab7fe640 1 -- 172.21.15.28:0/1013192895 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb49c09c0c0 con 0x7fb4b4101960 2024-05-09T08:26:16.851 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:16.438+0000 7ff0df322200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:26:16.857 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:16.447+0000 7fca9db38200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:26:16.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.888+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb4b41a04e0 con 0x7fb4b4101960 2024-05-09T08:26:16.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.888+0000 7fb4ab7fe640 1 -- 172.21.15.28:0/1013192895 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fb49c0a03c0 con 0x7fb4b4101960 2024-05-09T08:26:16.889 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:26:16.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.890+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb4a00a2be0 msgr2=0x7fb4a00a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:26:16.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.890+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb4a00a2be0 0x7fb4a00a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.890+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4101960 msgr2=0x7fb4b4198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:16.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.890+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4101960 0x7fb4b4198100 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00e9f0 tx=0x7fb49c00eeb0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.891+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 shutdown_connections 2024-05-09T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.891+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb4a00a2be0 0x7fb4a00a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.891+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4b4102320 0x7fb4b419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.891+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b4101960 0x7fb4b4198100 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.891+0000 7fb4b8c9e640 1 --2- 172.21.15.28:0/1013192895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4b4100770 0x7fb4b4197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.891+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 >> 172.21.15.28:0/1013192895 conn(0x7fb4b40fbce0 msgr2=0x7fb4b40689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.891+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 shutdown_connections 2024-05-09T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:16.891+0000 7fb4b8c9e640 1 -- 172.21.15.28:0/1013192895 wait complete. 2024-05-09T08:26:16.902 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:16.732+0000 7fc80319c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:26:16.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:26:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.054+0000 7fb83b310640 1 -- 172.21.15.28:0/1737066043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 msgr2=0x7fb834108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.054+0000 7fb83b310640 1 --2- 172.21.15.28:0/1737066043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 0x7fb834108bb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009ca0 tx=0x7fb82c02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.054+0000 7fb83b310640 1 -- 172.21.15.28:0/1737066043 shutdown_connections 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.054+0000 7fb83b310640 1 --2- 172.21.15.28:0/1737066043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 0x7fb834108bb0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.054+0000 7fb83b310640 1 --2- 172.21.15.28:0/1737066043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb834101960 0x7fb834101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.054+0000 7fb83b310640 1 --2- 172.21.15.28:0/1737066043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb834100770 0x7fb834100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 -- 172.21.15.28:0/1737066043 >> 172.21.15.28:0/1737066043 conn(0x7fb8340fbce0 msgr2=0x7fb8340fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 -- 172.21.15.28:0/1737066043 shutdown_connections 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 -- 172.21.15.28:0/1737066043 wait complete. 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 Processor -- start 2024-05-09T08:26:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 -- start start 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb834100770 0x7fb834199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb834101960 0x7fb83419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 0x7fb8341a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb83410d570 con 0x7fb834102320 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb83410d3f0 con 0x7fb834100770 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.055+0000 7fb83b310640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb83410d6f0 con 0x7fb834101960 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8398ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 0x7fb8341a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8398ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 0x7fb8341a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38052/0 (socket says 172.21.15.28:38052) 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8390ac640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb834100770 0x7fb834199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8398ad640 1 -- 172.21.15.28:0/3401304440 learned_addr learned my addr 172.21.15.28:0/3401304440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8388ab640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb834101960 0x7fb83419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8398ad640 1 -- 172.21.15.28:0/3401304440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb834101960 msgr2=0x7fb83419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8398ad640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb834101960 0x7fb83419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8398ad640 1 -- 172.21.15.28:0/3401304440 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb834100770 msgr2=0x7fb834199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8398ad640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb834100770 0x7fb834199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8398ad640 1 -- 172.21.15.28:0/3401304440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8341a1960 con 0x7fb834102320 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8398ad640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 0x7fb8341a12f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0319f0 tx=0x7fb82c02f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8267fc640 1 -- 172.21.15.28:0/3401304440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c003c70 con 0x7fb834102320 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8341a1bf0 con 0x7fb834102320 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8267fc640 1 -- 172.21.15.28:0/3401304440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb82c003e10 con 0x7fb834102320 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb8267fc640 1 -- 172.21.15.28:0/3401304440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c02fd30 con 0x7fb834102320 2024-05-09T08:26:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.056+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8341a2190 con 0x7fb834102320 2024-05-09T08:26:17.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.057+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8341a2420 con 0x7fb834102320 2024-05-09T08:26:17.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.059+0000 7fb8267fc640 1 -- 172.21.15.28:0/3401304440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fb82c004b40 con 0x7fb834102320 2024-05-09T08:26:17.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.059+0000 7fb8267fc640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8200a2be0 0x7fb8200a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.059+0000 7fb8390ac640 1 -- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8200a2be0 msgr2=0x7fb8200a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:17.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.060+0000 7fb8267fc640 1 -- 172.21.15.28:0/3401304440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fb82c0f0bb0 con 0x7fb834102320 2024-05-09T08:26:17.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.060+0000 7fb8390ac640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8200a2be0 0x7fb8200a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:26:17.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.063+0000 7fb8267fc640 1 -- 172.21.15.28:0/3401304440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb82c0bd7b0 con 0x7fb834102320 2024-05-09T08:26:17.182 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:17.058+0000 7fc80319c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.248+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb8341a25f0 con 0x7fb834102320 2024-05-09T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.248+0000 7fb8267fc640 1 -- 172.21.15.28:0/3401304440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fb82c0bd0b0 con 0x7fb834102320 2024-05-09T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8200a2be0 msgr2=0x7fb8200a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8200a2be0 0x7fb8200a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 msgr2=0x7fb8341a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 0x7fb8341a12f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0319f0 tx=0x7fb82c02f450 comp rx=0 tx=0).stop 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 shutdown_connections 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fb8200a2be0 0x7fb8200a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb834102320 0x7fb8341a12f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb834101960 0x7fb83419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 --2- 172.21.15.28:0/3401304440 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb834100770 0x7fb834199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 >> 172.21.15.28:0/3401304440 conn(0x7fb8340fbce0 msgr2=0x7fb834109570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 shutdown_connections 2024-05-09T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.251+0000 7fb83b310640 1 -- 172.21.15.28:0/3401304440 wait complete. 2024-05-09T08:26:17.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:26:17.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:17 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1969841540' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-09T08:26:17.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:17 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:26:17.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:17 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/947862782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:17.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:17 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1013192895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:17.352 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:17 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:17.040+0000 7ff0df322200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:26:17.352 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:26:17 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y[35924]: 2024-05-09T08:26:17.254+0000 7ff0df322200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:26:17.357 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:17 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:17.044+0000 7fca9db38200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:26:17.357 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:26:17 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z[36243]: 2024-05-09T08:26:17.260+0000 7fca9db38200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:26:17.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:17 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1969841540' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-09T08:26:17.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:17 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:26:17.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:17 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/947862782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:17.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:17 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1013192895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.374+0000 7f22006c3640 1 -- 172.21.15.28:0/3086382420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 msgr2=0x7f21f8100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.374+0000 7f22006c3640 1 --2- 172.21.15.28:0/3086382420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 0x7f21f8100b90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009900 tx=0x7f21f402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.374+0000 7f22006c3640 1 -- 172.21.15.28:0/3086382420 shutdown_connections 2024-05-09T08:26:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.374+0000 7f22006c3640 1 --2- 172.21.15.28:0/3086382420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f8102320 0x7f21f8108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.374+0000 7f22006c3640 1 --2- 172.21.15.28:0/3086382420 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f8101960 0x7f21f8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.374+0000 7f22006c3640 1 --2- 172.21.15.28:0/3086382420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 0x7f21f8100b90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.374+0000 7f22006c3640 1 -- 172.21.15.28:0/3086382420 >> 172.21.15.28:0/3086382420 conn(0x7f21f80fbce0 msgr2=0x7f21f80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.374+0000 7f22006c3640 1 -- 172.21.15.28:0/3086382420 shutdown_connections 2024-05-09T08:26:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.374+0000 7f22006c3640 1 -- 172.21.15.28:0/3086382420 wait complete. 2024-05-09T08:26:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f22006c3640 1 Processor -- start 2024-05-09T08:26:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f22006c3640 1 -- start start 2024-05-09T08:26:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f22006c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 0x7f21f8197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f22006c3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f8101960 0x7f21f8198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f22006c3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f8102320 0x7f21f819f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f22006c3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f21f810c3e0 con 0x7f21f8100770 2024-05-09T08:26:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f22006c3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f21f810c260 con 0x7f21f8102320 2024-05-09T08:26:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f22006c3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f21f810c560 con 0x7f21f8101960 2024-05-09T08:26:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fe45f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 0x7f21f8197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fe45f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 0x7f21f8197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38068/0 (socket says 172.21.15.28:38068) 2024-05-09T08:26:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fe45f640 1 -- 172.21.15.28:0/3322385977 learned_addr learned my addr 172.21.15.28:0/3322385977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fec60640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f8102320 0x7f21f819f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fdc5e640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f8101960 0x7f21f8198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fe45f640 1 -- 172.21.15.28:0/3322385977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f8101960 msgr2=0x7f21f8198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fe45f640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f8101960 0x7f21f8198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fe45f640 1 -- 172.21.15.28:0/3322385977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f8102320 msgr2=0x7f21f819f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fe45f640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f8102320 0x7f21f819f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fe45f640 1 -- 172.21.15.28:0/3322385977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f819f860 con 0x7f21f8100770 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.375+0000 7f21fe45f640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 0x7f21f8197c50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f21f4031610 tx=0x7f21f40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.376+0000 7f21eb7fe640 1 -- 172.21.15.28:0/3322385977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f403f030 con 0x7f21f8100770 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.376+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f819faf0 con 0x7f21f8100770 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.376+0000 7f21eb7fe640 1 -- 172.21.15.28:0/3322385977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21f4003ce0 con 0x7f21f8100770 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.376+0000 7f21eb7fe640 1 -- 172.21.15.28:0/3322385977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4035070 con 0x7f21f8100770 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.376+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f81a0030 con 0x7f21f8100770 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.377+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f81a0200 con 0x7f21f8100770 2024-05-09T08:26:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.378+0000 7f21eb7fe640 1 -- 172.21.15.28:0/3322385977 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f21f4042030 con 0x7f21f8100770 2024-05-09T08:26:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.379+0000 7f21eb7fe640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f21e40a2be0 0x7f21e40a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.379+0000 7f21fdc5e640 1 -- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f21e40a2be0 msgr2=0x7f21e40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.379+0000 7f21eb7fe640 1 -- 172.21.15.28:0/3322385977 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f21f40efd90 con 0x7f21f8100770 2024-05-09T08:26:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.379+0000 7f21fdc5e640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f21e40a2be0 0x7f21e40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:26:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.382+0000 7f21eb7fe640 1 -- 172.21.15.28:0/3322385977 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f21f40bc8a0 con 0x7f21f8100770 2024-05-09T08:26:17.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.569+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f21f81a03d0 con 0x7f21f8100770 2024-05-09T08:26:17.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.569+0000 7f21eb7fe640 1 -- 172.21.15.28:0/3322385977 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f21f40bc1a0 con 0x7f21f8100770 2024-05-09T08:26:17.570 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:26:17.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f21e40a2be0 msgr2=0x7f21e40a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:26:17.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f21e40a2be0 0x7f21e40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 msgr2=0x7f21f8197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 0x7f21f8197c50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f21f4031610 tx=0x7f21f40043b0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 shutdown_connections 2024-05-09T08:26:17.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7f21e40a2be0 0x7f21e40a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f8102320 0x7f21f819f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f21f8101960 0x7f21f8198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 --2- 172.21.15.28:0/3322385977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8100770 0x7f21f8197c50 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.571+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 >> 172.21.15.28:0/3322385977 conn(0x7f21f80fbce0 msgr2=0x7f21f80fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:17.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.572+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 shutdown_connections 2024-05-09T08:26:17.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.572+0000 7f22006c3640 1 -- 172.21.15.28:0/3322385977 wait complete. 2024-05-09T08:26:17.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1969841540' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-09T08:26:17.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:17 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:26:17.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/947862782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:17.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:17 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1013192895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:17.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:26:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.692+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/830236088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102350 msgr2=0x7fcfa0104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.692+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/830236088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102350 0x7fcfa0104740 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009900 tx=0x7fcf9002a410 comp rx=0 tx=0).stop 2024-05-09T08:26:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.692+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/830236088 shutdown_connections 2024-05-09T08:26:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.692+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/830236088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa01075b0 0x7fcfa01099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.692+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/830236088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa0104c80 0x7fcfa0107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.692+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/830236088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0102350 0x7fcfa0104740 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.692+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/830236088 >> 172.21.15.28:0/830236088 conn(0x7fcfa00fbce0 msgr2=0x7fcfa00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.692+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/830236088 shutdown_connections 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.692+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/830236088 wait complete. 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcfa4bd5640 1 Processor -- start 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcfa4bd5640 1 -- start start 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcfa4bd5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa0102350 0x7fcfa0101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcfa4bd5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa0104c80 0x7fcfa0100300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcfa4bd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa01075b0 0x7fcfa0100840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcfa4bd5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa010d510 con 0x7fcfa01075b0 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcfa4bd5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa010d390 con 0x7fcfa0104c80 2024-05-09T08:26:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcfa4bd5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa010d690 con 0x7fcfa0102350 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcf9ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa01075b0 0x7fcfa0100840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcf9ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa01075b0 0x7fcfa0100840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38086/0 (socket says 172.21.15.28:38086) 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcf9ed9d640 1 -- 172.21.15.28:0/2877220996 learned_addr learned my addr 172.21.15.28:0/2877220996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcf97fff640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa0104c80 0x7fcfa0100300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.693+0000 7fcf9e59c640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa0102350 0x7fcfa0101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf9ed9d640 1 -- 172.21.15.28:0/2877220996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa0102350 msgr2=0x7fcfa0101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf9ed9d640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa0102350 0x7fcfa0101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf9ed9d640 1 -- 172.21.15.28:0/2877220996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa0104c80 msgr2=0x7fcfa0100300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf9ed9d640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa0104c80 0x7fcfa0100300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf9ed9d640 1 -- 172.21.15.28:0/2877220996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa01010d0 con 0x7fcfa01075b0 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf97fff640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa0104c80 0x7fcfa0100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf9ed9d640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa01075b0 0x7fcfa0100840 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800eb10 tx=0x7fcf9800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf977fe640 1 -- 172.21.15.28:0/2877220996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98004b20 con 0x7fcfa01075b0 2024-05-09T08:26:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf977fe640 1 -- 172.21.15.28:0/2877220996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf98004cc0 con 0x7fcfa01075b0 2024-05-09T08:26:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa01a1d30 con 0x7fcfa01075b0 2024-05-09T08:26:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcf977fe640 1 -- 172.21.15.28:0/2877220996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98010760 con 0x7fcfa01075b0 2024-05-09T08:26:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.694+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa01a21d0 con 0x7fcfa01075b0 2024-05-09T08:26:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.695+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa00fd7d0 con 0x7fcfa01075b0 2024-05-09T08:26:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.696+0000 7fcf977fe640 1 -- 172.21.15.28:0/2877220996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fcf98005020 con 0x7fcfa01075b0 2024-05-09T08:26:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.697+0000 7fcf977fe640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcf8c0a2be0 0x7fcf8c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.697+0000 7fcf9e59c640 1 -- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcf8c0a2be0 msgr2=0x7fcf8c0a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/953899001 2024-05-09T08:26:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.697+0000 7fcf977fe640 1 -- 172.21.15.28:0/2877220996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fcf980cfae0 con 0x7fcfa01075b0 2024-05-09T08:26:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.697+0000 7fcf9e59c640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcf8c0a2be0 0x7fcf8c0a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:26:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.701+0000 7fcf977fe640 1 -- 172.21.15.28:0/2877220996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcf9809c5f0 con 0x7fcfa01075b0 2024-05-09T08:26:17.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.889+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcfa0062460 con 0x7fcfa01075b0 2024-05-09T08:26:17.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.890+0000 7fcf977fe640 1 -- 172.21.15.28:0/2877220996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fcf9809bef0 con 0x7fcfa01075b0 2024-05-09T08:26:17.890 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcf8c0a2be0 msgr2=0x7fcf8c0a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcf8c0a2be0 0x7fcf8c0a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa01075b0 msgr2=0x7fcfa0100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa01075b0 0x7fcfa0100840 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800eb10 tx=0x7fcf9800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 shutdown_connections 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:6800/953899001,v1:172.21.15.28:6801/953899001] conn(0x7fcf8c0a2be0 0x7fcf8c0a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa01075b0 0x7fcfa0100840 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa0104c80 0x7fcfa0100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 --2- 172.21.15.28:0/2877220996 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa0102350 0x7fcfa0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 >> 172.21.15.28:0/2877220996 conn(0x7fcfa00fbce0 msgr2=0x7fcfa0102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 shutdown_connections 2024-05-09T08:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:17.892+0000 7fcfa4bd5640 1 -- 172.21.15.28:0/2877220996 wait complete. 2024-05-09T08:26:17.902 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:17.725+0000 7fc80319c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:26:17.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:26:18.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.020+0000 7fc172eda640 1 -- 172.21.15.28:0/2602283049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 msgr2=0x7fc16c108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.020+0000 7fc172eda640 1 --2- 172.21.15.28:0/2602283049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 0x7fc16c108bb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc164009ca0 tx=0x7fc16402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.020+0000 7fc172eda640 1 -- 172.21.15.28:0/2602283049 shutdown_connections 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.020+0000 7fc172eda640 1 --2- 172.21.15.28:0/2602283049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 0x7fc16c108bb0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.020+0000 7fc172eda640 1 --2- 172.21.15.28:0/2602283049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc16c101960 0x7fc16c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.020+0000 7fc172eda640 1 --2- 172.21.15.28:0/2602283049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c100770 0x7fc16c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.020+0000 7fc172eda640 1 -- 172.21.15.28:0/2602283049 >> 172.21.15.28:0/2602283049 conn(0x7fc16c0fbce0 msgr2=0x7fc16c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.020+0000 7fc172eda640 1 -- 172.21.15.28:0/2602283049 shutdown_connections 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.020+0000 7fc172eda640 1 -- 172.21.15.28:0/2602283049 wait complete. 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc172eda640 1 Processor -- start 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc172eda640 1 -- start start 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc172eda640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c100770 0x7fc16c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc172eda640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc16c101960 0x7fc16c19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc172eda640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 0x7fc16c1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc172eda640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c10c3f0 con 0x7fc16c102320 2024-05-09T08:26:18.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc172eda640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c10c270 con 0x7fc16c100770 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc172eda640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c10c570 con 0x7fc16c101960 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc171477640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 0x7fc16c1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc171477640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 0x7fc16c1a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38144/0 (socket says 172.21.15.28:38144) 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc171477640 1 -- 172.21.15.28:0/2514934841 learned_addr learned my addr 172.21.15.28:0/2514934841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc170c76640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c100770 0x7fc16c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc16bfff640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc16c101960 0x7fc16c19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc171477640 1 -- 172.21.15.28:0/2514934841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc16c101960 msgr2=0x7fc16c19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc171477640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc16c101960 0x7fc16c19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc171477640 1 -- 172.21.15.28:0/2514934841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c100770 msgr2=0x7fc16c199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc171477640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c100770 0x7fc16c199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc171477640 1 -- 172.21.15.28:0/2514934841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c1a1980 con 0x7fc16c102320 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.021+0000 7fc16bfff640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc16c101960 0x7fc16c19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.022+0000 7fc171477640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 0x7fc16c1a1310 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc164031ae0 tx=0x7fc164031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:18.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.022+0000 7fc169ffb640 1 -- 172.21.15.28:0/2514934841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164003b70 con 0x7fc16c102320 2024-05-09T08:26:18.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.022+0000 7fc172eda640 1 -- 172.21.15.28:0/2514934841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc16c1a1c10 con 0x7fc16c102320 2024-05-09T08:26:18.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.022+0000 7fc169ffb640 1 -- 172.21.15.28:0/2514934841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc164003d10 con 0x7fc16c102320 2024-05-09T08:26:18.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.022+0000 7fc169ffb640 1 -- 172.21.15.28:0/2514934841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16402fc40 con 0x7fc16c102320 2024-05-09T08:26:18.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.022+0000 7fc172eda640 1 -- 172.21.15.28:0/2514934841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc16c1a21b0 con 0x7fc16c102320 2024-05-09T08:26:18.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.023+0000 7fc172eda640 1 -- 172.21.15.28:0/2514934841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc16c1a2440 con 0x7fc16c102320 2024-05-09T08:26:18.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.024+0000 7fc169ffb640 1 -- 172.21.15.28:0/2514934841 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7fc164004050 con 0x7fc16c102320 2024-05-09T08:26:18.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.025+0000 7fc169ffb640 1 -- 172.21.15.28:0/2514934841 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fc1640f00b0 con 0x7fc16c102320 2024-05-09T08:26:18.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.030+0000 7fc169ffb640 1 -- 172.21.15.28:0/2514934841 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc1640bcfe0 con 0x7fc16c102320 2024-05-09T08:26:18.182 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:17.962+0000 7fc80319c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:26:18.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.295+0000 7fc172eda640 1 -- 172.21.15.28:0/2514934841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc16c1a27d0 con 0x7fc16c102320 2024-05-09T08:26:18.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.297+0000 7fc169ffb640 1 -- 172.21.15.28:0/2514934841 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25274 (secure 0 0 0) 0x7fc16c1a27d0 con 0x7fc16c102320 2024-05-09T08:26:18.298 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:18.298 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:26:17.965032+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3330063293":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3745037139":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/2130261041":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/273816117":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:26:18.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.300+0000 7fc172eda640 1 -- 172.21.15.28:0/2514934841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 msgr2=0x7fc16c1a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.300+0000 7fc172eda640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 0x7fc16c1a1310 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc164031ae0 tx=0x7fc164031ee0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.300+0000 7fc172eda640 1 -- 172.21.15.28:0/2514934841 shutdown_connections 2024-05-09T08:26:18.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.300+0000 7fc172eda640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16c102320 0x7fc16c1a1310 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.300+0000 7fc172eda640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc16c101960 0x7fc16c19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.300+0000 7fc172eda640 1 --2- 172.21.15.28:0/2514934841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c100770 0x7fc16c199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.300+0000 7fc172eda640 1 -- 172.21.15.28:0/2514934841 >> 172.21.15.28:0/2514934841 conn(0x7fc16c0fbce0 msgr2=0x7fc16c0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:18.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.300+0000 7fc172eda640 1 -- 172.21.15.28:0/2514934841 shutdown_connections 2024-05-09T08:26:18.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.300+0000 7fc172eda640 1 -- 172.21.15.28:0/2514934841 wait complete. 2024-05-09T08:26:18.313 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:26:18.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:26:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.426+0000 7fefc0ea5640 1 -- 172.21.15.28:0/4102290558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefbc100aa0 msgr2=0x7fefbc100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.426+0000 7fefc0ea5640 1 --2- 172.21.15.28:0/4102290558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefbc100aa0 0x7fefbc100f00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009900 tx=0x7fefb002a410 comp rx=0 tx=0).stop 2024-05-09T08:26:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.426+0000 7fefc0ea5640 1 -- 172.21.15.28:0/4102290558 shutdown_connections 2024-05-09T08:26:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.426+0000 7fefc0ea5640 1 --2- 172.21.15.28:0/4102290558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefbc101440 0x7fefbc1018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.426+0000 7fefc0ea5640 1 --2- 172.21.15.28:0/4102290558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefbc100aa0 0x7fefbc100f00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.426+0000 7fefc0ea5640 1 --2- 172.21.15.28:0/4102290558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefbc100140 0x7fefbc100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.426+0000 7fefc0ea5640 1 -- 172.21.15.28:0/4102290558 >> 172.21.15.28:0/4102290558 conn(0x7fefbc0fbce0 msgr2=0x7fefbc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.426+0000 7fefc0ea5640 1 -- 172.21.15.28:0/4102290558 shutdown_connections 2024-05-09T08:26:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.426+0000 7fefc0ea5640 1 -- 172.21.15.28:0/4102290558 wait complete. 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.427+0000 7fefc0ea5640 1 Processor -- start 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.427+0000 7fefc0ea5640 1 -- start start 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.427+0000 7fefc0ea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefbc100140 0x7fefbc199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.427+0000 7fefc0ea5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefbc100aa0 0x7fefbc19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.427+0000 7fefc0ea5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefbc101440 0x7fefbc1a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefc0ea5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc10f6d0 con 0x7fefbc100140 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefc0ea5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc10f550 con 0x7fefbc101440 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefc0ea5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc10f850 con 0x7fefbc100aa0 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefbc100140 0x7fefbc199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefbc100140 0x7fefbc199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38242/0 (socket says 172.21.15.28:38242) 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefba59c640 1 -- 172.21.15.28:0/1606250729 learned_addr learned my addr 172.21.15.28:0/1606250729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefbad9d640 1 --2- 172.21.15.28:0/1606250729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefbc101440 0x7fefbc1a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefb9d9b640 1 --2- 172.21.15.28:0/1606250729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefbc100aa0 0x7fefbc19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefba59c640 1 -- 172.21.15.28:0/1606250729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefbc100aa0 msgr2=0x7fefbc19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefba59c640 1 --2- 172.21.15.28:0/1606250729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefbc100aa0 0x7fefbc19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefba59c640 1 -- 172.21.15.28:0/1606250729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefbc101440 msgr2=0x7fefbc1a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefba59c640 1 --2- 172.21.15.28:0/1606250729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefbc101440 0x7fefbc1a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefba59c640 1 -- 172.21.15.28:0/1606250729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefbc1a19f0 con 0x7fefbc100140 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefba59c640 1 --2- 172.21.15.28:0/1606250729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefbc100140 0x7fefbc199dd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fefac00e9f0 tx=0x7fefac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefab7fe640 1 -- 172.21.15.28:0/1606250729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac00ccc0 con 0x7fefbc100140 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefc0ea5640 1 -- 172.21.15.28:0/1606250729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefbc1a1ce0 con 0x7fefbc100140 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefab7fe640 1 -- 172.21.15.28:0/1606250729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefac00ce60 con 0x7fefbc100140 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.428+0000 7fefab7fe640 1 -- 172.21.15.28:0/1606250729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac010690 con 0x7fefbc100140 2024-05-09T08:26:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.429+0000 7fefc0ea5640 1 -- 172.21.15.28:0/1606250729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefbc1a21b0 con 0x7fefbc100140 2024-05-09T08:26:18.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.430+0000 7fefc0ea5640 1 -- 172.21.15.28:0/1606250729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefbc10b060 con 0x7fefbc100140 2024-05-09T08:26:18.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.431+0000 7fefab7fe640 1 -- 172.21.15.28:0/1606250729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7fefac0046c0 con 0x7fefbc100140 2024-05-09T08:26:18.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.432+0000 7fefab7fe640 1 -- 172.21.15.28:0/1606250729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fefac0cf6a0 con 0x7fefbc100140 2024-05-09T08:26:18.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.440+0000 7fefab7fe640 1 -- 172.21.15.28:0/1606250729 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fefac09c540 con 0x7fefbc100140 2024-05-09T08:26:18.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3401304440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:18.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: Standby manager daemon y restarted 2024-05-09T08:26:18.463 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: Standby manager daemon y started 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: Standby manager daemon z restarted 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: Standby manager daemon z started 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3322385977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2877220996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: Active manager daemon x restarted 2024-05-09T08:26:18.464 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: Activating manager daemon x 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: mgrmap e20: x(active, starting, since 0.0136952s), standbys: y, z 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:26:18.465 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:26:18.466 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:26:18.467 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: Manager daemon x is now available 2024-05-09T08:26:18.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:18.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:26:18.468 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.237+0000 7fc762280700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.237+0000 7fc762280700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.237+0000 7fc762280700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.237+0000 7fc762280700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.237+0000 7fc762280700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.238+0000 7fc765a87700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.238+0000 7fc765a87700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.238+0000 7fc765a87700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.238+0000 7fc765a87700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.476 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:26:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:26:18.238+0000 7fc765a87700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:26:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3401304440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: Standby manager daemon y restarted 2024-05-09T08:26:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: Standby manager daemon y started 2024-05-09T08:26:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:26:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:26:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:26:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:26:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: Standby manager daemon z restarted 2024-05-09T08:26:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: Standby manager daemon z started 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3322385977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2877220996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: Active manager daemon x restarted 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: Activating manager daemon x 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: mgrmap e20: x(active, starting, since 0.0136952s), standbys: y, z 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:18.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:26:18.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:26:18.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:26:18.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:26:18.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: Manager daemon x is now available 2024-05-09T08:26:18.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:18.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:26:18.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:26:18.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3401304440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:18.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: Standby manager daemon y restarted 2024-05-09T08:26:18.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: Standby manager daemon y started 2024-05-09T08:26:18.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:26:18.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:26:18.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:26:18.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.165:0/3812194265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:26:18.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: Standby manager daemon z restarted 2024-05-09T08:26:18.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: Standby manager daemon z started 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.? 172.21.15.184:0/2731794127' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3322385977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2877220996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: Active manager daemon x restarted 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: Activating manager daemon x 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: mgrmap e20: x(active, starting, since 0.0136952s), standbys: y, z 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:18.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:26:18.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:26:18.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: Manager daemon x is now available 2024-05-09T08:26:18.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:18.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:26:18.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:26:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.646+0000 7fefc0ea5640 1 -- 172.21.15.28:0/1606250729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fefbc02d400 con 0x7fefbc100140 2024-05-09T08:26:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.647+0000 7fefab7fe640 1 -- 172.21.15.28:0/1606250729 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fefac09be40 con 0x7fefbc100140 2024-05-09T08:26:18.648 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:26:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.650+0000 7fefc0ea5640 1 -- 172.21.15.28:0/1606250729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefbc100140 msgr2=0x7fefbc199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.650+0000 7fefc0ea5640 1 --2- 172.21.15.28:0/1606250729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefbc100140 0x7fefbc199dd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fefac00e9f0 tx=0x7fefac00edf0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.650+0000 7fefc0ea5640 1 -- 172.21.15.28:0/1606250729 shutdown_connections 2024-05-09T08:26:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.650+0000 7fefc0ea5640 1 --2- 172.21.15.28:0/1606250729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefbc101440 0x7fefbc1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.650+0000 7fefc0ea5640 1 --2- 172.21.15.28:0/1606250729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fefbc100aa0 0x7fefbc19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.650+0000 7fefc0ea5640 1 --2- 172.21.15.28:0/1606250729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefbc100140 0x7fefbc199dd0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.650+0000 7fefc0ea5640 1 -- 172.21.15.28:0/1606250729 >> 172.21.15.28:0/1606250729 conn(0x7fefbc0fbce0 msgr2=0x7fefbc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.650+0000 7fefc0ea5640 1 -- 172.21.15.28:0/1606250729 shutdown_connections 2024-05-09T08:26:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.650+0000 7fefc0ea5640 1 -- 172.21.15.28:0/1606250729 wait complete. 2024-05-09T08:26:18.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:26:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.772+0000 7f19ef861640 1 -- 172.21.15.28:0/4115589525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 msgr2=0x7f19e8100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.772+0000 7f19ef861640 1 --2- 172.21.15.28:0/4115589525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 0x7f19e8100bb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009900 tx=0x7f19e402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.773+0000 7f19ef861640 1 -- 172.21.15.28:0/4115589525 shutdown_connections 2024-05-09T08:26:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.773+0000 7f19ef861640 1 --2- 172.21.15.28:0/4115589525 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f19e8102340 0x7f19e8108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.773+0000 7f19ef861640 1 --2- 172.21.15.28:0/4115589525 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19e8101980 0x7f19e8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.773+0000 7f19ef861640 1 --2- 172.21.15.28:0/4115589525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 0x7f19e8100bb0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.773+0000 7f19ef861640 1 -- 172.21.15.28:0/4115589525 >> 172.21.15.28:0/4115589525 conn(0x7f19e80fbd20 msgr2=0x7f19e80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.773+0000 7f19ef861640 1 -- 172.21.15.28:0/4115589525 shutdown_connections 2024-05-09T08:26:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.773+0000 7f19ef861640 1 -- 172.21.15.28:0/4115589525 wait complete. 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.773+0000 7f19ef861640 1 Processor -- start 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ef861640 1 -- start start 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ef861640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 0x7f19e8197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ef861640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f19e8101980 0x7f19e81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ef861640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19e8102340 0x7f19e819f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ef861640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19e810d530 con 0x7f19e8100790 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ef861640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f19e810d3b0 con 0x7f19e8102340 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ef861640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f19e810d6b0 con 0x7f19e8101980 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ed5fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 0x7f19e8197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ed5fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 0x7f19e8197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38262/0 (socket says 172.21.15.28:38262) 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ed5fd640 1 -- 172.21.15.28:0/2180310481 learned_addr learned my addr 172.21.15.28:0/2180310481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ecdfc640 1 --2- 172.21.15.28:0/2180310481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f19e8101980 0x7f19e81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19eddfe640 1 --2- 172.21.15.28:0/2180310481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19e8102340 0x7f19e819f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ed5fd640 1 -- 172.21.15.28:0/2180310481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f19e8101980 msgr2=0x7f19e81981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ed5fd640 1 --2- 172.21.15.28:0/2180310481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f19e8101980 0x7f19e81981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ed5fd640 1 -- 172.21.15.28:0/2180310481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19e8102340 msgr2=0x7f19e819f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ed5fd640 1 --2- 172.21.15.28:0/2180310481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19e8102340 0x7f19e819f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.774+0000 7f19ed5fd640 1 -- 172.21.15.28:0/2180310481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e819f8c0 con 0x7f19e8100790 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.775+0000 7f19ed5fd640 1 --2- 172.21.15.28:0/2180310481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 0x7f19e8197c80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f19e4006010 tx=0x7f19e4031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.775+0000 7f19de7fc640 1 -- 172.21.15.28:0/2180310481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e403f030 con 0x7f19e8100790 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.775+0000 7f19de7fc640 1 -- 172.21.15.28:0/2180310481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19e40044a0 con 0x7f19e8100790 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.775+0000 7f19ef861640 1 -- 172.21.15.28:0/2180310481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e819fb50 con 0x7f19e8100790 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.775+0000 7f19de7fc640 1 -- 172.21.15.28:0/2180310481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e4035070 con 0x7f19e8100790 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.775+0000 7f19ef861640 1 -- 172.21.15.28:0/2180310481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e81a0090 con 0x7f19e8100790 2024-05-09T08:26:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.776+0000 7f19ef861640 1 -- 172.21.15.28:0/2180310481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e8107bd0 con 0x7f19e8100790 2024-05-09T08:26:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.778+0000 7f19de7fc640 1 -- 172.21.15.28:0/2180310481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7f19e4042030 con 0x7f19e8100790 2024-05-09T08:26:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.779+0000 7f19de7fc640 1 -- 172.21.15.28:0/2180310481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f19e40ef440 con 0x7f19e8100790 2024-05-09T08:26:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.784+0000 7f19de7fc640 1 -- 172.21.15.28:0/2180310481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f19e40bbc00 con 0x7f19e8100790 2024-05-09T08:26:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.970+0000 7f19ef861640 1 -- 172.21.15.28:0/2180310481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f19e8062430 con 0x7f19e8100790 2024-05-09T08:26:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.970+0000 7f19de7fc640 1 -- 172.21.15.28:0/2180310481 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f19e40bff00 con 0x7f19e8100790 2024-05-09T08:26:18.971 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.973+0000 7f19ef861640 1 -- 172.21.15.28:0/2180310481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 msgr2=0x7f19e8197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.973+0000 7f19ef861640 1 --2- 172.21.15.28:0/2180310481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 0x7f19e8197c80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f19e4006010 tx=0x7f19e4031b60 comp rx=0 tx=0).stop 2024-05-09T08:26:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.973+0000 7f19ef861640 1 -- 172.21.15.28:0/2180310481 shutdown_connections 2024-05-09T08:26:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.973+0000 7f19ef861640 1 --2- 172.21.15.28:0/2180310481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19e8102340 0x7f19e819f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.973+0000 7f19ef861640 1 --2- 172.21.15.28:0/2180310481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f19e8101980 0x7f19e81981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.973+0000 7f19ef861640 1 --2- 172.21.15.28:0/2180310481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8100790 0x7f19e8197c80 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.973+0000 7f19ef861640 1 -- 172.21.15.28:0/2180310481 >> 172.21.15.28:0/2180310481 conn(0x7f19e80fbd20 msgr2=0x7f19e8104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.973+0000 7f19ef861640 1 -- 172.21.15.28:0/2180310481 shutdown_connections 2024-05-09T08:26:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:18.973+0000 7f19ef861640 1 -- 172.21.15.28:0/2180310481 wait complete. 2024-05-09T08:26:18.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:26:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.113+0000 7fddc9e1c640 1 -- 172.21.15.28:0/721001584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4101960 msgr2=0x7fddc4101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.113+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/721001584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4101960 0x7fddc4101de0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009900 tx=0x7fddb802a410 comp rx=0 tx=0).stop 2024-05-09T08:26:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.113+0000 7fddc9e1c640 1 -- 172.21.15.28:0/721001584 shutdown_connections 2024-05-09T08:26:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.113+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/721001584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddc4102320 0x7fddc4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.113+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/721001584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4101960 0x7fddc4101de0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.113+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/721001584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4100770 0x7fddc4100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.113+0000 7fddc9e1c640 1 -- 172.21.15.28:0/721001584 >> 172.21.15.28:0/721001584 conn(0x7fddc40fbce0 msgr2=0x7fddc40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.113+0000 7fddc9e1c640 1 -- 172.21.15.28:0/721001584 shutdown_connections 2024-05-09T08:26:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.114+0000 7fddc9e1c640 1 -- 172.21.15.28:0/721001584 wait complete. 2024-05-09T08:26:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.114+0000 7fddc9e1c640 1 Processor -- start 2024-05-09T08:26:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.114+0000 7fddc9e1c640 1 -- start start 2024-05-09T08:26:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc9e1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4100770 0x7fddc4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc9e1c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddc4101960 0x7fddc41980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc9e1c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4102320 0x7fddc419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc9e1c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fddc410c3d0 con 0x7fddc4100770 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc9e1c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fddc410c250 con 0x7fddc4101960 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc9e1c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fddc410c550 con 0x7fddc4102320 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4100770 0x7fddc4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4100770 0x7fddc4197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38278/0 (socket says 172.21.15.28:38278) 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc37fe640 1 -- 172.21.15.28:0/1891663908 learned_addr learned my addr 172.21.15.28:0/1891663908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc3fff640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4102320 0x7fddc419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc2ffd640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddc4101960 0x7fddc41980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc37fe640 1 -- 172.21.15.28:0/1891663908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4102320 msgr2=0x7fddc419f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc37fe640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4102320 0x7fddc419f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc37fe640 1 -- 172.21.15.28:0/1891663908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddc4101960 msgr2=0x7fddc41980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc37fe640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddc4101960 0x7fddc41980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc37fe640 1 -- 172.21.15.28:0/1891663908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc419f7b0 con 0x7fddc4100770 2024-05-09T08:26:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc2ffd640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddc4101960 0x7fddc41980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc37fe640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4100770 0x7fddc4197b90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fddb400e9c0 tx=0x7fddb400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc0ff9640 1 -- 172.21.15.28:0/1891663908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb400cd80 con 0x7fddc4100770 2024-05-09T08:26:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc0ff9640 1 -- 172.21.15.28:0/1891663908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddb40035f0 con 0x7fddc4100770 2024-05-09T08:26:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc419faa0 con 0x7fddc4100770 2024-05-09T08:26:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.115+0000 7fddc0ff9640 1 -- 172.21.15.28:0/1891663908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4010690 con 0x7fddc4100770 2024-05-09T08:26:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.116+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc419ffc0 con 0x7fddc4100770 2024-05-09T08:26:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.117+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc4068650 con 0x7fddc4100770 2024-05-09T08:26:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.119+0000 7fddc0ff9640 1 -- 172.21.15.28:0/1891663908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7fddb40037a0 con 0x7fddc4100770 2024-05-09T08:26:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.120+0000 7fddc0ff9640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fddac0a2990 0x7fddac0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.120+0000 7fddc2ffd640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fddac0a2990 0x7fddac0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.120+0000 7fddc0ff9640 1 -- 172.21.15.28:0/1891663908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fddb40cf6b0 con 0x7fddc4100770 2024-05-09T08:26:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.121+0000 7fddc2ffd640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fddac0a2990 0x7fddac0a4e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fddb8000c00 tx=0x7fddb80095f0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.123+0000 7fddc0ff9640 1 -- 172.21.15.28:0/1891663908 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fddb409c460 con 0x7fddc4100770 2024-05-09T08:26:19.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.361+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fddc41a04a0 con 0x7fddc4100770 2024-05-09T08:26:19.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.361+0000 7fddc0ff9640 1 -- 172.21.15.28:0/1891663908 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fddb409bd60 con 0x7fddc4100770 2024-05-09T08:26:19.362 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:26:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fddac0a2990 msgr2=0x7fddac0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fddac0a2990 0x7fddac0a4e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fddb8000c00 tx=0x7fddb80095f0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4100770 msgr2=0x7fddc4197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4100770 0x7fddc4197b90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fddb400e9c0 tx=0x7fddb400ee80 comp rx=0 tx=0).stop 2024-05-09T08:26:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 shutdown_connections 2024-05-09T08:26:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fddac0a2990 0x7fddac0a4e50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4102320 0x7fddc419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddc4101960 0x7fddc41980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 --2- 172.21.15.28:0/1891663908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddc4100770 0x7fddc4197b90 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 >> 172.21.15.28:0/1891663908 conn(0x7fddc40fbce0 msgr2=0x7fddc40fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:19.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 shutdown_connections 2024-05-09T08:26:19.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.363+0000 7fddc9e1c640 1 -- 172.21.15.28:0/1891663908 wait complete. 2024-05-09T08:26:19.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:19 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2514934841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:19.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:19 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1606250729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:19.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:19 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2180310481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:19.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:19 smithi028 ceph-mon[30119]: mgrmap e21: x(active, since 1.02421s), standbys: y, z 2024-05-09T08:26:19.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:26:19.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:19 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2514934841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:19.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:19 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1606250729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:19.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:19 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2180310481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:19.479 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:19 smithi184 ceph-mon[34509]: mgrmap e21: x(active, since 1.02421s), standbys: y, z 2024-05-09T08:26:19.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.486+0000 7fcc44cc1640 1 -- 172.21.15.28:0/4213793919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc40107550 msgr2=0x7fcc401099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.486+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/4213793919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc40107550 0x7fcc401099a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcc38009ca0 tx=0x7fcc3802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.486+0000 7fcc44cc1640 1 -- 172.21.15.28:0/4213793919 shutdown_connections 2024-05-09T08:26:19.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.486+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/4213793919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc40107550 0x7fcc401099a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.486+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/4213793919 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc40104c20 0x7fcc40107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.486+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/4213793919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc401022f0 0x7fcc401046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.486+0000 7fcc44cc1640 1 -- 172.21.15.28:0/4213793919 >> 172.21.15.28:0/4213793919 conn(0x7fcc400fbce0 msgr2=0x7fcc400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:19.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.486+0000 7fcc44cc1640 1 -- 172.21.15.28:0/4213793919 shutdown_connections 2024-05-09T08:26:19.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.486+0000 7fcc44cc1640 1 -- 172.21.15.28:0/4213793919 wait complete. 2024-05-09T08:26:19.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.487+0000 7fcc44cc1640 1 Processor -- start 2024-05-09T08:26:19.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.487+0000 7fcc44cc1640 1 -- start start 2024-05-09T08:26:19.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.487+0000 7fcc44cc1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc401022f0 0x7fcc40197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.487+0000 7fcc44cc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc40104c20 0x7fcc401980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.487+0000 7fcc44cc1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc40107550 0x7fcc4019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.487+0000 7fcc44cc1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4010d540 con 0x7fcc40104c20 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.487+0000 7fcc44cc1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4010d3c0 con 0x7fcc40107550 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.487+0000 7fcc44cc1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4010d6c0 con 0x7fcc401022f0 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.487+0000 7fcc3dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc40104c20 0x7fcc401980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3e59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc401022f0 0x7fcc40197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc40104c20 0x7fcc401980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38296/0 (socket says 172.21.15.28:38296) 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3dd9b640 1 -- 172.21.15.28:0/3880600577 learned_addr learned my addr 172.21.15.28:0/3880600577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3ed9d640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc40107550 0x7fcc4019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3e59c640 1 -- 172.21.15.28:0/3880600577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc40107550 msgr2=0x7fcc4019f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3e59c640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc40107550 0x7fcc4019f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3e59c640 1 -- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc40104c20 msgr2=0x7fcc401980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3e59c640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc40104c20 0x7fcc401980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3e59c640 1 -- 172.21.15.28:0/3880600577 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4019f780 con 0x7fcc401022f0 2024-05-09T08:26:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3dd9b640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc40104c20 0x7fcc401980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc3e59c640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc401022f0 0x7fcc40197b60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000e9f0 tx=0x7fcc3000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc2f7fe640 1 -- 172.21.15.28:0/3880600577 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3000ccc0 con 0x7fcc401022f0 2024-05-09T08:26:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4019fa70 con 0x7fcc401022f0 2024-05-09T08:26:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc2f7fe640 1 -- 172.21.15.28:0/3880600577 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc30010910 con 0x7fcc401022f0 2024-05-09T08:26:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.488+0000 7fcc2f7fe640 1 -- 172.21.15.28:0/3880600577 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30018960 con 0x7fcc401022f0 2024-05-09T08:26:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.489+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc401a0010 con 0x7fcc401022f0 2024-05-09T08:26:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.490+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc400fd7d0 con 0x7fcc401022f0 2024-05-09T08:26:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.492+0000 7fcc2f7fe640 1 -- 172.21.15.28:0/3880600577 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7fcc30010ac0 con 0x7fcc401022f0 2024-05-09T08:26:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.492+0000 7fcc2f7fe640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcc280a2a10 0x7fcc280a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.493+0000 7fcc2f7fe640 1 -- 172.21.15.28:0/3880600577 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fcc300cf4a0 con 0x7fcc401022f0 2024-05-09T08:26:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.493+0000 7fcc3dd9b640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcc280a2a10 0x7fcc280a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.493+0000 7fcc3dd9b640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcc280a2a10 0x7fcc280a4ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc40199070 tx=0x7fcc3400b480 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.495+0000 7fcc2f7fe640 1 -- 172.21.15.28:0/3880600577 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcc3009bbb0 con 0x7fcc401022f0 2024-05-09T08:26:19.534 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:19 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2514934841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:19.534 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:19 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1606250729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:19.534 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:19 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2180310481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:19.534 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:19 smithi165 ceph-mon[34540]: mgrmap e21: x(active, since 1.02421s), standbys: y, z 2024-05-09T08:26:19.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.687+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcc4002d370 con 0x7fcc401022f0 2024-05-09T08:26:19.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.688+0000 7fcc2f7fe640 1 -- 172.21.15.28:0/3880600577 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fcc3009feb0 con 0x7fcc401022f0 2024-05-09T08:26:19.689 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcc280a2a10 msgr2=0x7fcc280a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcc280a2a10 0x7fcc280a4ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc40199070 tx=0x7fcc3400b480 comp rx=0 tx=0).stop 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc401022f0 msgr2=0x7fcc40197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc401022f0 0x7fcc40197b60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000e9f0 tx=0x7fcc3000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 shutdown_connections 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc40107550 0x7fcc4019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc40104c20 0x7fcc401980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcc280a2a10 0x7fcc280a4ed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 --2- 172.21.15.28:0/3880600577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc401022f0 0x7fcc40197b60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 >> 172.21.15.28:0/3880600577 conn(0x7fcc400fbce0 msgr2=0x7fcc40102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 shutdown_connections 2024-05-09T08:26:19.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.690+0000 7fcc44cc1640 1 -- 172.21.15.28:0/3880600577 wait complete. 2024-05-09T08:26:19.702 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-09T08:26:19.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.808+0000 7f62e05b0640 1 -- 172.21.15.28:0/3581897740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 msgr2=0x7f62d8104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.808+0000 7f62e05b0640 1 --2- 172.21.15.28:0/3581897740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 0x7f62d8104780 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009900 tx=0x7f62d402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.808+0000 7f62e05b0640 1 -- 172.21.15.28:0/3581897740 shutdown_connections 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.808+0000 7f62e05b0640 1 --2- 172.21.15.28:0/3581897740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62d81075f0 0x7f62d81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.808+0000 7f62e05b0640 1 --2- 172.21.15.28:0/3581897740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62d8104cc0 0x7f62d81070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.808+0000 7f62e05b0640 1 --2- 172.21.15.28:0/3581897740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 0x7f62d8104780 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.808+0000 7f62e05b0640 1 -- 172.21.15.28:0/3581897740 >> 172.21.15.28:0/3581897740 conn(0x7f62d80fbce0 msgr2=0x7f62d80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.808+0000 7f62e05b0640 1 -- 172.21.15.28:0/3581897740 shutdown_connections 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.808+0000 7f62e05b0640 1 -- 172.21.15.28:0/3581897740 wait complete. 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62e05b0640 1 Processor -- start 2024-05-09T08:26:19.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62e05b0640 1 -- start start 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62e05b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 0x7f62d8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62e05b0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62d8104cc0 0x7f62d81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62e05b0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62d81075f0 0x7f62d819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62e05b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62d810d510 con 0x7f62d8102390 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62e05b0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f62d810d390 con 0x7f62d81075f0 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62e05b0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62d810d690 con 0x7f62d8104cc0 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62de34c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 0x7f62d8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62de34c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 0x7f62d8197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38314/0 (socket says 172.21.15.28:38314) 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62de34c640 1 -- 172.21.15.28:0/1316673981 learned_addr learned my addr 172.21.15.28:0/1316673981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62ddb4b640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62d8104cc0 0x7f62d81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.809+0000 7f62deb4d640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62d81075f0 0x7f62d819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62de34c640 1 -- 172.21.15.28:0/1316673981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62d8104cc0 msgr2=0x7f62d81980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62de34c640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62d8104cc0 0x7f62d81980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62de34c640 1 -- 172.21.15.28:0/1316673981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62d81075f0 msgr2=0x7f62d819f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62de34c640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62d81075f0 0x7f62d819f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:19.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62de34c640 1 -- 172.21.15.28:0/1316673981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d819f7d0 con 0x7f62d8102390 2024-05-09T08:26:19.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62de34c640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 0x7f62d8197bb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009230 tx=0x7f62d40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:19.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62cf7fe640 1 -- 172.21.15.28:0/1316673981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d403f030 con 0x7f62d8102390 2024-05-09T08:26:19.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62cf7fe640 1 -- 172.21.15.28:0/1316673981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d4003ce0 con 0x7f62d8102390 2024-05-09T08:26:19.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d819fa60 con 0x7f62d8102390 2024-05-09T08:26:19.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62cf7fe640 1 -- 172.21.15.28:0/1316673981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4035070 con 0x7f62d8102390 2024-05-09T08:26:19.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.810+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d819ff50 con 0x7f62d8102390 2024-05-09T08:26:19.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.811+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d80fd7d0 con 0x7f62d8102390 2024-05-09T08:26:19.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.812+0000 7f62cf7fe640 1 -- 172.21.15.28:0/1316673981 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f62d4042030 con 0x7f62d8102390 2024-05-09T08:26:19.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.813+0000 7f62cf7fe640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62c40a2990 0x7f62c40a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:19.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.813+0000 7f62cf7fe640 1 -- 172.21.15.28:0/1316673981 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f62d40ef640 con 0x7f62d8102390 2024-05-09T08:26:19.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.813+0000 7f62ddb4b640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62c40a2990 0x7f62c40a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:19.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.813+0000 7f62ddb4b640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62c40a2990 0x7f62c40a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62d81990c0 tx=0x7f62c8005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:19.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:19.818+0000 7f62cf7fe640 1 -- 172.21.15.28:0/1316673981 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f62d40bbd10 con 0x7f62d8102390 2024-05-09T08:26:20.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.082+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62d8062430 con 0x7f62d8102390 2024-05-09T08:26:20.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.083+0000 7f62cf7fe640 1 -- 172.21.15.28:0/1316673981 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11932 (secure 0 0 0) 0x7f62d40c0010 con 0x7f62d8102390 2024-05-09T08:26:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:26:16.129765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24463,24512,24476,24481,24499]},"id":1}]} 2024-05-09T08:26:20.084 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-05-09T08:26:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.085+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62c40a2990 msgr2=0x7f62c40a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.085+0000 7f62e05b0640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62c40a2990 0x7f62c40a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62d81990c0 tx=0x7f62c8005f90 comp rx=0 tx=0).stop 2024-05-09T08:26:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.085+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 msgr2=0x7f62d8197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.085+0000 7f62e05b0640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 0x7f62d8197bb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009230 tx=0x7f62d40042f0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.085+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 shutdown_connections 2024-05-09T08:26:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.085+0000 7f62e05b0640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62d81075f0 0x7f62d819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.086+0000 7f62e05b0640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62c40a2990 0x7f62c40a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.086+0000 7f62e05b0640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62d8104cc0 0x7f62d81980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.086+0000 7f62e05b0640 1 --2- 172.21.15.28:0/1316673981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d8102390 0x7f62d8197bb0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.086+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 >> 172.21.15.28:0/1316673981 conn(0x7f62d80fbce0 msgr2=0x7f62d8102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.086+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 shutdown_connections 2024-05-09T08:26:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.086+0000 7f62e05b0640 1 -- 172.21.15.28:0/1316673981 wait complete. 2024-05-09T08:26:20.100 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-09T08:26:20.100 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-09T08:26:20.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:26:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.222+0000 7f7e6f742640 1 -- 172.21.15.28:0/1562558844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e68102a90 msgr2=0x7f7e68102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.222+0000 7f7e6f742640 1 --2- 172.21.15.28:0/1562558844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e68102a90 0x7f7e68102f10 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009900 tx=0x7f7e5802a410 comp rx=0 tx=0).stop 2024-05-09T08:26:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.222+0000 7f7e6f742640 1 -- 172.21.15.28:0/1562558844 shutdown_connections 2024-05-09T08:26:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.222+0000 7f7e6f742640 1 --2- 172.21.15.28:0/1562558844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e68103450 0x7f7e68109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.222+0000 7f7e6f742640 1 --2- 172.21.15.28:0/1562558844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e68102a90 0x7f7e68102f10 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.222+0000 7f7e6f742640 1 --2- 172.21.15.28:0/1562558844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e681018a0 0x7f7e68101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.222+0000 7f7e6f742640 1 -- 172.21.15.28:0/1562558844 >> 172.21.15.28:0/1562558844 conn(0x7f7e680fce30 msgr2=0x7f7e680ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.223+0000 7f7e6f742640 1 -- 172.21.15.28:0/1562558844 shutdown_connections 2024-05-09T08:26:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.223+0000 7f7e6f742640 1 -- 172.21.15.28:0/1562558844 wait complete. 2024-05-09T08:26:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.223+0000 7f7e6f742640 1 Processor -- start 2024-05-09T08:26:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.223+0000 7f7e6f742640 1 -- start start 2024-05-09T08:26:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6f742640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e681018a0 0x7f7e68197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6f742640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e68103450 0x7f7e681982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6f742640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e6819d350 0x7f7e6819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6f742640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6810d570 con 0x7f7e6819d350 2024-05-09T08:26:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6d4de640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e681018a0 0x7f7e68197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6d4de640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e681018a0 0x7f7e68197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:37414/0 (socket says 172.21.15.28:37414) 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6d4de640 1 -- 172.21.15.28:0/2701766396 learned_addr learned my addr 172.21.15.28:0/2701766396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6810d3f0 con 0x7f7e681018a0 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6810d6f0 con 0x7f7e68103450 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6ccdd640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e68103450 0x7f7e681982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.224+0000 7f7e6dcdf640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e6819d350 0x7f7e6819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e6d4de640 1 -- 172.21.15.28:0/2701766396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e68103450 msgr2=0x7f7e681982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e6d4de640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e68103450 0x7f7e681982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e6d4de640 1 -- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e6819d350 msgr2=0x7f7e6819f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e6d4de640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e6819d350 0x7f7e6819f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e6d4de640 1 -- 172.21.15.28:0/2701766396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6819fdb0 con 0x7f7e681018a0 2024-05-09T08:26:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e6d4de640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e681018a0 0x7f7e68197db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e9c0 tx=0x7f7e6400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e5e7fc640 1 -- 172.21.15.28:0/2701766396 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400cd80 con 0x7f7e681018a0 2024-05-09T08:26:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e5e7fc640 1 -- 172.21.15.28:0/2701766396 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e640035f0 con 0x7f7e681018a0 2024-05-09T08:26:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e5e7fc640 1 -- 172.21.15.28:0/2701766396 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64010690 con 0x7f7e681018a0 2024-05-09T08:26:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e681a00a0 con 0x7f7e681018a0 2024-05-09T08:26:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.225+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e681a05c0 con 0x7f7e681018a0 2024-05-09T08:26:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.227+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e68067dc0 con 0x7f7e681018a0 2024-05-09T08:26:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.228+0000 7f7e5e7fc640 1 -- 172.21.15.28:0/2701766396 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f7e640037a0 con 0x7f7e681018a0 2024-05-09T08:26:20.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.229+0000 7f7e5e7fc640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7e540a2e50 0x7f7e540a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.229+0000 7f7e5e7fc640 1 -- 172.21.15.28:0/2701766396 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f7e640cfaa0 con 0x7f7e681018a0 2024-05-09T08:26:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.229+0000 7f7e6ccdd640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7e540a2e50 0x7f7e540a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.230+0000 7f7e6ccdd640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7e540a2e50 0x7f7e540a5310 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009230 tx=0x7f7e58005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:20.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.234+0000 7f7e5e7fc640 1 -- 172.21.15.28:0/2701766396 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7e6409c2f0 con 0x7f7e681018a0 2024-05-09T08:26:20.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:20 smithi028 ceph-mon[30119]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:26:20.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:20 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1891663908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:20.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:20 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3880600577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:20.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:20 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1316673981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:20.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.448+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 --> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7e68061c70 con 0x7f7e540a2e50 2024-05-09T08:26:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.458+0000 7f7e5e7fc640 1 -- 172.21.15.28:0/2701766396 <== mgr.15528 v2:172.21.15.28:6800/160415970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339763 (secure 0 0 0) 0x7f7e68061c70 con 0x7f7e540a2e50 2024-05-09T08:26:20.463 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-09T08:26:20.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7e540a2e50 msgr2=0x7f7e540a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7e540a2e50 0x7f7e540a5310 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009230 tx=0x7f7e58005fd0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e681018a0 msgr2=0x7f7e68197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e681018a0 0x7f7e68197db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e9c0 tx=0x7f7e6400ee80 comp rx=0 tx=0).stop 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 shutdown_connections 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e6819d350 0x7f7e6819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7e540a2e50 0x7f7e540a5310 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e68103450 0x7f7e681982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 --2- 172.21.15.28:0/2701766396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e681018a0 0x7f7e68197db0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.461+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 >> 172.21.15.28:0/2701766396 conn(0x7f7e680fce30 msgr2=0x7f7e68105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.462+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 shutdown_connections 2024-05-09T08:26:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.462+0000 7f7e6f742640 1 -- 172.21.15.28:0/2701766396 wait complete. 2024-05-09T08:26:20.474 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-05-09T08:26:19.984209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1194703,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1460,"num_read_kb":899,"num_write":559,"num_write_kb":4413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1245015,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":944,"ondisk_log_size":944,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378620,"kb_used_data":12792,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563204,"statfs":{"total":1151940427776,"available":1151552720896,"internally_reserved":0,"allocated":13099008,"data_stored":9767031,"data_compressed":543546,"data_compressed_allocated":733184,"data_compressed_original":4247552,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.983070"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203766+0000","last_change":"2024-05-09T08:23:51.579564+0000","last_active":"2024-05-09T08:26:18.203766+0000","last_peered":"2024-05-09T08:26:18.203766+0000","last_clean":"2024-05-09T08:26:18.203766+0000","last_became_active":"2024-05-09T08:23:51.578884+0000","last_became_peered":"2024-05-09T08:23:51.578884+0000","last_unstale":"2024-05-09T08:26:18.203766+0000","last_undegraded":"2024-05-09T08:26:18.203766+0000","last_fullsized":"2024-05-09T08:26:18.203766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:40.473483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206222+0000","last_change":"2024-05-09T08:23:51.574187+0000","last_active":"2024-05-09T08:26:18.206222+0000","last_peered":"2024-05-09T08:26:18.206222+0000","last_clean":"2024-05-09T08:26:18.206222+0000","last_became_active":"2024-05-09T08:23:51.573837+0000","last_became_peered":"2024-05-09T08:23:51.573837+0000","last_unstale":"2024-05-09T08:26:18.206222+0000","last_undegraded":"2024-05-09T08:26:18.206222+0000","last_fullsized":"2024-05-09T08:26:18.206222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:31.448038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205938+0000","last_change":"2024-05-09T08:23:51.577224+0000","last_active":"2024-05-09T08:26:18.205938+0000","last_peered":"2024-05-09T08:26:18.205938+0000","last_clean":"2024-05-09T08:26:18.205938+0000","last_became_active":"2024-05-09T08:23:51.576822+0000","last_became_peered":"2024-05-09T08:23:51.576822+0000","last_unstale":"2024-05-09T08:26:18.205938+0000","last_undegraded":"2024-05-09T08:26:18.205938+0000","last_fullsized":"2024-05-09T08:26:18.205938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:04.902530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971321+0000","last_change":"2024-05-09T08:23:51.577172+0000","last_active":"2024-05-09T08:26:17.971321+0000","last_peered":"2024-05-09T08:26:17.971321+0000","last_clean":"2024-05-09T08:26:17.971321+0000","last_became_active":"2024-05-09T08:23:51.576861+0000","last_became_peered":"2024-05-09T08:23:51.576861+0000","last_unstale":"2024-05-09T08:26:17.971321+0000","last_undegraded":"2024-05-09T08:26:17.971321+0000","last_fullsized":"2024-05-09T08:26:17.971321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:10.925739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086698+0000","last_change":"2024-05-09T08:23:51.579408+0000","last_active":"2024-05-09T08:26:18.086698+0000","last_peered":"2024-05-09T08:26:18.086698+0000","last_clean":"2024-05-09T08:26:18.086698+0000","last_became_active":"2024-05-09T08:23:51.578229+0000","last_became_peered":"2024-05-09T08:23:51.578229+0000","last_unstale":"2024-05-09T08:26:18.086698+0000","last_undegraded":"2024-05-09T08:26:18.086698+0000","last_fullsized":"2024-05-09T08:26:18.086698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:47:27.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.095787+0000","last_change":"2024-05-09T08:23:51.576924+0000","last_active":"2024-05-09T08:26:18.095787+0000","last_peered":"2024-05-09T08:26:18.095787+0000","last_clean":"2024-05-09T08:26:18.095787+0000","last_became_active":"2024-05-09T08:23:51.576601+0000","last_became_peered":"2024-05-09T08:23:51.576601+0000","last_unstale":"2024-05-09T08:26:18.095787+0000","last_undegraded":"2024-05-09T08:26:18.095787+0000","last_fullsized":"2024-05-09T08:26:18.095787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:07:16.910271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187456+0000","last_change":"2024-05-09T08:25:52.714094+0000","last_active":"2024-05-09T08:26:18.187456+0000","last_peered":"2024-05-09T08:26:18.187456+0000","last_clean":"2024-05-09T08:26:18.187456+0000","last_became_active":"2024-05-09T08:25:52.713743+0000","last_became_peered":"2024-05-09T08:25:52.713743+0000","last_unstale":"2024-05-09T08:26:18.187456+0000","last_undegraded":"2024-05-09T08:26:18.187456+0000","last_fullsized":"2024-05-09T08:26:18.187456+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:59.810244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970350+0000","last_change":"2024-05-09T08:23:51.574001+0000","last_active":"2024-05-09T08:26:17.970350+0000","last_peered":"2024-05-09T08:26:17.970350+0000","last_clean":"2024-05-09T08:26:17.970350+0000","last_became_active":"2024-05-09T08:23:51.573632+0000","last_became_peered":"2024-05-09T08:23:51.573632+0000","last_unstale":"2024-05-09T08:26:17.970350+0000","last_undegraded":"2024-05-09T08:26:17.970350+0000","last_fullsized":"2024-05-09T08:26:17.970350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:55:16.418951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203492+0000","last_change":"2024-05-09T08:23:51.567635+0000","last_active":"2024-05-09T08:26:18.203492+0000","last_peered":"2024-05-09T08:26:18.203492+0000","last_clean":"2024-05-09T08:26:18.203492+0000","last_became_active":"2024-05-09T08:23:51.567283+0000","last_became_peered":"2024-05-09T08:23:51.567283+0000","last_unstale":"2024-05-09T08:26:18.203492+0000","last_undegraded":"2024-05-09T08:26:18.203492+0000","last_fullsized":"2024-05-09T08:26:18.203492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:42.346856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.096277+0000","last_change":"2024-05-09T08:23:51.576090+0000","last_active":"2024-05-09T08:26:18.096277+0000","last_peered":"2024-05-09T08:26:18.096277+0000","last_clean":"2024-05-09T08:26:18.096277+0000","last_became_active":"2024-05-09T08:23:51.575768+0000","last_became_peered":"2024-05-09T08:23:51.575768+0000","last_unstale":"2024-05-09T08:26:18.096277+0000","last_undegraded":"2024-05-09T08:26:18.096277+0000","last_fullsized":"2024-05-09T08:26:18.096277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:00:50.016924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.188104+0000","last_change":"2024-05-09T08:23:51.578957+0000","last_active":"2024-05-09T08:26:18.188104+0000","last_peered":"2024-05-09T08:26:18.188104+0000","last_clean":"2024-05-09T08:26:18.188104+0000","last_became_active":"2024-05-09T08:23:51.578210+0000","last_became_peered":"2024-05-09T08:23:51.578210+0000","last_unstale":"2024-05-09T08:26:18.188104+0000","last_undegraded":"2024-05-09T08:26:18.188104+0000","last_fullsized":"2024-05-09T08:26:18.188104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:28:28.953434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971480+0000","last_change":"2024-05-09T08:23:51.575389+0000","last_active":"2024-05-09T08:26:17.971480+0000","last_peered":"2024-05-09T08:26:17.971480+0000","last_clean":"2024-05-09T08:26:17.971480+0000","last_became_active":"2024-05-09T08:23:51.575071+0000","last_became_peered":"2024-05-09T08:23:51.575071+0000","last_unstale":"2024-05-09T08:26:17.971480+0000","last_undegraded":"2024-05-09T08:26:17.971480+0000","last_fullsized":"2024-05-09T08:26:17.971480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:34.844611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204819+0000","last_change":"2024-05-09T08:23:51.572574+0000","last_active":"2024-05-09T08:26:18.204819+0000","last_peered":"2024-05-09T08:26:18.204819+0000","last_clean":"2024-05-09T08:26:18.204819+0000","last_became_active":"2024-05-09T08:23:51.572243+0000","last_became_peered":"2024-05-09T08:23:51.572243+0000","last_unstale":"2024-05-09T08:26:18.204819+0000","last_undegraded":"2024-05-09T08:26:18.204819+0000","last_fullsized":"2024-05-09T08:26:18.204819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:52.534365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205049+0000","last_change":"2024-05-09T08:23:51.574963+0000","last_active":"2024-05-09T08:26:18.205049+0000","last_peered":"2024-05-09T08:26:18.205049+0000","last_clean":"2024-05-09T08:26:18.205049+0000","last_became_active":"2024-05-09T08:23:51.574597+0000","last_became_peered":"2024-05-09T08:23:51.574597+0000","last_unstale":"2024-05-09T08:26:18.205049+0000","last_undegraded":"2024-05-09T08:26:18.205049+0000","last_fullsized":"2024-05-09T08:26:18.205049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:33.348828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205526+0000","last_change":"2024-05-09T08:23:51.577777+0000","last_active":"2024-05-09T08:26:18.205526+0000","last_peered":"2024-05-09T08:26:18.205526+0000","last_clean":"2024-05-09T08:26:18.205526+0000","last_became_active":"2024-05-09T08:23:51.577073+0000","last_became_peered":"2024-05-09T08:23:51.577073+0000","last_unstale":"2024-05-09T08:26:18.205526+0000","last_undegraded":"2024-05-09T08:26:18.205526+0000","last_fullsized":"2024-05-09T08:26:18.205526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:26.249488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203039+0000","last_change":"2024-05-09T08:23:51.574476+0000","last_active":"2024-05-09T08:26:18.203039+0000","last_peered":"2024-05-09T08:26:18.203039+0000","last_clean":"2024-05-09T08:26:18.203039+0000","last_became_active":"2024-05-09T08:23:51.572748+0000","last_became_peered":"2024-05-09T08:23:51.572748+0000","last_unstale":"2024-05-09T08:26:18.203039+0000","last_undegraded":"2024-05-09T08:26:18.203039+0000","last_fullsized":"2024-05-09T08:26:18.203039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:43.515419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203660+0000","last_change":"2024-05-09T08:23:51.580568+0000","last_active":"2024-05-09T08:26:18.203660+0000","last_peered":"2024-05-09T08:26:18.203660+0000","last_clean":"2024-05-09T08:26:18.203660+0000","last_became_active":"2024-05-09T08:23:51.579626+0000","last_became_peered":"2024-05-09T08:23:51.579626+0000","last_unstale":"2024-05-09T08:26:18.203660+0000","last_undegraded":"2024-05-09T08:26:18.203660+0000","last_fullsized":"2024-05-09T08:26:18.203660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:28.756079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083992+0000","last_change":"2024-05-09T08:23:51.574363+0000","last_active":"2024-05-09T08:26:18.083992+0000","last_peered":"2024-05-09T08:26:18.083992+0000","last_clean":"2024-05-09T08:26:18.083992+0000","last_became_active":"2024-05-09T08:23:51.573858+0000","last_became_peered":"2024-05-09T08:23:51.573858+0000","last_unstale":"2024-05-09T08:26:18.083992+0000","last_undegraded":"2024-05-09T08:26:18.083992+0000","last_fullsized":"2024-05-09T08:26:18.083992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:30.752939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971361+0000","last_change":"2024-05-09T08:23:51.579361+0000","last_active":"2024-05-09T08:26:17.971361+0000","last_peered":"2024-05-09T08:26:17.971361+0000","last_clean":"2024-05-09T08:26:17.971361+0000","last_became_active":"2024-05-09T08:23:51.579025+0000","last_became_peered":"2024-05-09T08:23:51.579025+0000","last_unstale":"2024-05-09T08:26:17.971361+0000","last_undegraded":"2024-05-09T08:26:17.971361+0000","last_fullsized":"2024-05-09T08:26:17.971361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:02.273999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971573+0000","last_change":"2024-05-09T08:23:55.670107+0000","last_active":"2024-05-09T08:26:17.971573+0000","last_peered":"2024-05-09T08:26:17.971573+0000","last_clean":"2024-05-09T08:26:17.971573+0000","last_became_active":"2024-05-09T08:23:55.669621+0000","last_became_peered":"2024-05-09T08:23:55.669621+0000","last_unstale":"2024-05-09T08:26:17.971573+0000","last_undegraded":"2024-05-09T08:26:17.971573+0000","last_fullsized":"2024-05-09T08:26:17.971573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:00.547137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'14","reported_seq":189,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.253755+0000","last_change":"2024-05-09T08:24:51.498795+0000","last_active":"2024-05-09T08:26:18.253755+0000","last_peered":"2024-05-09T08:26:18.253755+0000","last_clean":"2024-05-09T08:26:18.253755+0000","last_became_active":"2024-05-09T08:24:51.487098+0000","last_became_peered":"2024-05-09T08:24:51.487098+0000","last_unstale":"2024-05-09T08:26:18.253755+0000","last_undegraded":"2024-05-09T08:26:18.253755+0000","last_fullsized":"2024-05-09T08:26:18.253755+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:10.646049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083803+0000","last_change":"2024-05-09T08:25:52.410940+0000","last_active":"2024-05-09T08:26:18.083803+0000","last_peered":"2024-05-09T08:26:18.083803+0000","last_clean":"2024-05-09T08:26:18.083803+0000","last_became_active":"2024-05-09T08:25:52.410514+0000","last_became_peered":"2024-05-09T08:25:52.410514+0000","last_unstale":"2024-05-09T08:26:18.083803+0000","last_undegraded":"2024-05-09T08:26:18.083803+0000","last_fullsized":"2024-05-09T08:26:18.083803+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:51.831125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971160+0000","last_change":"2024-05-09T08:23:55.670194+0000","last_active":"2024-05-09T08:26:17.971160+0000","last_peered":"2024-05-09T08:26:17.971160+0000","last_clean":"2024-05-09T08:26:17.971160+0000","last_became_active":"2024-05-09T08:23:55.669762+0000","last_became_peered":"2024-05-09T08:23:55.669762+0000","last_unstale":"2024-05-09T08:26:17.971160+0000","last_undegraded":"2024-05-09T08:26:17.971160+0000","last_fullsized":"2024-05-09T08:26:17.971160+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:22.879635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970723+0000","last_change":"2024-05-09T08:23:50.574421+0000","last_active":"2024-05-09T08:26:17.970723+0000","last_peered":"2024-05-09T08:26:17.970723+0000","last_clean":"2024-05-09T08:26:17.970723+0000","last_became_active":"2024-05-09T08:23:50.574070+0000","last_became_peered":"2024-05-09T08:23:50.574070+0000","last_unstale":"2024-05-09T08:26:17.970723+0000","last_undegraded":"2024-05-09T08:26:17.970723+0000","last_fullsized":"2024-05-09T08:26:17.970723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:22.996034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204912+0000","last_change":"2024-05-09T08:23:51.583275+0000","last_active":"2024-05-09T08:26:18.204912+0000","last_peered":"2024-05-09T08:26:18.204912+0000","last_clean":"2024-05-09T08:26:18.204912+0000","last_became_active":"2024-05-09T08:23:51.582847+0000","last_became_peered":"2024-05-09T08:23:51.582847+0000","last_unstale":"2024-05-09T08:26:18.204912+0000","last_undegraded":"2024-05-09T08:26:18.204912+0000","last_fullsized":"2024-05-09T08:26:18.204912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:03:03.512852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971336+0000","last_change":"2024-05-09T08:23:55.672256+0000","last_active":"2024-05-09T08:26:17.971336+0000","last_peered":"2024-05-09T08:26:17.971336+0000","last_clean":"2024-05-09T08:26:17.971336+0000","last_became_active":"2024-05-09T08:23:55.671855+0000","last_became_peered":"2024-05-09T08:23:55.671855+0000","last_unstale":"2024-05-09T08:26:17.971336+0000","last_undegraded":"2024-05-09T08:26:17.971336+0000","last_fullsized":"2024-05-09T08:26:17.971336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:45:12.199359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"94'13","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.087175+0000","last_change":"2024-05-09T08:23:50.566686+0000","last_active":"2024-05-09T08:26:18.087175+0000","last_peered":"2024-05-09T08:26:18.087175+0000","last_clean":"2024-05-09T08:26:18.087175+0000","last_became_active":"2024-05-09T08:23:50.565910+0000","last_became_peered":"2024-05-09T08:23:50.565910+0000","last_unstale":"2024-05-09T08:26:18.087175+0000","last_undegraded":"2024-05-09T08:26:18.087175+0000","last_fullsized":"2024-05-09T08:26:18.087175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:02.211551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203319+0000","last_change":"2024-05-09T08:23:51.570972+0000","last_active":"2024-05-09T08:26:18.203319+0000","last_peered":"2024-05-09T08:26:18.203319+0000","last_clean":"2024-05-09T08:26:18.203319+0000","last_became_active":"2024-05-09T08:23:51.570638+0000","last_became_peered":"2024-05-09T08:23:51.570638+0000","last_unstale":"2024-05-09T08:26:18.203319+0000","last_undegraded":"2024-05-09T08:26:18.203319+0000","last_fullsized":"2024-05-09T08:26:18.203319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:38.738820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971213+0000","last_change":"2024-05-09T08:23:55.671929+0000","last_active":"2024-05-09T08:26:17.971213+0000","last_peered":"2024-05-09T08:26:17.971213+0000","last_clean":"2024-05-09T08:26:17.971213+0000","last_became_active":"2024-05-09T08:23:55.671483+0000","last_became_peered":"2024-05-09T08:23:55.671483+0000","last_unstale":"2024-05-09T08:26:17.971213+0000","last_undegraded":"2024-05-09T08:26:17.971213+0000","last_fullsized":"2024-05-09T08:26:17.971213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:21.527266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086737+0000","last_change":"2024-05-09T08:23:50.565644+0000","last_active":"2024-05-09T08:26:18.086737+0000","last_peered":"2024-05-09T08:26:18.086737+0000","last_clean":"2024-05-09T08:26:18.086737+0000","last_became_active":"2024-05-09T08:23:50.565265+0000","last_became_peered":"2024-05-09T08:23:50.565265+0000","last_unstale":"2024-05-09T08:26:18.086737+0000","last_undegraded":"2024-05-09T08:26:18.086737+0000","last_fullsized":"2024-05-09T08:26:18.086737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:51:57.972535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092603+0000","last_change":"2024-05-09T08:23:51.571732+0000","last_active":"2024-05-09T08:26:18.092603+0000","last_peered":"2024-05-09T08:26:18.092603+0000","last_clean":"2024-05-09T08:26:18.092603+0000","last_became_active":"2024-05-09T08:23:51.570783+0000","last_became_peered":"2024-05-09T08:23:51.570783+0000","last_unstale":"2024-05-09T08:26:18.092603+0000","last_undegraded":"2024-05-09T08:26:18.092603+0000","last_fullsized":"2024-05-09T08:26:18.092603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:18.515905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971228+0000","last_change":"2024-05-09T08:23:55.666727+0000","last_active":"2024-05-09T08:26:17.971228+0000","last_peered":"2024-05-09T08:26:17.971228+0000","last_clean":"2024-05-09T08:26:17.971228+0000","last_became_active":"2024-05-09T08:23:55.666284+0000","last_became_peered":"2024-05-09T08:23:55.666284+0000","last_unstale":"2024-05-09T08:26:17.971228+0000","last_undegraded":"2024-05-09T08:26:17.971228+0000","last_fullsized":"2024-05-09T08:26:17.971228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:05.355404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.595864+0000","last_change":"2024-05-09T08:23:50.573308+0000","last_active":"2024-05-09T08:26:18.595864+0000","last_peered":"2024-05-09T08:26:18.595864+0000","last_clean":"2024-05-09T08:26:18.595864+0000","last_became_active":"2024-05-09T08:23:50.572946+0000","last_became_peered":"2024-05-09T08:23:50.572946+0000","last_unstale":"2024-05-09T08:26:18.595864+0000","last_undegraded":"2024-05-09T08:26:18.595864+0000","last_fullsized":"2024-05-09T08:26:18.595864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:44:35.816620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086752+0000","last_change":"2024-05-09T08:23:51.571748+0000","last_active":"2024-05-09T08:26:18.086752+0000","last_peered":"2024-05-09T08:26:18.086752+0000","last_clean":"2024-05-09T08:26:18.086752+0000","last_became_active":"2024-05-09T08:23:51.571388+0000","last_became_peered":"2024-05-09T08:23:51.571388+0000","last_unstale":"2024-05-09T08:26:18.086752+0000","last_undegraded":"2024-05-09T08:26:18.086752+0000","last_fullsized":"2024-05-09T08:26:18.086752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:34.916491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971219+0000","last_change":"2024-05-09T08:23:55.674022+0000","last_active":"2024-05-09T08:26:17.971219+0000","last_peered":"2024-05-09T08:26:17.971219+0000","last_clean":"2024-05-09T08:26:17.971219+0000","last_became_active":"2024-05-09T08:23:55.673322+0000","last_became_peered":"2024-05-09T08:23:55.673322+0000","last_unstale":"2024-05-09T08:26:17.971219+0000","last_undegraded":"2024-05-09T08:26:17.971219+0000","last_fullsized":"2024-05-09T08:26:17.971219+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:59:14.299886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.095836+0000","last_change":"2024-05-09T08:23:50.571235+0000","last_active":"2024-05-09T08:26:18.095836+0000","last_peered":"2024-05-09T08:26:18.095836+0000","last_clean":"2024-05-09T08:26:18.095836+0000","last_became_active":"2024-05-09T08:23:50.570694+0000","last_became_peered":"2024-05-09T08:23:50.570694+0000","last_unstale":"2024-05-09T08:26:18.095836+0000","last_undegraded":"2024-05-09T08:26:18.095836+0000","last_fullsized":"2024-05-09T08:26:18.095836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:22.850248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092662+0000","last_change":"2024-05-09T08:25:52.410264+0000","last_active":"2024-05-09T08:26:18.092662+0000","last_peered":"2024-05-09T08:26:18.092662+0000","last_clean":"2024-05-09T08:26:18.092662+0000","last_became_active":"2024-05-09T08:25:52.409851+0000","last_became_peered":"2024-05-09T08:25:52.409851+0000","last_unstale":"2024-05-09T08:26:18.092662+0000","last_undegraded":"2024-05-09T08:26:18.092662+0000","last_fullsized":"2024-05-09T08:26:18.092662+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:20:56.614365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203490+0000","last_change":"2024-05-09T08:24:51.489278+0000","last_active":"2024-05-09T08:26:18.203490+0000","last_peered":"2024-05-09T08:26:18.203490+0000","last_clean":"2024-05-09T08:26:18.203490+0000","last_became_active":"2024-05-09T08:24:51.488875+0000","last_became_peered":"2024-05-09T08:24:51.488875+0000","last_unstale":"2024-05-09T08:26:18.203490+0000","last_undegraded":"2024-05-09T08:26:18.203490+0000","last_fullsized":"2024-05-09T08:26:18.203490+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:47:45.306525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203847+0000","last_change":"2024-05-09T08:23:50.570040+0000","last_active":"2024-05-09T08:26:18.203847+0000","last_peered":"2024-05-09T08:26:18.203847+0000","last_clean":"2024-05-09T08:26:18.203847+0000","last_became_active":"2024-05-09T08:23:50.569665+0000","last_became_peered":"2024-05-09T08:23:50.569665+0000","last_unstale":"2024-05-09T08:26:18.203847+0000","last_undegraded":"2024-05-09T08:26:18.203847+0000","last_fullsized":"2024-05-09T08:26:18.203847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:22:29.639712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970223+0000","last_change":"2024-05-09T08:23:51.568561+0000","last_active":"2024-05-09T08:26:17.970223+0000","last_peered":"2024-05-09T08:26:17.970223+0000","last_clean":"2024-05-09T08:26:17.970223+0000","last_became_active":"2024-05-09T08:23:51.568196+0000","last_became_peered":"2024-05-09T08:23:51.568196+0000","last_unstale":"2024-05-09T08:26:17.970223+0000","last_undegraded":"2024-05-09T08:26:17.970223+0000","last_fullsized":"2024-05-09T08:26:17.970223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:52.132837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206099+0000","last_change":"2024-05-09T08:23:55.674960+0000","last_active":"2024-05-09T08:26:18.206099+0000","last_peered":"2024-05-09T08:26:18.206099+0000","last_clean":"2024-05-09T08:26:18.206099+0000","last_became_active":"2024-05-09T08:23:55.674509+0000","last_became_peered":"2024-05-09T08:23:55.674509+0000","last_unstale":"2024-05-09T08:26:18.206099+0000","last_undegraded":"2024-05-09T08:26:18.206099+0000","last_fullsized":"2024-05-09T08:26:18.206099+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:22.479867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092234+0000","last_change":"2024-05-09T08:23:50.569896+0000","last_active":"2024-05-09T08:26:18.092234+0000","last_peered":"2024-05-09T08:26:18.092234+0000","last_clean":"2024-05-09T08:26:18.092234+0000","last_became_active":"2024-05-09T08:23:50.569562+0000","last_became_peered":"2024-05-09T08:23:50.569562+0000","last_unstale":"2024-05-09T08:26:18.092234+0000","last_undegraded":"2024-05-09T08:26:18.092234+0000","last_fullsized":"2024-05-09T08:26:18.092234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:25:20.278726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083524+0000","last_change":"2024-05-09T08:23:51.570260+0000","last_active":"2024-05-09T08:26:18.083524+0000","last_peered":"2024-05-09T08:26:18.083524+0000","last_clean":"2024-05-09T08:26:18.083524+0000","last_became_active":"2024-05-09T08:23:51.569837+0000","last_became_peered":"2024-05-09T08:23:51.569837+0000","last_unstale":"2024-05-09T08:26:18.083524+0000","last_undegraded":"2024-05-09T08:26:18.083524+0000","last_fullsized":"2024-05-09T08:26:18.083524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:01:44.805271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970259+0000","last_change":"2024-05-09T08:23:55.676639+0000","last_active":"2024-05-09T08:26:17.970259+0000","last_peered":"2024-05-09T08:26:17.970259+0000","last_clean":"2024-05-09T08:26:17.970259+0000","last_became_active":"2024-05-09T08:23:55.673643+0000","last_became_peered":"2024-05-09T08:23:55.673643+0000","last_unstale":"2024-05-09T08:26:17.970259+0000","last_undegraded":"2024-05-09T08:26:17.970259+0000","last_fullsized":"2024-05-09T08:26:17.970259+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:02.377184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'6","reported_seq":280,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.595264+0000","last_change":"2024-05-09T08:23:50.565745+0000","last_active":"2024-05-09T08:26:18.595264+0000","last_peered":"2024-05-09T08:26:18.595264+0000","last_clean":"2024-05-09T08:26:18.595264+0000","last_became_active":"2024-05-09T08:23:50.565385+0000","last_became_peered":"2024-05-09T08:23:50.565385+0000","last_unstale":"2024-05-09T08:26:18.595264+0000","last_undegraded":"2024-05-09T08:26:18.595264+0000","last_fullsized":"2024-05-09T08:26:18.595264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:05:00.818398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":240,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203490+0000","last_change":"2024-05-09T08:23:51.572341+0000","last_active":"2024-05-09T08:26:18.203490+0000","last_peered":"2024-05-09T08:26:18.203490+0000","last_clean":"2024-05-09T08:26:18.203490+0000","last_became_active":"2024-05-09T08:23:51.571583+0000","last_became_peered":"2024-05-09T08:23:51.571583+0000","last_unstale":"2024-05-09T08:26:18.203490+0000","last_undegraded":"2024-05-09T08:26:18.203490+0000","last_fullsized":"2024-05-09T08:26:18.203490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:20.695995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083374+0000","last_change":"2024-05-09T08:23:55.677603+0000","last_active":"2024-05-09T08:26:18.083374+0000","last_peered":"2024-05-09T08:26:18.083374+0000","last_clean":"2024-05-09T08:26:18.083374+0000","last_became_active":"2024-05-09T08:23:55.677143+0000","last_became_peered":"2024-05-09T08:23:55.677143+0000","last_unstale":"2024-05-09T08:26:18.083374+0000","last_undegraded":"2024-05-09T08:26:18.083374+0000","last_fullsized":"2024-05-09T08:26:18.083374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:27.850997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.188009+0000","last_change":"2024-05-09T08:23:50.565497+0000","last_active":"2024-05-09T08:26:18.188009+0000","last_peered":"2024-05-09T08:26:18.188009+0000","last_clean":"2024-05-09T08:26:18.188009+0000","last_became_active":"2024-05-09T08:23:50.565120+0000","last_became_peered":"2024-05-09T08:23:50.565120+0000","last_unstale":"2024-05-09T08:26:18.188009+0000","last_undegraded":"2024-05-09T08:26:18.188009+0000","last_fullsized":"2024-05-09T08:26:18.188009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:56.141997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206840+0000","last_change":"2024-05-09T08:23:51.576489+0000","last_active":"2024-05-09T08:26:18.206840+0000","last_peered":"2024-05-09T08:26:18.206840+0000","last_clean":"2024-05-09T08:26:18.206840+0000","last_became_active":"2024-05-09T08:23:51.576145+0000","last_became_peered":"2024-05-09T08:23:51.576145+0000","last_unstale":"2024-05-09T08:26:18.206840+0000","last_undegraded":"2024-05-09T08:26:18.206840+0000","last_fullsized":"2024-05-09T08:26:18.206840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:37.650308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971493+0000","last_change":"2024-05-09T08:24:51.490957+0000","last_active":"2024-05-09T08:26:17.971493+0000","last_peered":"2024-05-09T08:26:17.971493+0000","last_clean":"2024-05-09T08:26:17.971493+0000","last_became_active":"2024-05-09T08:24:51.490541+0000","last_became_peered":"2024-05-09T08:24:51.490541+0000","last_unstale":"2024-05-09T08:26:17.971493+0000","last_undegraded":"2024-05-09T08:26:17.971493+0000","last_fullsized":"2024-05-09T08:26:17.971493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:03.168547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092549+0000","last_change":"2024-05-09T08:23:50.567182+0000","last_active":"2024-05-09T08:26:18.092549+0000","last_peered":"2024-05-09T08:26:18.092549+0000","last_clean":"2024-05-09T08:26:18.092549+0000","last_became_active":"2024-05-09T08:23:50.566851+0000","last_became_peered":"2024-05-09T08:23:50.566851+0000","last_unstale":"2024-05-09T08:26:18.092549+0000","last_undegraded":"2024-05-09T08:26:18.092549+0000","last_fullsized":"2024-05-09T08:26:18.092549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:26.540456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203499+0000","last_change":"2024-05-09T08:23:51.574074+0000","last_active":"2024-05-09T08:26:18.203499+0000","last_peered":"2024-05-09T08:26:18.203499+0000","last_clean":"2024-05-09T08:26:18.203499+0000","last_became_active":"2024-05-09T08:23:51.572587+0000","last_became_peered":"2024-05-09T08:23:51.572587+0000","last_unstale":"2024-05-09T08:26:18.203499+0000","last_undegraded":"2024-05-09T08:26:18.203499+0000","last_fullsized":"2024-05-09T08:26:18.203499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:46.186358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083525+0000","last_change":"2024-05-09T08:23:55.667170+0000","last_active":"2024-05-09T08:26:18.083525+0000","last_peered":"2024-05-09T08:26:18.083525+0000","last_clean":"2024-05-09T08:26:18.083525+0000","last_became_active":"2024-05-09T08:23:55.666707+0000","last_became_peered":"2024-05-09T08:23:55.666707+0000","last_unstale":"2024-05-09T08:26:18.083525+0000","last_undegraded":"2024-05-09T08:26:18.083525+0000","last_fullsized":"2024-05-09T08:26:18.083525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:48.332207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":345,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.609304+0000","last_change":"2024-05-09T08:24:51.509495+0000","last_active":"2024-05-09T08:26:18.609304+0000","last_peered":"2024-05-09T08:26:18.609304+0000","last_clean":"2024-05-09T08:26:18.609304+0000","last_became_active":"2024-05-09T08:24:51.491187+0000","last_became_peered":"2024-05-09T08:24:51.491187+0000","last_unstale":"2024-05-09T08:26:18.609304+0000","last_undegraded":"2024-05-09T08:26:18.609304+0000","last_fullsized":"2024-05-09T08:26:18.609304+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:55.320000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":241,"num_read_kb":190,"num_write":44,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":63356,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.087061+0000","last_change":"2024-05-09T08:23:51.577595+0000","last_active":"2024-05-09T08:26:18.087061+0000","last_peered":"2024-05-09T08:26:18.087061+0000","last_clean":"2024-05-09T08:26:18.087061+0000","last_became_active":"2024-05-09T08:23:51.577271+0000","last_became_peered":"2024-05-09T08:23:51.577271+0000","last_unstale":"2024-05-09T08:26:18.087061+0000","last_undegraded":"2024-05-09T08:26:18.087061+0000","last_fullsized":"2024-05-09T08:26:18.087061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:34.679111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092985+0000","last_change":"2024-05-09T08:23:55.671128+0000","last_active":"2024-05-09T08:26:18.092985+0000","last_peered":"2024-05-09T08:26:18.092985+0000","last_clean":"2024-05-09T08:26:18.092985+0000","last_became_active":"2024-05-09T08:23:55.670414+0000","last_became_peered":"2024-05-09T08:23:55.670414+0000","last_unstale":"2024-05-09T08:26:18.092985+0000","last_undegraded":"2024-05-09T08:26:18.092985+0000","last_fullsized":"2024-05-09T08:26:18.092985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:31:55.307412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971568+0000","last_change":"2024-05-09T08:23:50.568503+0000","last_active":"2024-05-09T08:26:17.971568+0000","last_peered":"2024-05-09T08:26:17.971568+0000","last_clean":"2024-05-09T08:26:17.971568+0000","last_became_active":"2024-05-09T08:23:50.568181+0000","last_became_peered":"2024-05-09T08:23:50.568181+0000","last_unstale":"2024-05-09T08:26:17.971568+0000","last_undegraded":"2024-05-09T08:26:17.971568+0000","last_fullsized":"2024-05-09T08:26:17.971568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:40.883792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204000+0000","last_change":"2024-05-09T08:23:51.573602+0000","last_active":"2024-05-09T08:26:18.204000+0000","last_peered":"2024-05-09T08:26:18.204000+0000","last_clean":"2024-05-09T08:26:18.204000+0000","last_became_active":"2024-05-09T08:23:51.572557+0000","last_became_peered":"2024-05-09T08:23:51.572557+0000","last_unstale":"2024-05-09T08:26:18.204000+0000","last_undegraded":"2024-05-09T08:26:18.204000+0000","last_fullsized":"2024-05-09T08:26:18.204000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.170962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203332+0000","last_change":"2024-05-09T08:23:55.673773+0000","last_active":"2024-05-09T08:26:18.203332+0000","last_peered":"2024-05-09T08:26:18.203332+0000","last_clean":"2024-05-09T08:26:18.203332+0000","last_became_active":"2024-05-09T08:23:55.673276+0000","last_became_peered":"2024-05-09T08:23:55.673276+0000","last_unstale":"2024-05-09T08:26:18.203332+0000","last_undegraded":"2024-05-09T08:26:18.203332+0000","last_fullsized":"2024-05-09T08:26:18.203332+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:59:00.629339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092468+0000","last_change":"2024-05-09T08:23:50.575894+0000","last_active":"2024-05-09T08:26:18.092468+0000","last_peered":"2024-05-09T08:26:18.092468+0000","last_clean":"2024-05-09T08:26:18.092468+0000","last_became_active":"2024-05-09T08:23:50.575555+0000","last_became_peered":"2024-05-09T08:23:50.575555+0000","last_unstale":"2024-05-09T08:26:18.092468+0000","last_undegraded":"2024-05-09T08:26:18.092468+0000","last_fullsized":"2024-05-09T08:26:18.092468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:19.070764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187948+0000","last_change":"2024-05-09T08:23:51.574125+0000","last_active":"2024-05-09T08:26:18.187948+0000","last_peered":"2024-05-09T08:26:18.187948+0000","last_clean":"2024-05-09T08:26:18.187948+0000","last_became_active":"2024-05-09T08:23:51.573794+0000","last_became_peered":"2024-05-09T08:23:51.573794+0000","last_unstale":"2024-05-09T08:26:18.187948+0000","last_undegraded":"2024-05-09T08:26:18.187948+0000","last_fullsized":"2024-05-09T08:26:18.187948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:19:26.634208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.084105+0000","last_change":"2024-05-09T08:23:55.677711+0000","last_active":"2024-05-09T08:26:18.084105+0000","last_peered":"2024-05-09T08:26:18.084105+0000","last_clean":"2024-05-09T08:26:18.084105+0000","last_became_active":"2024-05-09T08:23:55.677058+0000","last_became_peered":"2024-05-09T08:23:55.677058+0000","last_unstale":"2024-05-09T08:26:18.084105+0000","last_undegraded":"2024-05-09T08:26:18.084105+0000","last_fullsized":"2024-05-09T08:26:18.084105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:26.842145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'9","reported_seq":167,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.599726+0000","last_change":"2024-05-09T08:23:50.566413+0000","last_active":"2024-05-09T08:26:18.599726+0000","last_peered":"2024-05-09T08:26:18.599726+0000","last_clean":"2024-05-09T08:26:18.599726+0000","last_became_active":"2024-05-09T08:23:50.566047+0000","last_became_peered":"2024-05-09T08:23:50.566047+0000","last_unstale":"2024-05-09T08:26:18.599726+0000","last_undegraded":"2024-05-09T08:26:18.599726+0000","last_fullsized":"2024-05-09T08:26:18.599726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:15.340947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":5,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187600+0000","last_change":"2024-05-09T08:23:51.576779+0000","last_active":"2024-05-09T08:26:18.187600+0000","last_peered":"2024-05-09T08:26:18.187600+0000","last_clean":"2024-05-09T08:26:18.187600+0000","last_became_active":"2024-05-09T08:23:51.576119+0000","last_became_peered":"2024-05-09T08:23:51.576119+0000","last_unstale":"2024-05-09T08:26:18.187600+0000","last_undegraded":"2024-05-09T08:26:18.187600+0000","last_fullsized":"2024-05-09T08:26:18.187600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:02.250135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204904+0000","last_change":"2024-05-09T08:23:55.678564+0000","last_active":"2024-05-09T08:26:18.204904+0000","last_peered":"2024-05-09T08:26:18.204904+0000","last_clean":"2024-05-09T08:26:18.204904+0000","last_became_active":"2024-05-09T08:23:55.678128+0000","last_became_peered":"2024-05-09T08:23:55.678128+0000","last_unstale":"2024-05-09T08:26:18.204904+0000","last_undegraded":"2024-05-09T08:26:18.204904+0000","last_fullsized":"2024-05-09T08:26:18.204904+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:13:21.868267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086552+0000","last_change":"2024-05-09T08:24:51.501243+0000","last_active":"2024-05-09T08:26:18.086552+0000","last_peered":"2024-05-09T08:26:18.086552+0000","last_clean":"2024-05-09T08:26:18.086552+0000","last_became_active":"2024-05-09T08:24:51.490110+0000","last_became_peered":"2024-05-09T08:24:51.490110+0000","last_unstale":"2024-05-09T08:26:18.086552+0000","last_undegraded":"2024-05-09T08:26:18.086552+0000","last_fullsized":"2024-05-09T08:26:18.086552+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:03.086570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092330+0000","last_change":"2024-05-09T08:23:51.574914+0000","last_active":"2024-05-09T08:26:18.092330+0000","last_peered":"2024-05-09T08:26:18.092330+0000","last_clean":"2024-05-09T08:26:18.092330+0000","last_became_active":"2024-05-09T08:23:51.574380+0000","last_became_peered":"2024-05-09T08:23:51.574380+0000","last_unstale":"2024-05-09T08:26:18.092330+0000","last_undegraded":"2024-05-09T08:26:18.092330+0000","last_fullsized":"2024-05-09T08:26:18.092330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:34.698328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092194+0000","last_change":"2024-05-09T08:23:55.670198+0000","last_active":"2024-05-09T08:26:18.092194+0000","last_peered":"2024-05-09T08:26:18.092194+0000","last_clean":"2024-05-09T08:26:18.092194+0000","last_became_active":"2024-05-09T08:23:55.669798+0000","last_became_peered":"2024-05-09T08:23:55.669798+0000","last_unstale":"2024-05-09T08:26:18.092194+0000","last_undegraded":"2024-05-09T08:26:18.092194+0000","last_fullsized":"2024-05-09T08:26:18.092194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:09.474264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203593+0000","last_change":"2024-05-09T08:24:51.504286+0000","last_active":"2024-05-09T08:26:18.203593+0000","last_peered":"2024-05-09T08:26:18.203593+0000","last_clean":"2024-05-09T08:26:18.203593+0000","last_became_active":"2024-05-09T08:24:51.489111+0000","last_became_peered":"2024-05-09T08:24:51.489111+0000","last_unstale":"2024-05-09T08:26:18.203593+0000","last_undegraded":"2024-05-09T08:26:18.203593+0000","last_fullsized":"2024-05-09T08:26:18.203593+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:14.975959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203398+0000","last_change":"2024-05-09T08:23:51.573434+0000","last_active":"2024-05-09T08:26:18.203398+0000","last_peered":"2024-05-09T08:26:18.203398+0000","last_clean":"2024-05-09T08:26:18.203398+0000","last_became_active":"2024-05-09T08:23:51.573081+0000","last_became_peered":"2024-05-09T08:23:51.573081+0000","last_unstale":"2024-05-09T08:26:18.203398+0000","last_undegraded":"2024-05-09T08:26:18.203398+0000","last_fullsized":"2024-05-09T08:26:18.203398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:49.270355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086717+0000","last_change":"2024-05-09T08:23:55.853114+0000","last_active":"2024-05-09T08:26:18.086717+0000","last_peered":"2024-05-09T08:26:18.086717+0000","last_clean":"2024-05-09T08:26:18.086717+0000","last_became_active":"2024-05-09T08:23:55.852640+0000","last_became_peered":"2024-05-09T08:23:55.852640+0000","last_unstale":"2024-05-09T08:26:18.086717+0000","last_undegraded":"2024-05-09T08:26:18.086717+0000","last_fullsized":"2024-05-09T08:26:18.086717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:55:17.823585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083798+0000","last_change":"2024-05-09T08:23:50.567834+0000","last_active":"2024-05-09T08:26:18.083798+0000","last_peered":"2024-05-09T08:26:18.083798+0000","last_clean":"2024-05-09T08:26:18.083798+0000","last_became_active":"2024-05-09T08:23:50.567538+0000","last_became_peered":"2024-05-09T08:23:50.567538+0000","last_unstale":"2024-05-09T08:26:18.083798+0000","last_undegraded":"2024-05-09T08:26:18.083798+0000","last_fullsized":"2024-05-09T08:26:18.083798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.674352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.096142+0000","last_change":"2024-05-09T08:23:51.580219+0000","last_active":"2024-05-09T08:26:18.096142+0000","last_peered":"2024-05-09T08:26:18.096142+0000","last_clean":"2024-05-09T08:26:18.096142+0000","last_became_active":"2024-05-09T08:23:51.579835+0000","last_became_peered":"2024-05-09T08:23:51.579835+0000","last_unstale":"2024-05-09T08:26:18.096142+0000","last_undegraded":"2024-05-09T08:26:18.096142+0000","last_fullsized":"2024-05-09T08:26:18.096142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:15.232318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204599+0000","last_change":"2024-05-09T08:23:55.674324+0000","last_active":"2024-05-09T08:26:18.204599+0000","last_peered":"2024-05-09T08:26:18.204599+0000","last_clean":"2024-05-09T08:26:18.204599+0000","last_became_active":"2024-05-09T08:23:55.673254+0000","last_became_peered":"2024-05-09T08:23:55.673254+0000","last_unstale":"2024-05-09T08:26:18.204599+0000","last_undegraded":"2024-05-09T08:26:18.204599+0000","last_fullsized":"2024-05-09T08:26:18.204599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:05:23.303987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'6","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.617307+0000","last_change":"2024-05-09T08:23:50.571682+0000","last_active":"2024-05-09T08:26:18.617307+0000","last_peered":"2024-05-09T08:26:18.617307+0000","last_clean":"2024-05-09T08:26:18.617307+0000","last_became_active":"2024-05-09T08:23:50.571338+0000","last_became_peered":"2024-05-09T08:23:50.571338+0000","last_unstale":"2024-05-09T08:26:18.617307+0000","last_undegraded":"2024-05-09T08:26:18.617307+0000","last_fullsized":"2024-05-09T08:26:18.617307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:17.578071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.095802+0000","last_change":"2024-05-09T08:24:51.487208+0000","last_active":"2024-05-09T08:26:18.095802+0000","last_peered":"2024-05-09T08:26:18.095802+0000","last_clean":"2024-05-09T08:26:18.095802+0000","last_became_active":"2024-05-09T08:24:51.486665+0000","last_became_peered":"2024-05-09T08:24:51.486665+0000","last_unstale":"2024-05-09T08:26:18.095802+0000","last_undegraded":"2024-05-09T08:26:18.095802+0000","last_fullsized":"2024-05-09T08:26:18.095802+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:01:10.050846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205221+0000","last_change":"2024-05-09T08:23:55.853086+0000","last_active":"2024-05-09T08:26:18.205221+0000","last_peered":"2024-05-09T08:26:18.205221+0000","last_clean":"2024-05-09T08:26:18.205221+0000","last_became_active":"2024-05-09T08:23:55.852302+0000","last_became_peered":"2024-05-09T08:23:55.852302+0000","last_unstale":"2024-05-09T08:26:18.205221+0000","last_undegraded":"2024-05-09T08:26:18.205221+0000","last_fullsized":"2024-05-09T08:26:18.205221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:42:18.547623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092340+0000","last_change":"2024-05-09T08:23:51.575522+0000","last_active":"2024-05-09T08:26:18.092340+0000","last_peered":"2024-05-09T08:26:18.092340+0000","last_clean":"2024-05-09T08:26:18.092340+0000","last_became_active":"2024-05-09T08:23:51.575204+0000","last_became_peered":"2024-05-09T08:23:51.575204+0000","last_unstale":"2024-05-09T08:26:18.092340+0000","last_undegraded":"2024-05-09T08:26:18.092340+0000","last_fullsized":"2024-05-09T08:26:18.092340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:57.257435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'7","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.253210+0000","last_change":"2024-05-09T08:23:50.574996+0000","last_active":"2024-05-09T08:26:18.253210+0000","last_peered":"2024-05-09T08:26:18.253210+0000","last_clean":"2024-05-09T08:26:18.253210+0000","last_became_active":"2024-05-09T08:23:50.574619+0000","last_became_peered":"2024-05-09T08:23:50.574619+0000","last_unstale":"2024-05-09T08:26:18.253210+0000","last_undegraded":"2024-05-09T08:26:18.253210+0000","last_fullsized":"2024-05-09T08:26:18.253210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:44.681314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187892+0000","last_change":"2024-05-09T08:24:51.493390+0000","last_active":"2024-05-09T08:26:18.187892+0000","last_peered":"2024-05-09T08:26:18.187892+0000","last_clean":"2024-05-09T08:26:18.187892+0000","last_became_active":"2024-05-09T08:24:51.487797+0000","last_became_peered":"2024-05-09T08:24:51.487797+0000","last_unstale":"2024-05-09T08:26:18.187892+0000","last_undegraded":"2024-05-09T08:26:18.187892+0000","last_fullsized":"2024-05-09T08:26:18.187892+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:46.646471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083550+0000","last_change":"2024-05-09T08:25:52.411415+0000","last_active":"2024-05-09T08:26:18.083550+0000","last_peered":"2024-05-09T08:26:18.083550+0000","last_clean":"2024-05-09T08:26:18.083550+0000","last_became_active":"2024-05-09T08:25:52.410972+0000","last_became_peered":"2024-05-09T08:25:52.410972+0000","last_unstale":"2024-05-09T08:26:18.083550+0000","last_undegraded":"2024-05-09T08:26:18.083550+0000","last_fullsized":"2024-05-09T08:26:18.083550+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:31.045318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971384+0000","last_change":"2024-05-09T08:23:50.566415+0000","last_active":"2024-05-09T08:26:17.971384+0000","last_peered":"2024-05-09T08:26:17.971384+0000","last_clean":"2024-05-09T08:26:17.971384+0000","last_became_active":"2024-05-09T08:23:50.565506+0000","last_became_peered":"2024-05-09T08:23:50.565506+0000","last_unstale":"2024-05-09T08:26:17.971384+0000","last_undegraded":"2024-05-09T08:26:17.971384+0000","last_fullsized":"2024-05-09T08:26:17.971384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:15.741565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187381+0000","last_change":"2024-05-09T08:22:38.366126+0000","last_active":"2024-05-09T08:26:18.187381+0000","last_peered":"2024-05-09T08:26:18.187381+0000","last_clean":"2024-05-09T08:26:18.187381+0000","last_became_active":"2024-05-09T08:22:38.365683+0000","last_became_peered":"2024-05-09T08:22:38.365683+0000","last_unstale":"2024-05-09T08:26:18.187381+0000","last_undegraded":"2024-05-09T08:26:18.187381+0000","last_fullsized":"2024-05-09T08:26:18.187381+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_clean_scrub_stamp":"2024-05-09T08:19:12.435445+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:04.744060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092887+0000","last_change":"2024-05-09T08:23:55.853248+0000","last_active":"2024-05-09T08:26:18.092887+0000","last_peered":"2024-05-09T08:26:18.092887+0000","last_clean":"2024-05-09T08:26:18.092887+0000","last_became_active":"2024-05-09T08:23:55.852646+0000","last_became_peered":"2024-05-09T08:23:55.852646+0000","last_unstale":"2024-05-09T08:26:18.092887+0000","last_undegraded":"2024-05-09T08:26:18.092887+0000","last_fullsized":"2024-05-09T08:26:18.092887+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:19:54.378185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203614+0000","last_change":"2024-05-09T08:23:51.574967+0000","last_active":"2024-05-09T08:26:18.203614+0000","last_peered":"2024-05-09T08:26:18.203614+0000","last_clean":"2024-05-09T08:26:18.203614+0000","last_became_active":"2024-05-09T08:23:51.574600+0000","last_became_peered":"2024-05-09T08:23:51.574600+0000","last_unstale":"2024-05-09T08:26:18.203614+0000","last_undegraded":"2024-05-09T08:26:18.203614+0000","last_fullsized":"2024-05-09T08:26:18.203614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:03.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187411+0000","last_change":"2024-05-09T08:23:50.569886+0000","last_active":"2024-05-09T08:26:18.187411+0000","last_peered":"2024-05-09T08:26:18.187411+0000","last_clean":"2024-05-09T08:26:18.187411+0000","last_became_active":"2024-05-09T08:23:50.569018+0000","last_became_peered":"2024-05-09T08:23:50.569018+0000","last_unstale":"2024-05-09T08:26:18.187411+0000","last_undegraded":"2024-05-09T08:26:18.187411+0000","last_fullsized":"2024-05-09T08:26:18.187411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:20.711485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203347+0000","last_change":"2024-05-09T08:23:55.676408+0000","last_active":"2024-05-09T08:26:18.203347+0000","last_peered":"2024-05-09T08:26:18.203347+0000","last_clean":"2024-05-09T08:26:18.203347+0000","last_became_active":"2024-05-09T08:23:55.675964+0000","last_became_peered":"2024-05-09T08:23:55.675964+0000","last_unstale":"2024-05-09T08:26:18.203347+0000","last_undegraded":"2024-05-09T08:26:18.203347+0000","last_fullsized":"2024-05-09T08:26:18.203347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:29.210602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086991+0000","last_change":"2024-05-09T08:23:50.566946+0000","last_active":"2024-05-09T08:26:18.086991+0000","last_peered":"2024-05-09T08:26:18.086991+0000","last_clean":"2024-05-09T08:26:18.086991+0000","last_became_active":"2024-05-09T08:23:50.566366+0000","last_became_peered":"2024-05-09T08:23:50.566366+0000","last_unstale":"2024-05-09T08:26:18.086991+0000","last_undegraded":"2024-05-09T08:26:18.086991+0000","last_fullsized":"2024-05-09T08:26:18.086991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:37:30.720879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205041+0000","last_change":"2024-05-09T08:23:51.571520+0000","last_active":"2024-05-09T08:26:18.205041+0000","last_peered":"2024-05-09T08:26:18.205041+0000","last_clean":"2024-05-09T08:26:18.205041+0000","last_became_active":"2024-05-09T08:23:51.571166+0000","last_became_peered":"2024-05-09T08:23:51.571166+0000","last_unstale":"2024-05-09T08:26:18.205041+0000","last_undegraded":"2024-05-09T08:26:18.205041+0000","last_fullsized":"2024-05-09T08:26:18.205041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:02:12.560238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970123+0000","last_change":"2024-05-09T08:23:55.852649+0000","last_active":"2024-05-09T08:26:17.970123+0000","last_peered":"2024-05-09T08:26:17.970123+0000","last_clean":"2024-05-09T08:26:17.970123+0000","last_became_active":"2024-05-09T08:23:55.852180+0000","last_became_peered":"2024-05-09T08:23:55.852180+0000","last_unstale":"2024-05-09T08:26:17.970123+0000","last_undegraded":"2024-05-09T08:26:17.970123+0000","last_fullsized":"2024-05-09T08:26:17.970123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:33.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203198+0000","last_change":"2024-05-09T08:23:50.561639+0000","last_active":"2024-05-09T08:26:18.203198+0000","last_peered":"2024-05-09T08:26:18.203198+0000","last_clean":"2024-05-09T08:26:18.203198+0000","last_became_active":"2024-05-09T08:23:50.561306+0000","last_became_peered":"2024-05-09T08:23:50.561306+0000","last_unstale":"2024-05-09T08:26:18.203198+0000","last_undegraded":"2024-05-09T08:26:18.203198+0000","last_fullsized":"2024-05-09T08:26:18.203198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:46.897965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970871+0000","last_change":"2024-05-09T08:23:51.573765+0000","last_active":"2024-05-09T08:26:17.970871+0000","last_peered":"2024-05-09T08:26:17.970871+0000","last_clean":"2024-05-09T08:26:17.970871+0000","last_became_active":"2024-05-09T08:23:51.573134+0000","last_became_peered":"2024-05-09T08:23:51.573134+0000","last_unstale":"2024-05-09T08:26:17.970871+0000","last_undegraded":"2024-05-09T08:26:17.970871+0000","last_fullsized":"2024-05-09T08:26:17.970871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:25.334148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970989+0000","last_change":"2024-05-09T08:23:55.663480+0000","last_active":"2024-05-09T08:26:17.970989+0000","last_peered":"2024-05-09T08:26:17.970989+0000","last_clean":"2024-05-09T08:26:17.970989+0000","last_became_active":"2024-05-09T08:23:55.662823+0000","last_became_peered":"2024-05-09T08:23:55.662823+0000","last_unstale":"2024-05-09T08:26:17.970989+0000","last_undegraded":"2024-05-09T08:26:17.970989+0000","last_fullsized":"2024-05-09T08:26:17.970989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:18.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971781+0000","last_change":"2024-05-09T08:23:50.574641+0000","last_active":"2024-05-09T08:26:17.971781+0000","last_peered":"2024-05-09T08:26:17.971781+0000","last_clean":"2024-05-09T08:26:17.971781+0000","last_became_active":"2024-05-09T08:23:50.574266+0000","last_became_peered":"2024-05-09T08:23:50.574266+0000","last_unstale":"2024-05-09T08:26:17.971781+0000","last_undegraded":"2024-05-09T08:26:17.971781+0000","last_fullsized":"2024-05-09T08:26:17.971781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.575211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206374+0000","last_change":"2024-05-09T08:25:52.412012+0000","last_active":"2024-05-09T08:26:18.206374+0000","last_peered":"2024-05-09T08:26:18.206374+0000","last_clean":"2024-05-09T08:26:18.206374+0000","last_became_active":"2024-05-09T08:25:52.411479+0000","last_became_peered":"2024-05-09T08:25:52.411479+0000","last_unstale":"2024-05-09T08:26:18.206374+0000","last_undegraded":"2024-05-09T08:26:18.206374+0000","last_fullsized":"2024-05-09T08:26:18.206374+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:58.577359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971567+0000","last_change":"2024-05-09T08:23:55.853759+0000","last_active":"2024-05-09T08:26:17.971567+0000","last_peered":"2024-05-09T08:26:17.971567+0000","last_clean":"2024-05-09T08:26:17.971567+0000","last_became_active":"2024-05-09T08:23:55.853045+0000","last_became_peered":"2024-05-09T08:23:55.853045+0000","last_unstale":"2024-05-09T08:26:17.971567+0000","last_undegraded":"2024-05-09T08:26:17.971567+0000","last_fullsized":"2024-05-09T08:26:17.971567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:53.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203004+0000","last_change":"2024-05-09T08:23:50.568373+0000","last_active":"2024-05-09T08:26:18.203004+0000","last_peered":"2024-05-09T08:26:18.203004+0000","last_clean":"2024-05-09T08:26:18.203004+0000","last_became_active":"2024-05-09T08:23:50.568015+0000","last_became_peered":"2024-05-09T08:23:50.568015+0000","last_unstale":"2024-05-09T08:26:18.203004+0000","last_undegraded":"2024-05-09T08:26:18.203004+0000","last_fullsized":"2024-05-09T08:26:18.203004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:05:16.391673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086521+0000","last_change":"2024-05-09T08:23:51.576825+0000","last_active":"2024-05-09T08:26:18.086521+0000","last_peered":"2024-05-09T08:26:18.086521+0000","last_clean":"2024-05-09T08:26:18.086521+0000","last_became_active":"2024-05-09T08:23:51.573350+0000","last_became_peered":"2024-05-09T08:23:51.573350+0000","last_unstale":"2024-05-09T08:26:18.086521+0000","last_undegraded":"2024-05-09T08:26:18.086521+0000","last_fullsized":"2024-05-09T08:26:18.086521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:00.963281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.084500+0000","last_change":"2024-05-09T08:23:55.672145+0000","last_active":"2024-05-09T08:26:18.084500+0000","last_peered":"2024-05-09T08:26:18.084500+0000","last_clean":"2024-05-09T08:26:18.084500+0000","last_became_active":"2024-05-09T08:23:55.671743+0000","last_became_peered":"2024-05-09T08:23:55.671743+0000","last_unstale":"2024-05-09T08:26:18.084500+0000","last_undegraded":"2024-05-09T08:26:18.084500+0000","last_fullsized":"2024-05-09T08:26:18.084500+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:09:23.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970351+0000","last_change":"2024-05-09T08:23:50.568908+0000","last_active":"2024-05-09T08:26:17.970351+0000","last_peered":"2024-05-09T08:26:17.970351+0000","last_clean":"2024-05-09T08:26:17.970351+0000","last_became_active":"2024-05-09T08:23:50.567009+0000","last_became_peered":"2024-05-09T08:23:50.567009+0000","last_unstale":"2024-05-09T08:26:17.970351+0000","last_undegraded":"2024-05-09T08:26:17.970351+0000","last_fullsized":"2024-05-09T08:26:17.970351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:35.190156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203170+0000","last_change":"2024-05-09T08:23:51.580121+0000","last_active":"2024-05-09T08:26:18.203170+0000","last_peered":"2024-05-09T08:26:18.203170+0000","last_clean":"2024-05-09T08:26:18.203170+0000","last_became_active":"2024-05-09T08:23:51.579402+0000","last_became_peered":"2024-05-09T08:23:51.579402+0000","last_unstale":"2024-05-09T08:26:18.203170+0000","last_undegraded":"2024-05-09T08:26:18.203170+0000","last_fullsized":"2024-05-09T08:26:18.203170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:11:26.144344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971820+0000","last_change":"2024-05-09T08:23:55.664017+0000","last_active":"2024-05-09T08:26:17.971820+0000","last_peered":"2024-05-09T08:26:17.971820+0000","last_clean":"2024-05-09T08:26:17.971820+0000","last_became_active":"2024-05-09T08:23:55.663546+0000","last_became_peered":"2024-05-09T08:23:55.663546+0000","last_unstale":"2024-05-09T08:26:17.971820+0000","last_undegraded":"2024-05-09T08:26:17.971820+0000","last_fullsized":"2024-05-09T08:26:17.971820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:13:14.705971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203214+0000","last_change":"2024-05-09T08:24:51.493220+0000","last_active":"2024-05-09T08:26:18.203214+0000","last_peered":"2024-05-09T08:26:18.203214+0000","last_clean":"2024-05-09T08:26:18.203214+0000","last_became_active":"2024-05-09T08:24:51.486332+0000","last_became_peered":"2024-05-09T08:24:51.486332+0000","last_unstale":"2024-05-09T08:26:18.203214+0000","last_undegraded":"2024-05-09T08:26:18.203214+0000","last_fullsized":"2024-05-09T08:26:18.203214+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:23.694545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970561+0000","last_change":"2024-05-09T08:23:51.571414+0000","last_active":"2024-05-09T08:26:17.970561+0000","last_peered":"2024-05-09T08:26:17.970561+0000","last_clean":"2024-05-09T08:26:17.970561+0000","last_became_active":"2024-05-09T08:23:51.571085+0000","last_became_peered":"2024-05-09T08:23:51.571085+0000","last_unstale":"2024-05-09T08:26:17.970561+0000","last_undegraded":"2024-05-09T08:26:17.970561+0000","last_fullsized":"2024-05-09T08:26:17.970561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:37.513584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971413+0000","last_change":"2024-05-09T08:23:55.852785+0000","last_active":"2024-05-09T08:26:17.971413+0000","last_peered":"2024-05-09T08:26:17.971413+0000","last_clean":"2024-05-09T08:26:17.971413+0000","last_became_active":"2024-05-09T08:23:55.852252+0000","last_became_peered":"2024-05-09T08:23:55.852252+0000","last_unstale":"2024-05-09T08:26:17.971413+0000","last_undegraded":"2024-05-09T08:26:17.971413+0000","last_fullsized":"2024-05-09T08:26:17.971413+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:44.503295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086689+0000","last_change":"2024-05-09T08:23:50.564787+0000","last_active":"2024-05-09T08:26:18.086689+0000","last_peered":"2024-05-09T08:26:18.086689+0000","last_clean":"2024-05-09T08:26:18.086689+0000","last_became_active":"2024-05-09T08:23:50.564409+0000","last_became_peered":"2024-05-09T08:23:50.564409+0000","last_unstale":"2024-05-09T08:26:18.086689+0000","last_undegraded":"2024-05-09T08:26:18.086689+0000","last_fullsized":"2024-05-09T08:26:18.086689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:23.051221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187715+0000","last_change":"2024-05-09T08:23:51.576830+0000","last_active":"2024-05-09T08:26:18.187715+0000","last_peered":"2024-05-09T08:26:18.187715+0000","last_clean":"2024-05-09T08:26:18.187715+0000","last_became_active":"2024-05-09T08:23:51.576311+0000","last_became_peered":"2024-05-09T08:23:51.576311+0000","last_unstale":"2024-05-09T08:26:18.187715+0000","last_undegraded":"2024-05-09T08:26:18.187715+0000","last_fullsized":"2024-05-09T08:26:18.187715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:26:08.651061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203014+0000","last_change":"2024-05-09T08:23:55.672309+0000","last_active":"2024-05-09T08:26:18.203014+0000","last_peered":"2024-05-09T08:26:18.203014+0000","last_clean":"2024-05-09T08:26:18.203014+0000","last_became_active":"2024-05-09T08:23:55.671811+0000","last_became_peered":"2024-05-09T08:23:55.671811+0000","last_unstale":"2024-05-09T08:26:18.203014+0000","last_undegraded":"2024-05-09T08:26:18.203014+0000","last_fullsized":"2024-05-09T08:26:18.203014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:19.400949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'7","reported_seq":165,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.595790+0000","last_change":"2024-05-09T08:23:50.575317+0000","last_active":"2024-05-09T08:26:18.595790+0000","last_peered":"2024-05-09T08:26:18.595790+0000","last_clean":"2024-05-09T08:26:18.595790+0000","last_became_active":"2024-05-09T08:23:50.574995+0000","last_became_peered":"2024-05-09T08:23:50.574995+0000","last_unstale":"2024-05-09T08:26:18.595790+0000","last_undegraded":"2024-05-09T08:26:18.595790+0000","last_fullsized":"2024-05-09T08:26:18.595790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:32:54.801799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1869,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.188099+0000","last_change":"2024-05-09T08:23:51.575904+0000","last_active":"2024-05-09T08:26:18.188099+0000","last_peered":"2024-05-09T08:26:18.188099+0000","last_clean":"2024-05-09T08:26:18.188099+0000","last_became_active":"2024-05-09T08:23:51.575599+0000","last_became_peered":"2024-05-09T08:23:51.575599+0000","last_unstale":"2024-05-09T08:26:18.188099+0000","last_undegraded":"2024-05-09T08:26:18.188099+0000","last_fullsized":"2024-05-09T08:26:18.188099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:20:13.087096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205215+0000","last_change":"2024-05-09T08:23:55.680683+0000","last_active":"2024-05-09T08:26:18.205215+0000","last_peered":"2024-05-09T08:26:18.205215+0000","last_clean":"2024-05-09T08:26:18.205215+0000","last_became_active":"2024-05-09T08:23:55.680232+0000","last_became_peered":"2024-05-09T08:23:55.680232+0000","last_unstale":"2024-05-09T08:26:18.205215+0000","last_undegraded":"2024-05-09T08:26:18.205215+0000","last_fullsized":"2024-05-09T08:26:18.205215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:33.583552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.200717+0000","last_change":"2024-05-09T08:23:50.575802+0000","last_active":"2024-05-09T08:26:18.200717+0000","last_peered":"2024-05-09T08:26:18.200717+0000","last_clean":"2024-05-09T08:26:18.200717+0000","last_became_active":"2024-05-09T08:23:50.575416+0000","last_became_peered":"2024-05-09T08:23:50.575416+0000","last_unstale":"2024-05-09T08:26:18.200717+0000","last_undegraded":"2024-05-09T08:26:18.200717+0000","last_fullsized":"2024-05-09T08:26:18.200717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:24:04.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187831+0000","last_change":"2024-05-09T08:23:51.578189+0000","last_active":"2024-05-09T08:26:18.187831+0000","last_peered":"2024-05-09T08:26:18.187831+0000","last_clean":"2024-05-09T08:26:18.187831+0000","last_became_active":"2024-05-09T08:23:51.577500+0000","last_became_peered":"2024-05-09T08:23:51.577500+0000","last_unstale":"2024-05-09T08:26:18.187831+0000","last_undegraded":"2024-05-09T08:26:18.187831+0000","last_fullsized":"2024-05-09T08:26:18.187831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:26:24.718791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086471+0000","last_change":"2024-05-09T08:24:51.487659+0000","last_active":"2024-05-09T08:26:18.086471+0000","last_peered":"2024-05-09T08:26:18.086471+0000","last_clean":"2024-05-09T08:26:18.086471+0000","last_became_active":"2024-05-09T08:24:51.487212+0000","last_became_peered":"2024-05-09T08:24:51.487212+0000","last_unstale":"2024-05-09T08:26:18.086471+0000","last_undegraded":"2024-05-09T08:26:18.086471+0000","last_fullsized":"2024-05-09T08:26:18.086471+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:29:57.077744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205019+0000","last_change":"2024-05-09T08:23:50.578438+0000","last_active":"2024-05-09T08:26:18.205019+0000","last_peered":"2024-05-09T08:26:18.205019+0000","last_clean":"2024-05-09T08:26:18.205019+0000","last_became_active":"2024-05-09T08:23:50.578039+0000","last_became_peered":"2024-05-09T08:23:50.578039+0000","last_unstale":"2024-05-09T08:26:18.205019+0000","last_undegraded":"2024-05-09T08:26:18.205019+0000","last_fullsized":"2024-05-09T08:26:18.205019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:59.420193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187887+0000","last_change":"2024-05-09T08:23:51.568037+0000","last_active":"2024-05-09T08:26:18.187887+0000","last_peered":"2024-05-09T08:26:18.187887+0000","last_clean":"2024-05-09T08:26:18.187887+0000","last_became_active":"2024-05-09T08:23:51.567706+0000","last_became_peered":"2024-05-09T08:23:51.567706+0000","last_unstale":"2024-05-09T08:26:18.187887+0000","last_undegraded":"2024-05-09T08:26:18.187887+0000","last_fullsized":"2024-05-09T08:26:18.187887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:15:19.623805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203263+0000","last_change":"2024-05-09T08:23:51.579885+0000","last_active":"2024-05-09T08:26:18.203263+0000","last_peered":"2024-05-09T08:26:18.203263+0000","last_clean":"2024-05-09T08:26:18.203263+0000","last_became_active":"2024-05-09T08:23:51.578953+0000","last_became_peered":"2024-05-09T08:23:51.578953+0000","last_unstale":"2024-05-09T08:26:18.203263+0000","last_undegraded":"2024-05-09T08:26:18.203263+0000","last_fullsized":"2024-05-09T08:26:18.203263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:57:23.793119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971252+0000","last_change":"2024-05-09T08:23:51.583423+0000","last_active":"2024-05-09T08:26:17.971252+0000","last_peered":"2024-05-09T08:26:17.971252+0000","last_clean":"2024-05-09T08:26:17.971252+0000","last_became_active":"2024-05-09T08:23:51.583092+0000","last_became_peered":"2024-05-09T08:23:51.583092+0000","last_unstale":"2024-05-09T08:26:17.971252+0000","last_undegraded":"2024-05-09T08:26:17.971252+0000","last_fullsized":"2024-05-09T08:26:17.971252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:44:47.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971056+0000","last_change":"2024-05-09T08:23:51.582116+0000","last_active":"2024-05-09T08:26:17.971056+0000","last_peered":"2024-05-09T08:26:17.971056+0000","last_clean":"2024-05-09T08:26:17.971056+0000","last_became_active":"2024-05-09T08:23:51.581715+0000","last_became_peered":"2024-05-09T08:23:51.581715+0000","last_unstale":"2024-05-09T08:26:17.971056+0000","last_undegraded":"2024-05-09T08:26:17.971056+0000","last_fullsized":"2024-05-09T08:26:17.971056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:17.528758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205481+0000","last_change":"2024-05-09T08:23:51.575959+0000","last_active":"2024-05-09T08:26:18.205481+0000","last_peered":"2024-05-09T08:26:18.205481+0000","last_clean":"2024-05-09T08:26:18.205481+0000","last_became_active":"2024-05-09T08:23:51.575592+0000","last_became_peered":"2024-05-09T08:23:51.575592+0000","last_unstale":"2024-05-09T08:26:18.205481+0000","last_undegraded":"2024-05-09T08:26:18.205481+0000","last_fullsized":"2024-05-09T08:26:18.205481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:26.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205213+0000","last_change":"2024-05-09T08:23:51.570756+0000","last_active":"2024-05-09T08:26:18.205213+0000","last_peered":"2024-05-09T08:26:18.205213+0000","last_clean":"2024-05-09T08:26:18.205213+0000","last_became_active":"2024-05-09T08:23:51.570412+0000","last_became_peered":"2024-05-09T08:23:51.570412+0000","last_unstale":"2024-05-09T08:26:18.205213+0000","last_undegraded":"2024-05-09T08:26:18.205213+0000","last_fullsized":"2024-05-09T08:26:18.205213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:11:53.959559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203772+0000","last_change":"2024-05-09T08:23:51.572612+0000","last_active":"2024-05-09T08:26:18.203772+0000","last_peered":"2024-05-09T08:26:18.203772+0000","last_clean":"2024-05-09T08:26:18.203772+0000","last_became_active":"2024-05-09T08:23:51.572227+0000","last_became_peered":"2024-05-09T08:23:51.572227+0000","last_unstale":"2024-05-09T08:26:18.203772+0000","last_undegraded":"2024-05-09T08:26:18.203772+0000","last_fullsized":"2024-05-09T08:26:18.203772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:18.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971483+0000","last_change":"2024-05-09T08:23:51.573021+0000","last_active":"2024-05-09T08:26:17.971483+0000","last_peered":"2024-05-09T08:26:17.971483+0000","last_clean":"2024-05-09T08:26:17.971483+0000","last_became_active":"2024-05-09T08:23:51.572716+0000","last_became_peered":"2024-05-09T08:23:51.572716+0000","last_unstale":"2024-05-09T08:26:17.971483+0000","last_undegraded":"2024-05-09T08:26:17.971483+0000","last_fullsized":"2024-05-09T08:26:17.971483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:31.494329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971007+0000","last_change":"2024-05-09T08:23:51.572687+0000","last_active":"2024-05-09T08:26:17.971007+0000","last_peered":"2024-05-09T08:26:17.971007+0000","last_clean":"2024-05-09T08:26:17.971007+0000","last_became_active":"2024-05-09T08:23:51.572339+0000","last_became_peered":"2024-05-09T08:23:51.572339+0000","last_unstale":"2024-05-09T08:26:17.971007+0000","last_undegraded":"2024-05-09T08:26:17.971007+0000","last_fullsized":"2024-05-09T08:26:17.971007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:24.012360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092190+0000","last_change":"2024-05-09T08:23:51.569404+0000","last_active":"2024-05-09T08:26:18.092190+0000","last_peered":"2024-05-09T08:26:18.092190+0000","last_clean":"2024-05-09T08:26:18.092190+0000","last_became_active":"2024-05-09T08:23:51.569044+0000","last_became_peered":"2024-05-09T08:23:51.569044+0000","last_unstale":"2024-05-09T08:26:18.092190+0000","last_undegraded":"2024-05-09T08:26:18.092190+0000","last_fullsized":"2024-05-09T08:26:18.092190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:26:01.343606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970502+0000","last_change":"2024-05-09T08:23:51.577483+0000","last_active":"2024-05-09T08:26:17.970502+0000","last_peered":"2024-05-09T08:26:17.970502+0000","last_clean":"2024-05-09T08:26:17.970502+0000","last_became_active":"2024-05-09T08:23:51.577078+0000","last_became_peered":"2024-05-09T08:23:51.577078+0000","last_unstale":"2024-05-09T08:26:17.970502+0000","last_undegraded":"2024-05-09T08:26:17.970502+0000","last_fullsized":"2024-05-09T08:26:17.970502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:29.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203171+0000","last_change":"2024-05-09T08:23:51.579736+0000","last_active":"2024-05-09T08:26:18.203171+0000","last_peered":"2024-05-09T08:26:18.203171+0000","last_clean":"2024-05-09T08:26:18.203171+0000","last_became_active":"2024-05-09T08:23:51.579409+0000","last_became_peered":"2024-05-09T08:23:51.579409+0000","last_unstale":"2024-05-09T08:26:18.203171+0000","last_undegraded":"2024-05-09T08:26:18.203171+0000","last_fullsized":"2024-05-09T08:26:18.203171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:05.697642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204632+0000","last_change":"2024-05-09T08:23:51.569461+0000","last_active":"2024-05-09T08:26:18.204632+0000","last_peered":"2024-05-09T08:26:18.204632+0000","last_clean":"2024-05-09T08:26:18.204632+0000","last_became_active":"2024-05-09T08:23:51.569085+0000","last_became_peered":"2024-05-09T08:23:51.569085+0000","last_unstale":"2024-05-09T08:26:18.204632+0000","last_undegraded":"2024-05-09T08:26:18.204632+0000","last_fullsized":"2024-05-09T08:26:18.204632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:54.554949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203414+0000","last_change":"2024-05-09T08:23:51.570593+0000","last_active":"2024-05-09T08:26:18.203414+0000","last_peered":"2024-05-09T08:26:18.203414+0000","last_clean":"2024-05-09T08:26:18.203414+0000","last_became_active":"2024-05-09T08:23:51.570195+0000","last_became_peered":"2024-05-09T08:23:51.570195+0000","last_unstale":"2024-05-09T08:26:18.203414+0000","last_undegraded":"2024-05-09T08:26:18.203414+0000","last_fullsized":"2024-05-09T08:26:18.203414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:18.376268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108273,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604101,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1324,"num_read_kb":780,"num_write":376,"num_write_kb":1473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":64162,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":667648,"data_stored":1812303,"data_compressed":89522,"data_compressed_allocated":118784,"data_compressed_original":1626112,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":531150,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32344,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712808,"statfs":{"total":95995035648,"available":95961915392,"internally_reserved":0,"allocated":1191936,"data_stored":1185786,"data_compressed":67266,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":923575,"data_compressed":48720,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":923259,"data_compressed":48720,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1101824,"data_stored":1117422,"data_compressed":62684,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526254,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1118208,"data_stored":598024,"data_compressed":31175,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1064960,"data_stored":528018,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":527309,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1097728,"data_stored":1050387,"data_compressed":54876,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":601431,"data_compressed":39988,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1142784,"data_stored":1254416,"data_compressed":71821,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":72883,"data_compressed":1601,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":72194,"data_compressed":10414,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134811,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394022,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":204718,"data_compressed":7757,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:26:20.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:26:20.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:20 smithi165 ceph-mon[34540]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:26:20.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:20 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1891663908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:20.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:20 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3880600577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:20.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:20 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1316673981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:20.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:20 smithi184 ceph-mon[34509]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:26:20.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:20 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1891663908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:20.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:20 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3880600577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:20.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:20 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1316673981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:20.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.621+0000 7f6d28627640 1 -- 172.21.15.28:0/2565829541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d201075c0 msgr2=0x7f6d201099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.621+0000 7f6d28627640 1 --2- 172.21.15.28:0/2565829541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d201075c0 0x7f6d201099b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009ca0 tx=0x7f6d1802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.622+0000 7f6d28627640 1 -- 172.21.15.28:0/2565829541 shutdown_connections 2024-05-09T08:26:20.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.622+0000 7f6d28627640 1 --2- 172.21.15.28:0/2565829541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d201075c0 0x7f6d201099b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.622+0000 7f6d28627640 1 --2- 172.21.15.28:0/2565829541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d20104c90 0x7f6d20107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.622+0000 7f6d28627640 1 --2- 172.21.15.28:0/2565829541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d20102360 0x7f6d20104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.622+0000 7f6d28627640 1 -- 172.21.15.28:0/2565829541 >> 172.21.15.28:0/2565829541 conn(0x7f6d200fbd10 msgr2=0x7f6d200fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.622+0000 7f6d28627640 1 -- 172.21.15.28:0/2565829541 shutdown_connections 2024-05-09T08:26:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.622+0000 7f6d28627640 1 -- 172.21.15.28:0/2565829541 wait complete. 2024-05-09T08:26:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.622+0000 7f6d28627640 1 Processor -- start 2024-05-09T08:26:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d28627640 1 -- start start 2024-05-09T08:26:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d28627640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d20102360 0x7f6d20197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d28627640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d20104c90 0x7f6d201980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d28627640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d201075c0 0x7f6d2019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d28627640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2010d520 con 0x7f6d20104c90 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d28627640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2010d3a0 con 0x7f6d20102360 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d28627640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2010d6a0 con 0x7f6d201075c0 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d25bc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d20104c90 0x7f6d201980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d263c3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d20102360 0x7f6d20197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d25bc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d20104c90 0x7f6d201980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38354/0 (socket says 172.21.15.28:38354) 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d25bc2640 1 -- 172.21.15.28:0/3679027484 learned_addr learned my addr 172.21.15.28:0/3679027484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d26bc4640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d201075c0 0x7f6d2019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d25bc2640 1 -- 172.21.15.28:0/3679027484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d201075c0 msgr2=0x7f6d2019f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d25bc2640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d201075c0 0x7f6d2019f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d25bc2640 1 -- 172.21.15.28:0/3679027484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d20102360 msgr2=0x7f6d20197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d25bc2640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d20102360 0x7f6d20197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d25bc2640 1 -- 172.21.15.28:0/3679027484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2019f780 con 0x7f6d20104c90 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d263c3640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d20102360 0x7f6d20197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.623+0000 7f6d25bc2640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d20104c90 0x7f6d201980a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e9f0 tx=0x7f6d1000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.624+0000 7f6d177fe640 1 -- 172.21.15.28:0/3679027484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1000ccc0 con 0x7f6d20104c90 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.624+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2019fa70 con 0x7f6d20104c90 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.624+0000 7f6d177fe640 1 -- 172.21.15.28:0/3679027484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d10005500 con 0x7f6d20104c90 2024-05-09T08:26:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.624+0000 7f6d177fe640 1 -- 172.21.15.28:0/3679027484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10010690 con 0x7f6d20104c90 2024-05-09T08:26:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.624+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2019ffc0 con 0x7f6d20104c90 2024-05-09T08:26:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.625+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d200fd800 con 0x7f6d20104c90 2024-05-09T08:26:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.626+0000 7f6d177fe640 1 -- 172.21.15.28:0/3679027484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f6d100046c0 con 0x7f6d20104c90 2024-05-09T08:26:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.627+0000 7f6d177fe640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f6d0c0a2d00 0x7f6d0c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.627+0000 7f6d263c3640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f6d0c0a2d00 0x7f6d0c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.627+0000 7f6d177fe640 1 -- 172.21.15.28:0/3679027484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f6d100cfc10 con 0x7f6d20104c90 2024-05-09T08:26:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.627+0000 7f6d263c3640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f6d0c0a2d00 0x7f6d0c0a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009fc0 tx=0x7f6d1c005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.636+0000 7f6d177fe640 1 -- 172.21.15.28:0/3679027484 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d1009c400 con 0x7f6d20104c90 2024-05-09T08:26:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.826+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 --> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6d20061c70 con 0x7f6d0c0a2d00 2024-05-09T08:26:20.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.836+0000 7f6d177fe640 1 -- 172.21.15.28:0/3679027484 <== mgr.15528 v2:172.21.15.28:6800/160415970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339763 (secure 0 0 0) 0x7f6d20061c70 con 0x7f6d0c0a2d00 2024-05-09T08:26:20.841 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-09T08:26:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f6d0c0a2d00 msgr2=0x7f6d0c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f6d0c0a2d00 0x7f6d0c0a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009fc0 tx=0x7f6d1c005f90 comp rx=0 tx=0).stop 2024-05-09T08:26:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d20104c90 msgr2=0x7f6d201980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d20104c90 0x7f6d201980a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e9f0 tx=0x7f6d1000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 shutdown_connections 2024-05-09T08:26:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f6d0c0a2d00 0x7f6d0c0a51c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d201075c0 0x7f6d2019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d20104c90 0x7f6d201980a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 --2- 172.21.15.28:0/3679027484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d20102360 0x7f6d20197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.840+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 >> 172.21.15.28:0/3679027484 conn(0x7f6d200fbd10 msgr2=0x7f6d20102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.841+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 shutdown_connections 2024-05-09T08:26:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:20.841+0000 7f6d28627640 1 -- 172.21.15.28:0/3679027484 wait complete. 2024-05-09T08:26:20.853 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-05-09T08:26:19.984209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1194703,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1460,"num_read_kb":899,"num_write":559,"num_write_kb":4413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1245015,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":944,"ondisk_log_size":944,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378620,"kb_used_data":12792,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563204,"statfs":{"total":1151940427776,"available":1151552720896,"internally_reserved":0,"allocated":13099008,"data_stored":9767031,"data_compressed":543546,"data_compressed_allocated":733184,"data_compressed_original":4247552,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.983070"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203766+0000","last_change":"2024-05-09T08:23:51.579564+0000","last_active":"2024-05-09T08:26:18.203766+0000","last_peered":"2024-05-09T08:26:18.203766+0000","last_clean":"2024-05-09T08:26:18.203766+0000","last_became_active":"2024-05-09T08:23:51.578884+0000","last_became_peered":"2024-05-09T08:23:51.578884+0000","last_unstale":"2024-05-09T08:26:18.203766+0000","last_undegraded":"2024-05-09T08:26:18.203766+0000","last_fullsized":"2024-05-09T08:26:18.203766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:40.473483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206222+0000","last_change":"2024-05-09T08:23:51.574187+0000","last_active":"2024-05-09T08:26:18.206222+0000","last_peered":"2024-05-09T08:26:18.206222+0000","last_clean":"2024-05-09T08:26:18.206222+0000","last_became_active":"2024-05-09T08:23:51.573837+0000","last_became_peered":"2024-05-09T08:23:51.573837+0000","last_unstale":"2024-05-09T08:26:18.206222+0000","last_undegraded":"2024-05-09T08:26:18.206222+0000","last_fullsized":"2024-05-09T08:26:18.206222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:31.448038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205938+0000","last_change":"2024-05-09T08:23:51.577224+0000","last_active":"2024-05-09T08:26:18.205938+0000","last_peered":"2024-05-09T08:26:18.205938+0000","last_clean":"2024-05-09T08:26:18.205938+0000","last_became_active":"2024-05-09T08:23:51.576822+0000","last_became_peered":"2024-05-09T08:23:51.576822+0000","last_unstale":"2024-05-09T08:26:18.205938+0000","last_undegraded":"2024-05-09T08:26:18.205938+0000","last_fullsized":"2024-05-09T08:26:18.205938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:04.902530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971321+0000","last_change":"2024-05-09T08:23:51.577172+0000","last_active":"2024-05-09T08:26:17.971321+0000","last_peered":"2024-05-09T08:26:17.971321+0000","last_clean":"2024-05-09T08:26:17.971321+0000","last_became_active":"2024-05-09T08:23:51.576861+0000","last_became_peered":"2024-05-09T08:23:51.576861+0000","last_unstale":"2024-05-09T08:26:17.971321+0000","last_undegraded":"2024-05-09T08:26:17.971321+0000","last_fullsized":"2024-05-09T08:26:17.971321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:10.925739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086698+0000","last_change":"2024-05-09T08:23:51.579408+0000","last_active":"2024-05-09T08:26:18.086698+0000","last_peered":"2024-05-09T08:26:18.086698+0000","last_clean":"2024-05-09T08:26:18.086698+0000","last_became_active":"2024-05-09T08:23:51.578229+0000","last_became_peered":"2024-05-09T08:23:51.578229+0000","last_unstale":"2024-05-09T08:26:18.086698+0000","last_undegraded":"2024-05-09T08:26:18.086698+0000","last_fullsized":"2024-05-09T08:26:18.086698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:47:27.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.095787+0000","last_change":"2024-05-09T08:23:51.576924+0000","last_active":"2024-05-09T08:26:18.095787+0000","last_peered":"2024-05-09T08:26:18.095787+0000","last_clean":"2024-05-09T08:26:18.095787+0000","last_became_active":"2024-05-09T08:23:51.576601+0000","last_became_peered":"2024-05-09T08:23:51.576601+0000","last_unstale":"2024-05-09T08:26:18.095787+0000","last_undegraded":"2024-05-09T08:26:18.095787+0000","last_fullsized":"2024-05-09T08:26:18.095787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:07:16.910271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187456+0000","last_change":"2024-05-09T08:25:52.714094+0000","last_active":"2024-05-09T08:26:18.187456+0000","last_peered":"2024-05-09T08:26:18.187456+0000","last_clean":"2024-05-09T08:26:18.187456+0000","last_became_active":"2024-05-09T08:25:52.713743+0000","last_became_peered":"2024-05-09T08:25:52.713743+0000","last_unstale":"2024-05-09T08:26:18.187456+0000","last_undegraded":"2024-05-09T08:26:18.187456+0000","last_fullsized":"2024-05-09T08:26:18.187456+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:59.810244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970350+0000","last_change":"2024-05-09T08:23:51.574001+0000","last_active":"2024-05-09T08:26:17.970350+0000","last_peered":"2024-05-09T08:26:17.970350+0000","last_clean":"2024-05-09T08:26:17.970350+0000","last_became_active":"2024-05-09T08:23:51.573632+0000","last_became_peered":"2024-05-09T08:23:51.573632+0000","last_unstale":"2024-05-09T08:26:17.970350+0000","last_undegraded":"2024-05-09T08:26:17.970350+0000","last_fullsized":"2024-05-09T08:26:17.970350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:55:16.418951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203492+0000","last_change":"2024-05-09T08:23:51.567635+0000","last_active":"2024-05-09T08:26:18.203492+0000","last_peered":"2024-05-09T08:26:18.203492+0000","last_clean":"2024-05-09T08:26:18.203492+0000","last_became_active":"2024-05-09T08:23:51.567283+0000","last_became_peered":"2024-05-09T08:23:51.567283+0000","last_unstale":"2024-05-09T08:26:18.203492+0000","last_undegraded":"2024-05-09T08:26:18.203492+0000","last_fullsized":"2024-05-09T08:26:18.203492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:42.346856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.096277+0000","last_change":"2024-05-09T08:23:51.576090+0000","last_active":"2024-05-09T08:26:18.096277+0000","last_peered":"2024-05-09T08:26:18.096277+0000","last_clean":"2024-05-09T08:26:18.096277+0000","last_became_active":"2024-05-09T08:23:51.575768+0000","last_became_peered":"2024-05-09T08:23:51.575768+0000","last_unstale":"2024-05-09T08:26:18.096277+0000","last_undegraded":"2024-05-09T08:26:18.096277+0000","last_fullsized":"2024-05-09T08:26:18.096277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:00:50.016924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.188104+0000","last_change":"2024-05-09T08:23:51.578957+0000","last_active":"2024-05-09T08:26:18.188104+0000","last_peered":"2024-05-09T08:26:18.188104+0000","last_clean":"2024-05-09T08:26:18.188104+0000","last_became_active":"2024-05-09T08:23:51.578210+0000","last_became_peered":"2024-05-09T08:23:51.578210+0000","last_unstale":"2024-05-09T08:26:18.188104+0000","last_undegraded":"2024-05-09T08:26:18.188104+0000","last_fullsized":"2024-05-09T08:26:18.188104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:28:28.953434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971480+0000","last_change":"2024-05-09T08:23:51.575389+0000","last_active":"2024-05-09T08:26:17.971480+0000","last_peered":"2024-05-09T08:26:17.971480+0000","last_clean":"2024-05-09T08:26:17.971480+0000","last_became_active":"2024-05-09T08:23:51.575071+0000","last_became_peered":"2024-05-09T08:23:51.575071+0000","last_unstale":"2024-05-09T08:26:17.971480+0000","last_undegraded":"2024-05-09T08:26:17.971480+0000","last_fullsized":"2024-05-09T08:26:17.971480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:34.844611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204819+0000","last_change":"2024-05-09T08:23:51.572574+0000","last_active":"2024-05-09T08:26:18.204819+0000","last_peered":"2024-05-09T08:26:18.204819+0000","last_clean":"2024-05-09T08:26:18.204819+0000","last_became_active":"2024-05-09T08:23:51.572243+0000","last_became_peered":"2024-05-09T08:23:51.572243+0000","last_unstale":"2024-05-09T08:26:18.204819+0000","last_undegraded":"2024-05-09T08:26:18.204819+0000","last_fullsized":"2024-05-09T08:26:18.204819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:52.534365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205049+0000","last_change":"2024-05-09T08:23:51.574963+0000","last_active":"2024-05-09T08:26:18.205049+0000","last_peered":"2024-05-09T08:26:18.205049+0000","last_clean":"2024-05-09T08:26:18.205049+0000","last_became_active":"2024-05-09T08:23:51.574597+0000","last_became_peered":"2024-05-09T08:23:51.574597+0000","last_unstale":"2024-05-09T08:26:18.205049+0000","last_undegraded":"2024-05-09T08:26:18.205049+0000","last_fullsized":"2024-05-09T08:26:18.205049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:33.348828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205526+0000","last_change":"2024-05-09T08:23:51.577777+0000","last_active":"2024-05-09T08:26:18.205526+0000","last_peered":"2024-05-09T08:26:18.205526+0000","last_clean":"2024-05-09T08:26:18.205526+0000","last_became_active":"2024-05-09T08:23:51.577073+0000","last_became_peered":"2024-05-09T08:23:51.577073+0000","last_unstale":"2024-05-09T08:26:18.205526+0000","last_undegraded":"2024-05-09T08:26:18.205526+0000","last_fullsized":"2024-05-09T08:26:18.205526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:26.249488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203039+0000","last_change":"2024-05-09T08:23:51.574476+0000","last_active":"2024-05-09T08:26:18.203039+0000","last_peered":"2024-05-09T08:26:18.203039+0000","last_clean":"2024-05-09T08:26:18.203039+0000","last_became_active":"2024-05-09T08:23:51.572748+0000","last_became_peered":"2024-05-09T08:23:51.572748+0000","last_unstale":"2024-05-09T08:26:18.203039+0000","last_undegraded":"2024-05-09T08:26:18.203039+0000","last_fullsized":"2024-05-09T08:26:18.203039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:43.515419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203660+0000","last_change":"2024-05-09T08:23:51.580568+0000","last_active":"2024-05-09T08:26:18.203660+0000","last_peered":"2024-05-09T08:26:18.203660+0000","last_clean":"2024-05-09T08:26:18.203660+0000","last_became_active":"2024-05-09T08:23:51.579626+0000","last_became_peered":"2024-05-09T08:23:51.579626+0000","last_unstale":"2024-05-09T08:26:18.203660+0000","last_undegraded":"2024-05-09T08:26:18.203660+0000","last_fullsized":"2024-05-09T08:26:18.203660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:28.756079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083992+0000","last_change":"2024-05-09T08:23:51.574363+0000","last_active":"2024-05-09T08:26:18.083992+0000","last_peered":"2024-05-09T08:26:18.083992+0000","last_clean":"2024-05-09T08:26:18.083992+0000","last_became_active":"2024-05-09T08:23:51.573858+0000","last_became_peered":"2024-05-09T08:23:51.573858+0000","last_unstale":"2024-05-09T08:26:18.083992+0000","last_undegraded":"2024-05-09T08:26:18.083992+0000","last_fullsized":"2024-05-09T08:26:18.083992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:30.752939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971361+0000","last_change":"2024-05-09T08:23:51.579361+0000","last_active":"2024-05-09T08:26:17.971361+0000","last_peered":"2024-05-09T08:26:17.971361+0000","last_clean":"2024-05-09T08:26:17.971361+0000","last_became_active":"2024-05-09T08:23:51.579025+0000","last_became_peered":"2024-05-09T08:23:51.579025+0000","last_unstale":"2024-05-09T08:26:17.971361+0000","last_undegraded":"2024-05-09T08:26:17.971361+0000","last_fullsized":"2024-05-09T08:26:17.971361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:02.273999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971573+0000","last_change":"2024-05-09T08:23:55.670107+0000","last_active":"2024-05-09T08:26:17.971573+0000","last_peered":"2024-05-09T08:26:17.971573+0000","last_clean":"2024-05-09T08:26:17.971573+0000","last_became_active":"2024-05-09T08:23:55.669621+0000","last_became_peered":"2024-05-09T08:23:55.669621+0000","last_unstale":"2024-05-09T08:26:17.971573+0000","last_undegraded":"2024-05-09T08:26:17.971573+0000","last_fullsized":"2024-05-09T08:26:17.971573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:00.547137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'14","reported_seq":189,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.253755+0000","last_change":"2024-05-09T08:24:51.498795+0000","last_active":"2024-05-09T08:26:18.253755+0000","last_peered":"2024-05-09T08:26:18.253755+0000","last_clean":"2024-05-09T08:26:18.253755+0000","last_became_active":"2024-05-09T08:24:51.487098+0000","last_became_peered":"2024-05-09T08:24:51.487098+0000","last_unstale":"2024-05-09T08:26:18.253755+0000","last_undegraded":"2024-05-09T08:26:18.253755+0000","last_fullsized":"2024-05-09T08:26:18.253755+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:10.646049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083803+0000","last_change":"2024-05-09T08:25:52.410940+0000","last_active":"2024-05-09T08:26:18.083803+0000","last_peered":"2024-05-09T08:26:18.083803+0000","last_clean":"2024-05-09T08:26:18.083803+0000","last_became_active":"2024-05-09T08:25:52.410514+0000","last_became_peered":"2024-05-09T08:25:52.410514+0000","last_unstale":"2024-05-09T08:26:18.083803+0000","last_undegraded":"2024-05-09T08:26:18.083803+0000","last_fullsized":"2024-05-09T08:26:18.083803+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:51.831125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971160+0000","last_change":"2024-05-09T08:23:55.670194+0000","last_active":"2024-05-09T08:26:17.971160+0000","last_peered":"2024-05-09T08:26:17.971160+0000","last_clean":"2024-05-09T08:26:17.971160+0000","last_became_active":"2024-05-09T08:23:55.669762+0000","last_became_peered":"2024-05-09T08:23:55.669762+0000","last_unstale":"2024-05-09T08:26:17.971160+0000","last_undegraded":"2024-05-09T08:26:17.971160+0000","last_fullsized":"2024-05-09T08:26:17.971160+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:22.879635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970723+0000","last_change":"2024-05-09T08:23:50.574421+0000","last_active":"2024-05-09T08:26:17.970723+0000","last_peered":"2024-05-09T08:26:17.970723+0000","last_clean":"2024-05-09T08:26:17.970723+0000","last_became_active":"2024-05-09T08:23:50.574070+0000","last_became_peered":"2024-05-09T08:23:50.574070+0000","last_unstale":"2024-05-09T08:26:17.970723+0000","last_undegraded":"2024-05-09T08:26:17.970723+0000","last_fullsized":"2024-05-09T08:26:17.970723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:22.996034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204912+0000","last_change":"2024-05-09T08:23:51.583275+0000","last_active":"2024-05-09T08:26:18.204912+0000","last_peered":"2024-05-09T08:26:18.204912+0000","last_clean":"2024-05-09T08:26:18.204912+0000","last_became_active":"2024-05-09T08:23:51.582847+0000","last_became_peered":"2024-05-09T08:23:51.582847+0000","last_unstale":"2024-05-09T08:26:18.204912+0000","last_undegraded":"2024-05-09T08:26:18.204912+0000","last_fullsized":"2024-05-09T08:26:18.204912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:03:03.512852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971336+0000","last_change":"2024-05-09T08:23:55.672256+0000","last_active":"2024-05-09T08:26:17.971336+0000","last_peered":"2024-05-09T08:26:17.971336+0000","last_clean":"2024-05-09T08:26:17.971336+0000","last_became_active":"2024-05-09T08:23:55.671855+0000","last_became_peered":"2024-05-09T08:23:55.671855+0000","last_unstale":"2024-05-09T08:26:17.971336+0000","last_undegraded":"2024-05-09T08:26:17.971336+0000","last_fullsized":"2024-05-09T08:26:17.971336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:45:12.199359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"94'13","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.087175+0000","last_change":"2024-05-09T08:23:50.566686+0000","last_active":"2024-05-09T08:26:18.087175+0000","last_peered":"2024-05-09T08:26:18.087175+0000","last_clean":"2024-05-09T08:26:18.087175+0000","last_became_active":"2024-05-09T08:23:50.565910+0000","last_became_peered":"2024-05-09T08:23:50.565910+0000","last_unstale":"2024-05-09T08:26:18.087175+0000","last_undegraded":"2024-05-09T08:26:18.087175+0000","last_fullsized":"2024-05-09T08:26:18.087175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:02.211551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203319+0000","last_change":"2024-05-09T08:23:51.570972+0000","last_active":"2024-05-09T08:26:18.203319+0000","last_peered":"2024-05-09T08:26:18.203319+0000","last_clean":"2024-05-09T08:26:18.203319+0000","last_became_active":"2024-05-09T08:23:51.570638+0000","last_became_peered":"2024-05-09T08:23:51.570638+0000","last_unstale":"2024-05-09T08:26:18.203319+0000","last_undegraded":"2024-05-09T08:26:18.203319+0000","last_fullsized":"2024-05-09T08:26:18.203319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:38.738820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971213+0000","last_change":"2024-05-09T08:23:55.671929+0000","last_active":"2024-05-09T08:26:17.971213+0000","last_peered":"2024-05-09T08:26:17.971213+0000","last_clean":"2024-05-09T08:26:17.971213+0000","last_became_active":"2024-05-09T08:23:55.671483+0000","last_became_peered":"2024-05-09T08:23:55.671483+0000","last_unstale":"2024-05-09T08:26:17.971213+0000","last_undegraded":"2024-05-09T08:26:17.971213+0000","last_fullsized":"2024-05-09T08:26:17.971213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:21.527266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086737+0000","last_change":"2024-05-09T08:23:50.565644+0000","last_active":"2024-05-09T08:26:18.086737+0000","last_peered":"2024-05-09T08:26:18.086737+0000","last_clean":"2024-05-09T08:26:18.086737+0000","last_became_active":"2024-05-09T08:23:50.565265+0000","last_became_peered":"2024-05-09T08:23:50.565265+0000","last_unstale":"2024-05-09T08:26:18.086737+0000","last_undegraded":"2024-05-09T08:26:18.086737+0000","last_fullsized":"2024-05-09T08:26:18.086737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:51:57.972535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092603+0000","last_change":"2024-05-09T08:23:51.571732+0000","last_active":"2024-05-09T08:26:18.092603+0000","last_peered":"2024-05-09T08:26:18.092603+0000","last_clean":"2024-05-09T08:26:18.092603+0000","last_became_active":"2024-05-09T08:23:51.570783+0000","last_became_peered":"2024-05-09T08:23:51.570783+0000","last_unstale":"2024-05-09T08:26:18.092603+0000","last_undegraded":"2024-05-09T08:26:18.092603+0000","last_fullsized":"2024-05-09T08:26:18.092603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:18.515905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971228+0000","last_change":"2024-05-09T08:23:55.666727+0000","last_active":"2024-05-09T08:26:17.971228+0000","last_peered":"2024-05-09T08:26:17.971228+0000","last_clean":"2024-05-09T08:26:17.971228+0000","last_became_active":"2024-05-09T08:23:55.666284+0000","last_became_peered":"2024-05-09T08:23:55.666284+0000","last_unstale":"2024-05-09T08:26:17.971228+0000","last_undegraded":"2024-05-09T08:26:17.971228+0000","last_fullsized":"2024-05-09T08:26:17.971228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:05.355404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.595864+0000","last_change":"2024-05-09T08:23:50.573308+0000","last_active":"2024-05-09T08:26:18.595864+0000","last_peered":"2024-05-09T08:26:18.595864+0000","last_clean":"2024-05-09T08:26:18.595864+0000","last_became_active":"2024-05-09T08:23:50.572946+0000","last_became_peered":"2024-05-09T08:23:50.572946+0000","last_unstale":"2024-05-09T08:26:18.595864+0000","last_undegraded":"2024-05-09T08:26:18.595864+0000","last_fullsized":"2024-05-09T08:26:18.595864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:44:35.816620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086752+0000","last_change":"2024-05-09T08:23:51.571748+0000","last_active":"2024-05-09T08:26:18.086752+0000","last_peered":"2024-05-09T08:26:18.086752+0000","last_clean":"2024-05-09T08:26:18.086752+0000","last_became_active":"2024-05-09T08:23:51.571388+0000","last_became_peered":"2024-05-09T08:23:51.571388+0000","last_unstale":"2024-05-09T08:26:18.086752+0000","last_undegraded":"2024-05-09T08:26:18.086752+0000","last_fullsized":"2024-05-09T08:26:18.086752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:34.916491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971219+0000","last_change":"2024-05-09T08:23:55.674022+0000","last_active":"2024-05-09T08:26:17.971219+0000","last_peered":"2024-05-09T08:26:17.971219+0000","last_clean":"2024-05-09T08:26:17.971219+0000","last_became_active":"2024-05-09T08:23:55.673322+0000","last_became_peered":"2024-05-09T08:23:55.673322+0000","last_unstale":"2024-05-09T08:26:17.971219+0000","last_undegraded":"2024-05-09T08:26:17.971219+0000","last_fullsized":"2024-05-09T08:26:17.971219+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:59:14.299886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.095836+0000","last_change":"2024-05-09T08:23:50.571235+0000","last_active":"2024-05-09T08:26:18.095836+0000","last_peered":"2024-05-09T08:26:18.095836+0000","last_clean":"2024-05-09T08:26:18.095836+0000","last_became_active":"2024-05-09T08:23:50.570694+0000","last_became_peered":"2024-05-09T08:23:50.570694+0000","last_unstale":"2024-05-09T08:26:18.095836+0000","last_undegraded":"2024-05-09T08:26:18.095836+0000","last_fullsized":"2024-05-09T08:26:18.095836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:22.850248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092662+0000","last_change":"2024-05-09T08:25:52.410264+0000","last_active":"2024-05-09T08:26:18.092662+0000","last_peered":"2024-05-09T08:26:18.092662+0000","last_clean":"2024-05-09T08:26:18.092662+0000","last_became_active":"2024-05-09T08:25:52.409851+0000","last_became_peered":"2024-05-09T08:25:52.409851+0000","last_unstale":"2024-05-09T08:26:18.092662+0000","last_undegraded":"2024-05-09T08:26:18.092662+0000","last_fullsized":"2024-05-09T08:26:18.092662+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:20:56.614365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203490+0000","last_change":"2024-05-09T08:24:51.489278+0000","last_active":"2024-05-09T08:26:18.203490+0000","last_peered":"2024-05-09T08:26:18.203490+0000","last_clean":"2024-05-09T08:26:18.203490+0000","last_became_active":"2024-05-09T08:24:51.488875+0000","last_became_peered":"2024-05-09T08:24:51.488875+0000","last_unstale":"2024-05-09T08:26:18.203490+0000","last_undegraded":"2024-05-09T08:26:18.203490+0000","last_fullsized":"2024-05-09T08:26:18.203490+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:47:45.306525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203847+0000","last_change":"2024-05-09T08:23:50.570040+0000","last_active":"2024-05-09T08:26:18.203847+0000","last_peered":"2024-05-09T08:26:18.203847+0000","last_clean":"2024-05-09T08:26:18.203847+0000","last_became_active":"2024-05-09T08:23:50.569665+0000","last_became_peered":"2024-05-09T08:23:50.569665+0000","last_unstale":"2024-05-09T08:26:18.203847+0000","last_undegraded":"2024-05-09T08:26:18.203847+0000","last_fullsized":"2024-05-09T08:26:18.203847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:22:29.639712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970223+0000","last_change":"2024-05-09T08:23:51.568561+0000","last_active":"2024-05-09T08:26:17.970223+0000","last_peered":"2024-05-09T08:26:17.970223+0000","last_clean":"2024-05-09T08:26:17.970223+0000","last_became_active":"2024-05-09T08:23:51.568196+0000","last_became_peered":"2024-05-09T08:23:51.568196+0000","last_unstale":"2024-05-09T08:26:17.970223+0000","last_undegraded":"2024-05-09T08:26:17.970223+0000","last_fullsized":"2024-05-09T08:26:17.970223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:52.132837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206099+0000","last_change":"2024-05-09T08:23:55.674960+0000","last_active":"2024-05-09T08:26:18.206099+0000","last_peered":"2024-05-09T08:26:18.206099+0000","last_clean":"2024-05-09T08:26:18.206099+0000","last_became_active":"2024-05-09T08:23:55.674509+0000","last_became_peered":"2024-05-09T08:23:55.674509+0000","last_unstale":"2024-05-09T08:26:18.206099+0000","last_undegraded":"2024-05-09T08:26:18.206099+0000","last_fullsized":"2024-05-09T08:26:18.206099+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:22.479867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092234+0000","last_change":"2024-05-09T08:23:50.569896+0000","last_active":"2024-05-09T08:26:18.092234+0000","last_peered":"2024-05-09T08:26:18.092234+0000","last_clean":"2024-05-09T08:26:18.092234+0000","last_became_active":"2024-05-09T08:23:50.569562+0000","last_became_peered":"2024-05-09T08:23:50.569562+0000","last_unstale":"2024-05-09T08:26:18.092234+0000","last_undegraded":"2024-05-09T08:26:18.092234+0000","last_fullsized":"2024-05-09T08:26:18.092234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:25:20.278726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083524+0000","last_change":"2024-05-09T08:23:51.570260+0000","last_active":"2024-05-09T08:26:18.083524+0000","last_peered":"2024-05-09T08:26:18.083524+0000","last_clean":"2024-05-09T08:26:18.083524+0000","last_became_active":"2024-05-09T08:23:51.569837+0000","last_became_peered":"2024-05-09T08:23:51.569837+0000","last_unstale":"2024-05-09T08:26:18.083524+0000","last_undegraded":"2024-05-09T08:26:18.083524+0000","last_fullsized":"2024-05-09T08:26:18.083524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:01:44.805271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970259+0000","last_change":"2024-05-09T08:23:55.676639+0000","last_active":"2024-05-09T08:26:17.970259+0000","last_peered":"2024-05-09T08:26:17.970259+0000","last_clean":"2024-05-09T08:26:17.970259+0000","last_became_active":"2024-05-09T08:23:55.673643+0000","last_became_peered":"2024-05-09T08:23:55.673643+0000","last_unstale":"2024-05-09T08:26:17.970259+0000","last_undegraded":"2024-05-09T08:26:17.970259+0000","last_fullsized":"2024-05-09T08:26:17.970259+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:02.377184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'6","reported_seq":280,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.595264+0000","last_change":"2024-05-09T08:23:50.565745+0000","last_active":"2024-05-09T08:26:18.595264+0000","last_peered":"2024-05-09T08:26:18.595264+0000","last_clean":"2024-05-09T08:26:18.595264+0000","last_became_active":"2024-05-09T08:23:50.565385+0000","last_became_peered":"2024-05-09T08:23:50.565385+0000","last_unstale":"2024-05-09T08:26:18.595264+0000","last_undegraded":"2024-05-09T08:26:18.595264+0000","last_fullsized":"2024-05-09T08:26:18.595264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:05:00.818398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":240,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203490+0000","last_change":"2024-05-09T08:23:51.572341+0000","last_active":"2024-05-09T08:26:18.203490+0000","last_peered":"2024-05-09T08:26:18.203490+0000","last_clean":"2024-05-09T08:26:18.203490+0000","last_became_active":"2024-05-09T08:23:51.571583+0000","last_became_peered":"2024-05-09T08:23:51.571583+0000","last_unstale":"2024-05-09T08:26:18.203490+0000","last_undegraded":"2024-05-09T08:26:18.203490+0000","last_fullsized":"2024-05-09T08:26:18.203490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:20.695995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083374+0000","last_change":"2024-05-09T08:23:55.677603+0000","last_active":"2024-05-09T08:26:18.083374+0000","last_peered":"2024-05-09T08:26:18.083374+0000","last_clean":"2024-05-09T08:26:18.083374+0000","last_became_active":"2024-05-09T08:23:55.677143+0000","last_became_peered":"2024-05-09T08:23:55.677143+0000","last_unstale":"2024-05-09T08:26:18.083374+0000","last_undegraded":"2024-05-09T08:26:18.083374+0000","last_fullsized":"2024-05-09T08:26:18.083374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:27.850997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.188009+0000","last_change":"2024-05-09T08:23:50.565497+0000","last_active":"2024-05-09T08:26:18.188009+0000","last_peered":"2024-05-09T08:26:18.188009+0000","last_clean":"2024-05-09T08:26:18.188009+0000","last_became_active":"2024-05-09T08:23:50.565120+0000","last_became_peered":"2024-05-09T08:23:50.565120+0000","last_unstale":"2024-05-09T08:26:18.188009+0000","last_undegraded":"2024-05-09T08:26:18.188009+0000","last_fullsized":"2024-05-09T08:26:18.188009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:56.141997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206840+0000","last_change":"2024-05-09T08:23:51.576489+0000","last_active":"2024-05-09T08:26:18.206840+0000","last_peered":"2024-05-09T08:26:18.206840+0000","last_clean":"2024-05-09T08:26:18.206840+0000","last_became_active":"2024-05-09T08:23:51.576145+0000","last_became_peered":"2024-05-09T08:23:51.576145+0000","last_unstale":"2024-05-09T08:26:18.206840+0000","last_undegraded":"2024-05-09T08:26:18.206840+0000","last_fullsized":"2024-05-09T08:26:18.206840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:37.650308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971493+0000","last_change":"2024-05-09T08:24:51.490957+0000","last_active":"2024-05-09T08:26:17.971493+0000","last_peered":"2024-05-09T08:26:17.971493+0000","last_clean":"2024-05-09T08:26:17.971493+0000","last_became_active":"2024-05-09T08:24:51.490541+0000","last_became_peered":"2024-05-09T08:24:51.490541+0000","last_unstale":"2024-05-09T08:26:17.971493+0000","last_undegraded":"2024-05-09T08:26:17.971493+0000","last_fullsized":"2024-05-09T08:26:17.971493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:03.168547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092549+0000","last_change":"2024-05-09T08:23:50.567182+0000","last_active":"2024-05-09T08:26:18.092549+0000","last_peered":"2024-05-09T08:26:18.092549+0000","last_clean":"2024-05-09T08:26:18.092549+0000","last_became_active":"2024-05-09T08:23:50.566851+0000","last_became_peered":"2024-05-09T08:23:50.566851+0000","last_unstale":"2024-05-09T08:26:18.092549+0000","last_undegraded":"2024-05-09T08:26:18.092549+0000","last_fullsized":"2024-05-09T08:26:18.092549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:26.540456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203499+0000","last_change":"2024-05-09T08:23:51.574074+0000","last_active":"2024-05-09T08:26:18.203499+0000","last_peered":"2024-05-09T08:26:18.203499+0000","last_clean":"2024-05-09T08:26:18.203499+0000","last_became_active":"2024-05-09T08:23:51.572587+0000","last_became_peered":"2024-05-09T08:23:51.572587+0000","last_unstale":"2024-05-09T08:26:18.203499+0000","last_undegraded":"2024-05-09T08:26:18.203499+0000","last_fullsized":"2024-05-09T08:26:18.203499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:46.186358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083525+0000","last_change":"2024-05-09T08:23:55.667170+0000","last_active":"2024-05-09T08:26:18.083525+0000","last_peered":"2024-05-09T08:26:18.083525+0000","last_clean":"2024-05-09T08:26:18.083525+0000","last_became_active":"2024-05-09T08:23:55.666707+0000","last_became_peered":"2024-05-09T08:23:55.666707+0000","last_unstale":"2024-05-09T08:26:18.083525+0000","last_undegraded":"2024-05-09T08:26:18.083525+0000","last_fullsized":"2024-05-09T08:26:18.083525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:48.332207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":345,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.609304+0000","last_change":"2024-05-09T08:24:51.509495+0000","last_active":"2024-05-09T08:26:18.609304+0000","last_peered":"2024-05-09T08:26:18.609304+0000","last_clean":"2024-05-09T08:26:18.609304+0000","last_became_active":"2024-05-09T08:24:51.491187+0000","last_became_peered":"2024-05-09T08:24:51.491187+0000","last_unstale":"2024-05-09T08:26:18.609304+0000","last_undegraded":"2024-05-09T08:26:18.609304+0000","last_fullsized":"2024-05-09T08:26:18.609304+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:55.320000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":241,"num_read_kb":190,"num_write":44,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":63356,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.087061+0000","last_change":"2024-05-09T08:23:51.577595+0000","last_active":"2024-05-09T08:26:18.087061+0000","last_peered":"2024-05-09T08:26:18.087061+0000","last_clean":"2024-05-09T08:26:18.087061+0000","last_became_active":"2024-05-09T08:23:51.577271+0000","last_became_peered":"2024-05-09T08:23:51.577271+0000","last_unstale":"2024-05-09T08:26:18.087061+0000","last_undegraded":"2024-05-09T08:26:18.087061+0000","last_fullsized":"2024-05-09T08:26:18.087061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:34.679111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092985+0000","last_change":"2024-05-09T08:23:55.671128+0000","last_active":"2024-05-09T08:26:18.092985+0000","last_peered":"2024-05-09T08:26:18.092985+0000","last_clean":"2024-05-09T08:26:18.092985+0000","last_became_active":"2024-05-09T08:23:55.670414+0000","last_became_peered":"2024-05-09T08:23:55.670414+0000","last_unstale":"2024-05-09T08:26:18.092985+0000","last_undegraded":"2024-05-09T08:26:18.092985+0000","last_fullsized":"2024-05-09T08:26:18.092985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:31:55.307412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971568+0000","last_change":"2024-05-09T08:23:50.568503+0000","last_active":"2024-05-09T08:26:17.971568+0000","last_peered":"2024-05-09T08:26:17.971568+0000","last_clean":"2024-05-09T08:26:17.971568+0000","last_became_active":"2024-05-09T08:23:50.568181+0000","last_became_peered":"2024-05-09T08:23:50.568181+0000","last_unstale":"2024-05-09T08:26:17.971568+0000","last_undegraded":"2024-05-09T08:26:17.971568+0000","last_fullsized":"2024-05-09T08:26:17.971568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:40.883792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204000+0000","last_change":"2024-05-09T08:23:51.573602+0000","last_active":"2024-05-09T08:26:18.204000+0000","last_peered":"2024-05-09T08:26:18.204000+0000","last_clean":"2024-05-09T08:26:18.204000+0000","last_became_active":"2024-05-09T08:23:51.572557+0000","last_became_peered":"2024-05-09T08:23:51.572557+0000","last_unstale":"2024-05-09T08:26:18.204000+0000","last_undegraded":"2024-05-09T08:26:18.204000+0000","last_fullsized":"2024-05-09T08:26:18.204000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.170962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203332+0000","last_change":"2024-05-09T08:23:55.673773+0000","last_active":"2024-05-09T08:26:18.203332+0000","last_peered":"2024-05-09T08:26:18.203332+0000","last_clean":"2024-05-09T08:26:18.203332+0000","last_became_active":"2024-05-09T08:23:55.673276+0000","last_became_peered":"2024-05-09T08:23:55.673276+0000","last_unstale":"2024-05-09T08:26:18.203332+0000","last_undegraded":"2024-05-09T08:26:18.203332+0000","last_fullsized":"2024-05-09T08:26:18.203332+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:59:00.629339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092468+0000","last_change":"2024-05-09T08:23:50.575894+0000","last_active":"2024-05-09T08:26:18.092468+0000","last_peered":"2024-05-09T08:26:18.092468+0000","last_clean":"2024-05-09T08:26:18.092468+0000","last_became_active":"2024-05-09T08:23:50.575555+0000","last_became_peered":"2024-05-09T08:23:50.575555+0000","last_unstale":"2024-05-09T08:26:18.092468+0000","last_undegraded":"2024-05-09T08:26:18.092468+0000","last_fullsized":"2024-05-09T08:26:18.092468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:19.070764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187948+0000","last_change":"2024-05-09T08:23:51.574125+0000","last_active":"2024-05-09T08:26:18.187948+0000","last_peered":"2024-05-09T08:26:18.187948+0000","last_clean":"2024-05-09T08:26:18.187948+0000","last_became_active":"2024-05-09T08:23:51.573794+0000","last_became_peered":"2024-05-09T08:23:51.573794+0000","last_unstale":"2024-05-09T08:26:18.187948+0000","last_undegraded":"2024-05-09T08:26:18.187948+0000","last_fullsized":"2024-05-09T08:26:18.187948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:19:26.634208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.084105+0000","last_change":"2024-05-09T08:23:55.677711+0000","last_active":"2024-05-09T08:26:18.084105+0000","last_peered":"2024-05-09T08:26:18.084105+0000","last_clean":"2024-05-09T08:26:18.084105+0000","last_became_active":"2024-05-09T08:23:55.677058+0000","last_became_peered":"2024-05-09T08:23:55.677058+0000","last_unstale":"2024-05-09T08:26:18.084105+0000","last_undegraded":"2024-05-09T08:26:18.084105+0000","last_fullsized":"2024-05-09T08:26:18.084105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:26.842145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'9","reported_seq":167,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.599726+0000","last_change":"2024-05-09T08:23:50.566413+0000","last_active":"2024-05-09T08:26:18.599726+0000","last_peered":"2024-05-09T08:26:18.599726+0000","last_clean":"2024-05-09T08:26:18.599726+0000","last_became_active":"2024-05-09T08:23:50.566047+0000","last_became_peered":"2024-05-09T08:23:50.566047+0000","last_unstale":"2024-05-09T08:26:18.599726+0000","last_undegraded":"2024-05-09T08:26:18.599726+0000","last_fullsized":"2024-05-09T08:26:18.599726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:15.340947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":5,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187600+0000","last_change":"2024-05-09T08:23:51.576779+0000","last_active":"2024-05-09T08:26:18.187600+0000","last_peered":"2024-05-09T08:26:18.187600+0000","last_clean":"2024-05-09T08:26:18.187600+0000","last_became_active":"2024-05-09T08:23:51.576119+0000","last_became_peered":"2024-05-09T08:23:51.576119+0000","last_unstale":"2024-05-09T08:26:18.187600+0000","last_undegraded":"2024-05-09T08:26:18.187600+0000","last_fullsized":"2024-05-09T08:26:18.187600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:02.250135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204904+0000","last_change":"2024-05-09T08:23:55.678564+0000","last_active":"2024-05-09T08:26:18.204904+0000","last_peered":"2024-05-09T08:26:18.204904+0000","last_clean":"2024-05-09T08:26:18.204904+0000","last_became_active":"2024-05-09T08:23:55.678128+0000","last_became_peered":"2024-05-09T08:23:55.678128+0000","last_unstale":"2024-05-09T08:26:18.204904+0000","last_undegraded":"2024-05-09T08:26:18.204904+0000","last_fullsized":"2024-05-09T08:26:18.204904+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:13:21.868267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086552+0000","last_change":"2024-05-09T08:24:51.501243+0000","last_active":"2024-05-09T08:26:18.086552+0000","last_peered":"2024-05-09T08:26:18.086552+0000","last_clean":"2024-05-09T08:26:18.086552+0000","last_became_active":"2024-05-09T08:24:51.490110+0000","last_became_peered":"2024-05-09T08:24:51.490110+0000","last_unstale":"2024-05-09T08:26:18.086552+0000","last_undegraded":"2024-05-09T08:26:18.086552+0000","last_fullsized":"2024-05-09T08:26:18.086552+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:03.086570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092330+0000","last_change":"2024-05-09T08:23:51.574914+0000","last_active":"2024-05-09T08:26:18.092330+0000","last_peered":"2024-05-09T08:26:18.092330+0000","last_clean":"2024-05-09T08:26:18.092330+0000","last_became_active":"2024-05-09T08:23:51.574380+0000","last_became_peered":"2024-05-09T08:23:51.574380+0000","last_unstale":"2024-05-09T08:26:18.092330+0000","last_undegraded":"2024-05-09T08:26:18.092330+0000","last_fullsized":"2024-05-09T08:26:18.092330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:34.698328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092194+0000","last_change":"2024-05-09T08:23:55.670198+0000","last_active":"2024-05-09T08:26:18.092194+0000","last_peered":"2024-05-09T08:26:18.092194+0000","last_clean":"2024-05-09T08:26:18.092194+0000","last_became_active":"2024-05-09T08:23:55.669798+0000","last_became_peered":"2024-05-09T08:23:55.669798+0000","last_unstale":"2024-05-09T08:26:18.092194+0000","last_undegraded":"2024-05-09T08:26:18.092194+0000","last_fullsized":"2024-05-09T08:26:18.092194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:09.474264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203593+0000","last_change":"2024-05-09T08:24:51.504286+0000","last_active":"2024-05-09T08:26:18.203593+0000","last_peered":"2024-05-09T08:26:18.203593+0000","last_clean":"2024-05-09T08:26:18.203593+0000","last_became_active":"2024-05-09T08:24:51.489111+0000","last_became_peered":"2024-05-09T08:24:51.489111+0000","last_unstale":"2024-05-09T08:26:18.203593+0000","last_undegraded":"2024-05-09T08:26:18.203593+0000","last_fullsized":"2024-05-09T08:26:18.203593+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:14.975959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203398+0000","last_change":"2024-05-09T08:23:51.573434+0000","last_active":"2024-05-09T08:26:18.203398+0000","last_peered":"2024-05-09T08:26:18.203398+0000","last_clean":"2024-05-09T08:26:18.203398+0000","last_became_active":"2024-05-09T08:23:51.573081+0000","last_became_peered":"2024-05-09T08:23:51.573081+0000","last_unstale":"2024-05-09T08:26:18.203398+0000","last_undegraded":"2024-05-09T08:26:18.203398+0000","last_fullsized":"2024-05-09T08:26:18.203398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:49.270355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086717+0000","last_change":"2024-05-09T08:23:55.853114+0000","last_active":"2024-05-09T08:26:18.086717+0000","last_peered":"2024-05-09T08:26:18.086717+0000","last_clean":"2024-05-09T08:26:18.086717+0000","last_became_active":"2024-05-09T08:23:55.852640+0000","last_became_peered":"2024-05-09T08:23:55.852640+0000","last_unstale":"2024-05-09T08:26:18.086717+0000","last_undegraded":"2024-05-09T08:26:18.086717+0000","last_fullsized":"2024-05-09T08:26:18.086717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:55:17.823585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083798+0000","last_change":"2024-05-09T08:23:50.567834+0000","last_active":"2024-05-09T08:26:18.083798+0000","last_peered":"2024-05-09T08:26:18.083798+0000","last_clean":"2024-05-09T08:26:18.083798+0000","last_became_active":"2024-05-09T08:23:50.567538+0000","last_became_peered":"2024-05-09T08:23:50.567538+0000","last_unstale":"2024-05-09T08:26:18.083798+0000","last_undegraded":"2024-05-09T08:26:18.083798+0000","last_fullsized":"2024-05-09T08:26:18.083798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.674352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.096142+0000","last_change":"2024-05-09T08:23:51.580219+0000","last_active":"2024-05-09T08:26:18.096142+0000","last_peered":"2024-05-09T08:26:18.096142+0000","last_clean":"2024-05-09T08:26:18.096142+0000","last_became_active":"2024-05-09T08:23:51.579835+0000","last_became_peered":"2024-05-09T08:23:51.579835+0000","last_unstale":"2024-05-09T08:26:18.096142+0000","last_undegraded":"2024-05-09T08:26:18.096142+0000","last_fullsized":"2024-05-09T08:26:18.096142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:15.232318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204599+0000","last_change":"2024-05-09T08:23:55.674324+0000","last_active":"2024-05-09T08:26:18.204599+0000","last_peered":"2024-05-09T08:26:18.204599+0000","last_clean":"2024-05-09T08:26:18.204599+0000","last_became_active":"2024-05-09T08:23:55.673254+0000","last_became_peered":"2024-05-09T08:23:55.673254+0000","last_unstale":"2024-05-09T08:26:18.204599+0000","last_undegraded":"2024-05-09T08:26:18.204599+0000","last_fullsized":"2024-05-09T08:26:18.204599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:05:23.303987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'6","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.617307+0000","last_change":"2024-05-09T08:23:50.571682+0000","last_active":"2024-05-09T08:26:18.617307+0000","last_peered":"2024-05-09T08:26:18.617307+0000","last_clean":"2024-05-09T08:26:18.617307+0000","last_became_active":"2024-05-09T08:23:50.571338+0000","last_became_peered":"2024-05-09T08:23:50.571338+0000","last_unstale":"2024-05-09T08:26:18.617307+0000","last_undegraded":"2024-05-09T08:26:18.617307+0000","last_fullsized":"2024-05-09T08:26:18.617307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:17.578071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.095802+0000","last_change":"2024-05-09T08:24:51.487208+0000","last_active":"2024-05-09T08:26:18.095802+0000","last_peered":"2024-05-09T08:26:18.095802+0000","last_clean":"2024-05-09T08:26:18.095802+0000","last_became_active":"2024-05-09T08:24:51.486665+0000","last_became_peered":"2024-05-09T08:24:51.486665+0000","last_unstale":"2024-05-09T08:26:18.095802+0000","last_undegraded":"2024-05-09T08:26:18.095802+0000","last_fullsized":"2024-05-09T08:26:18.095802+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:01:10.050846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205221+0000","last_change":"2024-05-09T08:23:55.853086+0000","last_active":"2024-05-09T08:26:18.205221+0000","last_peered":"2024-05-09T08:26:18.205221+0000","last_clean":"2024-05-09T08:26:18.205221+0000","last_became_active":"2024-05-09T08:23:55.852302+0000","last_became_peered":"2024-05-09T08:23:55.852302+0000","last_unstale":"2024-05-09T08:26:18.205221+0000","last_undegraded":"2024-05-09T08:26:18.205221+0000","last_fullsized":"2024-05-09T08:26:18.205221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:42:18.547623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092340+0000","last_change":"2024-05-09T08:23:51.575522+0000","last_active":"2024-05-09T08:26:18.092340+0000","last_peered":"2024-05-09T08:26:18.092340+0000","last_clean":"2024-05-09T08:26:18.092340+0000","last_became_active":"2024-05-09T08:23:51.575204+0000","last_became_peered":"2024-05-09T08:23:51.575204+0000","last_unstale":"2024-05-09T08:26:18.092340+0000","last_undegraded":"2024-05-09T08:26:18.092340+0000","last_fullsized":"2024-05-09T08:26:18.092340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:57.257435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'7","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.253210+0000","last_change":"2024-05-09T08:23:50.574996+0000","last_active":"2024-05-09T08:26:18.253210+0000","last_peered":"2024-05-09T08:26:18.253210+0000","last_clean":"2024-05-09T08:26:18.253210+0000","last_became_active":"2024-05-09T08:23:50.574619+0000","last_became_peered":"2024-05-09T08:23:50.574619+0000","last_unstale":"2024-05-09T08:26:18.253210+0000","last_undegraded":"2024-05-09T08:26:18.253210+0000","last_fullsized":"2024-05-09T08:26:18.253210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:44.681314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187892+0000","last_change":"2024-05-09T08:24:51.493390+0000","last_active":"2024-05-09T08:26:18.187892+0000","last_peered":"2024-05-09T08:26:18.187892+0000","last_clean":"2024-05-09T08:26:18.187892+0000","last_became_active":"2024-05-09T08:24:51.487797+0000","last_became_peered":"2024-05-09T08:24:51.487797+0000","last_unstale":"2024-05-09T08:26:18.187892+0000","last_undegraded":"2024-05-09T08:26:18.187892+0000","last_fullsized":"2024-05-09T08:26:18.187892+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:46.646471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083550+0000","last_change":"2024-05-09T08:25:52.411415+0000","last_active":"2024-05-09T08:26:18.083550+0000","last_peered":"2024-05-09T08:26:18.083550+0000","last_clean":"2024-05-09T08:26:18.083550+0000","last_became_active":"2024-05-09T08:25:52.410972+0000","last_became_peered":"2024-05-09T08:25:52.410972+0000","last_unstale":"2024-05-09T08:26:18.083550+0000","last_undegraded":"2024-05-09T08:26:18.083550+0000","last_fullsized":"2024-05-09T08:26:18.083550+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:31.045318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971384+0000","last_change":"2024-05-09T08:23:50.566415+0000","last_active":"2024-05-09T08:26:17.971384+0000","last_peered":"2024-05-09T08:26:17.971384+0000","last_clean":"2024-05-09T08:26:17.971384+0000","last_became_active":"2024-05-09T08:23:50.565506+0000","last_became_peered":"2024-05-09T08:23:50.565506+0000","last_unstale":"2024-05-09T08:26:17.971384+0000","last_undegraded":"2024-05-09T08:26:17.971384+0000","last_fullsized":"2024-05-09T08:26:17.971384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:15.741565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187381+0000","last_change":"2024-05-09T08:22:38.366126+0000","last_active":"2024-05-09T08:26:18.187381+0000","last_peered":"2024-05-09T08:26:18.187381+0000","last_clean":"2024-05-09T08:26:18.187381+0000","last_became_active":"2024-05-09T08:22:38.365683+0000","last_became_peered":"2024-05-09T08:22:38.365683+0000","last_unstale":"2024-05-09T08:26:18.187381+0000","last_undegraded":"2024-05-09T08:26:18.187381+0000","last_fullsized":"2024-05-09T08:26:18.187381+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_clean_scrub_stamp":"2024-05-09T08:19:12.435445+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:04.744060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092887+0000","last_change":"2024-05-09T08:23:55.853248+0000","last_active":"2024-05-09T08:26:18.092887+0000","last_peered":"2024-05-09T08:26:18.092887+0000","last_clean":"2024-05-09T08:26:18.092887+0000","last_became_active":"2024-05-09T08:23:55.852646+0000","last_became_peered":"2024-05-09T08:23:55.852646+0000","last_unstale":"2024-05-09T08:26:18.092887+0000","last_undegraded":"2024-05-09T08:26:18.092887+0000","last_fullsized":"2024-05-09T08:26:18.092887+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:19:54.378185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203614+0000","last_change":"2024-05-09T08:23:51.574967+0000","last_active":"2024-05-09T08:26:18.203614+0000","last_peered":"2024-05-09T08:26:18.203614+0000","last_clean":"2024-05-09T08:26:18.203614+0000","last_became_active":"2024-05-09T08:23:51.574600+0000","last_became_peered":"2024-05-09T08:23:51.574600+0000","last_unstale":"2024-05-09T08:26:18.203614+0000","last_undegraded":"2024-05-09T08:26:18.203614+0000","last_fullsized":"2024-05-09T08:26:18.203614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:03.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187411+0000","last_change":"2024-05-09T08:23:50.569886+0000","last_active":"2024-05-09T08:26:18.187411+0000","last_peered":"2024-05-09T08:26:18.187411+0000","last_clean":"2024-05-09T08:26:18.187411+0000","last_became_active":"2024-05-09T08:23:50.569018+0000","last_became_peered":"2024-05-09T08:23:50.569018+0000","last_unstale":"2024-05-09T08:26:18.187411+0000","last_undegraded":"2024-05-09T08:26:18.187411+0000","last_fullsized":"2024-05-09T08:26:18.187411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:20.711485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203347+0000","last_change":"2024-05-09T08:23:55.676408+0000","last_active":"2024-05-09T08:26:18.203347+0000","last_peered":"2024-05-09T08:26:18.203347+0000","last_clean":"2024-05-09T08:26:18.203347+0000","last_became_active":"2024-05-09T08:23:55.675964+0000","last_became_peered":"2024-05-09T08:23:55.675964+0000","last_unstale":"2024-05-09T08:26:18.203347+0000","last_undegraded":"2024-05-09T08:26:18.203347+0000","last_fullsized":"2024-05-09T08:26:18.203347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:29.210602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086991+0000","last_change":"2024-05-09T08:23:50.566946+0000","last_active":"2024-05-09T08:26:18.086991+0000","last_peered":"2024-05-09T08:26:18.086991+0000","last_clean":"2024-05-09T08:26:18.086991+0000","last_became_active":"2024-05-09T08:23:50.566366+0000","last_became_peered":"2024-05-09T08:23:50.566366+0000","last_unstale":"2024-05-09T08:26:18.086991+0000","last_undegraded":"2024-05-09T08:26:18.086991+0000","last_fullsized":"2024-05-09T08:26:18.086991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:37:30.720879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205041+0000","last_change":"2024-05-09T08:23:51.571520+0000","last_active":"2024-05-09T08:26:18.205041+0000","last_peered":"2024-05-09T08:26:18.205041+0000","last_clean":"2024-05-09T08:26:18.205041+0000","last_became_active":"2024-05-09T08:23:51.571166+0000","last_became_peered":"2024-05-09T08:23:51.571166+0000","last_unstale":"2024-05-09T08:26:18.205041+0000","last_undegraded":"2024-05-09T08:26:18.205041+0000","last_fullsized":"2024-05-09T08:26:18.205041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:02:12.560238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970123+0000","last_change":"2024-05-09T08:23:55.852649+0000","last_active":"2024-05-09T08:26:17.970123+0000","last_peered":"2024-05-09T08:26:17.970123+0000","last_clean":"2024-05-09T08:26:17.970123+0000","last_became_active":"2024-05-09T08:23:55.852180+0000","last_became_peered":"2024-05-09T08:23:55.852180+0000","last_unstale":"2024-05-09T08:26:17.970123+0000","last_undegraded":"2024-05-09T08:26:17.970123+0000","last_fullsized":"2024-05-09T08:26:17.970123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:33.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203198+0000","last_change":"2024-05-09T08:23:50.561639+0000","last_active":"2024-05-09T08:26:18.203198+0000","last_peered":"2024-05-09T08:26:18.203198+0000","last_clean":"2024-05-09T08:26:18.203198+0000","last_became_active":"2024-05-09T08:23:50.561306+0000","last_became_peered":"2024-05-09T08:23:50.561306+0000","last_unstale":"2024-05-09T08:26:18.203198+0000","last_undegraded":"2024-05-09T08:26:18.203198+0000","last_fullsized":"2024-05-09T08:26:18.203198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:46.897965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970871+0000","last_change":"2024-05-09T08:23:51.573765+0000","last_active":"2024-05-09T08:26:17.970871+0000","last_peered":"2024-05-09T08:26:17.970871+0000","last_clean":"2024-05-09T08:26:17.970871+0000","last_became_active":"2024-05-09T08:23:51.573134+0000","last_became_peered":"2024-05-09T08:23:51.573134+0000","last_unstale":"2024-05-09T08:26:17.970871+0000","last_undegraded":"2024-05-09T08:26:17.970871+0000","last_fullsized":"2024-05-09T08:26:17.970871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:25.334148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970989+0000","last_change":"2024-05-09T08:23:55.663480+0000","last_active":"2024-05-09T08:26:17.970989+0000","last_peered":"2024-05-09T08:26:17.970989+0000","last_clean":"2024-05-09T08:26:17.970989+0000","last_became_active":"2024-05-09T08:23:55.662823+0000","last_became_peered":"2024-05-09T08:23:55.662823+0000","last_unstale":"2024-05-09T08:26:17.970989+0000","last_undegraded":"2024-05-09T08:26:17.970989+0000","last_fullsized":"2024-05-09T08:26:17.970989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:18.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971781+0000","last_change":"2024-05-09T08:23:50.574641+0000","last_active":"2024-05-09T08:26:17.971781+0000","last_peered":"2024-05-09T08:26:17.971781+0000","last_clean":"2024-05-09T08:26:17.971781+0000","last_became_active":"2024-05-09T08:23:50.574266+0000","last_became_peered":"2024-05-09T08:23:50.574266+0000","last_unstale":"2024-05-09T08:26:17.971781+0000","last_undegraded":"2024-05-09T08:26:17.971781+0000","last_fullsized":"2024-05-09T08:26:17.971781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.575211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206374+0000","last_change":"2024-05-09T08:25:52.412012+0000","last_active":"2024-05-09T08:26:18.206374+0000","last_peered":"2024-05-09T08:26:18.206374+0000","last_clean":"2024-05-09T08:26:18.206374+0000","last_became_active":"2024-05-09T08:25:52.411479+0000","last_became_peered":"2024-05-09T08:25:52.411479+0000","last_unstale":"2024-05-09T08:26:18.206374+0000","last_undegraded":"2024-05-09T08:26:18.206374+0000","last_fullsized":"2024-05-09T08:26:18.206374+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:58.577359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971567+0000","last_change":"2024-05-09T08:23:55.853759+0000","last_active":"2024-05-09T08:26:17.971567+0000","last_peered":"2024-05-09T08:26:17.971567+0000","last_clean":"2024-05-09T08:26:17.971567+0000","last_became_active":"2024-05-09T08:23:55.853045+0000","last_became_peered":"2024-05-09T08:23:55.853045+0000","last_unstale":"2024-05-09T08:26:17.971567+0000","last_undegraded":"2024-05-09T08:26:17.971567+0000","last_fullsized":"2024-05-09T08:26:17.971567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:53.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203004+0000","last_change":"2024-05-09T08:23:50.568373+0000","last_active":"2024-05-09T08:26:18.203004+0000","last_peered":"2024-05-09T08:26:18.203004+0000","last_clean":"2024-05-09T08:26:18.203004+0000","last_became_active":"2024-05-09T08:23:50.568015+0000","last_became_peered":"2024-05-09T08:23:50.568015+0000","last_unstale":"2024-05-09T08:26:18.203004+0000","last_undegraded":"2024-05-09T08:26:18.203004+0000","last_fullsized":"2024-05-09T08:26:18.203004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:05:16.391673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086521+0000","last_change":"2024-05-09T08:23:51.576825+0000","last_active":"2024-05-09T08:26:18.086521+0000","last_peered":"2024-05-09T08:26:18.086521+0000","last_clean":"2024-05-09T08:26:18.086521+0000","last_became_active":"2024-05-09T08:23:51.573350+0000","last_became_peered":"2024-05-09T08:23:51.573350+0000","last_unstale":"2024-05-09T08:26:18.086521+0000","last_undegraded":"2024-05-09T08:26:18.086521+0000","last_fullsized":"2024-05-09T08:26:18.086521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:00.963281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.084500+0000","last_change":"2024-05-09T08:23:55.672145+0000","last_active":"2024-05-09T08:26:18.084500+0000","last_peered":"2024-05-09T08:26:18.084500+0000","last_clean":"2024-05-09T08:26:18.084500+0000","last_became_active":"2024-05-09T08:23:55.671743+0000","last_became_peered":"2024-05-09T08:23:55.671743+0000","last_unstale":"2024-05-09T08:26:18.084500+0000","last_undegraded":"2024-05-09T08:26:18.084500+0000","last_fullsized":"2024-05-09T08:26:18.084500+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:09:23.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970351+0000","last_change":"2024-05-09T08:23:50.568908+0000","last_active":"2024-05-09T08:26:17.970351+0000","last_peered":"2024-05-09T08:26:17.970351+0000","last_clean":"2024-05-09T08:26:17.970351+0000","last_became_active":"2024-05-09T08:23:50.567009+0000","last_became_peered":"2024-05-09T08:23:50.567009+0000","last_unstale":"2024-05-09T08:26:17.970351+0000","last_undegraded":"2024-05-09T08:26:17.970351+0000","last_fullsized":"2024-05-09T08:26:17.970351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:35.190156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203170+0000","last_change":"2024-05-09T08:23:51.580121+0000","last_active":"2024-05-09T08:26:18.203170+0000","last_peered":"2024-05-09T08:26:18.203170+0000","last_clean":"2024-05-09T08:26:18.203170+0000","last_became_active":"2024-05-09T08:23:51.579402+0000","last_became_peered":"2024-05-09T08:23:51.579402+0000","last_unstale":"2024-05-09T08:26:18.203170+0000","last_undegraded":"2024-05-09T08:26:18.203170+0000","last_fullsized":"2024-05-09T08:26:18.203170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:11:26.144344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971820+0000","last_change":"2024-05-09T08:23:55.664017+0000","last_active":"2024-05-09T08:26:17.971820+0000","last_peered":"2024-05-09T08:26:17.971820+0000","last_clean":"2024-05-09T08:26:17.971820+0000","last_became_active":"2024-05-09T08:23:55.663546+0000","last_became_peered":"2024-05-09T08:23:55.663546+0000","last_unstale":"2024-05-09T08:26:17.971820+0000","last_undegraded":"2024-05-09T08:26:17.971820+0000","last_fullsized":"2024-05-09T08:26:17.971820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:13:14.705971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203214+0000","last_change":"2024-05-09T08:24:51.493220+0000","last_active":"2024-05-09T08:26:18.203214+0000","last_peered":"2024-05-09T08:26:18.203214+0000","last_clean":"2024-05-09T08:26:18.203214+0000","last_became_active":"2024-05-09T08:24:51.486332+0000","last_became_peered":"2024-05-09T08:24:51.486332+0000","last_unstale":"2024-05-09T08:26:18.203214+0000","last_undegraded":"2024-05-09T08:26:18.203214+0000","last_fullsized":"2024-05-09T08:26:18.203214+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:23.694545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970561+0000","last_change":"2024-05-09T08:23:51.571414+0000","last_active":"2024-05-09T08:26:17.970561+0000","last_peered":"2024-05-09T08:26:17.970561+0000","last_clean":"2024-05-09T08:26:17.970561+0000","last_became_active":"2024-05-09T08:23:51.571085+0000","last_became_peered":"2024-05-09T08:23:51.571085+0000","last_unstale":"2024-05-09T08:26:17.970561+0000","last_undegraded":"2024-05-09T08:26:17.970561+0000","last_fullsized":"2024-05-09T08:26:17.970561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:37.513584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971413+0000","last_change":"2024-05-09T08:23:55.852785+0000","last_active":"2024-05-09T08:26:17.971413+0000","last_peered":"2024-05-09T08:26:17.971413+0000","last_clean":"2024-05-09T08:26:17.971413+0000","last_became_active":"2024-05-09T08:23:55.852252+0000","last_became_peered":"2024-05-09T08:23:55.852252+0000","last_unstale":"2024-05-09T08:26:17.971413+0000","last_undegraded":"2024-05-09T08:26:17.971413+0000","last_fullsized":"2024-05-09T08:26:17.971413+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:44.503295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086689+0000","last_change":"2024-05-09T08:23:50.564787+0000","last_active":"2024-05-09T08:26:18.086689+0000","last_peered":"2024-05-09T08:26:18.086689+0000","last_clean":"2024-05-09T08:26:18.086689+0000","last_became_active":"2024-05-09T08:23:50.564409+0000","last_became_peered":"2024-05-09T08:23:50.564409+0000","last_unstale":"2024-05-09T08:26:18.086689+0000","last_undegraded":"2024-05-09T08:26:18.086689+0000","last_fullsized":"2024-05-09T08:26:18.086689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:23.051221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187715+0000","last_change":"2024-05-09T08:23:51.576830+0000","last_active":"2024-05-09T08:26:18.187715+0000","last_peered":"2024-05-09T08:26:18.187715+0000","last_clean":"2024-05-09T08:26:18.187715+0000","last_became_active":"2024-05-09T08:23:51.576311+0000","last_became_peered":"2024-05-09T08:23:51.576311+0000","last_unstale":"2024-05-09T08:26:18.187715+0000","last_undegraded":"2024-05-09T08:26:18.187715+0000","last_fullsized":"2024-05-09T08:26:18.187715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:26:08.651061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203014+0000","last_change":"2024-05-09T08:23:55.672309+0000","last_active":"2024-05-09T08:26:18.203014+0000","last_peered":"2024-05-09T08:26:18.203014+0000","last_clean":"2024-05-09T08:26:18.203014+0000","last_became_active":"2024-05-09T08:23:55.671811+0000","last_became_peered":"2024-05-09T08:23:55.671811+0000","last_unstale":"2024-05-09T08:26:18.203014+0000","last_undegraded":"2024-05-09T08:26:18.203014+0000","last_fullsized":"2024-05-09T08:26:18.203014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:19.400949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'7","reported_seq":165,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.595790+0000","last_change":"2024-05-09T08:23:50.575317+0000","last_active":"2024-05-09T08:26:18.595790+0000","last_peered":"2024-05-09T08:26:18.595790+0000","last_clean":"2024-05-09T08:26:18.595790+0000","last_became_active":"2024-05-09T08:23:50.574995+0000","last_became_peered":"2024-05-09T08:23:50.574995+0000","last_unstale":"2024-05-09T08:26:18.595790+0000","last_undegraded":"2024-05-09T08:26:18.595790+0000","last_fullsized":"2024-05-09T08:26:18.595790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:32:54.801799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1869,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.188099+0000","last_change":"2024-05-09T08:23:51.575904+0000","last_active":"2024-05-09T08:26:18.188099+0000","last_peered":"2024-05-09T08:26:18.188099+0000","last_clean":"2024-05-09T08:26:18.188099+0000","last_became_active":"2024-05-09T08:23:51.575599+0000","last_became_peered":"2024-05-09T08:23:51.575599+0000","last_unstale":"2024-05-09T08:26:18.188099+0000","last_undegraded":"2024-05-09T08:26:18.188099+0000","last_fullsized":"2024-05-09T08:26:18.188099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:20:13.087096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205215+0000","last_change":"2024-05-09T08:23:55.680683+0000","last_active":"2024-05-09T08:26:18.205215+0000","last_peered":"2024-05-09T08:26:18.205215+0000","last_clean":"2024-05-09T08:26:18.205215+0000","last_became_active":"2024-05-09T08:23:55.680232+0000","last_became_peered":"2024-05-09T08:23:55.680232+0000","last_unstale":"2024-05-09T08:26:18.205215+0000","last_undegraded":"2024-05-09T08:26:18.205215+0000","last_fullsized":"2024-05-09T08:26:18.205215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:33.583552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.200717+0000","last_change":"2024-05-09T08:23:50.575802+0000","last_active":"2024-05-09T08:26:18.200717+0000","last_peered":"2024-05-09T08:26:18.200717+0000","last_clean":"2024-05-09T08:26:18.200717+0000","last_became_active":"2024-05-09T08:23:50.575416+0000","last_became_peered":"2024-05-09T08:23:50.575416+0000","last_unstale":"2024-05-09T08:26:18.200717+0000","last_undegraded":"2024-05-09T08:26:18.200717+0000","last_fullsized":"2024-05-09T08:26:18.200717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:24:04.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187831+0000","last_change":"2024-05-09T08:23:51.578189+0000","last_active":"2024-05-09T08:26:18.187831+0000","last_peered":"2024-05-09T08:26:18.187831+0000","last_clean":"2024-05-09T08:26:18.187831+0000","last_became_active":"2024-05-09T08:23:51.577500+0000","last_became_peered":"2024-05-09T08:23:51.577500+0000","last_unstale":"2024-05-09T08:26:18.187831+0000","last_undegraded":"2024-05-09T08:26:18.187831+0000","last_fullsized":"2024-05-09T08:26:18.187831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:26:24.718791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086471+0000","last_change":"2024-05-09T08:24:51.487659+0000","last_active":"2024-05-09T08:26:18.086471+0000","last_peered":"2024-05-09T08:26:18.086471+0000","last_clean":"2024-05-09T08:26:18.086471+0000","last_became_active":"2024-05-09T08:24:51.487212+0000","last_became_peered":"2024-05-09T08:24:51.487212+0000","last_unstale":"2024-05-09T08:26:18.086471+0000","last_undegraded":"2024-05-09T08:26:18.086471+0000","last_fullsized":"2024-05-09T08:26:18.086471+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:29:57.077744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205019+0000","last_change":"2024-05-09T08:23:50.578438+0000","last_active":"2024-05-09T08:26:18.205019+0000","last_peered":"2024-05-09T08:26:18.205019+0000","last_clean":"2024-05-09T08:26:18.205019+0000","last_became_active":"2024-05-09T08:23:50.578039+0000","last_became_peered":"2024-05-09T08:23:50.578039+0000","last_unstale":"2024-05-09T08:26:18.205019+0000","last_undegraded":"2024-05-09T08:26:18.205019+0000","last_fullsized":"2024-05-09T08:26:18.205019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:59.420193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187887+0000","last_change":"2024-05-09T08:23:51.568037+0000","last_active":"2024-05-09T08:26:18.187887+0000","last_peered":"2024-05-09T08:26:18.187887+0000","last_clean":"2024-05-09T08:26:18.187887+0000","last_became_active":"2024-05-09T08:23:51.567706+0000","last_became_peered":"2024-05-09T08:23:51.567706+0000","last_unstale":"2024-05-09T08:26:18.187887+0000","last_undegraded":"2024-05-09T08:26:18.187887+0000","last_fullsized":"2024-05-09T08:26:18.187887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:15:19.623805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203263+0000","last_change":"2024-05-09T08:23:51.579885+0000","last_active":"2024-05-09T08:26:18.203263+0000","last_peered":"2024-05-09T08:26:18.203263+0000","last_clean":"2024-05-09T08:26:18.203263+0000","last_became_active":"2024-05-09T08:23:51.578953+0000","last_became_peered":"2024-05-09T08:23:51.578953+0000","last_unstale":"2024-05-09T08:26:18.203263+0000","last_undegraded":"2024-05-09T08:26:18.203263+0000","last_fullsized":"2024-05-09T08:26:18.203263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:57:23.793119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971252+0000","last_change":"2024-05-09T08:23:51.583423+0000","last_active":"2024-05-09T08:26:17.971252+0000","last_peered":"2024-05-09T08:26:17.971252+0000","last_clean":"2024-05-09T08:26:17.971252+0000","last_became_active":"2024-05-09T08:23:51.583092+0000","last_became_peered":"2024-05-09T08:23:51.583092+0000","last_unstale":"2024-05-09T08:26:17.971252+0000","last_undegraded":"2024-05-09T08:26:17.971252+0000","last_fullsized":"2024-05-09T08:26:17.971252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:44:47.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971056+0000","last_change":"2024-05-09T08:23:51.582116+0000","last_active":"2024-05-09T08:26:17.971056+0000","last_peered":"2024-05-09T08:26:17.971056+0000","last_clean":"2024-05-09T08:26:17.971056+0000","last_became_active":"2024-05-09T08:23:51.581715+0000","last_became_peered":"2024-05-09T08:23:51.581715+0000","last_unstale":"2024-05-09T08:26:17.971056+0000","last_undegraded":"2024-05-09T08:26:17.971056+0000","last_fullsized":"2024-05-09T08:26:17.971056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:17.528758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205481+0000","last_change":"2024-05-09T08:23:51.575959+0000","last_active":"2024-05-09T08:26:18.205481+0000","last_peered":"2024-05-09T08:26:18.205481+0000","last_clean":"2024-05-09T08:26:18.205481+0000","last_became_active":"2024-05-09T08:23:51.575592+0000","last_became_peered":"2024-05-09T08:23:51.575592+0000","last_unstale":"2024-05-09T08:26:18.205481+0000","last_undegraded":"2024-05-09T08:26:18.205481+0000","last_fullsized":"2024-05-09T08:26:18.205481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:26.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205213+0000","last_change":"2024-05-09T08:23:51.570756+0000","last_active":"2024-05-09T08:26:18.205213+0000","last_peered":"2024-05-09T08:26:18.205213+0000","last_clean":"2024-05-09T08:26:18.205213+0000","last_became_active":"2024-05-09T08:23:51.570412+0000","last_became_peered":"2024-05-09T08:23:51.570412+0000","last_unstale":"2024-05-09T08:26:18.205213+0000","last_undegraded":"2024-05-09T08:26:18.205213+0000","last_fullsized":"2024-05-09T08:26:18.205213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:11:53.959559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203772+0000","last_change":"2024-05-09T08:23:51.572612+0000","last_active":"2024-05-09T08:26:18.203772+0000","last_peered":"2024-05-09T08:26:18.203772+0000","last_clean":"2024-05-09T08:26:18.203772+0000","last_became_active":"2024-05-09T08:23:51.572227+0000","last_became_peered":"2024-05-09T08:23:51.572227+0000","last_unstale":"2024-05-09T08:26:18.203772+0000","last_undegraded":"2024-05-09T08:26:18.203772+0000","last_fullsized":"2024-05-09T08:26:18.203772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:18.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971483+0000","last_change":"2024-05-09T08:23:51.573021+0000","last_active":"2024-05-09T08:26:17.971483+0000","last_peered":"2024-05-09T08:26:17.971483+0000","last_clean":"2024-05-09T08:26:17.971483+0000","last_became_active":"2024-05-09T08:23:51.572716+0000","last_became_peered":"2024-05-09T08:23:51.572716+0000","last_unstale":"2024-05-09T08:26:17.971483+0000","last_undegraded":"2024-05-09T08:26:17.971483+0000","last_fullsized":"2024-05-09T08:26:17.971483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:31.494329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971007+0000","last_change":"2024-05-09T08:23:51.572687+0000","last_active":"2024-05-09T08:26:17.971007+0000","last_peered":"2024-05-09T08:26:17.971007+0000","last_clean":"2024-05-09T08:26:17.971007+0000","last_became_active":"2024-05-09T08:23:51.572339+0000","last_became_peered":"2024-05-09T08:23:51.572339+0000","last_unstale":"2024-05-09T08:26:17.971007+0000","last_undegraded":"2024-05-09T08:26:17.971007+0000","last_fullsized":"2024-05-09T08:26:17.971007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:24.012360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092190+0000","last_change":"2024-05-09T08:23:51.569404+0000","last_active":"2024-05-09T08:26:18.092190+0000","last_peered":"2024-05-09T08:26:18.092190+0000","last_clean":"2024-05-09T08:26:18.092190+0000","last_became_active":"2024-05-09T08:23:51.569044+0000","last_became_peered":"2024-05-09T08:23:51.569044+0000","last_unstale":"2024-05-09T08:26:18.092190+0000","last_undegraded":"2024-05-09T08:26:18.092190+0000","last_fullsized":"2024-05-09T08:26:18.092190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:26:01.343606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970502+0000","last_change":"2024-05-09T08:23:51.577483+0000","last_active":"2024-05-09T08:26:17.970502+0000","last_peered":"2024-05-09T08:26:17.970502+0000","last_clean":"2024-05-09T08:26:17.970502+0000","last_became_active":"2024-05-09T08:23:51.577078+0000","last_became_peered":"2024-05-09T08:23:51.577078+0000","last_unstale":"2024-05-09T08:26:17.970502+0000","last_undegraded":"2024-05-09T08:26:17.970502+0000","last_fullsized":"2024-05-09T08:26:17.970502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:29.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203171+0000","last_change":"2024-05-09T08:23:51.579736+0000","last_active":"2024-05-09T08:26:18.203171+0000","last_peered":"2024-05-09T08:26:18.203171+0000","last_clean":"2024-05-09T08:26:18.203171+0000","last_became_active":"2024-05-09T08:23:51.579409+0000","last_became_peered":"2024-05-09T08:23:51.579409+0000","last_unstale":"2024-05-09T08:26:18.203171+0000","last_undegraded":"2024-05-09T08:26:18.203171+0000","last_fullsized":"2024-05-09T08:26:18.203171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:05.697642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204632+0000","last_change":"2024-05-09T08:23:51.569461+0000","last_active":"2024-05-09T08:26:18.204632+0000","last_peered":"2024-05-09T08:26:18.204632+0000","last_clean":"2024-05-09T08:26:18.204632+0000","last_became_active":"2024-05-09T08:23:51.569085+0000","last_became_peered":"2024-05-09T08:23:51.569085+0000","last_unstale":"2024-05-09T08:26:18.204632+0000","last_undegraded":"2024-05-09T08:26:18.204632+0000","last_fullsized":"2024-05-09T08:26:18.204632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:54.554949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203414+0000","last_change":"2024-05-09T08:23:51.570593+0000","last_active":"2024-05-09T08:26:18.203414+0000","last_peered":"2024-05-09T08:26:18.203414+0000","last_clean":"2024-05-09T08:26:18.203414+0000","last_became_active":"2024-05-09T08:23:51.570195+0000","last_became_peered":"2024-05-09T08:23:51.570195+0000","last_unstale":"2024-05-09T08:26:18.203414+0000","last_undegraded":"2024-05-09T08:26:18.203414+0000","last_fullsized":"2024-05-09T08:26:18.203414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:18.376268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108273,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604101,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1324,"num_read_kb":780,"num_write":376,"num_write_kb":1473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":64162,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":667648,"data_stored":1812303,"data_compressed":89522,"data_compressed_allocated":118784,"data_compressed_original":1626112,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":531150,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32344,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712808,"statfs":{"total":95995035648,"available":95961915392,"internally_reserved":0,"allocated":1191936,"data_stored":1185786,"data_compressed":67266,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":923575,"data_compressed":48720,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":923259,"data_compressed":48720,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1101824,"data_stored":1117422,"data_compressed":62684,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526254,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1118208,"data_stored":598024,"data_compressed":31175,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1064960,"data_stored":528018,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":527309,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1097728,"data_stored":1050387,"data_compressed":54876,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":601431,"data_compressed":39988,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1142784,"data_stored":1254416,"data_compressed":71821,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":72883,"data_compressed":1601,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":72194,"data_compressed":10414,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134811,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394022,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":204718,"data_compressed":7757,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:26:20.858 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-09T08:26:20.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:26:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.000+0000 7efe1defc640 1 -- 172.21.15.28:0/832010238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe180fff10 msgr2=0x7efe18100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.000+0000 7efe1defc640 1 --2- 172.21.15.28:0/832010238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe180fff10 0x7efe18100330 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7efe08009900 tx=0x7efe0802a410 comp rx=0 tx=0).stop 2024-05-09T08:26:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.000+0000 7efe1defc640 1 -- 172.21.15.28:0/832010238 shutdown_connections 2024-05-09T08:26:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.000+0000 7efe1defc640 1 --2- 172.21.15.28:0/832010238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe18101a20 0x7efe18108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.000+0000 7efe1defc640 1 --2- 172.21.15.28:0/832010238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe18101060 0x7efe181014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.000+0000 7efe1defc640 1 --2- 172.21.15.28:0/832010238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe180fff10 0x7efe18100330 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.000+0000 7efe1defc640 1 -- 172.21.15.28:0/832010238 >> 172.21.15.28:0/832010238 conn(0x7efe180fb4a0 msgr2=0x7efe180fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.000+0000 7efe1defc640 1 -- 172.21.15.28:0/832010238 shutdown_connections 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.000+0000 7efe1defc640 1 -- 172.21.15.28:0/832010238 wait complete. 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe1defc640 1 Processor -- start 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe1defc640 1 -- start start 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe1defc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe180fff10 0x7efe18103e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe1defc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe18101060 0x7efe181024d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe1defc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe18101a20 0x7efe18102a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe1defc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe1810b330 con 0x7efe18101a20 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe1defc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efe1810b1b0 con 0x7efe18101060 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe1defc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efe1810b4b0 con 0x7efe180fff10 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe18101a20 0x7efe18102a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe18101a20 0x7efe18102a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38368/0 (socket says 172.21.15.28:38368) 2024-05-09T08:26:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe17fff640 1 -- 172.21.15.28:0/901780673 learned_addr learned my addr 172.21.15.28:0/901780673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe16ffd640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe18101060 0x7efe181024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.001+0000 7efe177fe640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe180fff10 0x7efe18103e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe17fff640 1 -- 172.21.15.28:0/901780673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe180fff10 msgr2=0x7efe18103e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe17fff640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe180fff10 0x7efe18103e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe17fff640 1 -- 172.21.15.28:0/901780673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe18101060 msgr2=0x7efe181024d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe17fff640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe18101060 0x7efe181024d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe17fff640 1 -- 172.21.15.28:0/901780673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe181032d0 con 0x7efe18101a20 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe16ffd640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe18101060 0x7efe181024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe17fff640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe18101a20 0x7efe18102a10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7efe1000eb10 tx=0x7efe1000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe14ff9640 1 -- 172.21.15.28:0/901780673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe10004b20 con 0x7efe18101a20 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe181a1d40 con 0x7efe18101a20 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe14ff9640 1 -- 172.21.15.28:0/901780673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe10004cc0 con 0x7efe18101a20 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe14ff9640 1 -- 172.21.15.28:0/901780673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe10010760 con 0x7efe18101a20 2024-05-09T08:26:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.002+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe181a21b0 con 0x7efe18101a20 2024-05-09T08:26:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.003+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe181a2440 con 0x7efe18101a20 2024-05-09T08:26:21.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.004+0000 7efe14ff9640 1 -- 172.21.15.28:0/901780673 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7efe10005020 con 0x7efe18101a20 2024-05-09T08:26:21.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.005+0000 7efe14ff9640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efe000a2d00 0x7efe000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.005+0000 7efe14ff9640 1 -- 172.21.15.28:0/901780673 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7efe100cfda0 con 0x7efe18101a20 2024-05-09T08:26:21.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.005+0000 7efe177fe640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efe000a2d00 0x7efe000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.005+0000 7efe177fe640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efe000a2d00 0x7efe000a51c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efe080314e0 tx=0x7efe08000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.009+0000 7efe14ff9640 1 -- 172.21.15.28:0/901780673 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efe1009c5f0 con 0x7efe18101a20 2024-05-09T08:26:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.192+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 --> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efe18107cc0 con 0x7efe000a2d00 2024-05-09T08:26:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.204+0000 7efe14ff9640 1 -- 172.21.15.28:0/901780673 <== mgr.15528 v2:172.21.15.28:6800/160415970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339763 (secure 0 0 0) 0x7efe18107cc0 con 0x7efe000a2d00 2024-05-09T08:26:21.209 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:21.211 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-09T08:26:21.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efe000a2d00 msgr2=0x7efe000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efe000a2d00 0x7efe000a51c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efe080314e0 tx=0x7efe08000bc0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe18101a20 msgr2=0x7efe18102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe18101a20 0x7efe18102a10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7efe1000eb10 tx=0x7efe1000c2d0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 shutdown_connections 2024-05-09T08:26:21.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe18101a20 0x7efe18102a10 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe18101060 0x7efe181024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efe000a2d00 0x7efe000a51c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 --2- 172.21.15.28:0/901780673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe180fff10 0x7efe18103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 >> 172.21.15.28:0/901780673 conn(0x7efe180fb4a0 msgr2=0x7efe181065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:21.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 shutdown_connections 2024-05-09T08:26:21.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.208+0000 7efe1defc640 1 -- 172.21.15.28:0/901780673 wait complete. 2024-05-09T08:26:21.221 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-05-09T08:26:19.984209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1194703,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1460,"num_read_kb":899,"num_write":559,"num_write_kb":4413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1245015,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":944,"ondisk_log_size":944,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378620,"kb_used_data":12792,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563204,"statfs":{"total":1151940427776,"available":1151552720896,"internally_reserved":0,"allocated":13099008,"data_stored":9767031,"data_compressed":543546,"data_compressed_allocated":733184,"data_compressed_original":4247552,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.983070"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203766+0000","last_change":"2024-05-09T08:23:51.579564+0000","last_active":"2024-05-09T08:26:18.203766+0000","last_peered":"2024-05-09T08:26:18.203766+0000","last_clean":"2024-05-09T08:26:18.203766+0000","last_became_active":"2024-05-09T08:23:51.578884+0000","last_became_peered":"2024-05-09T08:23:51.578884+0000","last_unstale":"2024-05-09T08:26:18.203766+0000","last_undegraded":"2024-05-09T08:26:18.203766+0000","last_fullsized":"2024-05-09T08:26:18.203766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:40.473483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206222+0000","last_change":"2024-05-09T08:23:51.574187+0000","last_active":"2024-05-09T08:26:18.206222+0000","last_peered":"2024-05-09T08:26:18.206222+0000","last_clean":"2024-05-09T08:26:18.206222+0000","last_became_active":"2024-05-09T08:23:51.573837+0000","last_became_peered":"2024-05-09T08:23:51.573837+0000","last_unstale":"2024-05-09T08:26:18.206222+0000","last_undegraded":"2024-05-09T08:26:18.206222+0000","last_fullsized":"2024-05-09T08:26:18.206222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:31.448038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205938+0000","last_change":"2024-05-09T08:23:51.577224+0000","last_active":"2024-05-09T08:26:18.205938+0000","last_peered":"2024-05-09T08:26:18.205938+0000","last_clean":"2024-05-09T08:26:18.205938+0000","last_became_active":"2024-05-09T08:23:51.576822+0000","last_became_peered":"2024-05-09T08:23:51.576822+0000","last_unstale":"2024-05-09T08:26:18.205938+0000","last_undegraded":"2024-05-09T08:26:18.205938+0000","last_fullsized":"2024-05-09T08:26:18.205938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:04.902530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971321+0000","last_change":"2024-05-09T08:23:51.577172+0000","last_active":"2024-05-09T08:26:17.971321+0000","last_peered":"2024-05-09T08:26:17.971321+0000","last_clean":"2024-05-09T08:26:17.971321+0000","last_became_active":"2024-05-09T08:23:51.576861+0000","last_became_peered":"2024-05-09T08:23:51.576861+0000","last_unstale":"2024-05-09T08:26:17.971321+0000","last_undegraded":"2024-05-09T08:26:17.971321+0000","last_fullsized":"2024-05-09T08:26:17.971321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:10.925739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086698+0000","last_change":"2024-05-09T08:23:51.579408+0000","last_active":"2024-05-09T08:26:18.086698+0000","last_peered":"2024-05-09T08:26:18.086698+0000","last_clean":"2024-05-09T08:26:18.086698+0000","last_became_active":"2024-05-09T08:23:51.578229+0000","last_became_peered":"2024-05-09T08:23:51.578229+0000","last_unstale":"2024-05-09T08:26:18.086698+0000","last_undegraded":"2024-05-09T08:26:18.086698+0000","last_fullsized":"2024-05-09T08:26:18.086698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:47:27.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.095787+0000","last_change":"2024-05-09T08:23:51.576924+0000","last_active":"2024-05-09T08:26:18.095787+0000","last_peered":"2024-05-09T08:26:18.095787+0000","last_clean":"2024-05-09T08:26:18.095787+0000","last_became_active":"2024-05-09T08:23:51.576601+0000","last_became_peered":"2024-05-09T08:23:51.576601+0000","last_unstale":"2024-05-09T08:26:18.095787+0000","last_undegraded":"2024-05-09T08:26:18.095787+0000","last_fullsized":"2024-05-09T08:26:18.095787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:07:16.910271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187456+0000","last_change":"2024-05-09T08:25:52.714094+0000","last_active":"2024-05-09T08:26:18.187456+0000","last_peered":"2024-05-09T08:26:18.187456+0000","last_clean":"2024-05-09T08:26:18.187456+0000","last_became_active":"2024-05-09T08:25:52.713743+0000","last_became_peered":"2024-05-09T08:25:52.713743+0000","last_unstale":"2024-05-09T08:26:18.187456+0000","last_undegraded":"2024-05-09T08:26:18.187456+0000","last_fullsized":"2024-05-09T08:26:18.187456+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:59.810244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970350+0000","last_change":"2024-05-09T08:23:51.574001+0000","last_active":"2024-05-09T08:26:17.970350+0000","last_peered":"2024-05-09T08:26:17.970350+0000","last_clean":"2024-05-09T08:26:17.970350+0000","last_became_active":"2024-05-09T08:23:51.573632+0000","last_became_peered":"2024-05-09T08:23:51.573632+0000","last_unstale":"2024-05-09T08:26:17.970350+0000","last_undegraded":"2024-05-09T08:26:17.970350+0000","last_fullsized":"2024-05-09T08:26:17.970350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:55:16.418951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203492+0000","last_change":"2024-05-09T08:23:51.567635+0000","last_active":"2024-05-09T08:26:18.203492+0000","last_peered":"2024-05-09T08:26:18.203492+0000","last_clean":"2024-05-09T08:26:18.203492+0000","last_became_active":"2024-05-09T08:23:51.567283+0000","last_became_peered":"2024-05-09T08:23:51.567283+0000","last_unstale":"2024-05-09T08:26:18.203492+0000","last_undegraded":"2024-05-09T08:26:18.203492+0000","last_fullsized":"2024-05-09T08:26:18.203492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:42.346856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.096277+0000","last_change":"2024-05-09T08:23:51.576090+0000","last_active":"2024-05-09T08:26:18.096277+0000","last_peered":"2024-05-09T08:26:18.096277+0000","last_clean":"2024-05-09T08:26:18.096277+0000","last_became_active":"2024-05-09T08:23:51.575768+0000","last_became_peered":"2024-05-09T08:23:51.575768+0000","last_unstale":"2024-05-09T08:26:18.096277+0000","last_undegraded":"2024-05-09T08:26:18.096277+0000","last_fullsized":"2024-05-09T08:26:18.096277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:00:50.016924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.188104+0000","last_change":"2024-05-09T08:23:51.578957+0000","last_active":"2024-05-09T08:26:18.188104+0000","last_peered":"2024-05-09T08:26:18.188104+0000","last_clean":"2024-05-09T08:26:18.188104+0000","last_became_active":"2024-05-09T08:23:51.578210+0000","last_became_peered":"2024-05-09T08:23:51.578210+0000","last_unstale":"2024-05-09T08:26:18.188104+0000","last_undegraded":"2024-05-09T08:26:18.188104+0000","last_fullsized":"2024-05-09T08:26:18.188104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:28:28.953434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971480+0000","last_change":"2024-05-09T08:23:51.575389+0000","last_active":"2024-05-09T08:26:17.971480+0000","last_peered":"2024-05-09T08:26:17.971480+0000","last_clean":"2024-05-09T08:26:17.971480+0000","last_became_active":"2024-05-09T08:23:51.575071+0000","last_became_peered":"2024-05-09T08:23:51.575071+0000","last_unstale":"2024-05-09T08:26:17.971480+0000","last_undegraded":"2024-05-09T08:26:17.971480+0000","last_fullsized":"2024-05-09T08:26:17.971480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:34.844611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204819+0000","last_change":"2024-05-09T08:23:51.572574+0000","last_active":"2024-05-09T08:26:18.204819+0000","last_peered":"2024-05-09T08:26:18.204819+0000","last_clean":"2024-05-09T08:26:18.204819+0000","last_became_active":"2024-05-09T08:23:51.572243+0000","last_became_peered":"2024-05-09T08:23:51.572243+0000","last_unstale":"2024-05-09T08:26:18.204819+0000","last_undegraded":"2024-05-09T08:26:18.204819+0000","last_fullsized":"2024-05-09T08:26:18.204819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:52.534365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205049+0000","last_change":"2024-05-09T08:23:51.574963+0000","last_active":"2024-05-09T08:26:18.205049+0000","last_peered":"2024-05-09T08:26:18.205049+0000","last_clean":"2024-05-09T08:26:18.205049+0000","last_became_active":"2024-05-09T08:23:51.574597+0000","last_became_peered":"2024-05-09T08:23:51.574597+0000","last_unstale":"2024-05-09T08:26:18.205049+0000","last_undegraded":"2024-05-09T08:26:18.205049+0000","last_fullsized":"2024-05-09T08:26:18.205049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:33.348828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205526+0000","last_change":"2024-05-09T08:23:51.577777+0000","last_active":"2024-05-09T08:26:18.205526+0000","last_peered":"2024-05-09T08:26:18.205526+0000","last_clean":"2024-05-09T08:26:18.205526+0000","last_became_active":"2024-05-09T08:23:51.577073+0000","last_became_peered":"2024-05-09T08:23:51.577073+0000","last_unstale":"2024-05-09T08:26:18.205526+0000","last_undegraded":"2024-05-09T08:26:18.205526+0000","last_fullsized":"2024-05-09T08:26:18.205526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:26.249488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203039+0000","last_change":"2024-05-09T08:23:51.574476+0000","last_active":"2024-05-09T08:26:18.203039+0000","last_peered":"2024-05-09T08:26:18.203039+0000","last_clean":"2024-05-09T08:26:18.203039+0000","last_became_active":"2024-05-09T08:23:51.572748+0000","last_became_peered":"2024-05-09T08:23:51.572748+0000","last_unstale":"2024-05-09T08:26:18.203039+0000","last_undegraded":"2024-05-09T08:26:18.203039+0000","last_fullsized":"2024-05-09T08:26:18.203039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:43.515419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203660+0000","last_change":"2024-05-09T08:23:51.580568+0000","last_active":"2024-05-09T08:26:18.203660+0000","last_peered":"2024-05-09T08:26:18.203660+0000","last_clean":"2024-05-09T08:26:18.203660+0000","last_became_active":"2024-05-09T08:23:51.579626+0000","last_became_peered":"2024-05-09T08:23:51.579626+0000","last_unstale":"2024-05-09T08:26:18.203660+0000","last_undegraded":"2024-05-09T08:26:18.203660+0000","last_fullsized":"2024-05-09T08:26:18.203660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:28.756079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083992+0000","last_change":"2024-05-09T08:23:51.574363+0000","last_active":"2024-05-09T08:26:18.083992+0000","last_peered":"2024-05-09T08:26:18.083992+0000","last_clean":"2024-05-09T08:26:18.083992+0000","last_became_active":"2024-05-09T08:23:51.573858+0000","last_became_peered":"2024-05-09T08:23:51.573858+0000","last_unstale":"2024-05-09T08:26:18.083992+0000","last_undegraded":"2024-05-09T08:26:18.083992+0000","last_fullsized":"2024-05-09T08:26:18.083992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:30.752939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971361+0000","last_change":"2024-05-09T08:23:51.579361+0000","last_active":"2024-05-09T08:26:17.971361+0000","last_peered":"2024-05-09T08:26:17.971361+0000","last_clean":"2024-05-09T08:26:17.971361+0000","last_became_active":"2024-05-09T08:23:51.579025+0000","last_became_peered":"2024-05-09T08:23:51.579025+0000","last_unstale":"2024-05-09T08:26:17.971361+0000","last_undegraded":"2024-05-09T08:26:17.971361+0000","last_fullsized":"2024-05-09T08:26:17.971361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:02.273999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971573+0000","last_change":"2024-05-09T08:23:55.670107+0000","last_active":"2024-05-09T08:26:17.971573+0000","last_peered":"2024-05-09T08:26:17.971573+0000","last_clean":"2024-05-09T08:26:17.971573+0000","last_became_active":"2024-05-09T08:23:55.669621+0000","last_became_peered":"2024-05-09T08:23:55.669621+0000","last_unstale":"2024-05-09T08:26:17.971573+0000","last_undegraded":"2024-05-09T08:26:17.971573+0000","last_fullsized":"2024-05-09T08:26:17.971573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:00.547137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'14","reported_seq":189,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.253755+0000","last_change":"2024-05-09T08:24:51.498795+0000","last_active":"2024-05-09T08:26:18.253755+0000","last_peered":"2024-05-09T08:26:18.253755+0000","last_clean":"2024-05-09T08:26:18.253755+0000","last_became_active":"2024-05-09T08:24:51.487098+0000","last_became_peered":"2024-05-09T08:24:51.487098+0000","last_unstale":"2024-05-09T08:26:18.253755+0000","last_undegraded":"2024-05-09T08:26:18.253755+0000","last_fullsized":"2024-05-09T08:26:18.253755+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:10.646049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083803+0000","last_change":"2024-05-09T08:25:52.410940+0000","last_active":"2024-05-09T08:26:18.083803+0000","last_peered":"2024-05-09T08:26:18.083803+0000","last_clean":"2024-05-09T08:26:18.083803+0000","last_became_active":"2024-05-09T08:25:52.410514+0000","last_became_peered":"2024-05-09T08:25:52.410514+0000","last_unstale":"2024-05-09T08:26:18.083803+0000","last_undegraded":"2024-05-09T08:26:18.083803+0000","last_fullsized":"2024-05-09T08:26:18.083803+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:51.831125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971160+0000","last_change":"2024-05-09T08:23:55.670194+0000","last_active":"2024-05-09T08:26:17.971160+0000","last_peered":"2024-05-09T08:26:17.971160+0000","last_clean":"2024-05-09T08:26:17.971160+0000","last_became_active":"2024-05-09T08:23:55.669762+0000","last_became_peered":"2024-05-09T08:23:55.669762+0000","last_unstale":"2024-05-09T08:26:17.971160+0000","last_undegraded":"2024-05-09T08:26:17.971160+0000","last_fullsized":"2024-05-09T08:26:17.971160+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:22.879635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970723+0000","last_change":"2024-05-09T08:23:50.574421+0000","last_active":"2024-05-09T08:26:17.970723+0000","last_peered":"2024-05-09T08:26:17.970723+0000","last_clean":"2024-05-09T08:26:17.970723+0000","last_became_active":"2024-05-09T08:23:50.574070+0000","last_became_peered":"2024-05-09T08:23:50.574070+0000","last_unstale":"2024-05-09T08:26:17.970723+0000","last_undegraded":"2024-05-09T08:26:17.970723+0000","last_fullsized":"2024-05-09T08:26:17.970723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:22.996034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204912+0000","last_change":"2024-05-09T08:23:51.583275+0000","last_active":"2024-05-09T08:26:18.204912+0000","last_peered":"2024-05-09T08:26:18.204912+0000","last_clean":"2024-05-09T08:26:18.204912+0000","last_became_active":"2024-05-09T08:23:51.582847+0000","last_became_peered":"2024-05-09T08:23:51.582847+0000","last_unstale":"2024-05-09T08:26:18.204912+0000","last_undegraded":"2024-05-09T08:26:18.204912+0000","last_fullsized":"2024-05-09T08:26:18.204912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:03:03.512852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971336+0000","last_change":"2024-05-09T08:23:55.672256+0000","last_active":"2024-05-09T08:26:17.971336+0000","last_peered":"2024-05-09T08:26:17.971336+0000","last_clean":"2024-05-09T08:26:17.971336+0000","last_became_active":"2024-05-09T08:23:55.671855+0000","last_became_peered":"2024-05-09T08:23:55.671855+0000","last_unstale":"2024-05-09T08:26:17.971336+0000","last_undegraded":"2024-05-09T08:26:17.971336+0000","last_fullsized":"2024-05-09T08:26:17.971336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:45:12.199359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"94'13","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.087175+0000","last_change":"2024-05-09T08:23:50.566686+0000","last_active":"2024-05-09T08:26:18.087175+0000","last_peered":"2024-05-09T08:26:18.087175+0000","last_clean":"2024-05-09T08:26:18.087175+0000","last_became_active":"2024-05-09T08:23:50.565910+0000","last_became_peered":"2024-05-09T08:23:50.565910+0000","last_unstale":"2024-05-09T08:26:18.087175+0000","last_undegraded":"2024-05-09T08:26:18.087175+0000","last_fullsized":"2024-05-09T08:26:18.087175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:02.211551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203319+0000","last_change":"2024-05-09T08:23:51.570972+0000","last_active":"2024-05-09T08:26:18.203319+0000","last_peered":"2024-05-09T08:26:18.203319+0000","last_clean":"2024-05-09T08:26:18.203319+0000","last_became_active":"2024-05-09T08:23:51.570638+0000","last_became_peered":"2024-05-09T08:23:51.570638+0000","last_unstale":"2024-05-09T08:26:18.203319+0000","last_undegraded":"2024-05-09T08:26:18.203319+0000","last_fullsized":"2024-05-09T08:26:18.203319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:38.738820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971213+0000","last_change":"2024-05-09T08:23:55.671929+0000","last_active":"2024-05-09T08:26:17.971213+0000","last_peered":"2024-05-09T08:26:17.971213+0000","last_clean":"2024-05-09T08:26:17.971213+0000","last_became_active":"2024-05-09T08:23:55.671483+0000","last_became_peered":"2024-05-09T08:23:55.671483+0000","last_unstale":"2024-05-09T08:26:17.971213+0000","last_undegraded":"2024-05-09T08:26:17.971213+0000","last_fullsized":"2024-05-09T08:26:17.971213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:21.527266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086737+0000","last_change":"2024-05-09T08:23:50.565644+0000","last_active":"2024-05-09T08:26:18.086737+0000","last_peered":"2024-05-09T08:26:18.086737+0000","last_clean":"2024-05-09T08:26:18.086737+0000","last_became_active":"2024-05-09T08:23:50.565265+0000","last_became_peered":"2024-05-09T08:23:50.565265+0000","last_unstale":"2024-05-09T08:26:18.086737+0000","last_undegraded":"2024-05-09T08:26:18.086737+0000","last_fullsized":"2024-05-09T08:26:18.086737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:51:57.972535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092603+0000","last_change":"2024-05-09T08:23:51.571732+0000","last_active":"2024-05-09T08:26:18.092603+0000","last_peered":"2024-05-09T08:26:18.092603+0000","last_clean":"2024-05-09T08:26:18.092603+0000","last_became_active":"2024-05-09T08:23:51.570783+0000","last_became_peered":"2024-05-09T08:23:51.570783+0000","last_unstale":"2024-05-09T08:26:18.092603+0000","last_undegraded":"2024-05-09T08:26:18.092603+0000","last_fullsized":"2024-05-09T08:26:18.092603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:18.515905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971228+0000","last_change":"2024-05-09T08:23:55.666727+0000","last_active":"2024-05-09T08:26:17.971228+0000","last_peered":"2024-05-09T08:26:17.971228+0000","last_clean":"2024-05-09T08:26:17.971228+0000","last_became_active":"2024-05-09T08:23:55.666284+0000","last_became_peered":"2024-05-09T08:23:55.666284+0000","last_unstale":"2024-05-09T08:26:17.971228+0000","last_undegraded":"2024-05-09T08:26:17.971228+0000","last_fullsized":"2024-05-09T08:26:17.971228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:05.355404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.595864+0000","last_change":"2024-05-09T08:23:50.573308+0000","last_active":"2024-05-09T08:26:18.595864+0000","last_peered":"2024-05-09T08:26:18.595864+0000","last_clean":"2024-05-09T08:26:18.595864+0000","last_became_active":"2024-05-09T08:23:50.572946+0000","last_became_peered":"2024-05-09T08:23:50.572946+0000","last_unstale":"2024-05-09T08:26:18.595864+0000","last_undegraded":"2024-05-09T08:26:18.595864+0000","last_fullsized":"2024-05-09T08:26:18.595864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:44:35.816620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086752+0000","last_change":"2024-05-09T08:23:51.571748+0000","last_active":"2024-05-09T08:26:18.086752+0000","last_peered":"2024-05-09T08:26:18.086752+0000","last_clean":"2024-05-09T08:26:18.086752+0000","last_became_active":"2024-05-09T08:23:51.571388+0000","last_became_peered":"2024-05-09T08:23:51.571388+0000","last_unstale":"2024-05-09T08:26:18.086752+0000","last_undegraded":"2024-05-09T08:26:18.086752+0000","last_fullsized":"2024-05-09T08:26:18.086752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:34.916491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971219+0000","last_change":"2024-05-09T08:23:55.674022+0000","last_active":"2024-05-09T08:26:17.971219+0000","last_peered":"2024-05-09T08:26:17.971219+0000","last_clean":"2024-05-09T08:26:17.971219+0000","last_became_active":"2024-05-09T08:23:55.673322+0000","last_became_peered":"2024-05-09T08:23:55.673322+0000","last_unstale":"2024-05-09T08:26:17.971219+0000","last_undegraded":"2024-05-09T08:26:17.971219+0000","last_fullsized":"2024-05-09T08:26:17.971219+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:59:14.299886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.095836+0000","last_change":"2024-05-09T08:23:50.571235+0000","last_active":"2024-05-09T08:26:18.095836+0000","last_peered":"2024-05-09T08:26:18.095836+0000","last_clean":"2024-05-09T08:26:18.095836+0000","last_became_active":"2024-05-09T08:23:50.570694+0000","last_became_peered":"2024-05-09T08:23:50.570694+0000","last_unstale":"2024-05-09T08:26:18.095836+0000","last_undegraded":"2024-05-09T08:26:18.095836+0000","last_fullsized":"2024-05-09T08:26:18.095836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:22.850248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092662+0000","last_change":"2024-05-09T08:25:52.410264+0000","last_active":"2024-05-09T08:26:18.092662+0000","last_peered":"2024-05-09T08:26:18.092662+0000","last_clean":"2024-05-09T08:26:18.092662+0000","last_became_active":"2024-05-09T08:25:52.409851+0000","last_became_peered":"2024-05-09T08:25:52.409851+0000","last_unstale":"2024-05-09T08:26:18.092662+0000","last_undegraded":"2024-05-09T08:26:18.092662+0000","last_fullsized":"2024-05-09T08:26:18.092662+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:20:56.614365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203490+0000","last_change":"2024-05-09T08:24:51.489278+0000","last_active":"2024-05-09T08:26:18.203490+0000","last_peered":"2024-05-09T08:26:18.203490+0000","last_clean":"2024-05-09T08:26:18.203490+0000","last_became_active":"2024-05-09T08:24:51.488875+0000","last_became_peered":"2024-05-09T08:24:51.488875+0000","last_unstale":"2024-05-09T08:26:18.203490+0000","last_undegraded":"2024-05-09T08:26:18.203490+0000","last_fullsized":"2024-05-09T08:26:18.203490+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:47:45.306525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203847+0000","last_change":"2024-05-09T08:23:50.570040+0000","last_active":"2024-05-09T08:26:18.203847+0000","last_peered":"2024-05-09T08:26:18.203847+0000","last_clean":"2024-05-09T08:26:18.203847+0000","last_became_active":"2024-05-09T08:23:50.569665+0000","last_became_peered":"2024-05-09T08:23:50.569665+0000","last_unstale":"2024-05-09T08:26:18.203847+0000","last_undegraded":"2024-05-09T08:26:18.203847+0000","last_fullsized":"2024-05-09T08:26:18.203847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:22:29.639712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970223+0000","last_change":"2024-05-09T08:23:51.568561+0000","last_active":"2024-05-09T08:26:17.970223+0000","last_peered":"2024-05-09T08:26:17.970223+0000","last_clean":"2024-05-09T08:26:17.970223+0000","last_became_active":"2024-05-09T08:23:51.568196+0000","last_became_peered":"2024-05-09T08:23:51.568196+0000","last_unstale":"2024-05-09T08:26:17.970223+0000","last_undegraded":"2024-05-09T08:26:17.970223+0000","last_fullsized":"2024-05-09T08:26:17.970223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:49:52.132837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206099+0000","last_change":"2024-05-09T08:23:55.674960+0000","last_active":"2024-05-09T08:26:18.206099+0000","last_peered":"2024-05-09T08:26:18.206099+0000","last_clean":"2024-05-09T08:26:18.206099+0000","last_became_active":"2024-05-09T08:23:55.674509+0000","last_became_peered":"2024-05-09T08:23:55.674509+0000","last_unstale":"2024-05-09T08:26:18.206099+0000","last_undegraded":"2024-05-09T08:26:18.206099+0000","last_fullsized":"2024-05-09T08:26:18.206099+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:22.479867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092234+0000","last_change":"2024-05-09T08:23:50.569896+0000","last_active":"2024-05-09T08:26:18.092234+0000","last_peered":"2024-05-09T08:26:18.092234+0000","last_clean":"2024-05-09T08:26:18.092234+0000","last_became_active":"2024-05-09T08:23:50.569562+0000","last_became_peered":"2024-05-09T08:23:50.569562+0000","last_unstale":"2024-05-09T08:26:18.092234+0000","last_undegraded":"2024-05-09T08:26:18.092234+0000","last_fullsized":"2024-05-09T08:26:18.092234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:25:20.278726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083524+0000","last_change":"2024-05-09T08:23:51.570260+0000","last_active":"2024-05-09T08:26:18.083524+0000","last_peered":"2024-05-09T08:26:18.083524+0000","last_clean":"2024-05-09T08:26:18.083524+0000","last_became_active":"2024-05-09T08:23:51.569837+0000","last_became_peered":"2024-05-09T08:23:51.569837+0000","last_unstale":"2024-05-09T08:26:18.083524+0000","last_undegraded":"2024-05-09T08:26:18.083524+0000","last_fullsized":"2024-05-09T08:26:18.083524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:01:44.805271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970259+0000","last_change":"2024-05-09T08:23:55.676639+0000","last_active":"2024-05-09T08:26:17.970259+0000","last_peered":"2024-05-09T08:26:17.970259+0000","last_clean":"2024-05-09T08:26:17.970259+0000","last_became_active":"2024-05-09T08:23:55.673643+0000","last_became_peered":"2024-05-09T08:23:55.673643+0000","last_unstale":"2024-05-09T08:26:17.970259+0000","last_undegraded":"2024-05-09T08:26:17.970259+0000","last_fullsized":"2024-05-09T08:26:17.970259+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:02.377184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'6","reported_seq":280,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.595264+0000","last_change":"2024-05-09T08:23:50.565745+0000","last_active":"2024-05-09T08:26:18.595264+0000","last_peered":"2024-05-09T08:26:18.595264+0000","last_clean":"2024-05-09T08:26:18.595264+0000","last_became_active":"2024-05-09T08:23:50.565385+0000","last_became_peered":"2024-05-09T08:23:50.565385+0000","last_unstale":"2024-05-09T08:26:18.595264+0000","last_undegraded":"2024-05-09T08:26:18.595264+0000","last_fullsized":"2024-05-09T08:26:18.595264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:05:00.818398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":240,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203490+0000","last_change":"2024-05-09T08:23:51.572341+0000","last_active":"2024-05-09T08:26:18.203490+0000","last_peered":"2024-05-09T08:26:18.203490+0000","last_clean":"2024-05-09T08:26:18.203490+0000","last_became_active":"2024-05-09T08:23:51.571583+0000","last_became_peered":"2024-05-09T08:23:51.571583+0000","last_unstale":"2024-05-09T08:26:18.203490+0000","last_undegraded":"2024-05-09T08:26:18.203490+0000","last_fullsized":"2024-05-09T08:26:18.203490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:20.695995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083374+0000","last_change":"2024-05-09T08:23:55.677603+0000","last_active":"2024-05-09T08:26:18.083374+0000","last_peered":"2024-05-09T08:26:18.083374+0000","last_clean":"2024-05-09T08:26:18.083374+0000","last_became_active":"2024-05-09T08:23:55.677143+0000","last_became_peered":"2024-05-09T08:23:55.677143+0000","last_unstale":"2024-05-09T08:26:18.083374+0000","last_undegraded":"2024-05-09T08:26:18.083374+0000","last_fullsized":"2024-05-09T08:26:18.083374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:27.850997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.188009+0000","last_change":"2024-05-09T08:23:50.565497+0000","last_active":"2024-05-09T08:26:18.188009+0000","last_peered":"2024-05-09T08:26:18.188009+0000","last_clean":"2024-05-09T08:26:18.188009+0000","last_became_active":"2024-05-09T08:23:50.565120+0000","last_became_peered":"2024-05-09T08:23:50.565120+0000","last_unstale":"2024-05-09T08:26:18.188009+0000","last_undegraded":"2024-05-09T08:26:18.188009+0000","last_fullsized":"2024-05-09T08:26:18.188009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:56.141997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206840+0000","last_change":"2024-05-09T08:23:51.576489+0000","last_active":"2024-05-09T08:26:18.206840+0000","last_peered":"2024-05-09T08:26:18.206840+0000","last_clean":"2024-05-09T08:26:18.206840+0000","last_became_active":"2024-05-09T08:23:51.576145+0000","last_became_peered":"2024-05-09T08:23:51.576145+0000","last_unstale":"2024-05-09T08:26:18.206840+0000","last_undegraded":"2024-05-09T08:26:18.206840+0000","last_fullsized":"2024-05-09T08:26:18.206840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:37.650308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971493+0000","last_change":"2024-05-09T08:24:51.490957+0000","last_active":"2024-05-09T08:26:17.971493+0000","last_peered":"2024-05-09T08:26:17.971493+0000","last_clean":"2024-05-09T08:26:17.971493+0000","last_became_active":"2024-05-09T08:24:51.490541+0000","last_became_peered":"2024-05-09T08:24:51.490541+0000","last_unstale":"2024-05-09T08:26:17.971493+0000","last_undegraded":"2024-05-09T08:26:17.971493+0000","last_fullsized":"2024-05-09T08:26:17.971493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:03.168547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092549+0000","last_change":"2024-05-09T08:23:50.567182+0000","last_active":"2024-05-09T08:26:18.092549+0000","last_peered":"2024-05-09T08:26:18.092549+0000","last_clean":"2024-05-09T08:26:18.092549+0000","last_became_active":"2024-05-09T08:23:50.566851+0000","last_became_peered":"2024-05-09T08:23:50.566851+0000","last_unstale":"2024-05-09T08:26:18.092549+0000","last_undegraded":"2024-05-09T08:26:18.092549+0000","last_fullsized":"2024-05-09T08:26:18.092549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:40:26.540456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203499+0000","last_change":"2024-05-09T08:23:51.574074+0000","last_active":"2024-05-09T08:26:18.203499+0000","last_peered":"2024-05-09T08:26:18.203499+0000","last_clean":"2024-05-09T08:26:18.203499+0000","last_became_active":"2024-05-09T08:23:51.572587+0000","last_became_peered":"2024-05-09T08:23:51.572587+0000","last_unstale":"2024-05-09T08:26:18.203499+0000","last_undegraded":"2024-05-09T08:26:18.203499+0000","last_fullsized":"2024-05-09T08:26:18.203499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:46.186358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083525+0000","last_change":"2024-05-09T08:23:55.667170+0000","last_active":"2024-05-09T08:26:18.083525+0000","last_peered":"2024-05-09T08:26:18.083525+0000","last_clean":"2024-05-09T08:26:18.083525+0000","last_became_active":"2024-05-09T08:23:55.666707+0000","last_became_peered":"2024-05-09T08:23:55.666707+0000","last_unstale":"2024-05-09T08:26:18.083525+0000","last_undegraded":"2024-05-09T08:26:18.083525+0000","last_fullsized":"2024-05-09T08:26:18.083525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:48.332207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":345,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.609304+0000","last_change":"2024-05-09T08:24:51.509495+0000","last_active":"2024-05-09T08:26:18.609304+0000","last_peered":"2024-05-09T08:26:18.609304+0000","last_clean":"2024-05-09T08:26:18.609304+0000","last_became_active":"2024-05-09T08:24:51.491187+0000","last_became_peered":"2024-05-09T08:24:51.491187+0000","last_unstale":"2024-05-09T08:26:18.609304+0000","last_undegraded":"2024-05-09T08:26:18.609304+0000","last_fullsized":"2024-05-09T08:26:18.609304+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:01:55.320000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":241,"num_read_kb":190,"num_write":44,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":63356,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.087061+0000","last_change":"2024-05-09T08:23:51.577595+0000","last_active":"2024-05-09T08:26:18.087061+0000","last_peered":"2024-05-09T08:26:18.087061+0000","last_clean":"2024-05-09T08:26:18.087061+0000","last_became_active":"2024-05-09T08:23:51.577271+0000","last_became_peered":"2024-05-09T08:23:51.577271+0000","last_unstale":"2024-05-09T08:26:18.087061+0000","last_undegraded":"2024-05-09T08:26:18.087061+0000","last_fullsized":"2024-05-09T08:26:18.087061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:34.679111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092985+0000","last_change":"2024-05-09T08:23:55.671128+0000","last_active":"2024-05-09T08:26:18.092985+0000","last_peered":"2024-05-09T08:26:18.092985+0000","last_clean":"2024-05-09T08:26:18.092985+0000","last_became_active":"2024-05-09T08:23:55.670414+0000","last_became_peered":"2024-05-09T08:23:55.670414+0000","last_unstale":"2024-05-09T08:26:18.092985+0000","last_undegraded":"2024-05-09T08:26:18.092985+0000","last_fullsized":"2024-05-09T08:26:18.092985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:31:55.307412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971568+0000","last_change":"2024-05-09T08:23:50.568503+0000","last_active":"2024-05-09T08:26:17.971568+0000","last_peered":"2024-05-09T08:26:17.971568+0000","last_clean":"2024-05-09T08:26:17.971568+0000","last_became_active":"2024-05-09T08:23:50.568181+0000","last_became_peered":"2024-05-09T08:23:50.568181+0000","last_unstale":"2024-05-09T08:26:17.971568+0000","last_undegraded":"2024-05-09T08:26:17.971568+0000","last_fullsized":"2024-05-09T08:26:17.971568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:40.883792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204000+0000","last_change":"2024-05-09T08:23:51.573602+0000","last_active":"2024-05-09T08:26:18.204000+0000","last_peered":"2024-05-09T08:26:18.204000+0000","last_clean":"2024-05-09T08:26:18.204000+0000","last_became_active":"2024-05-09T08:23:51.572557+0000","last_became_peered":"2024-05-09T08:23:51.572557+0000","last_unstale":"2024-05-09T08:26:18.204000+0000","last_undegraded":"2024-05-09T08:26:18.204000+0000","last_fullsized":"2024-05-09T08:26:18.204000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.170962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203332+0000","last_change":"2024-05-09T08:23:55.673773+0000","last_active":"2024-05-09T08:26:18.203332+0000","last_peered":"2024-05-09T08:26:18.203332+0000","last_clean":"2024-05-09T08:26:18.203332+0000","last_became_active":"2024-05-09T08:23:55.673276+0000","last_became_peered":"2024-05-09T08:23:55.673276+0000","last_unstale":"2024-05-09T08:26:18.203332+0000","last_undegraded":"2024-05-09T08:26:18.203332+0000","last_fullsized":"2024-05-09T08:26:18.203332+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:59:00.629339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092468+0000","last_change":"2024-05-09T08:23:50.575894+0000","last_active":"2024-05-09T08:26:18.092468+0000","last_peered":"2024-05-09T08:26:18.092468+0000","last_clean":"2024-05-09T08:26:18.092468+0000","last_became_active":"2024-05-09T08:23:50.575555+0000","last_became_peered":"2024-05-09T08:23:50.575555+0000","last_unstale":"2024-05-09T08:26:18.092468+0000","last_undegraded":"2024-05-09T08:26:18.092468+0000","last_fullsized":"2024-05-09T08:26:18.092468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:19.070764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187948+0000","last_change":"2024-05-09T08:23:51.574125+0000","last_active":"2024-05-09T08:26:18.187948+0000","last_peered":"2024-05-09T08:26:18.187948+0000","last_clean":"2024-05-09T08:26:18.187948+0000","last_became_active":"2024-05-09T08:23:51.573794+0000","last_became_peered":"2024-05-09T08:23:51.573794+0000","last_unstale":"2024-05-09T08:26:18.187948+0000","last_undegraded":"2024-05-09T08:26:18.187948+0000","last_fullsized":"2024-05-09T08:26:18.187948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:19:26.634208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.084105+0000","last_change":"2024-05-09T08:23:55.677711+0000","last_active":"2024-05-09T08:26:18.084105+0000","last_peered":"2024-05-09T08:26:18.084105+0000","last_clean":"2024-05-09T08:26:18.084105+0000","last_became_active":"2024-05-09T08:23:55.677058+0000","last_became_peered":"2024-05-09T08:23:55.677058+0000","last_unstale":"2024-05-09T08:26:18.084105+0000","last_undegraded":"2024-05-09T08:26:18.084105+0000","last_fullsized":"2024-05-09T08:26:18.084105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:26.842145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'9","reported_seq":167,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.599726+0000","last_change":"2024-05-09T08:23:50.566413+0000","last_active":"2024-05-09T08:26:18.599726+0000","last_peered":"2024-05-09T08:26:18.599726+0000","last_clean":"2024-05-09T08:26:18.599726+0000","last_became_active":"2024-05-09T08:23:50.566047+0000","last_became_peered":"2024-05-09T08:23:50.566047+0000","last_unstale":"2024-05-09T08:26:18.599726+0000","last_undegraded":"2024-05-09T08:26:18.599726+0000","last_fullsized":"2024-05-09T08:26:18.599726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:15.340947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":5,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187600+0000","last_change":"2024-05-09T08:23:51.576779+0000","last_active":"2024-05-09T08:26:18.187600+0000","last_peered":"2024-05-09T08:26:18.187600+0000","last_clean":"2024-05-09T08:26:18.187600+0000","last_became_active":"2024-05-09T08:23:51.576119+0000","last_became_peered":"2024-05-09T08:23:51.576119+0000","last_unstale":"2024-05-09T08:26:18.187600+0000","last_undegraded":"2024-05-09T08:26:18.187600+0000","last_fullsized":"2024-05-09T08:26:18.187600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:02.250135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204904+0000","last_change":"2024-05-09T08:23:55.678564+0000","last_active":"2024-05-09T08:26:18.204904+0000","last_peered":"2024-05-09T08:26:18.204904+0000","last_clean":"2024-05-09T08:26:18.204904+0000","last_became_active":"2024-05-09T08:23:55.678128+0000","last_became_peered":"2024-05-09T08:23:55.678128+0000","last_unstale":"2024-05-09T08:26:18.204904+0000","last_undegraded":"2024-05-09T08:26:18.204904+0000","last_fullsized":"2024-05-09T08:26:18.204904+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:13:21.868267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086552+0000","last_change":"2024-05-09T08:24:51.501243+0000","last_active":"2024-05-09T08:26:18.086552+0000","last_peered":"2024-05-09T08:26:18.086552+0000","last_clean":"2024-05-09T08:26:18.086552+0000","last_became_active":"2024-05-09T08:24:51.490110+0000","last_became_peered":"2024-05-09T08:24:51.490110+0000","last_unstale":"2024-05-09T08:26:18.086552+0000","last_undegraded":"2024-05-09T08:26:18.086552+0000","last_fullsized":"2024-05-09T08:26:18.086552+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:03.086570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092330+0000","last_change":"2024-05-09T08:23:51.574914+0000","last_active":"2024-05-09T08:26:18.092330+0000","last_peered":"2024-05-09T08:26:18.092330+0000","last_clean":"2024-05-09T08:26:18.092330+0000","last_became_active":"2024-05-09T08:23:51.574380+0000","last_became_peered":"2024-05-09T08:23:51.574380+0000","last_unstale":"2024-05-09T08:26:18.092330+0000","last_undegraded":"2024-05-09T08:26:18.092330+0000","last_fullsized":"2024-05-09T08:26:18.092330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:34.698328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092194+0000","last_change":"2024-05-09T08:23:55.670198+0000","last_active":"2024-05-09T08:26:18.092194+0000","last_peered":"2024-05-09T08:26:18.092194+0000","last_clean":"2024-05-09T08:26:18.092194+0000","last_became_active":"2024-05-09T08:23:55.669798+0000","last_became_peered":"2024-05-09T08:23:55.669798+0000","last_unstale":"2024-05-09T08:26:18.092194+0000","last_undegraded":"2024-05-09T08:26:18.092194+0000","last_fullsized":"2024-05-09T08:26:18.092194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:09.474264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203593+0000","last_change":"2024-05-09T08:24:51.504286+0000","last_active":"2024-05-09T08:26:18.203593+0000","last_peered":"2024-05-09T08:26:18.203593+0000","last_clean":"2024-05-09T08:26:18.203593+0000","last_became_active":"2024-05-09T08:24:51.489111+0000","last_became_peered":"2024-05-09T08:24:51.489111+0000","last_unstale":"2024-05-09T08:26:18.203593+0000","last_undegraded":"2024-05-09T08:26:18.203593+0000","last_fullsized":"2024-05-09T08:26:18.203593+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:14.975959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203398+0000","last_change":"2024-05-09T08:23:51.573434+0000","last_active":"2024-05-09T08:26:18.203398+0000","last_peered":"2024-05-09T08:26:18.203398+0000","last_clean":"2024-05-09T08:26:18.203398+0000","last_became_active":"2024-05-09T08:23:51.573081+0000","last_became_peered":"2024-05-09T08:23:51.573081+0000","last_unstale":"2024-05-09T08:26:18.203398+0000","last_undegraded":"2024-05-09T08:26:18.203398+0000","last_fullsized":"2024-05-09T08:26:18.203398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:49.270355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086717+0000","last_change":"2024-05-09T08:23:55.853114+0000","last_active":"2024-05-09T08:26:18.086717+0000","last_peered":"2024-05-09T08:26:18.086717+0000","last_clean":"2024-05-09T08:26:18.086717+0000","last_became_active":"2024-05-09T08:23:55.852640+0000","last_became_peered":"2024-05-09T08:23:55.852640+0000","last_unstale":"2024-05-09T08:26:18.086717+0000","last_undegraded":"2024-05-09T08:26:18.086717+0000","last_fullsized":"2024-05-09T08:26:18.086717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:55:17.823585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083798+0000","last_change":"2024-05-09T08:23:50.567834+0000","last_active":"2024-05-09T08:26:18.083798+0000","last_peered":"2024-05-09T08:26:18.083798+0000","last_clean":"2024-05-09T08:26:18.083798+0000","last_became_active":"2024-05-09T08:23:50.567538+0000","last_became_peered":"2024-05-09T08:23:50.567538+0000","last_unstale":"2024-05-09T08:26:18.083798+0000","last_undegraded":"2024-05-09T08:26:18.083798+0000","last_fullsized":"2024-05-09T08:26:18.083798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.674352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.096142+0000","last_change":"2024-05-09T08:23:51.580219+0000","last_active":"2024-05-09T08:26:18.096142+0000","last_peered":"2024-05-09T08:26:18.096142+0000","last_clean":"2024-05-09T08:26:18.096142+0000","last_became_active":"2024-05-09T08:23:51.579835+0000","last_became_peered":"2024-05-09T08:23:51.579835+0000","last_unstale":"2024-05-09T08:26:18.096142+0000","last_undegraded":"2024-05-09T08:26:18.096142+0000","last_fullsized":"2024-05-09T08:26:18.096142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:15.232318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204599+0000","last_change":"2024-05-09T08:23:55.674324+0000","last_active":"2024-05-09T08:26:18.204599+0000","last_peered":"2024-05-09T08:26:18.204599+0000","last_clean":"2024-05-09T08:26:18.204599+0000","last_became_active":"2024-05-09T08:23:55.673254+0000","last_became_peered":"2024-05-09T08:23:55.673254+0000","last_unstale":"2024-05-09T08:26:18.204599+0000","last_undegraded":"2024-05-09T08:26:18.204599+0000","last_fullsized":"2024-05-09T08:26:18.204599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:05:23.303987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'6","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.617307+0000","last_change":"2024-05-09T08:23:50.571682+0000","last_active":"2024-05-09T08:26:18.617307+0000","last_peered":"2024-05-09T08:26:18.617307+0000","last_clean":"2024-05-09T08:26:18.617307+0000","last_became_active":"2024-05-09T08:23:50.571338+0000","last_became_peered":"2024-05-09T08:23:50.571338+0000","last_unstale":"2024-05-09T08:26:18.617307+0000","last_undegraded":"2024-05-09T08:26:18.617307+0000","last_fullsized":"2024-05-09T08:26:18.617307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:17.578071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.095802+0000","last_change":"2024-05-09T08:24:51.487208+0000","last_active":"2024-05-09T08:26:18.095802+0000","last_peered":"2024-05-09T08:26:18.095802+0000","last_clean":"2024-05-09T08:26:18.095802+0000","last_became_active":"2024-05-09T08:24:51.486665+0000","last_became_peered":"2024-05-09T08:24:51.486665+0000","last_unstale":"2024-05-09T08:26:18.095802+0000","last_undegraded":"2024-05-09T08:26:18.095802+0000","last_fullsized":"2024-05-09T08:26:18.095802+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:01:10.050846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205221+0000","last_change":"2024-05-09T08:23:55.853086+0000","last_active":"2024-05-09T08:26:18.205221+0000","last_peered":"2024-05-09T08:26:18.205221+0000","last_clean":"2024-05-09T08:26:18.205221+0000","last_became_active":"2024-05-09T08:23:55.852302+0000","last_became_peered":"2024-05-09T08:23:55.852302+0000","last_unstale":"2024-05-09T08:26:18.205221+0000","last_undegraded":"2024-05-09T08:26:18.205221+0000","last_fullsized":"2024-05-09T08:26:18.205221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:42:18.547623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092340+0000","last_change":"2024-05-09T08:23:51.575522+0000","last_active":"2024-05-09T08:26:18.092340+0000","last_peered":"2024-05-09T08:26:18.092340+0000","last_clean":"2024-05-09T08:26:18.092340+0000","last_became_active":"2024-05-09T08:23:51.575204+0000","last_became_peered":"2024-05-09T08:23:51.575204+0000","last_unstale":"2024-05-09T08:26:18.092340+0000","last_undegraded":"2024-05-09T08:26:18.092340+0000","last_fullsized":"2024-05-09T08:26:18.092340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:57.257435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'7","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.253210+0000","last_change":"2024-05-09T08:23:50.574996+0000","last_active":"2024-05-09T08:26:18.253210+0000","last_peered":"2024-05-09T08:26:18.253210+0000","last_clean":"2024-05-09T08:26:18.253210+0000","last_became_active":"2024-05-09T08:23:50.574619+0000","last_became_peered":"2024-05-09T08:23:50.574619+0000","last_unstale":"2024-05-09T08:26:18.253210+0000","last_undegraded":"2024-05-09T08:26:18.253210+0000","last_fullsized":"2024-05-09T08:26:18.253210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:44.681314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187892+0000","last_change":"2024-05-09T08:24:51.493390+0000","last_active":"2024-05-09T08:26:18.187892+0000","last_peered":"2024-05-09T08:26:18.187892+0000","last_clean":"2024-05-09T08:26:18.187892+0000","last_became_active":"2024-05-09T08:24:51.487797+0000","last_became_peered":"2024-05-09T08:24:51.487797+0000","last_unstale":"2024-05-09T08:26:18.187892+0000","last_undegraded":"2024-05-09T08:26:18.187892+0000","last_fullsized":"2024-05-09T08:26:18.187892+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:46.646471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.083550+0000","last_change":"2024-05-09T08:25:52.411415+0000","last_active":"2024-05-09T08:26:18.083550+0000","last_peered":"2024-05-09T08:26:18.083550+0000","last_clean":"2024-05-09T08:26:18.083550+0000","last_became_active":"2024-05-09T08:25:52.410972+0000","last_became_peered":"2024-05-09T08:25:52.410972+0000","last_unstale":"2024-05-09T08:26:18.083550+0000","last_undegraded":"2024-05-09T08:26:18.083550+0000","last_fullsized":"2024-05-09T08:26:18.083550+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:31.045318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971384+0000","last_change":"2024-05-09T08:23:50.566415+0000","last_active":"2024-05-09T08:26:17.971384+0000","last_peered":"2024-05-09T08:26:17.971384+0000","last_clean":"2024-05-09T08:26:17.971384+0000","last_became_active":"2024-05-09T08:23:50.565506+0000","last_became_peered":"2024-05-09T08:23:50.565506+0000","last_unstale":"2024-05-09T08:26:17.971384+0000","last_undegraded":"2024-05-09T08:26:17.971384+0000","last_fullsized":"2024-05-09T08:26:17.971384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:15.741565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187381+0000","last_change":"2024-05-09T08:22:38.366126+0000","last_active":"2024-05-09T08:26:18.187381+0000","last_peered":"2024-05-09T08:26:18.187381+0000","last_clean":"2024-05-09T08:26:18.187381+0000","last_became_active":"2024-05-09T08:22:38.365683+0000","last_became_peered":"2024-05-09T08:22:38.365683+0000","last_unstale":"2024-05-09T08:26:18.187381+0000","last_undegraded":"2024-05-09T08:26:18.187381+0000","last_fullsized":"2024-05-09T08:26:18.187381+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:19:12.435445+0000","last_clean_scrub_stamp":"2024-05-09T08:19:12.435445+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:46:04.744060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092887+0000","last_change":"2024-05-09T08:23:55.853248+0000","last_active":"2024-05-09T08:26:18.092887+0000","last_peered":"2024-05-09T08:26:18.092887+0000","last_clean":"2024-05-09T08:26:18.092887+0000","last_became_active":"2024-05-09T08:23:55.852646+0000","last_became_peered":"2024-05-09T08:23:55.852646+0000","last_unstale":"2024-05-09T08:26:18.092887+0000","last_undegraded":"2024-05-09T08:26:18.092887+0000","last_fullsized":"2024-05-09T08:26:18.092887+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:19:54.378185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203614+0000","last_change":"2024-05-09T08:23:51.574967+0000","last_active":"2024-05-09T08:26:18.203614+0000","last_peered":"2024-05-09T08:26:18.203614+0000","last_clean":"2024-05-09T08:26:18.203614+0000","last_became_active":"2024-05-09T08:23:51.574600+0000","last_became_peered":"2024-05-09T08:23:51.574600+0000","last_unstale":"2024-05-09T08:26:18.203614+0000","last_undegraded":"2024-05-09T08:26:18.203614+0000","last_fullsized":"2024-05-09T08:26:18.203614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:03.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187411+0000","last_change":"2024-05-09T08:23:50.569886+0000","last_active":"2024-05-09T08:26:18.187411+0000","last_peered":"2024-05-09T08:26:18.187411+0000","last_clean":"2024-05-09T08:26:18.187411+0000","last_became_active":"2024-05-09T08:23:50.569018+0000","last_became_peered":"2024-05-09T08:23:50.569018+0000","last_unstale":"2024-05-09T08:26:18.187411+0000","last_undegraded":"2024-05-09T08:26:18.187411+0000","last_fullsized":"2024-05-09T08:26:18.187411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:20.711485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203347+0000","last_change":"2024-05-09T08:23:55.676408+0000","last_active":"2024-05-09T08:26:18.203347+0000","last_peered":"2024-05-09T08:26:18.203347+0000","last_clean":"2024-05-09T08:26:18.203347+0000","last_became_active":"2024-05-09T08:23:55.675964+0000","last_became_peered":"2024-05-09T08:23:55.675964+0000","last_unstale":"2024-05-09T08:26:18.203347+0000","last_undegraded":"2024-05-09T08:26:18.203347+0000","last_fullsized":"2024-05-09T08:26:18.203347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:29.210602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086991+0000","last_change":"2024-05-09T08:23:50.566946+0000","last_active":"2024-05-09T08:26:18.086991+0000","last_peered":"2024-05-09T08:26:18.086991+0000","last_clean":"2024-05-09T08:26:18.086991+0000","last_became_active":"2024-05-09T08:23:50.566366+0000","last_became_peered":"2024-05-09T08:23:50.566366+0000","last_unstale":"2024-05-09T08:26:18.086991+0000","last_undegraded":"2024-05-09T08:26:18.086991+0000","last_fullsized":"2024-05-09T08:26:18.086991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:37:30.720879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205041+0000","last_change":"2024-05-09T08:23:51.571520+0000","last_active":"2024-05-09T08:26:18.205041+0000","last_peered":"2024-05-09T08:26:18.205041+0000","last_clean":"2024-05-09T08:26:18.205041+0000","last_became_active":"2024-05-09T08:23:51.571166+0000","last_became_peered":"2024-05-09T08:23:51.571166+0000","last_unstale":"2024-05-09T08:26:18.205041+0000","last_undegraded":"2024-05-09T08:26:18.205041+0000","last_fullsized":"2024-05-09T08:26:18.205041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:02:12.560238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970123+0000","last_change":"2024-05-09T08:23:55.852649+0000","last_active":"2024-05-09T08:26:17.970123+0000","last_peered":"2024-05-09T08:26:17.970123+0000","last_clean":"2024-05-09T08:26:17.970123+0000","last_became_active":"2024-05-09T08:23:55.852180+0000","last_became_peered":"2024-05-09T08:23:55.852180+0000","last_unstale":"2024-05-09T08:26:17.970123+0000","last_undegraded":"2024-05-09T08:26:17.970123+0000","last_fullsized":"2024-05-09T08:26:17.970123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:33.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203198+0000","last_change":"2024-05-09T08:23:50.561639+0000","last_active":"2024-05-09T08:26:18.203198+0000","last_peered":"2024-05-09T08:26:18.203198+0000","last_clean":"2024-05-09T08:26:18.203198+0000","last_became_active":"2024-05-09T08:23:50.561306+0000","last_became_peered":"2024-05-09T08:23:50.561306+0000","last_unstale":"2024-05-09T08:26:18.203198+0000","last_undegraded":"2024-05-09T08:26:18.203198+0000","last_fullsized":"2024-05-09T08:26:18.203198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:46.897965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970871+0000","last_change":"2024-05-09T08:23:51.573765+0000","last_active":"2024-05-09T08:26:17.970871+0000","last_peered":"2024-05-09T08:26:17.970871+0000","last_clean":"2024-05-09T08:26:17.970871+0000","last_became_active":"2024-05-09T08:23:51.573134+0000","last_became_peered":"2024-05-09T08:23:51.573134+0000","last_unstale":"2024-05-09T08:26:17.970871+0000","last_undegraded":"2024-05-09T08:26:17.970871+0000","last_fullsized":"2024-05-09T08:26:17.970871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:25.334148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970989+0000","last_change":"2024-05-09T08:23:55.663480+0000","last_active":"2024-05-09T08:26:17.970989+0000","last_peered":"2024-05-09T08:26:17.970989+0000","last_clean":"2024-05-09T08:26:17.970989+0000","last_became_active":"2024-05-09T08:23:55.662823+0000","last_became_peered":"2024-05-09T08:23:55.662823+0000","last_unstale":"2024-05-09T08:26:17.970989+0000","last_undegraded":"2024-05-09T08:26:17.970989+0000","last_fullsized":"2024-05-09T08:26:17.970989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:18.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971781+0000","last_change":"2024-05-09T08:23:50.574641+0000","last_active":"2024-05-09T08:26:17.971781+0000","last_peered":"2024-05-09T08:26:17.971781+0000","last_clean":"2024-05-09T08:26:17.971781+0000","last_became_active":"2024-05-09T08:23:50.574266+0000","last_became_peered":"2024-05-09T08:23:50.574266+0000","last_unstale":"2024-05-09T08:26:17.971781+0000","last_undegraded":"2024-05-09T08:26:17.971781+0000","last_fullsized":"2024-05-09T08:26:17.971781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.575211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.206374+0000","last_change":"2024-05-09T08:25:52.412012+0000","last_active":"2024-05-09T08:26:18.206374+0000","last_peered":"2024-05-09T08:26:18.206374+0000","last_clean":"2024-05-09T08:26:18.206374+0000","last_became_active":"2024-05-09T08:25:52.411479+0000","last_became_peered":"2024-05-09T08:25:52.411479+0000","last_unstale":"2024-05-09T08:26:18.206374+0000","last_undegraded":"2024-05-09T08:26:18.206374+0000","last_fullsized":"2024-05-09T08:26:18.206374+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:58.577359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971567+0000","last_change":"2024-05-09T08:23:55.853759+0000","last_active":"2024-05-09T08:26:17.971567+0000","last_peered":"2024-05-09T08:26:17.971567+0000","last_clean":"2024-05-09T08:26:17.971567+0000","last_became_active":"2024-05-09T08:23:55.853045+0000","last_became_peered":"2024-05-09T08:23:55.853045+0000","last_unstale":"2024-05-09T08:26:17.971567+0000","last_undegraded":"2024-05-09T08:26:17.971567+0000","last_fullsized":"2024-05-09T08:26:17.971567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:53.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203004+0000","last_change":"2024-05-09T08:23:50.568373+0000","last_active":"2024-05-09T08:26:18.203004+0000","last_peered":"2024-05-09T08:26:18.203004+0000","last_clean":"2024-05-09T08:26:18.203004+0000","last_became_active":"2024-05-09T08:23:50.568015+0000","last_became_peered":"2024-05-09T08:23:50.568015+0000","last_unstale":"2024-05-09T08:26:18.203004+0000","last_undegraded":"2024-05-09T08:26:18.203004+0000","last_fullsized":"2024-05-09T08:26:18.203004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:05:16.391673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086521+0000","last_change":"2024-05-09T08:23:51.576825+0000","last_active":"2024-05-09T08:26:18.086521+0000","last_peered":"2024-05-09T08:26:18.086521+0000","last_clean":"2024-05-09T08:26:18.086521+0000","last_became_active":"2024-05-09T08:23:51.573350+0000","last_became_peered":"2024-05-09T08:23:51.573350+0000","last_unstale":"2024-05-09T08:26:18.086521+0000","last_undegraded":"2024-05-09T08:26:18.086521+0000","last_fullsized":"2024-05-09T08:26:18.086521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:34:00.963281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.084500+0000","last_change":"2024-05-09T08:23:55.672145+0000","last_active":"2024-05-09T08:26:18.084500+0000","last_peered":"2024-05-09T08:26:18.084500+0000","last_clean":"2024-05-09T08:26:18.084500+0000","last_became_active":"2024-05-09T08:23:55.671743+0000","last_became_peered":"2024-05-09T08:23:55.671743+0000","last_unstale":"2024-05-09T08:26:18.084500+0000","last_undegraded":"2024-05-09T08:26:18.084500+0000","last_fullsized":"2024-05-09T08:26:18.084500+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:09:23.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970351+0000","last_change":"2024-05-09T08:23:50.568908+0000","last_active":"2024-05-09T08:26:17.970351+0000","last_peered":"2024-05-09T08:26:17.970351+0000","last_clean":"2024-05-09T08:26:17.970351+0000","last_became_active":"2024-05-09T08:23:50.567009+0000","last_became_peered":"2024-05-09T08:23:50.567009+0000","last_unstale":"2024-05-09T08:26:17.970351+0000","last_undegraded":"2024-05-09T08:26:17.970351+0000","last_fullsized":"2024-05-09T08:26:17.970351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:35.190156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203170+0000","last_change":"2024-05-09T08:23:51.580121+0000","last_active":"2024-05-09T08:26:18.203170+0000","last_peered":"2024-05-09T08:26:18.203170+0000","last_clean":"2024-05-09T08:26:18.203170+0000","last_became_active":"2024-05-09T08:23:51.579402+0000","last_became_peered":"2024-05-09T08:23:51.579402+0000","last_unstale":"2024-05-09T08:26:18.203170+0000","last_undegraded":"2024-05-09T08:26:18.203170+0000","last_fullsized":"2024-05-09T08:26:18.203170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:11:26.144344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971820+0000","last_change":"2024-05-09T08:23:55.664017+0000","last_active":"2024-05-09T08:26:17.971820+0000","last_peered":"2024-05-09T08:26:17.971820+0000","last_clean":"2024-05-09T08:26:17.971820+0000","last_became_active":"2024-05-09T08:23:55.663546+0000","last_became_peered":"2024-05-09T08:23:55.663546+0000","last_unstale":"2024-05-09T08:26:17.971820+0000","last_undegraded":"2024-05-09T08:26:17.971820+0000","last_fullsized":"2024-05-09T08:26:17.971820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:13:14.705971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203214+0000","last_change":"2024-05-09T08:24:51.493220+0000","last_active":"2024-05-09T08:26:18.203214+0000","last_peered":"2024-05-09T08:26:18.203214+0000","last_clean":"2024-05-09T08:26:18.203214+0000","last_became_active":"2024-05-09T08:24:51.486332+0000","last_became_peered":"2024-05-09T08:24:51.486332+0000","last_unstale":"2024-05-09T08:26:18.203214+0000","last_undegraded":"2024-05-09T08:26:18.203214+0000","last_fullsized":"2024-05-09T08:26:18.203214+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:23.694545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970561+0000","last_change":"2024-05-09T08:23:51.571414+0000","last_active":"2024-05-09T08:26:17.970561+0000","last_peered":"2024-05-09T08:26:17.970561+0000","last_clean":"2024-05-09T08:26:17.970561+0000","last_became_active":"2024-05-09T08:23:51.571085+0000","last_became_peered":"2024-05-09T08:23:51.571085+0000","last_unstale":"2024-05-09T08:26:17.970561+0000","last_undegraded":"2024-05-09T08:26:17.970561+0000","last_fullsized":"2024-05-09T08:26:17.970561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:37.513584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971413+0000","last_change":"2024-05-09T08:23:55.852785+0000","last_active":"2024-05-09T08:26:17.971413+0000","last_peered":"2024-05-09T08:26:17.971413+0000","last_clean":"2024-05-09T08:26:17.971413+0000","last_became_active":"2024-05-09T08:23:55.852252+0000","last_became_peered":"2024-05-09T08:23:55.852252+0000","last_unstale":"2024-05-09T08:26:17.971413+0000","last_undegraded":"2024-05-09T08:26:17.971413+0000","last_fullsized":"2024-05-09T08:26:17.971413+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:44.503295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086689+0000","last_change":"2024-05-09T08:23:50.564787+0000","last_active":"2024-05-09T08:26:18.086689+0000","last_peered":"2024-05-09T08:26:18.086689+0000","last_clean":"2024-05-09T08:26:18.086689+0000","last_became_active":"2024-05-09T08:23:50.564409+0000","last_became_peered":"2024-05-09T08:23:50.564409+0000","last_unstale":"2024-05-09T08:26:18.086689+0000","last_undegraded":"2024-05-09T08:26:18.086689+0000","last_fullsized":"2024-05-09T08:26:18.086689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:23.051221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187715+0000","last_change":"2024-05-09T08:23:51.576830+0000","last_active":"2024-05-09T08:26:18.187715+0000","last_peered":"2024-05-09T08:26:18.187715+0000","last_clean":"2024-05-09T08:26:18.187715+0000","last_became_active":"2024-05-09T08:23:51.576311+0000","last_became_peered":"2024-05-09T08:23:51.576311+0000","last_unstale":"2024-05-09T08:26:18.187715+0000","last_undegraded":"2024-05-09T08:26:18.187715+0000","last_fullsized":"2024-05-09T08:26:18.187715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:26:08.651061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203014+0000","last_change":"2024-05-09T08:23:55.672309+0000","last_active":"2024-05-09T08:26:18.203014+0000","last_peered":"2024-05-09T08:26:18.203014+0000","last_clean":"2024-05-09T08:26:18.203014+0000","last_became_active":"2024-05-09T08:23:55.671811+0000","last_became_peered":"2024-05-09T08:23:55.671811+0000","last_unstale":"2024-05-09T08:26:18.203014+0000","last_undegraded":"2024-05-09T08:26:18.203014+0000","last_fullsized":"2024-05-09T08:26:18.203014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:19.400949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'7","reported_seq":165,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.595790+0000","last_change":"2024-05-09T08:23:50.575317+0000","last_active":"2024-05-09T08:26:18.595790+0000","last_peered":"2024-05-09T08:26:18.595790+0000","last_clean":"2024-05-09T08:26:18.595790+0000","last_became_active":"2024-05-09T08:23:50.574995+0000","last_became_peered":"2024-05-09T08:23:50.574995+0000","last_unstale":"2024-05-09T08:26:18.595790+0000","last_undegraded":"2024-05-09T08:26:18.595790+0000","last_fullsized":"2024-05-09T08:26:18.595790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:32:54.801799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1869,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.188099+0000","last_change":"2024-05-09T08:23:51.575904+0000","last_active":"2024-05-09T08:26:18.188099+0000","last_peered":"2024-05-09T08:26:18.188099+0000","last_clean":"2024-05-09T08:26:18.188099+0000","last_became_active":"2024-05-09T08:23:51.575599+0000","last_became_peered":"2024-05-09T08:23:51.575599+0000","last_unstale":"2024-05-09T08:26:18.188099+0000","last_undegraded":"2024-05-09T08:26:18.188099+0000","last_fullsized":"2024-05-09T08:26:18.188099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:20:13.087096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205215+0000","last_change":"2024-05-09T08:23:55.680683+0000","last_active":"2024-05-09T08:26:18.205215+0000","last_peered":"2024-05-09T08:26:18.205215+0000","last_clean":"2024-05-09T08:26:18.205215+0000","last_became_active":"2024-05-09T08:23:55.680232+0000","last_became_peered":"2024-05-09T08:23:55.680232+0000","last_unstale":"2024-05-09T08:26:18.205215+0000","last_undegraded":"2024-05-09T08:26:18.205215+0000","last_fullsized":"2024-05-09T08:26:18.205215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:33.583552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.200717+0000","last_change":"2024-05-09T08:23:50.575802+0000","last_active":"2024-05-09T08:26:18.200717+0000","last_peered":"2024-05-09T08:26:18.200717+0000","last_clean":"2024-05-09T08:26:18.200717+0000","last_became_active":"2024-05-09T08:23:50.575416+0000","last_became_peered":"2024-05-09T08:23:50.575416+0000","last_unstale":"2024-05-09T08:26:18.200717+0000","last_undegraded":"2024-05-09T08:26:18.200717+0000","last_fullsized":"2024-05-09T08:26:18.200717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:24:04.873575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187831+0000","last_change":"2024-05-09T08:23:51.578189+0000","last_active":"2024-05-09T08:26:18.187831+0000","last_peered":"2024-05-09T08:26:18.187831+0000","last_clean":"2024-05-09T08:26:18.187831+0000","last_became_active":"2024-05-09T08:23:51.577500+0000","last_became_peered":"2024-05-09T08:23:51.577500+0000","last_unstale":"2024-05-09T08:26:18.187831+0000","last_undegraded":"2024-05-09T08:26:18.187831+0000","last_fullsized":"2024-05-09T08:26:18.187831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:26:24.718791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.086471+0000","last_change":"2024-05-09T08:24:51.487659+0000","last_active":"2024-05-09T08:26:18.086471+0000","last_peered":"2024-05-09T08:26:18.086471+0000","last_clean":"2024-05-09T08:26:18.086471+0000","last_became_active":"2024-05-09T08:24:51.487212+0000","last_became_peered":"2024-05-09T08:24:51.487212+0000","last_unstale":"2024-05-09T08:26:18.086471+0000","last_undegraded":"2024-05-09T08:26:18.086471+0000","last_fullsized":"2024-05-09T08:26:18.086471+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:54.638709+0000","last_clean_scrub_stamp":"2024-05-09T08:23:54.638709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:29:57.077744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205019+0000","last_change":"2024-05-09T08:23:50.578438+0000","last_active":"2024-05-09T08:26:18.205019+0000","last_peered":"2024-05-09T08:26:18.205019+0000","last_clean":"2024-05-09T08:26:18.205019+0000","last_became_active":"2024-05-09T08:23:50.578039+0000","last_became_peered":"2024-05-09T08:23:50.578039+0000","last_unstale":"2024-05-09T08:26:18.205019+0000","last_undegraded":"2024-05-09T08:26:18.205019+0000","last_fullsized":"2024-05-09T08:26:18.205019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:49.547136+0000","last_clean_scrub_stamp":"2024-05-09T08:23:49.547136+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:59.420193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.187887+0000","last_change":"2024-05-09T08:23:51.568037+0000","last_active":"2024-05-09T08:26:18.187887+0000","last_peered":"2024-05-09T08:26:18.187887+0000","last_clean":"2024-05-09T08:26:18.187887+0000","last_became_active":"2024-05-09T08:23:51.567706+0000","last_became_peered":"2024-05-09T08:23:51.567706+0000","last_unstale":"2024-05-09T08:26:18.187887+0000","last_undegraded":"2024-05-09T08:26:18.187887+0000","last_fullsized":"2024-05-09T08:26:18.187887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:15:19.623805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203263+0000","last_change":"2024-05-09T08:23:51.579885+0000","last_active":"2024-05-09T08:26:18.203263+0000","last_peered":"2024-05-09T08:26:18.203263+0000","last_clean":"2024-05-09T08:26:18.203263+0000","last_became_active":"2024-05-09T08:23:51.578953+0000","last_became_peered":"2024-05-09T08:23:51.578953+0000","last_unstale":"2024-05-09T08:26:18.203263+0000","last_undegraded":"2024-05-09T08:26:18.203263+0000","last_fullsized":"2024-05-09T08:26:18.203263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:57:23.793119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971252+0000","last_change":"2024-05-09T08:23:51.583423+0000","last_active":"2024-05-09T08:26:17.971252+0000","last_peered":"2024-05-09T08:26:17.971252+0000","last_clean":"2024-05-09T08:26:17.971252+0000","last_became_active":"2024-05-09T08:23:51.583092+0000","last_became_peered":"2024-05-09T08:23:51.583092+0000","last_unstale":"2024-05-09T08:26:17.971252+0000","last_undegraded":"2024-05-09T08:26:17.971252+0000","last_fullsized":"2024-05-09T08:26:17.971252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:44:47.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971056+0000","last_change":"2024-05-09T08:23:51.582116+0000","last_active":"2024-05-09T08:26:17.971056+0000","last_peered":"2024-05-09T08:26:17.971056+0000","last_clean":"2024-05-09T08:26:17.971056+0000","last_became_active":"2024-05-09T08:23:51.581715+0000","last_became_peered":"2024-05-09T08:23:51.581715+0000","last_unstale":"2024-05-09T08:26:17.971056+0000","last_undegraded":"2024-05-09T08:26:17.971056+0000","last_fullsized":"2024-05-09T08:26:17.971056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:17.528758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205481+0000","last_change":"2024-05-09T08:23:51.575959+0000","last_active":"2024-05-09T08:26:18.205481+0000","last_peered":"2024-05-09T08:26:18.205481+0000","last_clean":"2024-05-09T08:26:18.205481+0000","last_became_active":"2024-05-09T08:23:51.575592+0000","last_became_peered":"2024-05-09T08:23:51.575592+0000","last_unstale":"2024-05-09T08:26:18.205481+0000","last_undegraded":"2024-05-09T08:26:18.205481+0000","last_fullsized":"2024-05-09T08:26:18.205481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:26.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.205213+0000","last_change":"2024-05-09T08:23:51.570756+0000","last_active":"2024-05-09T08:26:18.205213+0000","last_peered":"2024-05-09T08:26:18.205213+0000","last_clean":"2024-05-09T08:26:18.205213+0000","last_became_active":"2024-05-09T08:23:51.570412+0000","last_became_peered":"2024-05-09T08:23:51.570412+0000","last_unstale":"2024-05-09T08:26:18.205213+0000","last_undegraded":"2024-05-09T08:26:18.205213+0000","last_fullsized":"2024-05-09T08:26:18.205213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:11:53.959559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203772+0000","last_change":"2024-05-09T08:23:51.572612+0000","last_active":"2024-05-09T08:26:18.203772+0000","last_peered":"2024-05-09T08:26:18.203772+0000","last_clean":"2024-05-09T08:26:18.203772+0000","last_became_active":"2024-05-09T08:23:51.572227+0000","last_became_peered":"2024-05-09T08:23:51.572227+0000","last_unstale":"2024-05-09T08:26:18.203772+0000","last_undegraded":"2024-05-09T08:26:18.203772+0000","last_fullsized":"2024-05-09T08:26:18.203772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:18.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971483+0000","last_change":"2024-05-09T08:23:51.573021+0000","last_active":"2024-05-09T08:26:17.971483+0000","last_peered":"2024-05-09T08:26:17.971483+0000","last_clean":"2024-05-09T08:26:17.971483+0000","last_became_active":"2024-05-09T08:23:51.572716+0000","last_became_peered":"2024-05-09T08:23:51.572716+0000","last_unstale":"2024-05-09T08:26:17.971483+0000","last_undegraded":"2024-05-09T08:26:17.971483+0000","last_fullsized":"2024-05-09T08:26:17.971483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:31.494329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.971007+0000","last_change":"2024-05-09T08:23:51.572687+0000","last_active":"2024-05-09T08:26:17.971007+0000","last_peered":"2024-05-09T08:26:17.971007+0000","last_clean":"2024-05-09T08:26:17.971007+0000","last_became_active":"2024-05-09T08:23:51.572339+0000","last_became_peered":"2024-05-09T08:23:51.572339+0000","last_unstale":"2024-05-09T08:26:17.971007+0000","last_undegraded":"2024-05-09T08:26:17.971007+0000","last_fullsized":"2024-05-09T08:26:17.971007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:24.012360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.092190+0000","last_change":"2024-05-09T08:23:51.569404+0000","last_active":"2024-05-09T08:26:18.092190+0000","last_peered":"2024-05-09T08:26:18.092190+0000","last_clean":"2024-05-09T08:26:18.092190+0000","last_became_active":"2024-05-09T08:23:51.569044+0000","last_became_peered":"2024-05-09T08:23:51.569044+0000","last_unstale":"2024-05-09T08:26:18.092190+0000","last_undegraded":"2024-05-09T08:26:18.092190+0000","last_fullsized":"2024-05-09T08:26:18.092190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:26:01.343606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:17.970502+0000","last_change":"2024-05-09T08:23:51.577483+0000","last_active":"2024-05-09T08:26:17.970502+0000","last_peered":"2024-05-09T08:26:17.970502+0000","last_clean":"2024-05-09T08:26:17.970502+0000","last_became_active":"2024-05-09T08:23:51.577078+0000","last_became_peered":"2024-05-09T08:23:51.577078+0000","last_unstale":"2024-05-09T08:26:17.970502+0000","last_undegraded":"2024-05-09T08:26:17.970502+0000","last_fullsized":"2024-05-09T08:26:17.970502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:29.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203171+0000","last_change":"2024-05-09T08:23:51.579736+0000","last_active":"2024-05-09T08:26:18.203171+0000","last_peered":"2024-05-09T08:26:18.203171+0000","last_clean":"2024-05-09T08:26:18.203171+0000","last_became_active":"2024-05-09T08:23:51.579409+0000","last_became_peered":"2024-05-09T08:23:51.579409+0000","last_unstale":"2024-05-09T08:26:18.203171+0000","last_undegraded":"2024-05-09T08:26:18.203171+0000","last_fullsized":"2024-05-09T08:26:18.203171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:05.697642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.204632+0000","last_change":"2024-05-09T08:23:51.569461+0000","last_active":"2024-05-09T08:26:18.204632+0000","last_peered":"2024-05-09T08:26:18.204632+0000","last_clean":"2024-05-09T08:26:18.204632+0000","last_became_active":"2024-05-09T08:23:51.569085+0000","last_became_peered":"2024-05-09T08:23:51.569085+0000","last_unstale":"2024-05-09T08:26:18.204632+0000","last_undegraded":"2024-05-09T08:26:18.204632+0000","last_fullsized":"2024-05-09T08:26:18.204632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:45:54.554949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:26:18.203414+0000","last_change":"2024-05-09T08:23:51.570593+0000","last_active":"2024-05-09T08:26:18.203414+0000","last_peered":"2024-05-09T08:26:18.203414+0000","last_clean":"2024-05-09T08:26:18.203414+0000","last_became_active":"2024-05-09T08:23:51.570195+0000","last_became_peered":"2024-05-09T08:23:51.570195+0000","last_unstale":"2024-05-09T08:26:18.203414+0000","last_undegraded":"2024-05-09T08:26:18.203414+0000","last_fullsized":"2024-05-09T08:26:18.203414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:23:50.548640+0000","last_clean_scrub_stamp":"2024-05-09T08:23:50.548640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:18.376268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108273,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604101,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1324,"num_read_kb":780,"num_write":376,"num_write_kb":1473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":64162,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":667648,"data_stored":1812303,"data_compressed":89522,"data_compressed_allocated":118784,"data_compressed_original":1626112,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":531150,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32344,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712808,"statfs":{"total":95995035648,"available":95961915392,"internally_reserved":0,"allocated":1191936,"data_stored":1185786,"data_compressed":67266,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":923575,"data_compressed":48720,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":923259,"data_compressed":48720,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1101824,"data_stored":1117422,"data_compressed":62684,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526254,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1118208,"data_stored":598024,"data_compressed":31175,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1064960,"data_stored":528018,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":527309,"data_compressed":29574,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1097728,"data_stored":1050387,"data_compressed":54876,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":601431,"data_compressed":39988,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1142784,"data_stored":1254416,"data_compressed":71821,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36091,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":72883,"data_compressed":1601,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":72194,"data_compressed":10414,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":134811,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394022,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":204718,"data_compressed":7757,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:26:21.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:26:21.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.373+0000 7f69e13d5640 1 -- 172.21.15.28:0/3209257156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc109730 msgr2=0x7f69dc10bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.373+0000 7f69e13d5640 1 --2- 172.21.15.28:0/3209257156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc109730 0x7f69dc10bbd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009ca0 tx=0x7f69d402a950 comp rx=0 tx=0).stop 2024-05-09T08:26:21.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.373+0000 7f69e13d5640 1 -- 172.21.15.28:0/3209257156 shutdown_connections 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.373+0000 7f69e13d5640 1 --2- 172.21.15.28:0/3209257156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc109730 0x7f69dc10bbd0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.373+0000 7f69e13d5640 1 --2- 172.21.15.28:0/3209257156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69dc074230 0x7f69dc074690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.373+0000 7f69e13d5640 1 --2- 172.21.15.28:0/3209257156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69dc075810 0x7f69dc073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.373+0000 7f69e13d5640 1 -- 172.21.15.28:0/3209257156 >> 172.21.15.28:0/3209257156 conn(0x7f69dc0fce30 msgr2=0x7f69dc0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 -- 172.21.15.28:0/3209257156 shutdown_connections 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 -- 172.21.15.28:0/3209257156 wait complete. 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 Processor -- start 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 -- start start 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69dc074230 0x7f69dc199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc075810 0x7f69dc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69dc109730 0x7f69dc1a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10f720 con 0x7f69dc075810 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10f5a0 con 0x7f69dc074230 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69e13d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10f8a0 con 0x7f69dc109730 2024-05-09T08:26:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc075810 0x7f69dc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc075810 0x7f69dc19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38378/0 (socket says 172.21.15.28:38378) 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.374+0000 7f69d3fff640 1 -- 172.21.15.28:0/616146100 learned_addr learned my addr 172.21.15.28:0/616146100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69db7fe640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69dc109730 0x7f69dc1a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69daffd640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69dc074230 0x7f69dc199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69d3fff640 1 -- 172.21.15.28:0/616146100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69dc109730 msgr2=0x7f69dc1a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69d3fff640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69dc109730 0x7f69dc1a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69d3fff640 1 -- 172.21.15.28:0/616146100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69dc074230 msgr2=0x7f69dc199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69d3fff640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69dc074230 0x7f69dc199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69d3fff640 1 -- 172.21.15.28:0/616146100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69dc1a1960 con 0x7f69dc075810 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69daffd640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69dc074230 0x7f69dc199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69d3fff640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc075810 0x7f69dc19a290 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f69c400e9f0 tx=0x7f69c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69d8ff9640 1 -- 172.21.15.28:0/616146100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c400cd80 con 0x7f69dc075810 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69d8ff9640 1 -- 172.21.15.28:0/616146100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c4010910 con 0x7f69dc075810 2024-05-09T08:26:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69dc1a1c50 con 0x7f69dc075810 2024-05-09T08:26:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69d8ff9640 1 -- 172.21.15.28:0/616146100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4018990 con 0x7f69dc075810 2024-05-09T08:26:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.375+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69dc1a21a0 con 0x7f69dc075810 2024-05-09T08:26:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.376+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69dc1084f0 con 0x7f69dc075810 2024-05-09T08:26:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.377+0000 7f69d8ff9640 1 -- 172.21.15.28:0/616146100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f69c4010ac0 con 0x7f69dc075810 2024-05-09T08:26:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.378+0000 7f69d8ff9640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f69c80a2d00 0x7f69c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.378+0000 7f69daffd640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f69c80a2d00 0x7f69c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.378+0000 7f69d8ff9640 1 -- 172.21.15.28:0/616146100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f69c40cfd40 con 0x7f69dc075810 2024-05-09T08:26:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.379+0000 7f69daffd640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f69c80a2d00 0x7f69c80a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f69dc19b220 tx=0x7f69cc005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.383+0000 7f69d8ff9640 1 -- 172.21.15.28:0/616146100 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f69c409c590 con 0x7f69dc075810 2024-05-09T08:26:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.589+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69dc02d400 con 0x7f69dc075810 2024-05-09T08:26:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.591+0000 7f69d8ff9640 1 -- 172.21.15.28:0/616146100 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25274 (secure 0 0 0) 0x7f69c409be90 con 0x7f69dc075810 2024-05-09T08:26:21.592 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:21.592 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:26:17.965032+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3330063293":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3745037139":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/2130261041":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/273816117":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:26:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f69c80a2d00 msgr2=0x7f69c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f69c80a2d00 0x7f69c80a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f69dc19b220 tx=0x7f69cc005f90 comp rx=0 tx=0).stop 2024-05-09T08:26:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc075810 msgr2=0x7f69dc19a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc075810 0x7f69dc19a290 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f69c400e9f0 tx=0x7f69c400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 shutdown_connections 2024-05-09T08:26:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f69c80a2d00 0x7f69c80a51c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69dc109730 0x7f69dc1a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69dc075810 0x7f69dc19a290 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 --2- 172.21.15.28:0/616146100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69dc074230 0x7f69dc199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 >> 172.21.15.28:0/616146100 conn(0x7f69dc0fce30 msgr2=0x7f69dc1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:21.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 shutdown_connections 2024-05-09T08:26:21.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.593+0000 7f69e13d5640 1 -- 172.21.15.28:0/616146100 wait complete. 2024-05-09T08:26:21.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: [09/May/2024:08:26:20] ENGINE Bus STARTING 2024-05-09T08:26:21.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-05-09T08:26:21.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: [09/May/2024:08:26:20] ENGINE Serving on http://172.21.15.28:8765 2024-05-09T08:26:21.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: [09/May/2024:08:26:20] ENGINE Serving on https://172.21.15.28:7150 2024-05-09T08:26:21.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: [09/May/2024:08:26:20] ENGINE Bus STARTED 2024-05-09T08:26:21.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:21 smithi165 ceph-mon[34540]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:26:21.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: [09/May/2024:08:26:20] ENGINE Bus STARTING 2024-05-09T08:26:21.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-05-09T08:26:21.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: [09/May/2024:08:26:20] ENGINE Serving on http://172.21.15.28:8765 2024-05-09T08:26:21.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: [09/May/2024:08:26:20] ENGINE Serving on https://172.21.15.28:7150 2024-05-09T08:26:21.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: [09/May/2024:08:26:20] ENGINE Bus STARTED 2024-05-09T08:26:21.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:21 smithi184 ceph-mon[34509]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:26:21.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: [09/May/2024:08:26:20] ENGINE Bus STARTING 2024-05-09T08:26:21.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-05-09T08:26:21.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: [09/May/2024:08:26:20] ENGINE Serving on http://172.21.15.28:8765 2024-05-09T08:26:21.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:21.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: [09/May/2024:08:26:20] ENGINE Serving on https://172.21.15.28:7150 2024-05-09T08:26:21.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: [09/May/2024:08:26:20] ENGINE Bus STARTED 2024-05-09T08:26:21.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:21 smithi028 ceph-mon[30119]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:26:21.610 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:26:21.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:26:21.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.720+0000 7fdd1c570640 1 -- 172.21.15.28:0/1935699960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14101960 msgr2=0x7fdd14101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.720+0000 7fdd1c570640 1 --2- 172.21.15.28:0/1935699960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14101960 0x7fdd14101de0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009900 tx=0x7fdd0402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:21.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.720+0000 7fdd1c570640 1 -- 172.21.15.28:0/1935699960 shutdown_connections 2024-05-09T08:26:21.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.720+0000 7fdd1c570640 1 --2- 172.21.15.28:0/1935699960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd14102320 0x7fdd14108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.720+0000 7fdd1c570640 1 --2- 172.21.15.28:0/1935699960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14101960 0x7fdd14101de0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.720+0000 7fdd1c570640 1 --2- 172.21.15.28:0/1935699960 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd14100770 0x7fdd14100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.720+0000 7fdd1c570640 1 -- 172.21.15.28:0/1935699960 >> 172.21.15.28:0/1935699960 conn(0x7fdd140fbce0 msgr2=0x7fdd140fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:21.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.720+0000 7fdd1c570640 1 -- 172.21.15.28:0/1935699960 shutdown_connections 2024-05-09T08:26:21.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.720+0000 7fdd1c570640 1 -- 172.21.15.28:0/1935699960 wait complete. 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1c570640 1 Processor -- start 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1c570640 1 -- start start 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1c570640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14100770 0x7fdd14197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1c570640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd14101960 0x7fdd14198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1c570640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd14102320 0x7fdd1419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1c570640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1410c3d0 con 0x7fdd14100770 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1c570640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1410c250 con 0x7fdd14102320 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1c570640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1410c550 con 0x7fdd14101960 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1a30c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14100770 0x7fdd14197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1a30c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14100770 0x7fdd14197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38404/0 (socket says 172.21.15.28:38404) 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1a30c640 1 -- 172.21.15.28:0/336162600 learned_addr learned my addr 172.21.15.28:0/336162600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.721+0000 7fdd1ab0d640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd14102320 0x7fdd1419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd19b0b640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd14101960 0x7fdd14198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd1a30c640 1 -- 172.21.15.28:0/336162600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd14101960 msgr2=0x7fdd14198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd1a30c640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd14101960 0x7fdd14198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd1a30c640 1 -- 172.21.15.28:0/336162600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd14102320 msgr2=0x7fdd1419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd1a30c640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd14102320 0x7fdd1419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd1a30c640 1 -- 172.21.15.28:0/336162600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1419f720 con 0x7fdd14100770 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd1a30c640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14100770 0x7fdd14197b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000c970 tx=0x7fdd1000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd0b7fe640 1 -- 172.21.15.28:0/336162600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10007c10 con 0x7fdd14100770 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1419fa10 con 0x7fdd14100770 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd0b7fe640 1 -- 172.21.15.28:0/336162600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd10007db0 con 0x7fdd14100770 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd0b7fe640 1 -- 172.21.15.28:0/336162600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1000f8a0 con 0x7fdd14100770 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.722+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd1419ff80 con 0x7fdd14100770 2024-05-09T08:26:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.723+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd141a0210 con 0x7fdd14100770 2024-05-09T08:26:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.725+0000 7fdd0b7fe640 1 -- 172.21.15.28:0/336162600 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7fdd1000fa40 con 0x7fdd14100770 2024-05-09T08:26:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.725+0000 7fdd0b7fe640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd000a2d00 0x7fdd000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.725+0000 7fdd0b7fe640 1 -- 172.21.15.28:0/336162600 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fdd100cfe10 con 0x7fdd14100770 2024-05-09T08:26:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.725+0000 7fdd19b0b640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd000a2d00 0x7fdd000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.726+0000 7fdd19b0b640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd000a2d00 0x7fdd000a51c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd04000c00 tx=0x7fdd040315d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.732+0000 7fdd0b7fe640 1 -- 172.21.15.28:0/336162600 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdd1009c720 con 0x7fdd14100770 2024-05-09T08:26:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.935+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdd141a03e0 con 0x7fdd14100770 2024-05-09T08:26:21.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.936+0000 7fdd0b7fe640 1 -- 172.21.15.28:0/336162600 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fdd1009c020 con 0x7fdd14100770 2024-05-09T08:26:21.936 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:26:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.937+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd000a2d00 msgr2=0x7fdd000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.937+0000 7fdd1c570640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd000a2d00 0x7fdd000a51c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd04000c00 tx=0x7fdd040315d0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14100770 msgr2=0x7fdd14197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14100770 0x7fdd14197b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000c970 tx=0x7fdd1000ce30 comp rx=0 tx=0).stop 2024-05-09T08:26:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 shutdown_connections 2024-05-09T08:26:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd14102320 0x7fdd1419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd000a2d00 0x7fdd000a51c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd14101960 0x7fdd14198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 --2- 172.21.15.28:0/336162600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14100770 0x7fdd14197b00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 >> 172.21.15.28:0/336162600 conn(0x7fdd140fbce0 msgr2=0x7fdd140fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 shutdown_connections 2024-05-09T08:26:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:21.938+0000 7fdd1c570640 1 -- 172.21.15.28:0/336162600 wait complete. 2024-05-09T08:26:21.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:26:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.062+0000 7f96b37e7640 1 -- 172.21.15.28:0/3994103754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac101a40 msgr2=0x7f96ac108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.062+0000 7f96b37e7640 1 --2- 172.21.15.28:0/3994103754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac101a40 0x7f96ac108360 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f96a4009be0 tx=0x7f96a402a6c0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.062+0000 7f96b37e7640 1 -- 172.21.15.28:0/3994103754 shutdown_connections 2024-05-09T08:26:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.062+0000 7f96b37e7640 1 --2- 172.21.15.28:0/3994103754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac101a40 0x7f96ac108360 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.062+0000 7f96b37e7640 1 --2- 172.21.15.28:0/3994103754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96ac101080 0x7f96ac101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.062+0000 7f96b37e7640 1 --2- 172.21.15.28:0/3994103754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ac0ffea0 0x7f96ac1002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.062+0000 7f96b37e7640 1 -- 172.21.15.28:0/3994103754 >> 172.21.15.28:0/3994103754 conn(0x7f96ac0fb460 msgr2=0x7f96ac0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.062+0000 7f96b37e7640 1 -- 172.21.15.28:0/3994103754 shutdown_connections 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.062+0000 7f96b37e7640 1 -- 172.21.15.28:0/3994103754 wait complete. 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b37e7640 1 Processor -- start 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b37e7640 1 -- start start 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b37e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac0ffea0 0x7f96ac196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b37e7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ac101080 0x7f96ac196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b37e7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96ac101a40 0x7f96ac19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b37e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96ac10c3a0 con 0x7f96ac0ffea0 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b37e7640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f96ac10c220 con 0x7f96ac101080 2024-05-09T08:26:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b37e7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f96ac10c520 con 0x7f96ac101a40 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b1583640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac0ffea0 0x7f96ac196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b1583640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac0ffea0 0x7f96ac196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38420/0 (socket says 172.21.15.28:38420) 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b1583640 1 -- 172.21.15.28:0/1309757131 learned_addr learned my addr 172.21.15.28:0/1309757131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b1d84640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96ac101a40 0x7f96ac19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b0d82640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ac101080 0x7f96ac196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b1583640 1 -- 172.21.15.28:0/1309757131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96ac101a40 msgr2=0x7f96ac19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b1583640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96ac101a40 0x7f96ac19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b1583640 1 -- 172.21.15.28:0/1309757131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ac101080 msgr2=0x7f96ac196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b1583640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ac101080 0x7f96ac196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.063+0000 7f96b1583640 1 -- 172.21.15.28:0/1309757131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ac19e630 con 0x7f96ac0ffea0 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.064+0000 7f96b1583640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac0ffea0 0x7f96ac196a10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f96a800e9f0 tx=0x7f96a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.064+0000 7f96a27fc640 1 -- 172.21.15.28:0/1309757131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a800ccc0 con 0x7f96ac0ffea0 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.064+0000 7f96a27fc640 1 -- 172.21.15.28:0/1309757131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a80035f0 con 0x7f96ac0ffea0 2024-05-09T08:26:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.064+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96ac19e920 con 0x7f96ac0ffea0 2024-05-09T08:26:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.064+0000 7f96a27fc640 1 -- 172.21.15.28:0/1309757131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a800ce60 con 0x7f96ac0ffea0 2024-05-09T08:26:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.064+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96ac19eec0 con 0x7f96ac0ffea0 2024-05-09T08:26:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.065+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96ac067dc0 con 0x7f96ac0ffea0 2024-05-09T08:26:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.066+0000 7f96a27fc640 1 -- 172.21.15.28:0/1309757131 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f96a80037a0 con 0x7f96ac0ffea0 2024-05-09T08:26:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.067+0000 7f96a27fc640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f96980a2d80 0x7f96980a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.067+0000 7f96a27fc640 1 -- 172.21.15.28:0/1309757131 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f96a80cf9e0 con 0x7f96ac0ffea0 2024-05-09T08:26:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.067+0000 7f96b0d82640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f96980a2d80 0x7f96980a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.068+0000 7f96b0d82640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f96980a2d80 0x7f96980a5240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f96ac197f20 tx=0x7f969c005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.072+0000 7f96a27fc640 1 -- 172.21.15.28:0/1309757131 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f96a809c230 con 0x7f96ac0ffea0 2024-05-09T08:26:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.262+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f96ac02d240 con 0x7f96ac0ffea0 2024-05-09T08:26:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.262+0000 7f96a27fc640 1 -- 172.21.15.28:0/1309757131 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f96a809bb30 con 0x7f96ac0ffea0 2024-05-09T08:26:22.263 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:22.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f96980a2d80 msgr2=0x7f96980a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f96980a2d80 0x7f96980a5240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f96ac197f20 tx=0x7f969c005f90 comp rx=0 tx=0).stop 2024-05-09T08:26:22.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac0ffea0 msgr2=0x7f96ac196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac0ffea0 0x7f96ac196a10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f96a800e9f0 tx=0x7f96a800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 shutdown_connections 2024-05-09T08:26:22.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f96980a2d80 0x7f96980a5240 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f96ac101a40 0x7f96ac19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ac101080 0x7f96ac196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 --2- 172.21.15.28:0/1309757131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96ac0ffea0 0x7f96ac196a10 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.264+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 >> 172.21.15.28:0/1309757131 conn(0x7f96ac0fb460 msgr2=0x7f96ac104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:22.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.265+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 shutdown_connections 2024-05-09T08:26:22.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.265+0000 7f96b37e7640 1 -- 172.21.15.28:0/1309757131 wait complete. 2024-05-09T08:26:22.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:26:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.389+0000 7f5a95f09640 1 -- 172.21.15.28:0/1769140320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a9010e240 msgr2=0x7f5a90110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.389+0000 7f5a95f09640 1 --2- 172.21.15.28:0/1769140320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a9010e240 0x7f5a90110630 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5a88009ca0 tx=0x7f5a8802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.389+0000 7f5a95f09640 1 -- 172.21.15.28:0/1769140320 shutdown_connections 2024-05-09T08:26:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.389+0000 7f5a95f09640 1 --2- 172.21.15.28:0/1769140320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a9010e240 0x7f5a90110630 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.389+0000 7f5a95f09640 1 --2- 172.21.15.28:0/1769140320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a9006a7d0 0x7f5a9010dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.389+0000 7f5a95f09640 1 --2- 172.21.15.28:0/1769140320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a90069eb0 0x7f5a9006a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.389+0000 7f5a95f09640 1 -- 172.21.15.28:0/1769140320 >> 172.21.15.28:0/1769140320 conn(0x7f5a90077790 msgr2=0x7f5a90077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.389+0000 7f5a95f09640 1 -- 172.21.15.28:0/1769140320 shutdown_connections 2024-05-09T08:26:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.389+0000 7f5a95f09640 1 -- 172.21.15.28:0/1769140320 wait complete. 2024-05-09T08:26:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a95f09640 1 Processor -- start 2024-05-09T08:26:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a95f09640 1 -- start start 2024-05-09T08:26:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a95f09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a90069eb0 0x7f5a9019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a95f09640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a9006a7d0 0x7f5a9019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a95f09640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a9010e240 0x7f5a9019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a95f09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a90113e00 con 0x7f5a90069eb0 2024-05-09T08:26:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a95f09640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5a90113c80 con 0x7f5a9006a7d0 2024-05-09T08:26:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a95f09640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a90113f80 con 0x7f5a9010e240 2024-05-09T08:26:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a90069eb0 0x7f5a9019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a90069eb0 0x7f5a9019a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38432/0 (socket says 172.21.15.28:38432) 2024-05-09T08:26:22.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a8f7fe640 1 -- 172.21.15.28:0/4130361989 learned_addr learned my addr 172.21.15.28:0/4130361989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:22.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.390+0000 7f5a87fff640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a9006a7d0 0x7f5a9019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a8f7fe640 1 -- 172.21.15.28:0/4130361989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a9010e240 msgr2=0x7f5a9019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a8f7fe640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a9010e240 0x7f5a9019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a8f7fe640 1 -- 172.21.15.28:0/4130361989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a9006a7d0 msgr2=0x7f5a9019a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a8f7fe640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a9006a7d0 0x7f5a9019a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a8f7fe640 1 -- 172.21.15.28:0/4130361989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9019f320 con 0x7f5a90069eb0 2024-05-09T08:26:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a8f7fe640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a90069eb0 0x7f5a9019a380 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000d950 tx=0x7f5a8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a8d7fa640 1 -- 172.21.15.28:0/4130361989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80017030 con 0x7f5a90069eb0 2024-05-09T08:26:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a8d7fa640 1 -- 172.21.15.28:0/4130361989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a80004020 con 0x7f5a90069eb0 2024-05-09T08:26:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a8d7fa640 1 -- 172.21.15.28:0/4130361989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a800047a0 con 0x7f5a90069eb0 2024-05-09T08:26:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9019f610 con 0x7f5a90069eb0 2024-05-09T08:26:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.391+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a901a6ea0 con 0x7f5a90069eb0 2024-05-09T08:26:22.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.392+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a90068aa0 con 0x7f5a90069eb0 2024-05-09T08:26:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.393+0000 7f5a8d7fa640 1 -- 172.21.15.28:0/4130361989 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f5a800041d0 con 0x7f5a90069eb0 2024-05-09T08:26:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.394+0000 7f5a8d7fa640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a7c0a2d80 0x7f5a7c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.394+0000 7f5a87fff640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a7c0a2d80 0x7f5a7c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.394+0000 7f5a8d7fa640 1 -- 172.21.15.28:0/4130361989 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f5a800cf180 con 0x7f5a90069eb0 2024-05-09T08:26:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.394+0000 7f5a87fff640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a7c0a2d80 0x7f5a7c0a5240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5a90101c10 tx=0x7f5a78005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.398+0000 7f5a8d7fa640 1 -- 172.21.15.28:0/4130361989 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5a8009b9d0 con 0x7f5a90069eb0 2024-05-09T08:26:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.565+0000 7f5a8d7fa640 1 -- 172.21.15.28:0/4130361989 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f5a800cc780 con 0x7f5a90069eb0 2024-05-09T08:26:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:26:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='client.15681 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:26:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi184 to 176.2M 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: Unable to set osd_memory_target on smithi184 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi165 to 176.2M 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: Unable to set osd_memory_target on smithi165 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-09T08:26:22.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/616146100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/336162600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:22.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:22 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1309757131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:22.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.606+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5a9002d1c0 con 0x7f5a90069eb0 2024-05-09T08:26:22.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.607+0000 7f5a8d7fa640 1 -- 172.21.15.28:0/4130361989 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f5a800c0d50 con 0x7f5a90069eb0 2024-05-09T08:26:22.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:26:22.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='client.15681 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:26:22.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi184 to 176.2M 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: Unable to set osd_memory_target on smithi184 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.610 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi165 to 176.2M 2024-05-09T08:26:22.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: Unable to set osd_memory_target on smithi165 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-09T08:26:22.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.611 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.612 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:22.612 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:22.612 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/616146100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:22.612 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/336162600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:22.612 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:22 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1309757131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a7c0a2d80 msgr2=0x7f5a7c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a7c0a2d80 0x7f5a7c0a5240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5a90101c10 tx=0x7f5a78005f90 comp rx=0 tx=0).stop 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a90069eb0 msgr2=0x7f5a9019a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a90069eb0 0x7f5a9019a380 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000d950 tx=0x7f5a8000de10 comp rx=0 tx=0).stop 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 shutdown_connections 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a7c0a2d80 0x7f5a7c0a5240 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a9010e240 0x7f5a9019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a9006a7d0 0x7f5a9019a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 --2- 172.21.15.28:0/4130361989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a90069eb0 0x7f5a9019a380 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 >> 172.21.15.28:0/4130361989 conn(0x7f5a90077790 msgr2=0x7f5a9010f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 shutdown_connections 2024-05-09T08:26:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.609+0000 7f5a95f09640 1 -- 172.21.15.28:0/4130361989 wait complete. 2024-05-09T08:26:22.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:26:22.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='client.15681 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:26:22.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi184 to 176.2M 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: Unable to set osd_memory_target on smithi184 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi165 to 176.2M 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: Unable to set osd_memory_target on smithi165 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-09T08:26:22.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/616146100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/336162600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:26:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:22 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1309757131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:26:22.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:26:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 -- 172.21.15.28:0/3299433511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c1010c0 msgr2=0x7fe59c10d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3299433511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c1010c0 0x7fe59c10d5d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe594009cd0 tx=0x7fe59402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 -- 172.21.15.28:0/3299433511 shutdown_connections 2024-05-09T08:26:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3299433511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c1010c0 0x7fe59c10d5d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3299433511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe59c100700 0x7fe59c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3299433511 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe59c106710 0x7fe59c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 -- 172.21.15.28:0/3299433511 >> 172.21.15.28:0/3299433511 conn(0x7fe59c077830 msgr2=0x7fe59c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 -- 172.21.15.28:0/3299433511 shutdown_connections 2024-05-09T08:26:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 -- 172.21.15.28:0/3299433511 wait complete. 2024-05-09T08:26:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.746+0000 7fe5a131d640 1 Processor -- start 2024-05-09T08:26:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe5a131d640 1 -- start start 2024-05-09T08:26:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe5a131d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c100700 0x7fe59c198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe5a131d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe59c1010c0 0x7fe59c199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe5a131d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe59c106710 0x7fe59c1a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe5a131d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c110ae0 con 0x7fe59c100700 2024-05-09T08:26:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe5a131d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c110960 con 0x7fe59c1010c0 2024-05-09T08:26:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe5a131d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c110c60 con 0x7fe59c106710 2024-05-09T08:26:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c100700 0x7fe59c198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c100700 0x7fe59c198f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38442/0 (socket says 172.21.15.28:38442) 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59affd640 1 -- 172.21.15.28:0/3878709543 learned_addr learned my addr 172.21.15.28:0/3878709543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59a7fc640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe59c1010c0 0x7fe59c199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59b7fe640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe59c106710 0x7fe59c1a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59affd640 1 -- 172.21.15.28:0/3878709543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe59c106710 msgr2=0x7fe59c1a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59affd640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe59c106710 0x7fe59c1a3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59affd640 1 -- 172.21.15.28:0/3878709543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe59c1010c0 msgr2=0x7fe59c199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59affd640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe59c1010c0 0x7fe59c199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59affd640 1 -- 172.21.15.28:0/3878709543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe59c1a44c0 con 0x7fe59c100700 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.747+0000 7fe59a7fc640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe59c1010c0 0x7fe59c199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.748+0000 7fe59affd640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c100700 0x7fe59c198f00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe58c00c890 tx=0x7fe58c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.748+0000 7fe593fff640 1 -- 172.21.15.28:0/3878709543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe58c004220 con 0x7fe59c100700 2024-05-09T08:26:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.748+0000 7fe593fff640 1 -- 172.21.15.28:0/3878709543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe58c0043c0 con 0x7fe59c100700 2024-05-09T08:26:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.748+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe59c1a4690 con 0x7fe59c100700 2024-05-09T08:26:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.748+0000 7fe593fff640 1 -- 172.21.15.28:0/3878709543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe58c0049e0 con 0x7fe59c100700 2024-05-09T08:26:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.748+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe59c1a4bb0 con 0x7fe59c100700 2024-05-09T08:26:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.749+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe59c068650 con 0x7fe59c100700 2024-05-09T08:26:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.750+0000 7fe593fff640 1 -- 172.21.15.28:0/3878709543 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7fe58c0079b0 con 0x7fe59c100700 2024-05-09T08:26:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.751+0000 7fe593fff640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe5880a2d90 0x7fe5880a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.751+0000 7fe59a7fc640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe5880a2d90 0x7fe5880a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.751+0000 7fe593fff640 1 -- 172.21.15.28:0/3878709543 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fe58c0cfa90 con 0x7fe59c100700 2024-05-09T08:26:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.752+0000 7fe59a7fc640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe5880a2d90 0x7fe5880a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe59c069510 tx=0x7fe584005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:22.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.756+0000 7fe593fff640 1 -- 172.21.15.28:0/3878709543 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe58c014030 con 0x7fe59c100700 2024-05-09T08:26:22.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.958+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe59c108c90 con 0x7fe59c100700 2024-05-09T08:26:22.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.959+0000 7fe593fff640 1 -- 172.21.15.28:0/3878709543 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fe58c09c170 con 0x7fe59c100700 2024-05-09T08:26:22.959 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe5880a2d90 msgr2=0x7fe5880a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe5880a2d90 0x7fe5880a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe59c069510 tx=0x7fe584005f90 comp rx=0 tx=0).stop 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c100700 msgr2=0x7fe59c198f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c100700 0x7fe59c198f00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe58c00c890 tx=0x7fe58c00cd50 comp rx=0 tx=0).stop 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 shutdown_connections 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe5880a2d90 0x7fe5880a5250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe59c106710 0x7fe59c1a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe59c1010c0 0x7fe59c199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 --2- 172.21.15.28:0/3878709543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c100700 0x7fe59c198f00 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 >> 172.21.15.28:0/3878709543 conn(0x7fe59c077830 msgr2=0x7fe59c102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 shutdown_connections 2024-05-09T08:26:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:22.962+0000 7fe5a131d640 1 -- 172.21.15.28:0/3878709543 wait complete. 2024-05-09T08:26:22.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:26:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.079+0000 7f0e3c770640 1 -- 172.21.15.28:0/397615185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e34100f60 msgr2=0x7f0e3410dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.079+0000 7f0e3c770640 1 --2- 172.21.15.28:0/397615185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e34100f60 0x7f0e3410dcb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009900 tx=0x7f0e2402a410 comp rx=0 tx=0).stop 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 -- 172.21.15.28:0/397615185 shutdown_connections 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 --2- 172.21.15.28:0/397615185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e3410e1f0 0x7f0e341105e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 --2- 172.21.15.28:0/397615185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e34100f60 0x7f0e3410dcb0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 --2- 172.21.15.28:0/397615185 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e34100640 0x7f0e34100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 -- 172.21.15.28:0/397615185 >> 172.21.15.28:0/397615185 conn(0x7f0e34077830 msgr2=0x7f0e34077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 -- 172.21.15.28:0/397615185 shutdown_connections 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 -- 172.21.15.28:0/397615185 wait complete. 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 Processor -- start 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 -- start start 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e34100640 0x7f0e3419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e34100f60 0x7f0e3419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3410e1f0 0x7f0e3419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e34113e20 con 0x7f0e3410e1f0 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0e34113ca0 con 0x7f0e34100f60 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.080+0000 7f0e3c770640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0e34113fa0 con 0x7f0e34100640 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3ad0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3410e1f0 0x7f0e3419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3ad0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3410e1f0 0x7f0e3419eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38464/0 (socket says 172.21.15.28:38464) 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3ad0d640 1 -- 172.21.15.28:0/2514949988 learned_addr learned my addr 172.21.15.28:0/2514949988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e39d0b640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e34100f60 0x7f0e3419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3a50c640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e34100640 0x7f0e3419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3ad0d640 1 -- 172.21.15.28:0/2514949988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e34100640 msgr2=0x7f0e3419a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3ad0d640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e34100640 0x7f0e3419a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3ad0d640 1 -- 172.21.15.28:0/2514949988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e34100f60 msgr2=0x7f0e3419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3ad0d640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e34100f60 0x7f0e3419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3ad0d640 1 -- 172.21.15.28:0/2514949988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3419f2a0 con 0x7f0e3410e1f0 2024-05-09T08:26:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3a50c640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e34100640 0x7f0e3419a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e39d0b640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e34100f60 0x7f0e3419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3ad0d640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3410e1f0 0x7f0e3419eb70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00ccb0 tx=0x7f0e2c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e2b7fe640 1 -- 172.21.15.28:0/2514949988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c003970 con 0x7f0e3410e1f0 2024-05-09T08:26:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3419f590 con 0x7f0e3410e1f0 2024-05-09T08:26:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e2b7fe640 1 -- 172.21.15.28:0/2514949988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e2c004030 con 0x7f0e3410e1f0 2024-05-09T08:26:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e2b7fe640 1 -- 172.21.15.28:0/2514949988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00f720 con 0x7f0e3410e1f0 2024-05-09T08:26:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.081+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e341a6e20 con 0x7f0e3410e1f0 2024-05-09T08:26:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.082+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e34068650 con 0x7f0e3410e1f0 2024-05-09T08:26:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.083+0000 7f0e2b7fe640 1 -- 172.21.15.28:0/2514949988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f0e2c0041e0 con 0x7f0e3410e1f0 2024-05-09T08:26:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.084+0000 7f0e2b7fe640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0e200a2d90 0x7f0e200a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.085+0000 7f0e3a50c640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0e200a2d90 0x7f0e200a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.085+0000 7f0e2b7fe640 1 -- 172.21.15.28:0/2514949988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0e2c0cfb10 con 0x7f0e3410e1f0 2024-05-09T08:26:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.085+0000 7f0e3a50c640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0e200a2d90 0x7f0e200a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e300058c0 tx=0x7f0e30005850 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.089+0000 7f0e2b7fe640 1 -- 172.21.15.28:0/2514949988 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0e2c014070 con 0x7f0e3410e1f0 2024-05-09T08:26:23.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.312+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e341093e0 con 0x7f0e3410e1f0 2024-05-09T08:26:23.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.313+0000 7f0e2b7fe640 1 -- 172.21.15.28:0/2514949988 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11932 (secure 0 0 0) 0x7f0e2c09c190 con 0x7f0e3410e1f0 2024-05-09T08:26:23.314 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:23.314 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:26:16.129765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24463,24512,24476,24481,24499]},"id":1}]} 2024-05-09T08:26:23.314 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-05-09T08:26:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0e200a2d90 msgr2=0x7f0e200a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0e200a2d90 0x7f0e200a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e300058c0 tx=0x7f0e30005850 comp rx=0 tx=0).stop 2024-05-09T08:26:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3410e1f0 msgr2=0x7f0e3419eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3410e1f0 0x7f0e3419eb70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00ccb0 tx=0x7f0e2c00a620 comp rx=0 tx=0).stop 2024-05-09T08:26:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 shutdown_connections 2024-05-09T08:26:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3410e1f0 0x7f0e3419eb70 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e34100f60 0x7f0e3419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0e200a2d90 0x7f0e200a5250 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 --2- 172.21.15.28:0/2514949988 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e34100640 0x7f0e3419a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 >> 172.21.15.28:0/2514949988 conn(0x7f0e34077830 msgr2=0x7f0e3410f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 shutdown_connections 2024-05-09T08:26:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.315+0000 7f0e3c770640 1 -- 172.21.15.28:0/2514949988 wait complete. 2024-05-09T08:26:23.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.433+0000 7f643d86b640 1 -- 172.21.15.28:0/2749799515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438102250 msgr2=0x7f643810e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.433+0000 7f643d86b640 1 --2- 172.21.15.28:0/2749799515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438102250 0x7f643810e690 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6430009ca0 tx=0x7f643002a950 comp rx=0 tx=0).stop 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.433+0000 7f643d86b640 1 -- 172.21.15.28:0/2749799515 shutdown_connections 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.433+0000 7f643d86b640 1 --2- 172.21.15.28:0/2749799515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438102250 0x7f643810e690 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.433+0000 7f643d86b640 1 --2- 172.21.15.28:0/2749799515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64381018b0 0x7f6438101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.433+0000 7f643d86b640 1 --2- 172.21.15.28:0/2749799515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6438107840 0x7f6438107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.433+0000 7f643d86b640 1 -- 172.21.15.28:0/2749799515 >> 172.21.15.28:0/2749799515 conn(0x7f64380789c0 msgr2=0x7f6438078dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.433+0000 7f643d86b640 1 -- 172.21.15.28:0/2749799515 shutdown_connections 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.433+0000 7f643d86b640 1 -- 172.21.15.28:0/2749799515 wait complete. 2024-05-09T08:26:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f643d86b640 1 Processor -- start 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f643d86b640 1 -- start start 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f643d86b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64381018b0 0x7f64381980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f643d86b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6438102250 0x7f6438198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f643d86b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6438107840 0x7f643819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f643d86b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6438111c00 con 0x7f64381018b0 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f643d86b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6438111a80 con 0x7f6438102250 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f643d86b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6438111d80 con 0x7f6438107840 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f6436ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64381018b0 0x7f64381980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f6436ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64381018b0 0x7f64381980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38486/0 (socket says 172.21.15.28:38486) 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f6436ffd640 1 -- 172.21.15.28:0/1986092944 learned_addr learned my addr 172.21.15.28:0/1986092944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f642ffff640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6438102250 0x7f6438198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f64377fe640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6438107840 0x7f643819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f6436ffd640 1 -- 172.21.15.28:0/1986092944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6438107840 msgr2=0x7f643819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f6436ffd640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6438107840 0x7f643819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f6436ffd640 1 -- 172.21.15.28:0/1986092944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6438102250 msgr2=0x7f6438198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f6436ffd640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6438102250 0x7f6438198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f6436ffd640 1 -- 172.21.15.28:0/1986092944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643819d080 con 0x7f64381018b0 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f64377fe640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6438107840 0x7f643819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.434+0000 7f642ffff640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6438102250 0x7f6438198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.435+0000 7f6436ffd640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64381018b0 0x7f64381980e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f642800d950 tx=0x7f642800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.435+0000 7f643c869640 1 -- 172.21.15.28:0/1986092944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428017070 con 0x7f64381018b0 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.435+0000 7f643c869640 1 -- 172.21.15.28:0/1986092944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6428009c30 con 0x7f64381018b0 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.435+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f643819d370 con 0x7f64381018b0 2024-05-09T08:26:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.435+0000 7f643c869640 1 -- 172.21.15.28:0/1986092944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642800f6b0 con 0x7f64381018b0 2024-05-09T08:26:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.435+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64381a4c00 con 0x7f64381018b0 2024-05-09T08:26:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.436+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6438068aa0 con 0x7f64381018b0 2024-05-09T08:26:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.437+0000 7f643c869640 1 -- 172.21.15.28:0/1986092944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f64280109d0 con 0x7f64381018b0 2024-05-09T08:26:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.438+0000 7f643c869640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f64240a2e10 0x7f64240a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.438+0000 7f643c869640 1 -- 172.21.15.28:0/1986092944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f64280cf590 con 0x7f64381018b0 2024-05-09T08:26:23.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.438+0000 7f642ffff640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f64240a2e10 0x7f64240a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.438+0000 7f642ffff640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f64240a2e10 0x7f64240a52d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f64200058c0 tx=0x7f6420005850 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.442+0000 7f643c869640 1 -- 172.21.15.28:0/1986092944 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6428009de0 con 0x7f64381018b0 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Unable to set osd_memory_target on smithi028 to 185879859: error parsing value: Value '185879859' is below minimum 939524096 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi165:/etc/ceph/ceph.conf 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi184:/etc/ceph/ceph.conf 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:26:23.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:26:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:26:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:26:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:26:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-05-09T08:26:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4130361989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:23.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:23.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:23.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3878709543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:23.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:23 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2514949988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:23.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-05-09T08:26:23.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Unable to set osd_memory_target on smithi028 to 185879859: error parsing value: Value '185879859' is below minimum 939524096 2024-05-09T08:26:23.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-09T08:26:23.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi165:/etc/ceph/ceph.conf 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi184:/etc/ceph/ceph.conf 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4130361989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3878709543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:23.609 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:23 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2514949988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.633+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6438109d40 con 0x7f64381018b0 2024-05-09T08:26:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.634+0000 7f643c869640 1 -- 172.21.15.28:0/1986092944 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25274 (secure 0 0 0) 0x7f64280bb020 con 0x7f64381018b0 2024-05-09T08:26:23.635 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:23.636 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:26:17.965032+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3330063293":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3745037139":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/2130261041":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/273816117":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f64240a2e10 msgr2=0x7f64240a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f64240a2e10 0x7f64240a52d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f64200058c0 tx=0x7f6420005850 comp rx=0 tx=0).stop 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64381018b0 msgr2=0x7f64381980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64381018b0 0x7f64381980e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f642800d950 tx=0x7f642800de10 comp rx=0 tx=0).stop 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 shutdown_connections 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f64240a2e10 0x7f64240a52d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6438107840 0x7f643819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6438102250 0x7f6438198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 --2- 172.21.15.28:0/1986092944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64381018b0 0x7f64381980e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 >> 172.21.15.28:0/1986092944 conn(0x7f64380789c0 msgr2=0x7f64381005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 shutdown_connections 2024-05-09T08:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.637+0000 7f643d86b640 1 -- 172.21.15.28:0/1986092944 wait complete. 2024-05-09T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-05-09T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Unable to set osd_memory_target on smithi028 to 185879859: error parsing value: Value '185879859' is below minimum 939524096 2024-05-09T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-09T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi165:/etc/ceph/ceph.conf 2024-05-09T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi184:/etc/ceph/ceph.conf 2024-05-09T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: Updating smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4130361989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:23.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:23.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:23.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:26:23.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3878709543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:26:23.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:23 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2514949988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:23.652 DEBUG:tasks.fwd_scrub:Yielding 2024-05-09T08:26:23.652 INFO:teuthology.run_tasks:Running task exec... 2024-05-09T08:26:23.664 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:26:23.664 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:26:23.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-09T08:26:23.664 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-09T08:26:23.664 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:26:23.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:23.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1361219311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4101120 msgr2=0x7f3ce410d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1361219311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4101120 0x7f3ce410d5e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009930 tx=0x7f3cdc02a420 comp rx=0 tx=0).stop 2024-05-09T08:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1361219311 shutdown_connections 2024-05-09T08:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1361219311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4101120 0x7f3ce410d5e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1361219311 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce4100760 0x7f3ce4100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1361219311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ce4106720 0x7f3ce4106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1361219311 >> 172.21.15.28:0/1361219311 conn(0x7f3ce40778b0 msgr2=0x7f3ce4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1361219311 shutdown_connections 2024-05-09T08:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f8bcac7d640 1 -- 172.21.15.28:0/2594850652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4106710 msgr2=0x7f8bc4106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/2594850652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4106710 0x7f8bc4106af0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009930 tx=0x7f8bb802a420 comp rx=0 tx=0).stop 2024-05-09T08:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1361219311 wait complete. 2024-05-09T08:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f8bcac7d640 1 -- 172.21.15.28:0/2594850652 shutdown_connections 2024-05-09T08:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/2594850652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bc41010c0 0x7f8bc410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/2594850652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bc4100700 0x7f8bc4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/2594850652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4106710 0x7f8bc4106af0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f8bcac7d640 1 -- 172.21.15.28:0/2594850652 >> 172.21.15.28:0/2594850652 conn(0x7f8bc4077830 msgr2=0x7f8bc4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f8bcac7d640 1 -- 172.21.15.28:0/2594850652 shutdown_connections 2024-05-09T08:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.773+0000 7f8bcac7d640 1 -- 172.21.15.28:0/2594850652 wait complete. 2024-05-09T08:26:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce9b42640 1 Processor -- start 2024-05-09T08:26:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bcac7d640 1 Processor -- start 2024-05-09T08:26:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bcac7d640 1 -- start start 2024-05-09T08:26:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce9b42640 1 -- start start 2024-05-09T08:26:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bcac7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4100700 0x7f8bc4197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bcac7d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bc41010c0 0x7f8bc4197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bcac7d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bc4106710 0x7f8bc419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bcac7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc4110b20 con 0x7f8bc4100700 2024-05-09T08:26:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bcac7d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41109a0 con 0x7f8bc4106710 2024-05-09T08:26:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bcac7d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc4110ca0 con 0x7f8bc41010c0 2024-05-09T08:26:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bc8a19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4100700 0x7f8bc4197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bc8a19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4100700 0x7f8bc4197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38510/0 (socket says 172.21.15.28:38510) 2024-05-09T08:26:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce9b42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4100760 0x7f3ce4198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bc8a19640 1 -- 172.21.15.28:0/3475427859 learned_addr learned my addr 172.21.15.28:0/3475427859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce9b42640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce4101120 0x7f3ce4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce9b42640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ce4106720 0x7f3ce419c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce9b42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4110b80 con 0x7f3ce4100760 2024-05-09T08:26:23.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce9b42640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4110a00 con 0x7f3ce4106720 2024-05-09T08:26:23.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce9b42640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4110d00 con 0x7f3ce4101120 2024-05-09T08:26:23.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bc41010c0 0x7f8bc4197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4100760 0x7f3ce4198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f8bc921a640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bc4106710 0x7f8bc419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4100760 0x7f3ce4198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38518/0 (socket says 172.21.15.28:38518) 2024-05-09T08:26:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce37fe640 1 -- 172.21.15.28:0/1095652418 learned_addr learned my addr 172.21.15.28:0/1095652418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce2ffd640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce4101120 0x7f3ce4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.774+0000 7f3ce3fff640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ce4106720 0x7f3ce419c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bc8a19640 1 -- 172.21.15.28:0/3475427859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bc41010c0 msgr2=0x7f8bc4197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bc41010c0 0x7f8bc4197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bc8a19640 1 -- 172.21.15.28:0/3475427859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bc4106710 msgr2=0x7f8bc419b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bc4106710 0x7f8bc419b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bc8a19640 1 -- 172.21.15.28:0/3475427859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc419c030 con 0x7f8bc4100700 2024-05-09T08:26:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce37fe640 1 -- 172.21.15.28:0/1095652418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce4101120 msgr2=0x7f3ce4198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce37fe640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce4101120 0x7f3ce4198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce37fe640 1 -- 172.21.15.28:0/1095652418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ce4106720 msgr2=0x7f3ce419c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce37fe640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ce4106720 0x7f3ce419c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce37fe640 1 -- 172.21.15.28:0/1095652418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce419d100 con 0x7f3ce4100760 2024-05-09T08:26:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4100700 0x7f8bc4197040 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009230 tx=0x7f8bb8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce37fe640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4100760 0x7f3ce4198110 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400ebc0 tx=0x7f3cd400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bc1ffb640 1 -- 172.21.15.28:0/3475427859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb803f030 con 0x7f8bc4100700 2024-05-09T08:26:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bc1ffb640 1 -- 172.21.15.28:0/3475427859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bb8003ad0 con 0x7f8bc4100700 2024-05-09T08:26:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bcac7d640 1 -- 172.21.15.28:0/3475427859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc419c2c0 con 0x7f8bc4100700 2024-05-09T08:26:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bc1ffb640 1 -- 172.21.15.28:0/3475427859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8035070 con 0x7f8bc4100700 2024-05-09T08:26:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce0ff9640 1 -- 172.21.15.28:0/1095652418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4010630 con 0x7f3ce4100760 2024-05-09T08:26:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce0ff9640 1 -- 172.21.15.28:0/1095652418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cd4010cb0 con 0x7f3ce4100760 2024-05-09T08:26:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce0ff9640 1 -- 172.21.15.28:0/1095652418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4018bd0 con 0x7f3ce4100760 2024-05-09T08:26:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce419d3f0 con 0x7f3ce4100760 2024-05-09T08:26:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce41a4c80 con 0x7f3ce4100760 2024-05-09T08:26:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.775+0000 7f8bcac7d640 1 -- 172.21.15.28:0/3475427859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc41a3bb0 con 0x7f8bc4100700 2024-05-09T08:26:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.776+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce404f5f0 con 0x7f3ce4100760 2024-05-09T08:26:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.778+0000 7f3ce0ff9640 1 -- 172.21.15.28:0/1095652418 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f3cd40107d0 con 0x7f3ce4100760 2024-05-09T08:26:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.778+0000 7f8bc1ffb640 1 -- 172.21.15.28:0/3475427859 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f8bb8042030 con 0x7f8bc4100700 2024-05-09T08:26:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.778+0000 7f8bcd98c740 1 Processor -- start 2024-05-09T08:26:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.778+0000 7f8bcd98c740 1 -- start start 2024-05-09T08:26:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x5587944793f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5587944799c0 0x558794479e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558794486950 0x558794488d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5587943de990 con 0x55879447b0b0 2024-05-09T08:26:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5587943865b0 con 0x5587944799c0 2024-05-09T08:26:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5587943da950 con 0x558794486950 2024-05-09T08:26:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f3ce0ff9640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3ccc0a2ec0 0x7f3ccc0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc921a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x5587944793f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f3ce2ffd640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3ccc0a2ec0 0x7f3ccc0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc8a19640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5587944799c0 0x558794479e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc921a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x5587944793f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38528/0 (socket says 172.21.15.28:38528) 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc921a640 1 -- 172.21.15.28:0/1506400054 learned_addr learned my addr 172.21.15.28:0/1506400054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc1ffb640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8bb40a71a0 0x7f8bb40a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558794486950 0x558794488d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc921a640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8bb40a71a0 0x7f8bb40a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc921a640 1 -- 172.21.15.28:0/1506400054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558794486950 msgr2=0x558794488d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc921a640 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558794486950 0x558794488d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc921a640 1 -- 172.21.15.28:0/1506400054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5587944799c0 msgr2=0x558794479e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc1ffb640 1 -- 172.21.15.28:0/3475427859 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f8bb80efd70 con 0x7f8bc4100700 2024-05-09T08:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f3ce0ff9640 1 -- 172.21.15.28:0/1095652418 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3cd40d0100 con 0x7f3ce4100760 2024-05-09T08:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc921a640 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5587944799c0 0x558794479e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.779+0000 7f8bc921a640 1 -- 172.21.15.28:0/1506400054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587941ba7e0 con 0x55879447b0b0 2024-05-09T08:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.780+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558794486950 0x558794488d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.780+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5587944799c0 0x558794479e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.780+0000 7f3ce2ffd640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3ccc0a2ec0 0x7f3ccc0a5380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4077fe0 tx=0x7f3cd800b410 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.780+0000 7f8bc921a640 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x5587944793f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc005f40 tx=0x7f8bbc012f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc5ec490f2275e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.780+0000 7f8bc921a640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8bb40a71a0 0x7f8bb40a9660 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc021000 tx=0x7f8bbc01f000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8baeffd640 1 -- 172.21.15.28:0/1506400054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc0124b0 con 0x55879447b0b0 2024-05-09T08:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8baeffd640 1 -- 172.21.15.28:0/1506400054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bbc022070 con 0x55879447b0b0 2024-05-09T08:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8baeffd640 1 -- 172.21.15.28:0/1506400054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00fe60 con 0x55879447b0b0 2024-05-09T08:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f3ce0ff9640 1 -- 172.21.15.28:0/1095652418 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3cd409c170 con 0x7f3ce4100760 2024-05-09T08:26:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1506400054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 msgr2=0x5587944793f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x5587944793f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc005f40 tx=0x7f8bbc012f00 comp rx=0 tx=0).stop 2024-05-09T08:26:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1506400054 shutdown_connections 2024-05-09T08:26:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558794486950 0x558794488d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5587944799c0 0x558794479e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1506400054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x5587944793f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1506400054 >> 172.21.15.28:0/1506400054 conn(0x5587944420a0 msgr2=0x558794475f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1506400054 shutdown_connections 2024-05-09T08:26:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.783+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1506400054 wait complete. 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bcd98c740 1 -- start start 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bcd98c740 1 Processor -- start 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 0x558794481ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55879447b0b0 0x558794482430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558794486950 0x5587944905e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55879447c890 con 0x558794486950 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558794399b40 con 0x55879447b0b0 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55879447ca10 con 0x5587944799c0 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bcd98c740 10 client.0 _ll_register_callbacks cb 0x55879438e0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bc921a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558794486950 0x5587944905e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bc921a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558794486950 0x5587944905e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38542/0 (socket says 172.21.15.28:38542) 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.784+0000 7f8bc921a640 1 -- 172.21.15.28:0/1291136385 learned_addr learned my addr 172.21.15.28:0/1291136385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 0x558794481ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55879447b0b0 0x558794482430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bc921a640 1 -- 172.21.15.28:0/1291136385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 msgr2=0x558794481ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bc921a640 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 0x558794481ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bc921a640 1 -- 172.21.15.28:0/1291136385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55879447b0b0 msgr2=0x558794482430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bc921a640 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55879447b0b0 0x558794482430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bc921a640 1 -- 172.21.15.28:0/1291136385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55879421af70 con 0x558794486950 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bc921a640 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558794486950 0x5587944905e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc006010 tx=0x7f8bbc00fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc5880be5baea6a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8ba3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1291136385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc012490 con 0x558794486950 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1291136385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bbc022070 con 0x558794486950 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1291136385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f8bbc01db60 con 0x558794486950 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1291136385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8bbc039050 con 0x558794486950 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bcd98c740 10 client.15741 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.785+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1291136385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5587940b2170 con 0x558794486950 2024-05-09T08:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.786+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1291136385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bbc022070 con 0x558794486950 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.786+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1291136385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8bbc026800 con 0x558794486950 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.786+0000 7f8bcd98c740 10 client.15741 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.786+0000 7f8bcd98c740 10 client.15741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.786+0000 7f8bcd98c740 20 client.15741 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.786+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5587944947f0 0x558794496be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.786+0000 7f8bcd98c740 4 client.15741 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.786+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1291136385 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587943ec0c0 con 0x5587944947f0 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.786+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5587944947f0 0x558794496be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8ba3fff640 10 client.15741 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5587944947f0 0x558794496be0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54c1ad0e9b8bcf3a server_cookie=b099a3dd2b463738 in_seq=0 out_seq=0 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1291136385 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587943ec0c0 con 0x5587944947f0 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8ba3fff640 10 client.15741 handle_command_reply: tid=0 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bcd98c740 1 client.15741 shutdown 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1291136385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558794486950 msgr2=0x5587944905e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558794486950 0x5587944905e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc006010 tx=0x7f8bbc00fe10 comp rx=0 tx=0).stop 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1291136385 shutdown_connections 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5587944947f0 0x558794496be0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558794486950 0x5587944905e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55879447b0b0 0x558794482430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8ba3fff640 0 client.15741 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1291136385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 0x558794481ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.787+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1291136385 >> 172.21.15.28:0/1291136385 conn(0x5587944420a0 msgr2=0x55879447bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.788+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1291136385 shutdown_connections 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.788+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1291136385 wait complete. 2024-05-09T08:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.788+0000 7f8bcd98c740 20 client.15741 trim_cache size 0 max 16384 2024-05-09T08:26:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.805+0000 7f8bcd98c740 1 Processor -- start 2024-05-09T08:26:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.805+0000 7f8bcd98c740 1 -- start start 2024-05-09T08:26:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 0x55879449c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x55879449c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 0x5587944905e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41acc80 con 0x55879447b0b0 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41ae7c0 con 0x558794486950 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41ad350 con 0x5587944799c0 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x55879449c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x55879449c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38548/0 (socket says 172.21.15.28:38548) 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc3fff640 1 -- 172.21.15.28:0/4120676458 learned_addr learned my addr 172.21.15.28:0/4120676458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc921a640 1 --2- 172.21.15.28:0/4120676458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 0x5587944905e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/4120676458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 0x55879449c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc3fff640 1 -- 172.21.15.28:0/4120676458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 msgr2=0x55879449c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/4120676458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 0x55879449c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc3fff640 1 -- 172.21.15.28:0/4120676458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 msgr2=0x5587944905e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/4120676458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 0x5587944905e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc3fff640 1 -- 172.21.15.28:0/4120676458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558794497170 con 0x55879447b0b0 2024-05-09T08:26:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.806+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/4120676458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x55879449c8e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009d60 tx=0x7f8bb0010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d305f7608d534c57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bae7fc640 1 -- 172.21.15.28:0/4120676458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0004310 con 0x55879447b0b0 2024-05-09T08:26:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bae7fc640 1 -- 172.21.15.28:0/4120676458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bb00044b0 con 0x55879447b0b0 2024-05-09T08:26:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bae7fc640 1 -- 172.21.15.28:0/4120676458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb001c9a0 con 0x55879447b0b0 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bcd98c740 1 -- 172.21.15.28:0/4120676458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 msgr2=0x55879449c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/4120676458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x55879449c8e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009d60 tx=0x7f8bb0010800 comp rx=0 tx=0).stop 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bcd98c740 1 -- 172.21.15.28:0/4120676458 shutdown_connections 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/4120676458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 0x5587944905e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/4120676458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55879447b0b0 0x55879449c8e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/4120676458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5587944799c0 0x55879449c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bcd98c740 1 -- 172.21.15.28:0/4120676458 >> 172.21.15.28:0/4120676458 conn(0x5587944420a0 msgr2=0x558794475080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bcd98c740 1 -- 172.21.15.28:0/4120676458 shutdown_connections 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.807+0000 7f8bcd98c740 1 -- 172.21.15.28:0/4120676458 wait complete. 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bcd98c740 1 -- start start 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bcd98c740 1 Processor -- start 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587944799c0 0x55879448ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55879447b0b0 0x55879448bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bcd98c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 0x55879448c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41ae0f0 con 0x5587944799c0 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41ada20 con 0x558794486950 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bcd98c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41aac60 con 0x55879447b0b0 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bcd98c740 10 client.0 _ll_register_callbacks cb 0x7f8bb0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bc8a19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587944799c0 0x55879448ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bc8a19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587944799c0 0x55879448ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38558/0 (socket says 172.21.15.28:38558) 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bc8a19640 1 -- 172.21.15.28:0/1377212277 learned_addr learned my addr 172.21.15.28:0/1377212277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.808+0000 7f8bc921a640 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 0x55879448c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55879447b0b0 0x55879448bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bc8a19640 1 -- 172.21.15.28:0/1377212277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55879447b0b0 msgr2=0x55879448bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55879447b0b0 0x55879448bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bc8a19640 1 -- 172.21.15.28:0/1377212277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 msgr2=0x55879448c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 0x55879448c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bc8a19640 1 -- 172.21.15.28:0/1377212277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587944f9c10 con 0x5587944799c0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bc8a19640 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587944799c0 0x55879448ba70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80b8260 tx=0x7f8bb80f2490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a129aba4ec37a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8ba3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1377212277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8049040 con 0x5587944799c0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1377212277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f8bb80ba100 con 0x5587944799c0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1377212277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bb00044b0 con 0x5587944799c0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1377212277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8bb80c0100 con 0x5587944799c0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bcd98c740 10 client.15747 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1377212277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5587944faf90 con 0x5587944799c0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1377212277 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bb00044b0 con 0x5587944799c0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.809+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1377212277 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8bb80ed2e0 con 0x5587944799c0 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.810+0000 7f8bcd98c740 10 client.15747 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.810+0000 7f8bcd98c740 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.810+0000 7f8bcd98c740 20 client.15747 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.810+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5587944fb710 0x5587944fdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.810+0000 7f8bcd98c740 4 client.15747 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.810+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1377212277 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5587943e6410 con 0x5587944fb710 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.810+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5587944fb710 0x5587944fdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.810+0000 7f8bc3fff640 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5587944fb710 0x5587944fdb00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b48b013351d4e30 server_cookie=4252464fc5c44c1b in_seq=0 out_seq=0 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.810+0000 7f8ba3fff640 10 client.15747 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.811+0000 7f8ba3fff640 1 -- 172.21.15.28:0/1377212277 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5587943e6410 con 0x5587944fb710 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.811+0000 7f8ba3fff640 10 client.15747 handle_command_reply: tid=0 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.811+0000 7f8bcd98c740 1 client.15747 shutdown 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.811+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1377212277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587944799c0 msgr2=0x55879448ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.811+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587944799c0 0x55879448ba70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80b8260 tx=0x7f8bb80f2490 comp rx=0 tx=0).stop 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.811+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1377212277 shutdown_connections 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.811+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5587944fb710 0x5587944fdb00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.811+0000 7f8ba3fff640 0 client.15747 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.811+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558794486950 0x55879448c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55879447b0b0 0x55879448bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcd98c740 1 --2- 172.21.15.28:0/1377212277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587944799c0 0x55879448ba70 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1377212277 >> 172.21.15.28:0/1377212277 conn(0x5587944420a0 msgr2=0x55879447b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1377212277 shutdown_connections 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcd98c740 1 -- 172.21.15.28:0/1377212277 wait complete. 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcd98c740 20 client.15747 trim_cache size 0 max 16384 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "ccea587c-a3a8-4523-b712-2c36df00d164", 2024-05-09T08:26:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-09T08:26:23.819 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 -- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8bb40a71a0 msgr2=0x7f8bb40a9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8bb40a71a0 0x7f8bb40a9660 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc021000 tx=0x7f8bbc01f000 comp rx=0 tx=0).stop 2024-05-09T08:26:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 -- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4100700 msgr2=0x7f8bc4197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4100700 0x7f8bc4197040 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009230 tx=0x7f8bb8004270 comp rx=0 tx=0).stop 2024-05-09T08:26:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 -- 172.21.15.28:0/3475427859 shutdown_connections 2024-05-09T08:26:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bc4106710 0x7f8bc419b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8bb40a71a0 0x7f8bb40a9660 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8bc41010c0 0x7f8bc4197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 --2- 172.21.15.28:0/3475427859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bc4100700 0x7f8bc4197040 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.812+0000 7f8bcac7d640 1 -- 172.21.15.28:0/3475427859 >> 172.21.15.28:0/3475427859 conn(0x7f8bc4077830 msgr2=0x7f8bc4102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.813+0000 7f8bcac7d640 1 -- 172.21.15.28:0/3475427859 shutdown_connections 2024-05-09T08:26:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.813+0000 7f8bcac7d640 1 -- 172.21.15.28:0/3475427859 wait complete. 2024-05-09T08:26:23.824 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:26:23.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:26:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.932+0000 7f3f1625d640 1 -- 172.21.15.28:0/834383514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10102240 msgr2=0x7f3f1010e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.932+0000 7f3f1625d640 1 --2- 172.21.15.28:0/834383514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10102240 0x7f3f1010e750 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3f08009cd0 tx=0x7f3f0802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.932+0000 7f3f1625d640 1 -- 172.21.15.28:0/834383514 shutdown_connections 2024-05-09T08:26:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.932+0000 7f3f1625d640 1 --2- 172.21.15.28:0/834383514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10102240 0x7f3f1010e750 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.932+0000 7f3f1625d640 1 --2- 172.21.15.28:0/834383514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f10101880 0x7f3f10101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.932+0000 7f3f1625d640 1 --2- 172.21.15.28:0/834383514 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f10107890 0x7f3f10107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.932+0000 7f3f1625d640 1 -- 172.21.15.28:0/834383514 >> 172.21.15.28:0/834383514 conn(0x7f3f100fd390 msgr2=0x7f3f100ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.932+0000 7f3f1625d640 1 -- 172.21.15.28:0/834383514 shutdown_connections 2024-05-09T08:26:23.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.932+0000 7f3f1625d640 1 -- 172.21.15.28:0/834383514 wait complete. 2024-05-09T08:26:23.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f1625d640 1 Processor -- start 2024-05-09T08:26:23.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f1625d640 1 -- start start 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f1625d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10101880 0x7f3f10198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f1625d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f10102240 0x7f3f101986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f1625d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f10107890 0x7f3f1019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f1625d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f10111ca0 con 0x7f3f10101880 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f1625d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f10111b20 con 0x7f3f10107890 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f1625d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f10111e20 con 0x7f3f10102240 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10101880 0x7f3f10198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10101880 0x7f3f10198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38576/0 (socket says 172.21.15.28:38576) 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f0f7fe640 1 -- 172.21.15.28:0/2902795551 learned_addr learned my addr 172.21.15.28:0/2902795551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f10107890 0x7f3f1019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f10102240 0x7f3f101986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f0f7fe640 1 -- 172.21.15.28:0/2902795551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f10102240 msgr2=0x7f3f101986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f10102240 0x7f3f101986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f0f7fe640 1 -- 172.21.15.28:0/2902795551 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f10107890 msgr2=0x7f3f1019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f10107890 0x7f3f1019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f0f7fe640 1 -- 172.21.15.28:0/2902795551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1019d170 con 0x7f3f10101880 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.933+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f10102240 0x7f3f101986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.934+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10101880 0x7f3f10198180 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000d950 tx=0x7f3f0000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.934+0000 7f3f0d7fa640 1 -- 172.21.15.28:0/2902795551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00017070 con 0x7f3f10101880 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.934+0000 7f3f0d7fa640 1 -- 172.21.15.28:0/2902795551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f00009c30 con 0x7f3f10101880 2024-05-09T08:26:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.934+0000 7f3f1625d640 1 -- 172.21.15.28:0/2902795551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1019d460 con 0x7f3f10101880 2024-05-09T08:26:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.934+0000 7f3f0d7fa640 1 -- 172.21.15.28:0/2902795551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0000f6b0 con 0x7f3f10101880 2024-05-09T08:26:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.934+0000 7f3f1625d640 1 -- 172.21.15.28:0/2902795551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f101a4cf0 con 0x7f3f10101880 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.936+0000 7f3f0d7fa640 1 -- 172.21.15.28:0/2902795551 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f3f0000f850 con 0x7f3f10101880 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.936+0000 7f3f18f6b740 1 Processor -- start 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.936+0000 7f3f18f6b740 1 -- start start 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822462660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 0x56282246f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5628223c7d00 con 0x562822462ba0 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56282236f4c0 con 0x562822462280 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5628223c3cc0 con 0x56282246d270 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38582/0 (socket says 172.21.15.28:38582) 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0f7fe640 1 -- 172.21.15.28:0/2200825291 learned_addr learned my addr 172.21.15.28:0/2200825291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/2200825291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822462660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/2200825291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 0x56282246f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0d7fa640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3efc0a2d90 0x7f3efc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0ffff640 1 -- 172.21.15.28:0/2200825291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 msgr2=0x56282246f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/2200825291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 0x56282246f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0ffff640 1 -- 172.21.15.28:0/2200825291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 msgr2=0x56282246cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/2200825291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0ffff640 1 -- 172.21.15.28:0/2200825291 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628221a57d0 con 0x562822462280 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0d7fa640 1 -- 172.21.15.28:0/2902795551 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3f000cf670 con 0x7f3f10101880 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.937+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3efc0a2d90 0x7f3efc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.938+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/2200825291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822462660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3f08009ca0 tx=0x7f3f080315b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38bef4f71e8e338d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.938+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3efc0a2d90 0x7f3efc0a5250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x56282246fcb0 tx=0x7f3f0804c000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.938+0000 7f3f05ffb640 1 -- 172.21.15.28:0/2200825291 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f08041040 con 0x562822462280 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.938+0000 7f3f05ffb640 1 -- 172.21.15.28:0/2200825291 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f0804c070 con 0x562822462280 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.938+0000 7f3f05ffb640 1 -- 172.21.15.28:0/2200825291 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f08030d60 con 0x562822462280 2024-05-09T08:26:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 -- 172.21.15.28:0/2200825291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 msgr2=0x562822462660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/2200825291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822462660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3f08009ca0 tx=0x7f3f080315b0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 -- 172.21.15.28:0/2200825291 shutdown_connections 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/2200825291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 0x56282246f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/2200825291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/2200825291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822462660 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 -- 172.21.15.28:0/2200825291 >> 172.21.15.28:0/2200825291 conn(0x56282245e5a0 msgr2=0x56282245e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 -- 172.21.15.28:0/2200825291 shutdown_connections 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 -- 172.21.15.28:0/2200825291 wait complete. 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 -- start start 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 Processor -- start 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562822462280 0x56282246a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56282246d270 0x56282246ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562822470060 con 0x562822462ba0 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x562822382b90 con 0x56282246d270 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.939+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5628224701e0 con 0x562822462280 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f18f6b740 10 client.0 _ll_register_callbacks cb 0x5628223beda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38598/0 (socket says 172.21.15.28:38598) 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f06ffd640 1 -- 172.21.15.28:0/1695168232 learned_addr learned my addr 172.21.15.28:0/1695168232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56282246d270 0x56282246ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562822462280 0x56282246a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f06ffd640 1 -- 172.21.15.28:0/1695168232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562822462280 msgr2=0x56282246a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562822462280 0x56282246a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f06ffd640 1 -- 172.21.15.28:0/1695168232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56282246d270 msgr2=0x56282246ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56282246d270 0x56282246ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f06ffd640 1 -- 172.21.15.28:0/1695168232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628220db3f0 con 0x562822462ba0 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246a730 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8012480 tx=0x7f3ef8012940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec5670d08cf37d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3eeeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1695168232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef801f060 con 0x562822462ba0 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1695168232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3ef8010aa0 con 0x562822462ba0 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.940+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1695168232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f0804c070 con 0x562822462ba0 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1695168232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3ef8026050 con 0x562822462ba0 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3f18f6b740 10 client.15759 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1695168232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56282209b0f0 con 0x562822462ba0 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1695168232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f0804c070 con 0x562822462ba0 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1695168232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3ef8019030 con 0x562822462ba0 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3f18f6b740 10 client.15759 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3f18f6b740 10 client.15759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3f18f6b740 20 client.15759 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56282247db50 0x56282247ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3f18f6b740 4 client.15759 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1695168232 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628223d5430 con 0x56282247db50 2024-05-09T08:26:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.941+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56282247db50 0x56282247ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.942+0000 7f3eeeffd640 10 client.15759 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.942+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56282247db50 0x56282247ffb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91ed6ffd2ef3375c server_cookie=3cfaefd109420743 in_seq=0 out_seq=0 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.942+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1695168232 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628223d5430 con 0x56282247db50 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.942+0000 7f3eeeffd640 10 client.15759 handle_command_reply: tid=0 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.942+0000 7f3f18f6b740 1 client.15759 shutdown 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.942+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1695168232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 msgr2=0x56282246a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.942+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246a730 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8012480 tx=0x7f3ef8012940 comp rx=0 tx=0).stop 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1695168232 shutdown_connections 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56282247db50 0x56282247ffb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56282246d270 0x56282246ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x56282246a730 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3eeeffd640 0 client.15759 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1695168232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562822462280 0x56282246a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1695168232 >> 172.21.15.28:0/1695168232 conn(0x56282245e5a0 msgr2=0x56282245d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1695168232 shutdown_connections 2024-05-09T08:26:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1695168232 wait complete. 2024-05-09T08:26:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.943+0000 7f3f18f6b740 20 client.15759 trim_cache size 0 max 16384 2024-05-09T08:26:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.960+0000 7f3f18f6b740 1 Processor -- start 2024-05-09T08:26:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.960+0000 7f3f18f6b740 1 -- start start 2024-05-09T08:26:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.960+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562822462280 0x562822459670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.960+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462ba0 0x562822485660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.960+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282246d270 0x562822485ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.960+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f101aed40 con 0x56282246d270 2024-05-09T08:26:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.960+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f101b0880 con 0x562822462ba0 2024-05-09T08:26:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.960+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f101af410 con 0x562822462280 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282246d270 0x562822485ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282246d270 0x562822485ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38608/0 (socket says 172.21.15.28:38608) 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f0ffff640 1 -- 172.21.15.28:0/3348123754 learned_addr learned my addr 172.21.15.28:0/3348123754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/3348123754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462ba0 0x562822485660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f0ffff640 1 -- 172.21.15.28:0/3348123754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562822462280 msgr2=0x562822459670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/3348123754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562822462280 0x562822459670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f0ffff640 1 -- 172.21.15.28:0/3348123754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462ba0 msgr2=0x562822485660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/3348123754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462ba0 0x562822485660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f0ffff640 1 -- 172.21.15.28:0/3348123754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628224691a0 con 0x56282246d270 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/3348123754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282246d270 0x562822485ba0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3f0802ae00 tx=0x7f3f0802a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12615899207a3bbb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f057fa640 1 -- 172.21.15.28:0/3348123754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0805f070 con 0x56282246d270 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f057fa640 1 -- 172.21.15.28:0/3348123754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f08043070 con 0x56282246d270 2024-05-09T08:26:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.961+0000 7f3f057fa640 1 -- 172.21.15.28:0/3348123754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f08041040 con 0x56282246d270 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 -- 172.21.15.28:0/3348123754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282246d270 msgr2=0x562822485ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/3348123754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282246d270 0x562822485ba0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3f0802ae00 tx=0x7f3f0802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 -- 172.21.15.28:0/3348123754 shutdown_connections 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/3348123754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282246d270 0x562822485ba0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/3348123754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462ba0 0x562822485660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/3348123754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562822462280 0x562822459670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 -- 172.21.15.28:0/3348123754 >> 172.21.15.28:0/3348123754 conn(0x56282245e5a0 msgr2=0x56282246aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 -- 172.21.15.28:0/3348123754 shutdown_connections 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 -- 172.21.15.28:0/3348123754 wait complete. 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.962+0000 7f3f18f6b740 1 -- start start 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f18f6b740 1 Processor -- start 2024-05-09T08:26:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822483240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x562822483780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f18f6b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 0x562822483cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f101b01b0 con 0x562822462ba0 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f101afae0 con 0x562822462280 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f18f6b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f101acd20 con 0x56282246d270 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f18f6b740 10 client.0 _ll_register_callbacks cb 0x7f3ef800b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x562822483780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x562822483780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38610/0 (socket says 172.21.15.28:38610) 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f06ffd640 1 -- 172.21.15.28:0/1242009251 learned_addr learned my addr 172.21.15.28:0/1242009251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f0ffff640 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 0x562822483cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822483240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f06ffd640 1 -- 172.21.15.28:0/1242009251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 msgr2=0x562822483cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 0x562822483cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f06ffd640 1 -- 172.21.15.28:0/1242009251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 msgr2=0x562822483240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822483240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f06ffd640 1 -- 172.21.15.28:0/1242009251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628224e3220 con 0x562822462ba0 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822483240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3f06ffd640 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x562822483780 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x562822469a50 tx=0x7f3ef800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab58237f21a6fca8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.963+0000 7f3eeeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1242009251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef802e070 con 0x562822462ba0 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1242009251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3ef8010cb0 con 0x562822462ba0 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1242009251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f08043070 con 0x562822462ba0 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1242009251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3ef803a050 con 0x562822462ba0 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f18f6b740 10 client.15765 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1242009251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5628224e44f0 con 0x562822462ba0 2024-05-09T08:26:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1242009251 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f08043070 con 0x562822462ba0 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1242009251 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3ef8019040 con 0x562822462ba0 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f18f6b740 10 client.15765 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f18f6b740 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f18f6b740 20 client.15765 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5628224e4c10 0x5628224e7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f18f6b740 4 client.15765 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1242009251 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5628223cf780 con 0x5628224e4c10 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.964+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5628224e4c10 0x5628224e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f0f7fe640 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5628224e4c10 0x5628224e7030 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e82340eb246cc52e server_cookie=a70815f5eb95f3a2 in_seq=0 out_seq=0 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3eeeffd640 10 client.15765 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3eeeffd640 1 -- 172.21.15.28:0/1242009251 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5628223cf780 con 0x5628224e4c10 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3eeeffd640 10 client.15765 handle_command_reply: tid=0 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 client.15765 shutdown 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1242009251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 msgr2=0x562822483780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x562822483780 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x562822469a50 tx=0x7f3ef800b840 comp rx=0 tx=0).stop 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1242009251 shutdown_connections 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5628224e4c10 0x5628224e7030 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56282246d270 0x562822483cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562822462ba0 0x562822483780 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3eeeffd640 0 client.15765 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 --2- 172.21.15.28:0/1242009251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562822462280 0x562822483240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1242009251 >> 172.21.15.28:0/1242009251 conn(0x56282245e5a0 msgr2=0x5628224639d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1242009251 shutdown_connections 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.965+0000 7f3f18f6b740 1 -- 172.21.15.28:0/1242009251 wait complete. 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f18f6b740 20 client.15765 trim_cache size 0 max 16384 2024-05-09T08:26:23.972 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:26:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-09T08:26:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "ccea587c-a3a8-4523-b712-2c36df00d164": { 2024-05-09T08:26:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-09T08:26:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "ccea587c-a3a8-4523-b712-2c36df00d164", 2024-05-09T08:26:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-09T08:26:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:26:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:26:23.973 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 -- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3efc0a2d90 msgr2=0x7f3efc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3efc0a2d90 0x7f3efc0a5250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x56282246fcb0 tx=0x7f3f0804c000 comp rx=0 tx=0).stop 2024-05-09T08:26:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 -- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10101880 msgr2=0x7f3f10198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10101880 0x7f3f10198180 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000d950 tx=0x7f3f0000de10 comp rx=0 tx=0).stop 2024-05-09T08:26:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 -- 172.21.15.28:0/2902795551 shutdown_connections 2024-05-09T08:26:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f10107890 0x7f3f1019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3efc0a2d90 0x7f3efc0a5250 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f10102240 0x7f3f101986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 --2- 172.21.15.28:0/2902795551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f10101880 0x7f3f10198180 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 -- 172.21.15.28:0/2902795551 >> 172.21.15.28:0/2902795551 conn(0x7f3f100fd390 msgr2=0x7f3f100fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 -- 172.21.15.28:0/2902795551 shutdown_connections 2024-05-09T08:26:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:23.966+0000 7f3f1625d640 1 -- 172.21.15.28:0/2902795551 wait complete. 2024-05-09T08:26:23.978 INFO:tasks.cephfs.filesystem:scrub status for tag:ccea587c-a3a8-4523-b712-2c36df00d164 - {'path': '/', 'tag': 'ccea587c-a3a8-4523-b712-2c36df00d164', 'options': 'recursive,force'} 2024-05-09T08:26:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.001+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 --> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f3ce4061c70 con 0x7f3ccc0a2ec0 2024-05-09T08:26:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.064+0000 7f3ce0ff9640 1 -- 172.21.15.28:0/1095652418 <== mgr.15528 v2:172.21.15.28:6800/160415970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3ce4061c70 con 0x7f3ccc0a2ec0 2024-05-09T08:26:24.066 INFO:teuthology.orchestra.run.smithi028.stdout:Schedule set for path / 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3ccc0a2ec0 msgr2=0x7f3ccc0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3ccc0a2ec0 0x7f3ccc0a5380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4077fe0 tx=0x7f3cd800b410 comp rx=0 tx=0).stop 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4100760 msgr2=0x7f3ce4198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4100760 0x7f3ce4198110 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400ebc0 tx=0x7f3cd400c6a0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 shutdown_connections 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ce4106720 0x7f3ce419c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3ccc0a2ec0 0x7f3ccc0a5380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ce4101120 0x7f3ce4198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 --2- 172.21.15.28:0/1095652418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce4100760 0x7f3ce4198110 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 >> 172.21.15.28:0/1095652418 conn(0x7f3ce40778b0 msgr2=0x7f3ce4069220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 shutdown_connections 2024-05-09T08:26:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.066+0000 7f3ce9b42640 1 -- 172.21.15.28:0/1095652418 wait complete. 2024-05-09T08:26:24.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-09T08:26:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 -- 172.21.15.28:0/4230914677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 msgr2=0x7fe020107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 --2- 172.21.15.28:0/4230914677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 0x7fe020107c30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe010009cd0 tx=0x7fe01002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 -- 172.21.15.28:0/4230914677 shutdown_connections 2024-05-09T08:26:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 --2- 172.21.15.28:0/4230914677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe020102200 0x7fe02010e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 --2- 172.21.15.28:0/4230914677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe020101840 0x7fe020101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 --2- 172.21.15.28:0/4230914677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 0x7fe020107c30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 -- 172.21.15.28:0/4230914677 >> 172.21.15.28:0/4230914677 conn(0x7fe0200fd330 msgr2=0x7fe0200ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 -- 172.21.15.28:0/4230914677 shutdown_connections 2024-05-09T08:26:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 -- 172.21.15.28:0/4230914677 wait complete. 2024-05-09T08:26:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 Processor -- start 2024-05-09T08:26:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.226+0000 7fe024b5c640 1 -- start start 2024-05-09T08:26:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe024b5c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe020101840 0x7fe0201980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe024b5c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe020102200 0x7fe020198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe024b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 0x7fe02019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe024b5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe020111c60 con 0x7fe020107850 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe024b5c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe020111ae0 con 0x7fe020101840 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe024b5c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe020111de0 con 0x7fe020102200 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 0x7fe02019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 0x7fe02019c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38614/0 (socket says 172.21.15.28:38614) 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01ed9d640 1 -- 172.21.15.28:0/773523839 learned_addr learned my addr 172.21.15.28:0/773523839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01dd9b640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe020102200 0x7fe020198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01e59c640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe020101840 0x7fe0201980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01ed9d640 1 -- 172.21.15.28:0/773523839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe020102200 msgr2=0x7fe020198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01ed9d640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe020102200 0x7fe020198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01ed9d640 1 -- 172.21.15.28:0/773523839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe020101840 msgr2=0x7fe0201980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01ed9d640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe020101840 0x7fe0201980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01ed9d640 1 -- 172.21.15.28:0/773523839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02019d080 con 0x7fe020107850 2024-05-09T08:26:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe01ed9d640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 0x7fe02019c950 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe01800d750 tx=0x7fe01800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe0177fe640 1 -- 172.21.15.28:0/773523839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018017070 con 0x7fe020107850 2024-05-09T08:26:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe0177fe640 1 -- 172.21.15.28:0/773523839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe018005500 con 0x7fe020107850 2024-05-09T08:26:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe02019d370 con 0x7fe020107850 2024-05-09T08:26:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.227+0000 7fe0177fe640 1 -- 172.21.15.28:0/773523839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018004d90 con 0x7fe020107850 2024-05-09T08:26:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.228+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0201a4c00 con 0x7fe020107850 2024-05-09T08:26:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.229+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe02004f5f0 con 0x7fe020107850 2024-05-09T08:26:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.230+0000 7fe0177fe640 1 -- 172.21.15.28:0/773523839 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7fe018004f30 con 0x7fe020107850 2024-05-09T08:26:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.230+0000 7fe0177fe640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe00c0a2e40 0x7fe00c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.230+0000 7fe01e59c640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe00c0a2e40 0x7fe00c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.230+0000 7fe0177fe640 1 -- 172.21.15.28:0/773523839 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fe0180cf210 con 0x7fe020107850 2024-05-09T08:26:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.231+0000 7fe01e59c640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe00c0a2e40 0x7fe00c0a5300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe010006010 tx=0x7fe010009360 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.235+0000 7fe0177fe640 1 -- 172.21.15.28:0/773523839 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe01809b2a0 con 0x7fe020107850 2024-05-09T08:26:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.445+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 --> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fe020061c70 con 0x7fe00c0a2e40 2024-05-09T08:26:24.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.506+0000 7fe0177fe640 1 -- 172.21.15.28:0/773523839 <== mgr.15528 v2:172.21.15.28:6800/160415970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fe020061c70 con 0x7fe00c0a2e40 2024-05-09T08:26:24.507 INFO:teuthology.orchestra.run.smithi028.stdout:Retention added to path / 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe00c0a2e40 msgr2=0x7fe00c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe00c0a2e40 0x7fe00c0a5300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe010006010 tx=0x7fe010009360 comp rx=0 tx=0).stop 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 msgr2=0x7fe02019c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 0x7fe02019c950 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe01800d750 tx=0x7fe01800dc10 comp rx=0 tx=0).stop 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 shutdown_connections 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020107850 0x7fe02019c950 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fe00c0a2e40 0x7fe00c0a5300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe020102200 0x7fe020198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 --2- 172.21.15.28:0/773523839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe020101840 0x7fe0201980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 >> 172.21.15.28:0/773523839 conn(0x7fe0200fd330 msgr2=0x7fe0200fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 shutdown_connections 2024-05-09T08:26:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.508+0000 7fe024b5c640 1 -- 172.21.15.28:0/773523839 wait complete. 2024-05-09T08:26:24.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-09T08:26:24.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:24 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1986092944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:24.608 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:24 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1986092944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.625+0000 7fc18029e640 1 -- 172.21.15.28:0/3240761874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 msgr2=0x7fc178102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.625+0000 7fc18029e640 1 --2- 172.21.15.28:0/3240761874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 0x7fc178102ec0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc174009900 tx=0x7fc17402a480 comp rx=0 tx=0).stop 2024-05-09T08:26:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.625+0000 7fc18029e640 1 -- 172.21.15.28:0/3240761874 shutdown_connections 2024-05-09T08:26:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.625+0000 7fc18029e640 1 --2- 172.21.15.28:0/3240761874 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc17810beb0 0x7fc17810e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.625+0000 7fc18029e640 1 --2- 172.21.15.28:0/3240761874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc178103400 0x7fc17810b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.625+0000 7fc18029e640 1 --2- 172.21.15.28:0/3240761874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 0x7fc178102ec0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.625+0000 7fc18029e640 1 -- 172.21.15.28:0/3240761874 >> 172.21.15.28:0/3240761874 conn(0x7fc178076ab0 msgr2=0x7fc178076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 -- 172.21.15.28:0/3240761874 shutdown_connections 2024-05-09T08:26:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 -- 172.21.15.28:0/3240761874 wait complete. 2024-05-09T08:26:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 Processor -- start 2024-05-09T08:26:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 -- start start 2024-05-09T08:26:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 0x7fc1781980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc178103400 0x7fc178198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc17810beb0 0x7fc17819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc178111c40 con 0x7fc178102ae0 2024-05-09T08:26:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc178111ac0 con 0x7fc17810beb0 2024-05-09T08:26:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc18029e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc178111dc0 con 0x7fc178103400 2024-05-09T08:26:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.626+0000 7fc17e03a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 0x7fc1781980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc17e03a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 0x7fc1781980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48456/0 (socket says 172.21.15.28:48456) 2024-05-09T08:26:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc17e03a640 1 -- 172.21.15.28:0/1931016443 learned_addr learned my addr 172.21.15.28:0/1931016443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc17e03a640 1 -- 172.21.15.28:0/1931016443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc178103400 msgr2=0x7fc178198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc17e03a640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc178103400 0x7fc178198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc17e03a640 1 -- 172.21.15.28:0/1931016443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc17810beb0 msgr2=0x7fc17819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc17e03a640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc17810beb0 0x7fc17819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc17e03a640 1 -- 172.21.15.28:0/1931016443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17819d060 con 0x7fc178102ae0 2024-05-09T08:26:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc17e03a640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 0x7fc1781980c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc174006010 tx=0x7fc1740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc16f7fe640 1 -- 172.21.15.28:0/1931016443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17403f030 con 0x7fc178102ae0 2024-05-09T08:26:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17819d2f0 con 0x7fc178102ae0 2024-05-09T08:26:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc16f7fe640 1 -- 172.21.15.28:0/1931016443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc174003ce0 con 0x7fc178102ae0 2024-05-09T08:26:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc16f7fe640 1 -- 172.21.15.28:0/1931016443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174035070 con 0x7fc178102ae0 2024-05-09T08:26:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.627+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1781a4be0 con 0x7fc178102ae0 2024-05-09T08:26:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.628+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc17804f5f0 con 0x7fc178102ae0 2024-05-09T08:26:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.629+0000 7fc16f7fe640 1 -- 172.21.15.28:0/1931016443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7fc174042030 con 0x7fc178102ae0 2024-05-09T08:26:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.630+0000 7fc16f7fe640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1640a2e40 0x7fc1640a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.630+0000 7fc16f7fe640 1 -- 172.21.15.28:0/1931016443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fc1740f00b0 con 0x7fc178102ae0 2024-05-09T08:26:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.630+0000 7fc17d839640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1640a2e40 0x7fc1640a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.630+0000 7fc17d839640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1640a2e40 0x7fc1640a5300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc1680058c0 tx=0x7fc168005850 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.635+0000 7fc16f7fe640 1 -- 172.21.15.28:0/1931016443 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc1740bc7f0 con 0x7fc178102ae0 2024-05-09T08:26:24.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:24 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1986092944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:26:24.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.842+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 --> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fc1780fee80 con 0x7fc1640a2e40 2024-05-09T08:26:24.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.849+0000 7fc16f7fe640 1 -- 172.21.15.28:0/1931016443 <== mgr.15528 v2:172.21.15.28:6800/160415970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fc1780fee80 con 0x7fc1640a2e40 2024-05-09T08:26:24.850 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-09T00:00:00", "created": "2024-05-09T08:26:24", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-09T08:26:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1640a2e40 msgr2=0x7fc1640a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1640a2e40 0x7fc1640a5300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc1680058c0 tx=0x7fc168005850 comp rx=0 tx=0).stop 2024-05-09T08:26:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 msgr2=0x7fc1781980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 0x7fc1781980c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc174006010 tx=0x7fc1740042f0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 shutdown_connections 2024-05-09T08:26:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc17810beb0 0x7fc17819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1640a2e40 0x7fc1640a5300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc178103400 0x7fc178198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 --2- 172.21.15.28:0/1931016443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178102ae0 0x7fc1781980c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 >> 172.21.15.28:0/1931016443 conn(0x7fc178076ab0 msgr2=0x7fc1780fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 shutdown_connections 2024-05-09T08:26:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.851+0000 7fc18029e640 1 -- 172.21.15.28:0/1931016443 wait complete. 2024-05-09T08:26:24.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-09T08:26:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.970+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3246042009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc4102ac0 msgr2=0x7f8cc4102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.970+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3246042009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc4102ac0 0x7f8cc4102ea0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009900 tx=0x7f8cb402a510 comp rx=0 tx=0).stop 2024-05-09T08:26:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.970+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3246042009 shutdown_connections 2024-05-09T08:26:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.970+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3246042009 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc410be00 0x7f8cc410e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.970+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3246042009 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc41033e0 0x7f8cc410b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.970+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3246042009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc4102ac0 0x7f8cc4102ea0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.970+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3246042009 >> 172.21.15.28:0/3246042009 conn(0x7f8cc40778b0 msgr2=0x7f8cc4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.970+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3246042009 shutdown_connections 2024-05-09T08:26:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.970+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3246042009 wait complete. 2024-05-09T08:26:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc88a5640 1 Processor -- start 2024-05-09T08:26:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc88a5640 1 -- start start 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc88a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc4102ac0 0x7f8cc41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc88a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc41033e0 0x7f8cc4198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc88a5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc410be00 0x7f8cc419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc88a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc4111c00 con 0x7f8cc41033e0 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc88a5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc4111a80 con 0x7f8cc410be00 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc88a5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc4111d80 con 0x7f8cc4102ac0 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc41033e0 0x7f8cc4198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc41033e0 0x7f8cc4198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48478/0 (socket says 172.21.15.28:48478) 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc1d9b640 1 -- 172.21.15.28:0/3852768 learned_addr learned my addr 172.21.15.28:0/3852768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc259c640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc4102ac0 0x7f8cc41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc2d9d640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc410be00 0x7f8cc419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc1d9b640 1 -- 172.21.15.28:0/3852768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc4102ac0 msgr2=0x7f8cc41980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc1d9b640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc4102ac0 0x7f8cc41980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc1d9b640 1 -- 172.21.15.28:0/3852768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc410be00 msgr2=0x7f8cc419c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc1d9b640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc410be00 0x7f8cc419c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc1d9b640 1 -- 172.21.15.28:0/3852768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc419d0d0 con 0x7f8cc41033e0 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.971+0000 7f8cc1d9b640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc41033e0 0x7f8cc4198620 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800db80 tx=0x7f8cb800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.972+0000 7f8cb37fe640 1 -- 172.21.15.28:0/3852768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8017070 con 0x7f8cc41033e0 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.972+0000 7f8cb37fe640 1 -- 172.21.15.28:0/3852768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cb8003b70 con 0x7f8cc41033e0 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.972+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc419d3c0 con 0x7f8cc41033e0 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.972+0000 7f8cb37fe640 1 -- 172.21.15.28:0/3852768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb80106c0 con 0x7f8cc41033e0 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.972+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc41a4c50 con 0x7f8cc41033e0 2024-05-09T08:26:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.973+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc404f5f0 con 0x7f8cc41033e0 2024-05-09T08:26:24.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.974+0000 7f8cb37fe640 1 -- 172.21.15.28:0/3852768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f8cb8004020 con 0x7f8cc41033e0 2024-05-09T08:26:24.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.974+0000 7f8cb37fe640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8cac0a2ec0 0x7f8cac0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:24.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.975+0000 7f8cc259c640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8cac0a2ec0 0x7f8cac0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:24.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.975+0000 7f8cb37fe640 1 -- 172.21.15.28:0/3852768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f8cb80cf7b0 con 0x7f8cc41033e0 2024-05-09T08:26:24.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.975+0000 7f8cc259c640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8cac0a2ec0 0x7f8cac0a5380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009230 tx=0x7f8cb40058d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:24.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:24.979+0000 7f8cb37fe640 1 -- 172.21.15.28:0/3852768 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8cb8020070 con 0x7f8cc41033e0 2024-05-09T08:26:25.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.186+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 --> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8cc4061c70 con 0x7f8cac0a2ec0 2024-05-09T08:26:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.193+0000 7f8cb37fe640 1 -- 172.21.15.28:0/3852768 <== mgr.15528 v2:172.21.15.28:6800/160415970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8cc4061c70 con 0x7f8cac0a2ec0 2024-05-09T08:26:25.193 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-05-09T08:26:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8cac0a2ec0 msgr2=0x7f8cac0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8cac0a2ec0 0x7f8cac0a5380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009230 tx=0x7f8cb40058d0 comp rx=0 tx=0).stop 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc41033e0 msgr2=0x7f8cc4198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc41033e0 0x7f8cc4198620 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800db80 tx=0x7f8cb800b630 comp rx=0 tx=0).stop 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 shutdown_connections 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc410be00 0x7f8cc419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cc41033e0 0x7f8cc4198620 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f8cac0a2ec0 0x7f8cac0a5380 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 --2- 172.21.15.28:0/3852768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc4102ac0 0x7f8cc41980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 >> 172.21.15.28:0/3852768 conn(0x7f8cc40778b0 msgr2=0x7f8cc40fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 shutdown_connections 2024-05-09T08:26:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:25.195+0000 7f8cc88a5640 1 -- 172.21.15.28:0/3852768 wait complete. 2024-05-09T08:26:25.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-09T08:26:25.234 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-09T08:26:25.245 INFO:teuthology.run_tasks:Running task background_exec... 2024-05-09T08:26:25.273 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:26:25.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-05-09T08:26:25.276 INFO:teuthology.run_tasks:Running task workunit... 2024-05-09T08:26:25.317 INFO:tasks.workunit:Pulling workunits from ref be85c202969d25af6e130678e7d73a20e9c4ca11 2024-05-09T08:26:25.317 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-09T08:26:25.317 INFO:tasks.workunit:timeout=3h 2024-05-09T08:26:25.317 INFO:tasks.workunit:cleanup=True 2024-05-09T08:26:25.317 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-09T08:26:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-09T08:26:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-09T08:26:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-05-09T08:26:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:26:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-09T08:26:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-09 08:24:18.845672808 +0000 2024-05-09T08:26:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-09 08:24:18.845672808 +0000 2024-05-09T08:26:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-09 08:26:15.420582980 +0000 2024-05-09T08:26:25.335 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-09 08:24:18.845672808 +0000 2024-05-09T08:26:25.335 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-09T08:26:25.335 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-09T08:26:25.406 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout be85c202969d25af6e130678e7d73a20e9c4ca11 2024-05-09T08:26:25.463 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-09T08:26:25.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:25 smithi165 ceph-mon[34540]: scrub queued for path: / 2024-05-09T08:26:25.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:25 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:25.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:25 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:26:25.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:25 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:25.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:25 smithi165 ceph-mon[34540]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:26:25.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:25 smithi165 ceph-mon[34540]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:25.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:25 smithi165 ceph-mon[34540]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:25.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:25 smithi184 ceph-mon[34509]: scrub queued for path: / 2024-05-09T08:26:25.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:25 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:25.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:25 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:26:25.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:25 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:25.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:25 smithi184 ceph-mon[34509]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:26:25.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:25 smithi184 ceph-mon[34509]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:25.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:25 smithi184 ceph-mon[34509]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:25.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:25 smithi028 ceph-mon[30119]: scrub queued for path: / 2024-05-09T08:26:25.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:25 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:25.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:25 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:26:25.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:25 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:25.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:25 smithi028 ceph-mon[30119]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:26:25.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:25 smithi028 ceph-mon[30119]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:25.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:25 smithi028 ceph-mon[30119]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:26 smithi165 ceph-mon[34540]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:26 smithi165 ceph-mon[34540]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:26.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:26 smithi184 ceph-mon[34509]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:26.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:26 smithi184 ceph-mon[34509]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:26 smithi028 ceph-mon[30119]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:26 smithi028 ceph-mon[30119]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:27.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:27 smithi165 ceph-mon[34540]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 0 B/s wr, 24 op/s 2024-05-09T08:26:27.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:27 smithi184 ceph-mon[34509]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 0 B/s wr, 24 op/s 2024-05-09T08:26:27.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:27 smithi028 ceph-mon[30119]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 0 B/s wr, 24 op/s 2024-05-09T08:26:29.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:29 smithi165 ceph-mon[34540]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 19 op/s 2024-05-09T08:26:29.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:29 smithi184 ceph-mon[34509]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 19 op/s 2024-05-09T08:26:29.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:29 smithi028 ceph-mon[30119]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 19 op/s 2024-05-09T08:26:31.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:31 smithi165 ceph-mon[34540]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-05-09T08:26:31.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:31 smithi184 ceph-mon[34509]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-05-09T08:26:31.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:31 smithi028 ceph-mon[30119]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-05-09T08:26:33.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:33 smithi165 ceph-mon[34540]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-05-09T08:26:33.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:33 smithi184 ceph-mon[34509]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-05-09T08:26:33.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:33 smithi028 ceph-mon[30119]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-05-09T08:26:35.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:35 smithi165 ceph-mon[34540]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-05-09T08:26:35.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:35 smithi184 ceph-mon[34509]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-05-09T08:26:35.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:35 smithi028 ceph-mon[30119]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-05-09T08:26:36.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:36 smithi165 ceph-mon[34540]: scrub summary: idle 2024-05-09T08:26:36.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:36 smithi184 ceph-mon[34509]: scrub summary: idle 2024-05-09T08:26:36.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:36 smithi028 ceph-mon[30119]: scrub summary: idle 2024-05-09T08:26:37.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:37 smithi165 ceph-mon[34540]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-05-09T08:26:37.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:37 smithi184 ceph-mon[34509]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-05-09T08:26:37.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:37 smithi028 ceph-mon[30119]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-05-09T08:26:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.421+0000 7f9c12c64640 1 -- 172.21.15.28:0/1081372415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 msgr2=0x7f9c0c0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.421+0000 7f9c12c64640 1 --2- 172.21.15.28:0/1081372415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 0x7f9c0c0ffed0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9c000093a0 tx=0x7f9c0002a420 comp rx=0 tx=0).stop 2024-05-09T08:26:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.421+0000 7f9c12c64640 1 -- 172.21.15.28:0/1081372415 shutdown_connections 2024-05-09T08:26:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.421+0000 7f9c12c64640 1 --2- 172.21.15.28:0/1081372415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c0c100410 0x7f9c0c10c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.421+0000 7f9c12c64640 1 --2- 172.21.15.28:0/1081372415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 0x7f9c0c0ffed0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.421+0000 7f9c12c64640 1 --2- 172.21.15.28:0/1081372415 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c0c105a60 0x7f9c0c105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.421+0000 7f9c12c64640 1 -- 172.21.15.28:0/1081372415 >> 172.21.15.28:0/1081372415 conn(0x7f9c0c076830 msgr2=0x7f9c0c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.421+0000 7f9c12c64640 1 -- 172.21.15.28:0/1081372415 shutdown_connections 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.421+0000 7f9c12c64640 1 -- 172.21.15.28:0/1081372415 wait complete. 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c12c64640 1 Processor -- start 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c12c64640 1 -- start start 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c12c64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 0x7f9c0c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c12c64640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c0c100410 0x7f9c0c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c12c64640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c0c105a60 0x7f9c0c19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c12c64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c10fa70 con 0x7f9c0c0ffa50 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c12c64640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c10f8f0 con 0x7f9c0c100410 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c12c64640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c10fbf0 con 0x7f9c0c105a60 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c10a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 0x7f9c0c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c10a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 0x7f9c0c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34898/0 (socket says 172.21.15.28:34898) 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c10a00640 1 -- 172.21.15.28:0/2681022281 learned_addr learned my addr 172.21.15.28:0/2681022281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:38.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c11201640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c0c105a60 0x7f9c0c19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c0c100410 0x7f9c0c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c10a00640 1 -- 172.21.15.28:0/2681022281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c0c105a60 msgr2=0x7f9c0c19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c10a00640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c0c105a60 0x7f9c0c19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c10a00640 1 -- 172.21.15.28:0/2681022281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c0c100410 msgr2=0x7f9c0c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c10a00640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c0c100410 0x7f9c0c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c10a00640 1 -- 172.21.15.28:0/2681022281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0c19d050 con 0x7f9c0c0ffa50 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c11201640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c0c105a60 0x7f9c0c19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.422+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c0c100410 0x7f9c0c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.423+0000 7f9c10a00640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 0x7f9c0c198060 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400de60 tx=0x7f9c0400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.423+0000 7f9c09ffb640 1 -- 172.21.15.28:0/2681022281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0400e860 con 0x7f9c0c0ffa50 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.423+0000 7f9c09ffb640 1 -- 172.21.15.28:0/2681022281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c04016430 con 0x7f9c0c0ffa50 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.423+0000 7f9c12c64640 1 -- 172.21.15.28:0/2681022281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c0c19d340 con 0x7f9c0c0ffa50 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.423+0000 7f9c09ffb640 1 -- 172.21.15.28:0/2681022281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04015640 con 0x7f9c0c0ffa50 2024-05-09T08:26:38.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.423+0000 7f9c12c64640 1 -- 172.21.15.28:0/2681022281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c0c1a4bd0 con 0x7f9c0c0ffa50 2024-05-09T08:26:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.425+0000 7f9c09ffb640 1 -- 172.21.15.28:0/2681022281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f9c040165e0 con 0x7f9c0c0ffa50 2024-05-09T08:26:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.426+0000 7f9c15973740 1 Processor -- start 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.426+0000 7f9c15973740 1 -- start start 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.426+0000 7f9c15973740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae39830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.426+0000 7f9c15973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae440f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.426+0000 7f9c15973740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae44630 0x556c7ae46a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.426+0000 7f9c15973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c7ad9e340 con 0x556c7ae39d70 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.426+0000 7f9c15973740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x556c7ad46290 con 0x556c7ae44630 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c15973740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556c7ad9a300 con 0x556c7ae39450 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c09ffb640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9bf40a2e40 0x7f9bf40a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c10a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c10a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae440f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34900/0 (socket says 172.21.15.28:34900) 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c10a00640 1 -- 172.21.15.28:0/3745137926 learned_addr learned my addr 172.21.15.28:0/3745137926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c11201640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9bf40a2e40 0x7f9bf40a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c11201640 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae39830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae44630 0x556c7ae46a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c09ffb640 1 -- 172.21.15.28:0/2681022281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f9c040cff20 con 0x7f9c0c0ffa50 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c10a00640 1 -- 172.21.15.28:0/3745137926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 msgr2=0x556c7ae39830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c10a00640 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae39830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c10a00640 1 -- 172.21.15.28:0/3745137926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae44630 msgr2=0x556c7ae46a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c10a00640 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae44630 0x556c7ae46a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c10a00640 1 -- 172.21.15.28:0/3745137926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c7ad6a8e0 con 0x556c7ae39d70 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae44630 0x556c7ae46a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c10a00640 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae440f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9c040cd200 tx=0x7f9c04000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d677ccbd4c8d38f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.427+0000 7f9c11201640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9bf40a2e40 0x7f9bf40a5300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc013960 tx=0x7f9bfc005f70 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9bf2ffd640 1 -- 172.21.15.28:0/3745137926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0409b910 con 0x556c7ae39d70 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9bf2ffd640 1 -- 172.21.15.28:0/3745137926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c0409c6b0 con 0x556c7ae39d70 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9bf2ffd640 1 -- 172.21.15.28:0/3745137926 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04098620 con 0x556c7ae39d70 2024-05-09T08:26:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9c15973740 1 -- 172.21.15.28:0/3745137926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 msgr2=0x556c7ae440f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9c15973740 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae440f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9c040cd200 tx=0x7f9c04000f40 comp rx=0 tx=0).stop 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9c15973740 1 -- 172.21.15.28:0/3745137926 shutdown_connections 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9c15973740 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae44630 0x556c7ae46a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9c15973740 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae440f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9c15973740 1 --2- 172.21.15.28:0/3745137926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae39830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9c15973740 1 -- 172.21.15.28:0/3745137926 >> 172.21.15.28:0/3745137926 conn(0x556c7ae35960 msgr2=0x556c7ae35d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9c15973740 1 -- 172.21.15.28:0/3745137926 shutdown_connections 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.428+0000 7f9c15973740 1 -- 172.21.15.28:0/3745137926 wait complete. 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c15973740 1 -- start start 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c15973740 1 Processor -- start 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c15973740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39450 0x556c7ae41590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c15973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae41ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c15973740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae44630 0x556c7ae42010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c15973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c7ae471a0 con 0x556c7ae39d70 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c15973740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x556c7ad59790 con 0x556c7ae39450 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c15973740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556c7ae47320 con 0x556c7ae44630 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c15973740 10 client.0 _ll_register_callbacks cb 0x556c7ad94a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.429+0000 7f9c0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae41ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae41ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34912/0 (socket says 172.21.15.28:34912) 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c0bfff640 1 -- 172.21.15.28:0/2070731157 learned_addr learned my addr 172.21.15.28:0/2070731157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c10a00640 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39450 0x556c7ae41590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c11201640 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae44630 0x556c7ae42010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c0bfff640 1 -- 172.21.15.28:0/2070731157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae44630 msgr2=0x556c7ae42010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae44630 0x556c7ae42010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c0bfff640 1 -- 172.21.15.28:0/2070731157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39450 msgr2=0x556c7ae41590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39450 0x556c7ae41590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c0bfff640 1 -- 172.21.15.28:0/2070731157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c7aab2210 con 0x556c7ae39d70 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c11201640 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae44630 0x556c7ae42010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae41ad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9c000093a0 tx=0x7f9c000052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec133b2e14db6996 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9bebfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9bebfff640 1 -- 172.21.15.28:0/2070731157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00040070 con 0x556c7ae39d70 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9c15973740 1 -- 172.21.15.28:0/2070731157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c0409c6b0 con 0x556c7ae39d70 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.430+0000 7f9bebfff640 1 -- 172.21.15.28:0/2070731157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f9c000048c0 con 0x556c7ae39d70 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9bebfff640 1 -- 172.21.15.28:0/2070731157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9c0004c050 con 0x556c7ae39d70 2024-05-09T08:26:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9c15973740 10 client.15795 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9c15973740 1 -- 172.21.15.28:0/2070731157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c7ab7e620 con 0x556c7ae39d70 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9bebfff640 1 -- 172.21.15.28:0/2070731157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c0409c6b0 con 0x556c7ae39d70 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9bebfff640 1 -- 172.21.15.28:0/2070731157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9c00031ce0 con 0x556c7ae39d70 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9c15973740 10 client.15795 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9c15973740 10 client.15795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9c15973740 20 client.15795 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9c15973740 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556c7ae54320 0x556c7ae56750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9c15973740 4 client.15795 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9c15973740 1 -- 172.21.15.28:0/2070731157 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c7adaba70 con 0x556c7ae54320 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.431+0000 7f9c10a00640 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556c7ae54320 0x556c7ae56750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.432+0000 7f9c10a00640 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556c7ae54320 0x556c7ae56750 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56f56af1b8ac1fc0 server_cookie=f88a587c23491a91 in_seq=0 out_seq=0 2024-05-09T08:26:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.432+0000 7f9bebfff640 10 client.15795 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9bebfff640 1 -- 172.21.15.28:0/2070731157 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c7adaba70 con 0x556c7ae54320 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9bebfff640 10 client.15795 handle_command_reply: tid=0 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 client.15795 shutdown 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 -- 172.21.15.28:0/2070731157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 msgr2=0x556c7ae41ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae41ad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9c000093a0 tx=0x7f9c000052e0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 -- 172.21.15.28:0/2070731157 shutdown_connections 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556c7ae54320 0x556c7ae56750 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae44630 0x556c7ae42010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9bebfff640 0 client.15795 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae39d70 0x556c7ae41ad0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 --2- 172.21.15.28:0/2070731157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39450 0x556c7ae41590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 -- 172.21.15.28:0/2070731157 >> 172.21.15.28:0/2070731157 conn(0x556c7ae35960 msgr2=0x556c7ae3b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 -- 172.21.15.28:0/2070731157 shutdown_connections 2024-05-09T08:26:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 1 -- 172.21.15.28:0/2070731157 wait complete. 2024-05-09T08:26:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.433+0000 7f9c15973740 20 client.15795 trim_cache size 0 max 16384 2024-05-09T08:26:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c15973740 1 Processor -- start 2024-05-09T08:26:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c15973740 1 -- start start 2024-05-09T08:26:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c15973740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae50060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c15973740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 0x556c7ae3be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c15973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae35d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c15973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c1aed90 con 0x556c7ae44630 2024-05-09T08:26:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c15973740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c1b08d0 con 0x556c7ae39d70 2024-05-09T08:26:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c15973740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c1af460 con 0x556c7ae39450 2024-05-09T08:26:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c11201640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae35d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c11201640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae35d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34920/0 (socket says 172.21.15.28:34920) 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.448+0000 7f9c11201640 1 -- 172.21.15.28:0/1568090644 learned_addr learned my addr 172.21.15.28:0/1568090644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/1568090644 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 0x556c7ae3be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c11201640 1 -- 172.21.15.28:0/1568090644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 msgr2=0x556c7ae50060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c11201640 1 --2- 172.21.15.28:0/1568090644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae50060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c11201640 1 -- 172.21.15.28:0/1568090644 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 msgr2=0x556c7ae3be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c11201640 1 --2- 172.21.15.28:0/1568090644 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 0x556c7ae3be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c11201640 1 -- 172.21.15.28:0/1568090644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c7ae40540 con 0x556c7ae44630 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/1568090644 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 0x556c7ae3be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c11201640 1 --2- 172.21.15.28:0/1568090644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae35d40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc013150 tx=0x7f9bfc005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b66fd1ddf4d4b7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9bf27fc640 1 -- 172.21.15.28:0/1568090644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc00ecf0 con 0x556c7ae44630 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9bf27fc640 1 -- 172.21.15.28:0/1568090644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bfc00ee90 con 0x556c7ae44630 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9bf27fc640 1 -- 172.21.15.28:0/1568090644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc022900 con 0x556c7ae44630 2024-05-09T08:26:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c15973740 1 -- 172.21.15.28:0/1568090644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 msgr2=0x556c7ae35d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.449+0000 7f9c15973740 1 --2- 172.21.15.28:0/1568090644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae35d40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc013150 tx=0x7f9bfc005d60 comp rx=0 tx=0).stop 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 -- 172.21.15.28:0/1568090644 shutdown_connections 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 --2- 172.21.15.28:0/1568090644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae35d40 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 --2- 172.21.15.28:0/1568090644 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 0x556c7ae3be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 --2- 172.21.15.28:0/1568090644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae50060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 -- 172.21.15.28:0/1568090644 >> 172.21.15.28:0/1568090644 conn(0x556c7ae35960 msgr2=0x556c7ae34a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 -- 172.21.15.28:0/1568090644 shutdown_connections 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 -- 172.21.15.28:0/1568090644 wait complete. 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 -- start start 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 Processor -- start 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae4c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 0x556c7ae4cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae49540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c1b0200 con 0x556c7ae44630 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c1afb30 con 0x556c7ae39d70 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.450+0000 7f9c15973740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c1acd70 con 0x556c7ae39450 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c15973740 10 client.0 _ll_register_callbacks cb 0x7f9c00009270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c11201640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae49540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c11201640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae49540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34924/0 (socket says 172.21.15.28:34924) 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c11201640 1 -- 172.21.15.28:0/911050112 learned_addr learned my addr 172.21.15.28:0/911050112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c0bfff640 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 0x556c7ae4cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c10a00640 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae4c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c11201640 1 -- 172.21.15.28:0/911050112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 msgr2=0x556c7ae4c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c11201640 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae4c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c11201640 1 -- 172.21.15.28:0/911050112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 msgr2=0x556c7ae4cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c11201640 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 0x556c7ae4cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c11201640 1 -- 172.21.15.28:0/911050112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c7ae49ad0 con 0x556c7ae44630 2024-05-09T08:26:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c11201640 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae49540 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc013040 tx=0x7f9bfc00be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3cb917e244e4e09 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9bebfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9bebfff640 1 -- 172.21.15.28:0/911050112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc039060 con 0x556c7ae44630 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9bebfff640 1 -- 172.21.15.28:0/911050112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f9bfc005040 con 0x556c7ae44630 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.451+0000 7f9c15973740 1 -- 172.21.15.28:0/911050112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bfc00ee90 con 0x556c7ae44630 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9bebfff640 1 -- 172.21.15.28:0/911050112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9bfc040050 con 0x556c7ae44630 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9c15973740 10 client.15801 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9c15973740 1 -- 172.21.15.28:0/911050112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c7aebb0e0 con 0x556c7ae44630 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9bebfff640 1 -- 172.21.15.28:0/911050112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bfc00ee90 con 0x556c7ae44630 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9bebfff640 1 -- 172.21.15.28:0/911050112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9bfc01f050 con 0x556c7ae44630 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9c15973740 10 client.15801 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9c15973740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9c15973740 20 client.15801 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9c15973740 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556c7aebb6f0 0x556c7aebdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9c15973740 4 client.15801 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:26:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9c15973740 1 -- 172.21.15.28:0/911050112 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556c7ada5dc0 con 0x556c7aebb6f0 2024-05-09T08:26:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.452+0000 7f9c10a00640 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556c7aebb6f0 0x556c7aebdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.453+0000 7f9bebfff640 10 client.15801 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.453+0000 7f9c10a00640 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556c7aebb6f0 0x556c7aebdae0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0047150e7489a47 server_cookie=6176c5fbf3a6996a in_seq=0 out_seq=0 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.466+0000 7f9bebfff640 1 -- 172.21.15.28:0/911050112 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556c7ada5dc0 con 0x556c7aebb6f0 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.466+0000 7f9bebfff640 10 client.15801 handle_command_reply: tid=0 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.466+0000 7f9c15973740 1 client.15801 shutdown 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 -- 172.21.15.28:0/911050112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 msgr2=0x556c7ae49540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae49540 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc013040 tx=0x7f9bfc00be90 comp rx=0 tx=0).stop 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 -- 172.21.15.28:0/911050112 shutdown_connections 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556c7aebb6f0 0x556c7aebdae0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c7ae44630 0x556c7ae49540 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9bebfff640 0 client.15801 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556c7ae39d70 0x556c7ae4cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 --2- 172.21.15.28:0/911050112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556c7ae39450 0x556c7ae4c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 -- 172.21.15.28:0/911050112 >> 172.21.15.28:0/911050112 conn(0x556c7ae35960 msgr2=0x556c7ae34a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:38.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 -- 172.21.15.28:0/911050112 shutdown_connections 2024-05-09T08:26:38.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 1 -- 172.21.15.28:0/911050112 wait complete. 2024-05-09T08:26:38.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.467+0000 7f9c15973740 20 client.15801 trim_cache size 0 max 16384 2024-05-09T08:26:38.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:26:38.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:26:38.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:26:38.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:26:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 -- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9bf40a2e40 msgr2=0x7f9bf40a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9bf40a2e40 0x7f9bf40a5300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc013960 tx=0x7f9bfc005f70 comp rx=0 tx=0).stop 2024-05-09T08:26:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 -- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 msgr2=0x7f9c0c198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 0x7f9c0c198060 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400de60 tx=0x7f9c0400b730 comp rx=0 tx=0).stop 2024-05-09T08:26:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 -- 172.21.15.28:0/2681022281 shutdown_connections 2024-05-09T08:26:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9bf40a2e40 0x7f9bf40a5300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c0c105a60 0x7f9c0c19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c0c100410 0x7f9c0c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 --2- 172.21.15.28:0/2681022281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0c0ffa50 0x7f9c0c198060 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 -- 172.21.15.28:0/2681022281 >> 172.21.15.28:0/2681022281 conn(0x7f9c0c076830 msgr2=0x7f9c0c102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 -- 172.21.15.28:0/2681022281 shutdown_connections 2024-05-09T08:26:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:38.468+0000 7f9c12c64640 1 -- 172.21.15.28:0/2681022281 wait complete. 2024-05-09T08:26:39.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:39 smithi184 ceph-mon[34509]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-09T08:26:39.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:39 smithi028 ceph-mon[30119]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-09T08:26:39.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:39 smithi165 ceph-mon[34540]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-09T08:26:41.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:41 smithi184 ceph-mon[34509]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 78 KiB/s wr, 49 op/s 2024-05-09T08:26:41.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:41 smithi028 ceph-mon[30119]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 78 KiB/s wr, 49 op/s 2024-05-09T08:26:41.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:41 smithi165 ceph-mon[34540]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 78 KiB/s wr, 49 op/s 2024-05-09T08:26:43.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:43 smithi028 ceph-mon[30119]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-05-09T08:26:43.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:43 smithi165 ceph-mon[34540]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-05-09T08:26:43.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:43 smithi184 ceph-mon[34509]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-05-09T08:26:45.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:45 smithi028 ceph-mon[30119]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-05-09T08:26:45.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:45 smithi165 ceph-mon[34540]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-05-09T08:26:45.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:45 smithi184 ceph-mon[34509]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-05-09T08:26:47.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:47 smithi028 ceph-mon[30119]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-05-09T08:26:47.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:47 smithi165 ceph-mon[34540]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-05-09T08:26:47.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:47 smithi184 ceph-mon[34509]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-05-09T08:26:49.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:49 smithi028 ceph-mon[30119]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-05-09T08:26:49.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:49 smithi165 ceph-mon[34540]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-05-09T08:26:49.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:49 smithi184 ceph-mon[34509]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.569+0000 7f8131a6b640 1 -- 172.21.15.28:0/855102076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c10be30 msgr2=0x7f812c10e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.569+0000 7f8131a6b640 1 --2- 172.21.15.28:0/855102076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c10be30 0x7f812c10e2b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8124009ca0 tx=0x7f812402a950 comp rx=0 tx=0).stop 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.570+0000 7f8131a6b640 1 -- 172.21.15.28:0/855102076 shutdown_connections 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.570+0000 7f8131a6b640 1 --2- 172.21.15.28:0/855102076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c10be30 0x7f812c10e2b0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.570+0000 7f8131a6b640 1 --2- 172.21.15.28:0/855102076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f812c103380 0x7f812c10b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.570+0000 7f8131a6b640 1 --2- 172.21.15.28:0/855102076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812c102a60 0x7f812c102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.570+0000 7f8131a6b640 1 -- 172.21.15.28:0/855102076 >> 172.21.15.28:0/855102076 conn(0x7f812c0778b0 msgr2=0x7f812c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.570+0000 7f8131a6b640 1 -- 172.21.15.28:0/855102076 shutdown_connections 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.570+0000 7f8131a6b640 1 -- 172.21.15.28:0/855102076 wait complete. 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.570+0000 7f8131a6b640 1 Processor -- start 2024-05-09T08:26:51.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.570+0000 7f8131a6b640 1 -- start start 2024-05-09T08:26:51.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f8131a6b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f812c102a60 0x7f812c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f8131a6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c103380 0x7f812c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f8131a6b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812c10be30 0x7f812c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f8131a6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f812c111c10 con 0x7f812c103380 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f8131a6b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f812c111a90 con 0x7f812c102a60 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f8131a6b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f812c111d90 con 0x7f812c10be30 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c103380 0x7f812c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c103380 0x7f812c1984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54352/0 (socket says 172.21.15.28:54352) 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812affd640 1 -- 172.21.15.28:0/2387980043 learned_addr learned my addr 172.21.15.28:0/2387980043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812b7fe640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f812c102a60 0x7f812c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812bfff640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812c10be30 0x7f812c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812affd640 1 -- 172.21.15.28:0/2387980043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812c10be30 msgr2=0x7f812c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812affd640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812c10be30 0x7f812c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812affd640 1 -- 172.21.15.28:0/2387980043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f812c102a60 msgr2=0x7f812c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812affd640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f812c102a60 0x7f812c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812affd640 1 -- 172.21.15.28:0/2387980043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f812c19cf60 con 0x7f812c103380 2024-05-09T08:26:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f812affd640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c103380 0x7f812c1984b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f812000c920 tx=0x7f812000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:51.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.571+0000 7f8128ff9640 1 -- 172.21.15.28:0/2387980043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120007c70 con 0x7f812c103380 2024-05-09T08:26:51.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.572+0000 7f8128ff9640 1 -- 172.21.15.28:0/2387980043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8120007e10 con 0x7f812c103380 2024-05-09T08:26:51.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.572+0000 7f8131a6b640 1 -- 172.21.15.28:0/2387980043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f812c19d250 con 0x7f812c103380 2024-05-09T08:26:51.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.572+0000 7f8128ff9640 1 -- 172.21.15.28:0/2387980043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120018890 con 0x7f812c103380 2024-05-09T08:26:51.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.572+0000 7f8131a6b640 1 -- 172.21.15.28:0/2387980043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f812c1a4ae0 con 0x7f812c103380 2024-05-09T08:26:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.574+0000 7f8128ff9640 1 -- 172.21.15.28:0/2387980043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f8120018a70 con 0x7f812c103380 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.574+0000 7f813477a740 1 Processor -- start 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f813477a740 1 -- start start 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f813477a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f813477a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcbc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f813477a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 0x55a9bbcbea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f813477a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a9bbc16340 con 0x55a9bbcb1450 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f813477a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a9bbbbe290 con 0x55a9bbcb1d70 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f813477a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a9bbc12300 con 0x55a9bbcbc630 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f812bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f812bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54360/0 (socket says 172.21.15.28:54360) 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f812bfff640 1 -- 172.21.15.28:0/3813362975 learned_addr learned my addr 172.21.15.28:0/3813362975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f812b7fe640 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcbc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f812affd640 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 0x55a9bbcbea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f8128ff9640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f81140a2ec0 0x7f81140a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f812bfff640 1 -- 172.21.15.28:0/3813362975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 msgr2=0x55a9bbcbea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f812bfff640 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 0x55a9bbcbea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f812bfff640 1 -- 172.21.15.28:0/3813362975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1d70 msgr2=0x55a9bbcbc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.575+0000 7f812bfff640 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcbc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.576+0000 7f812bfff640 1 -- 172.21.15.28:0/3813362975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9bbbe28e0 con 0x55a9bbcb1450 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.576+0000 7f812b7fe640 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcbc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.576+0000 7f8128ff9640 1 -- 172.21.15.28:0/2387980043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f81200cfac0 con 0x7f812c103380 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.576+0000 7f812bfff640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f81140a2ec0 0x7f81140a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.576+0000 7f812bfff640 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb1830 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f812402aef0 tx=0x7f812402f940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc02f3065b3c754 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.576+0000 7f8111ffb640 1 -- 172.21.15.28:0/3813362975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8124041040 con 0x55a9bbcb1450 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.576+0000 7f8111ffb640 1 -- 172.21.15.28:0/3813362975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8124004520 con 0x55a9bbcb1450 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.576+0000 7f8111ffb640 1 -- 172.21.15.28:0/3813362975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8124034040 con 0x55a9bbcb1450 2024-05-09T08:26:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 -- 172.21.15.28:0/3813362975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 msgr2=0x55a9bbcb1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f812bfff640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f81140a2ec0 0x7f81140a5380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f812404c000 tx=0x7f812404d000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb1830 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f812402aef0 tx=0x7f812402f940 comp rx=0 tx=0).stop 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 -- 172.21.15.28:0/3813362975 shutdown_connections 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 0x55a9bbcbea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcbc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 --2- 172.21.15.28:0/3813362975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb1830 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 -- 172.21.15.28:0/3813362975 >> 172.21.15.28:0/3813362975 conn(0x55a9bbcad960 msgr2=0x55a9bbcadd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 -- 172.21.15.28:0/3813362975 shutdown_connections 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 -- 172.21.15.28:0/3813362975 wait complete. 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.577+0000 7f813477a740 1 -- start start 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f813477a740 1 Processor -- start 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f813477a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f813477a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcb9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f813477a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcbc630 0x55a9bbcb9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f813477a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a9bbcbf260 con 0x55a9bbcb1450 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f813477a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a9bbbd1790 con 0x55a9bbcbc630 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f813477a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a9bbcbf3e0 con 0x55a9bbcb1d70 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f813477a740 10 client.0 _ll_register_callbacks cb 0x55a9bbc0ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54372/0 (socket says 172.21.15.28:54372) 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812b7fe640 1 -- 172.21.15.28:0/3946760352 learned_addr learned my addr 172.21.15.28:0/3946760352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812affd640 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcb9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812bfff640 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcbc630 0x55a9bbcb9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812b7fe640 1 -- 172.21.15.28:0/3946760352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 msgr2=0x55a9bbcb9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812b7fe640 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcb9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812b7fe640 1 -- 172.21.15.28:0/3946760352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcbc630 msgr2=0x55a9bbcb9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812b7fe640 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcbc630 0x55a9bbcb9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812b7fe640 1 -- 172.21.15.28:0/3946760352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9bb92a210 con 0x55a9bbcb1450 2024-05-09T08:26:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812bfff640 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcbc630 0x55a9bbcb9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f812b7fe640 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb94f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f811c0126b0 tx=0x7f811c012ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fffb15d57ac468ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.578+0000 7f810affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f810affd640 1 -- 172.21.15.28:0/3946760352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c0118d0 con 0x55a9bbcb1450 2024-05-09T08:26:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f810affd640 1 -- 172.21.15.28:0/3946760352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f811c011a70 con 0x55a9bbcb1450 2024-05-09T08:26:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f813477a740 1 -- 172.21.15.28:0/3946760352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8124004520 con 0x55a9bbcb1450 2024-05-09T08:26:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f810affd640 1 -- 172.21.15.28:0/3946760352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f811c011c70 con 0x55a9bbcb1450 2024-05-09T08:26:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f813477a740 10 client.15813 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:51.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f813477a740 1 -- 172.21.15.28:0/3946760352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a9bb9f6620 con 0x55a9bbcb1450 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f810affd640 1 -- 172.21.15.28:0/3946760352 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8124004520 con 0x55a9bbcb1450 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f810affd640 1 -- 172.21.15.28:0/3946760352 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f811c004cd0 con 0x55a9bbcb1450 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f813477a740 10 client.15813 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f813477a740 10 client.15813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f813477a740 20 client.15813 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f813477a740 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a9bbccc360 0x55a9bbcce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f813477a740 4 client.15813 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.579+0000 7f813477a740 1 -- 172.21.15.28:0/3946760352 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9bbc23a70 con 0x55a9bbccc360 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.580+0000 7f812affd640 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a9bbccc360 0x55a9bbcce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.580+0000 7f810affd640 10 client.15813 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.580+0000 7f812affd640 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a9bbccc360 0x55a9bbcce750 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d7e9cceb0d95bdf server_cookie=7ceb6c35146e7059 in_seq=0 out_seq=0 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f810affd640 1 -- 172.21.15.28:0/3946760352 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9bbc23a70 con 0x55a9bbccc360 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f810affd640 10 client.15813 handle_command_reply: tid=0 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 client.15813 shutdown 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 -- 172.21.15.28:0/3946760352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 msgr2=0x55a9bbcb94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb94f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f811c0126b0 tx=0x7f811c012ab0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 -- 172.21.15.28:0/3946760352 shutdown_connections 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a9bbccc360 0x55a9bbcce750 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcbc630 0x55a9bbcb9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcb9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f810affd640 0 client.15813 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 --2- 172.21.15.28:0/3946760352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1450 0x55a9bbcb94f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 -- 172.21.15.28:0/3946760352 >> 172.21.15.28:0/3946760352 conn(0x55a9bbcad960 msgr2=0x55a9bbcb36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 -- 172.21.15.28:0/3946760352 shutdown_connections 2024-05-09T08:26:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 1 -- 172.21.15.28:0/3946760352 wait complete. 2024-05-09T08:26:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.581+0000 7f813477a740 20 client.15813 trim_cache size 0 max 16384 2024-05-09T08:26:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f813477a740 1 Processor -- start 2024-05-09T08:26:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f813477a740 1 -- start start 2024-05-09T08:26:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f813477a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f813477a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcb3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f813477a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcbc630 0x55a9bbcd4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f813477a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f812c1aec70 con 0x55a9bbcbc630 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f813477a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f812c1b07b0 con 0x55a9bbcb1450 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f813477a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f812c1af340 con 0x55a9bbcb1d70 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcbc630 0x55a9bbcd4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcbc630 0x55a9bbcd4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54382/0 (socket says 172.21.15.28:54382) 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812bfff640 1 -- 172.21.15.28:0/4007012986 learned_addr learned my addr 172.21.15.28:0/4007012986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812b7fe640 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812affd640 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcb3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812bfff640 1 -- 172.21.15.28:0/4007012986 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 msgr2=0x55a9bbcb3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812bfff640 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcb3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812bfff640 1 -- 172.21.15.28:0/4007012986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 msgr2=0x55a9bbcc8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812bfff640 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812bfff640 1 -- 172.21.15.28:0/4007012986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9bbcb1ba0 con 0x55a9bbcbc630 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812b7fe640 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812affd640 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcb3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.597+0000 7f812bfff640 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcbc630 0x55a9bbcd4060 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8124009ca0 tx=0x7f8124009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc9892e097792d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f81117fa640 1 -- 172.21.15.28:0/4007012986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812405e070 con 0x55a9bbcbc630 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f81117fa640 1 -- 172.21.15.28:0/4007012986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8124009800 con 0x55a9bbcbc630 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f81117fa640 1 -- 172.21.15.28:0/4007012986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8124041040 con 0x55a9bbcbc630 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f813477a740 1 -- 172.21.15.28:0/4007012986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcbc630 msgr2=0x55a9bbcd4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f813477a740 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcbc630 0x55a9bbcd4060 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8124009ca0 tx=0x7f8124009560 comp rx=0 tx=0).stop 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f813477a740 1 -- 172.21.15.28:0/4007012986 shutdown_connections 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f813477a740 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcbc630 0x55a9bbcd4060 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f813477a740 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcb3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f813477a740 1 --2- 172.21.15.28:0/4007012986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f813477a740 1 -- 172.21.15.28:0/4007012986 >> 172.21.15.28:0/4007012986 conn(0x55a9bbcad960 msgr2=0x55a9bbcac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f813477a740 1 -- 172.21.15.28:0/4007012986 shutdown_connections 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.598+0000 7f813477a740 1 -- 172.21.15.28:0/4007012986 wait complete. 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f813477a740 1 -- start start 2024-05-09T08:26:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f813477a740 1 Processor -- start 2024-05-09T08:26:51.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f813477a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f813477a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcc0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f813477a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 0x55a9bbd35ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f813477a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f812c1b00e0 con 0x55a9bbcb1d70 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f813477a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f812c1afa10 con 0x55a9bbcb1450 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f813477a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f812c1acc50 con 0x55a9bbcbc630 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f813477a740 10 client.0 _ll_register_callbacks cb 0x7f811c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f812affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcc0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f812affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcc0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54396/0 (socket says 172.21.15.28:54396) 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.599+0000 7f812affd640 1 -- 172.21.15.28:0/2130587963 learned_addr learned my addr 172.21.15.28:0/2130587963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f812b7fe640 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f812bfff640 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 0x55a9bbd35ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f812affd640 1 -- 172.21.15.28:0/2130587963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 msgr2=0x55a9bbd35ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f812affd640 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 0x55a9bbd35ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f812affd640 1 -- 172.21.15.28:0/2130587963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 msgr2=0x55a9bbcc0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f812affd640 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f812affd640 1 -- 172.21.15.28:0/2130587963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9bbd362f0 con 0x55a9bbcb1d70 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f812b7fe640 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f812affd640 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcc0d80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f812009b7a0 tx=0x7f8120097e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0b3b4570960cc2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f810affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f810affd640 1 -- 172.21.15.28:0/2130587963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812009c1a0 con 0x55a9bbcb1d70 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f810affd640 1 -- 172.21.15.28:0/2130587963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f812000fb30 con 0x55a9bbcb1d70 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f813477a740 1 -- 172.21.15.28:0/2130587963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8124009800 con 0x55a9bbcb1d70 2024-05-09T08:26:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f810affd640 1 -- 172.21.15.28:0/2130587963 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f812000de20 con 0x55a9bbcb1d70 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f813477a740 10 client.15819 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f813477a740 1 -- 172.21.15.28:0/2130587963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a9bbd375d0 con 0x55a9bbcb1d70 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.600+0000 7f810affd640 1 -- 172.21.15.28:0/2130587963 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8124009800 con 0x55a9bbcb1d70 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f810affd640 1 -- 172.21.15.28:0/2130587963 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f81200cd030 con 0x55a9bbcb1d70 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f813477a740 10 client.15819 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f813477a740 10 client.15819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f813477a740 20 client.15819 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f813477a740 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a9bbd37cb0 0x55a9bbd3a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f813477a740 4 client.15819 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f813477a740 1 -- 172.21.15.28:0/2130587963 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a9bbc1ddc0 con 0x55a9bbd37cb0 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f812b7fe640 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a9bbd37cb0 0x55a9bbd3a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f810affd640 10 client.15819 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.601+0000 7f812b7fe640 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a9bbd37cb0 0x55a9bbd3a0d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d6c76ee02368aa1 server_cookie=6fa2d4e99806c2c0 in_seq=0 out_seq=0 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f810affd640 1 -- 172.21.15.28:0/2130587963 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a9bbc1ddc0 con 0x55a9bbd37cb0 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f810affd640 10 client.15819 handle_command_reply: tid=0 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 client.15819 shutdown 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 -- 172.21.15.28:0/2130587963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1d70 msgr2=0x55a9bbcc0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcc0d80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f812009b7a0 tx=0x7f8120097e40 comp rx=0 tx=0).stop 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 -- 172.21.15.28:0/2130587963 shutdown_connections 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a9bbd37cb0 0x55a9bbd3a0d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9bbcbc630 0x55a9bbd35ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9bbcb1d70 0x55a9bbcc0d80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 --2- 172.21.15.28:0/2130587963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a9bbcb1450 0x55a9bbcc0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f810affd640 0 client.15819 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 -- 172.21.15.28:0/2130587963 >> 172.21.15.28:0/2130587963 conn(0x55a9bbcad960 msgr2=0x55a9bbcb92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 -- 172.21.15.28:0/2130587963 shutdown_connections 2024-05-09T08:26:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 1 -- 172.21.15.28:0/2130587963 wait complete. 2024-05-09T08:26:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.605+0000 7f813477a740 20 client.15819 trim_cache size 0 max 16384 2024-05-09T08:26:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:26:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:26:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:26:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:26:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 -- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f81140a2ec0 msgr2=0x7f81140a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f81140a2ec0 0x7f81140a5380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f812404c000 tx=0x7f812404d000 comp rx=0 tx=0).stop 2024-05-09T08:26:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 -- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c103380 msgr2=0x7f812c1984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c103380 0x7f812c1984b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f812000c920 tx=0x7f812000cde0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 -- 172.21.15.28:0/2387980043 shutdown_connections 2024-05-09T08:26:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f81140a2ec0 0x7f81140a5380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f812c10be30 0x7f812c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f812c103380 0x7f812c1984b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 --2- 172.21.15.28:0/2387980043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f812c102a60 0x7f812c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 -- 172.21.15.28:0/2387980043 >> 172.21.15.28:0/2387980043 conn(0x7f812c0778b0 msgr2=0x7f812c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 -- 172.21.15.28:0/2387980043 shutdown_connections 2024-05-09T08:26:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:26:51.606+0000 7f8131a6b640 1 -- 172.21.15.28:0/2387980043 wait complete. 2024-05-09T08:26:51.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:51 smithi028 ceph-mon[30119]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-05-09T08:26:51.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:51 smithi165 ceph-mon[34540]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-05-09T08:26:51.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:51 smithi184 ceph-mon[34509]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-05-09T08:26:53.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:53 smithi028 ceph-mon[30119]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:26:53.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:53 smithi165 ceph-mon[34540]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:26:53.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:53 smithi184 ceph-mon[34509]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:26:53.979 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:26:53.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:26:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.102+0000 7f5a8d542640 1 -- 172.21.15.28:0/1227201202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 msgr2=0x7f5a88106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.102+0000 7f5a8d542640 1 --2- 172.21.15.28:0/1227201202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 0x7f5a88106af0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009930 tx=0x7f5a7802a420 comp rx=0 tx=0).stop 2024-05-09T08:26:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.102+0000 7f5a8d542640 1 -- 172.21.15.28:0/1227201202 shutdown_connections 2024-05-09T08:26:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.102+0000 7f5a8d542640 1 --2- 172.21.15.28:0/1227201202 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a881010c0 0x7f5a8810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.102+0000 7f5a8d542640 1 --2- 172.21.15.28:0/1227201202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a88100700 0x7f5a88100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.102+0000 7f5a8d542640 1 --2- 172.21.15.28:0/1227201202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 0x7f5a88106af0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.102+0000 7f5a8d542640 1 -- 172.21.15.28:0/1227201202 >> 172.21.15.28:0/1227201202 conn(0x7f5a88077830 msgr2=0x7f5a88077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.102+0000 7f5a8d542640 1 -- 172.21.15.28:0/1227201202 shutdown_connections 2024-05-09T08:26:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.102+0000 7f5a8d542640 1 -- 172.21.15.28:0/1227201202 wait complete. 2024-05-09T08:26:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.103+0000 7f5a8d542640 1 Processor -- start 2024-05-09T08:26:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.103+0000 7f5a8d542640 1 -- start start 2024-05-09T08:26:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.103+0000 7f5a8d542640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a88100700 0x7f5a8819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.103+0000 7f5a8d542640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a881010c0 0x7f5a8819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.103+0000 7f5a8d542640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 0x7f5a8819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.103+0000 7f5a8d542640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88110b30 con 0x7f5a88106710 2024-05-09T08:26:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.103+0000 7f5a8d542640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881109b0 con 0x7f5a88100700 2024-05-09T08:26:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.103+0000 7f5a8d542640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88110cb0 con 0x7f5a881010c0 2024-05-09T08:26:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 0x7f5a8819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 0x7f5a8819eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54424/0 (socket says 172.21.15.28:54424) 2024-05-09T08:26:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a877fe640 1 -- 172.21.15.28:0/2365904484 learned_addr learned my addr 172.21.15.28:0/2365904484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a88100700 0x7f5a8819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a867fc640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a881010c0 0x7f5a8819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a877fe640 1 -- 172.21.15.28:0/2365904484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a881010c0 msgr2=0x7f5a8819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a877fe640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a881010c0 0x7f5a8819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a877fe640 1 -- 172.21.15.28:0/2365904484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a88100700 msgr2=0x7f5a8819a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a877fe640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a88100700 0x7f5a8819a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a877fe640 1 -- 172.21.15.28:0/2365904484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a8819f260 con 0x7f5a88106710 2024-05-09T08:26:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a877fe640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 0x7f5a8819eb30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009fd0 tx=0x7f5a8000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a6ffff640 1 -- 172.21.15.28:0/2365904484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80017070 con 0x7f5a88106710 2024-05-09T08:26:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a6ffff640 1 -- 172.21.15.28:0/2365904484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a80003e50 con 0x7f5a88106710 2024-05-09T08:26:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a8d542640 1 -- 172.21.15.28:0/2365904484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a8819f550 con 0x7f5a88106710 2024-05-09T08:26:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a6ffff640 1 -- 172.21.15.28:0/2365904484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a800106c0 con 0x7f5a88106710 2024-05-09T08:26:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.104+0000 7f5a8d542640 1 -- 172.21.15.28:0/2365904484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a881a6de0 con 0x7f5a88106710 2024-05-09T08:26:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.107+0000 7f5a6ffff640 1 -- 172.21.15.28:0/2365904484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f5a80003970 con 0x7f5a88106710 2024-05-09T08:26:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.107+0000 7f5a90253740 1 Processor -- start 2024-05-09T08:26:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.107+0000 7f5a90253740 1 -- start start 2024-05-09T08:26:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.107+0000 7f5a90253740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406280 0x5624eb406660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.107+0000 7f5a90253740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406ba0 0x5624eb410d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.107+0000 7f5a90253740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 0x5624eb413660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.107+0000 7f5a90253740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624eb36bd00 con 0x5624eb406280 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.107+0000 7f5a90253740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5624eb3134c0 con 0x5624eb406ba0 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.107+0000 7f5a90253740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5624eb367cc0 con 0x5624eb411270 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406280 0x5624eb406660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406280 0x5624eb406660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54438/0 (socket says 172.21.15.28:54438) 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 -- 172.21.15.28:0/4002165533 learned_addr learned my addr 172.21.15.28:0/4002165533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a700a2d90 0x7f5a700a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/4002165533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406ba0 0x5624eb410d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 --2- 172.21.15.28:0/4002165533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 0x5624eb413660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 -- 172.21.15.28:0/4002165533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 msgr2=0x5624eb413660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 --2- 172.21.15.28:0/4002165533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 0x5624eb413660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 -- 172.21.15.28:0/4002165533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406ba0 msgr2=0x5624eb410d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 --2- 172.21.15.28:0/4002165533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406ba0 0x5624eb410d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 -- 172.21.15.28:0/4002165533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624eb1497d0 con 0x5624eb406280 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a877fe640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a700a2d90 0x7f5a700a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a6ffff640 1 -- 172.21.15.28:0/2365904484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f5a800cf6f0 con 0x7f5a88106710 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a867fc640 1 --2- 172.21.15.28:0/4002165533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406280 0x5624eb406660 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00cfa0 tx=0x7f5a7c00c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9515a52d2bca6c1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/4002165533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c018030 con 0x5624eb406280 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/4002165533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a7c005500 con 0x5624eb406280 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.108+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/4002165533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c0108a0 con 0x5624eb406280 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 -- 172.21.15.28:0/4002165533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406280 msgr2=0x5624eb406660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a877fe640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a700a2d90 0x7f5a700a5250 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a800cca10 tx=0x7f5a800667a0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 --2- 172.21.15.28:0/4002165533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406280 0x5624eb406660 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00cfa0 tx=0x7f5a7c00c620 comp rx=0 tx=0).stop 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 -- 172.21.15.28:0/4002165533 shutdown_connections 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 --2- 172.21.15.28:0/4002165533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 0x5624eb413660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 --2- 172.21.15.28:0/4002165533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406ba0 0x5624eb410d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 --2- 172.21.15.28:0/4002165533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406280 0x5624eb406660 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 -- 172.21.15.28:0/4002165533 >> 172.21.15.28:0/4002165533 conn(0x5624eb4025a0 msgr2=0x5624eb402980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 -- 172.21.15.28:0/4002165533 shutdown_connections 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 -- 172.21.15.28:0/4002165533 wait complete. 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.109+0000 7f5a90253740 1 -- start start 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a90253740 1 Processor -- start 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a90253740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 0x5624eb40e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a90253740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb40e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a90253740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 0x5624eb40ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a90253740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624eb414060 con 0x5624eb406ba0 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a90253740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5624eb326b90 con 0x5624eb406280 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a90253740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5624eb4141e0 con 0x5624eb411270 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a90253740 10 client.0 _ll_register_callbacks cb 0x5624eb362da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a86ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb40e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a86ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb40e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54446/0 (socket says 172.21.15.28:54446) 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a86ffd640 1 -- 172.21.15.28:0/4285259190 learned_addr learned my addr 172.21.15.28:0/4285259190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a867fc640 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 0x5624eb40e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a867fc640 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 0x5624eb40ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a86ffd640 1 -- 172.21.15.28:0/4285259190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 msgr2=0x5624eb40ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 0x5624eb40ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a86ffd640 1 -- 172.21.15.28:0/4285259190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 msgr2=0x5624eb40e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 0x5624eb40e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a86ffd640 1 -- 172.21.15.28:0/4285259190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624eb07f3f0 con 0x5624eb406ba0 2024-05-09T08:26:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.110+0000 7f5a867fc640 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 0x5624eb40e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb40e6e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009930 tx=0x7f5a78004480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ff9300981c16b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a6aff5640 1 -- 172.21.15.28:0/4285259190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78035040 con 0x5624eb406ba0 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a6aff5640 1 -- 172.21.15.28:0/4285259190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f5a78005510 con 0x5624eb406ba0 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a90253740 1 -- 172.21.15.28:0/4285259190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a7c005500 con 0x5624eb406ba0 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a6aff5640 1 -- 172.21.15.28:0/4285259190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5a780046c0 con 0x5624eb406ba0 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a90253740 10 client.15831 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a90253740 1 -- 172.21.15.28:0/4285259190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5624eb03f0f0 con 0x5624eb406ba0 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a6aff5640 1 -- 172.21.15.28:0/4285259190 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a7c005500 con 0x5624eb406ba0 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a6aff5640 1 -- 172.21.15.28:0/4285259190 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5a78031aa0 con 0x5624eb406ba0 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a90253740 10 client.15831 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a90253740 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a90253740 20 client.15831 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a90253740 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5624eb421590 0x5624eb4239c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a90253740 4 client.15831 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.111+0000 7f5a90253740 1 -- 172.21.15.28:0/4285259190 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624eb379430 con 0x5624eb421590 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.112+0000 7f5a867fc640 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5624eb421590 0x5624eb4239c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.112+0000 7f5a867fc640 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5624eb421590 0x5624eb4239c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fe251a50735b9e3 server_cookie=11235d498cf5e6fa in_seq=0 out_seq=0 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.112+0000 7f5a6aff5640 10 client.15831 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a6aff5640 1 -- 172.21.15.28:0/4285259190 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624eb379430 con 0x5624eb421590 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a6aff5640 10 client.15831 handle_command_reply: tid=0 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 client.15831 shutdown 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 -- 172.21.15.28:0/4285259190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 msgr2=0x5624eb40e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb40e6e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009930 tx=0x7f5a78004480 comp rx=0 tx=0).stop 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 -- 172.21.15.28:0/4285259190 shutdown_connections 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5624eb421590 0x5624eb4239c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb411270 0x5624eb40ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a6aff5640 0 client.15831 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb40e6e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 --2- 172.21.15.28:0/4285259190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 0x5624eb40e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 -- 172.21.15.28:0/4285259190 >> 172.21.15.28:0/4285259190 conn(0x5624eb4025a0 msgr2=0x5624eb4017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 -- 172.21.15.28:0/4285259190 shutdown_connections 2024-05-09T08:26:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 1 -- 172.21.15.28:0/4285259190 wait complete. 2024-05-09T08:26:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.113+0000 7f5a90253740 20 client.15831 trim_cache size 0 max 16384 2024-05-09T08:26:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a90253740 1 Processor -- start 2024-05-09T08:26:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a90253740 1 -- start start 2024-05-09T08:26:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a90253740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb406280 0x5624eb40e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a90253740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb4089e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a90253740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb411270 0x5624eb4292d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a90253740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881b1030 con 0x5624eb406ba0 2024-05-09T08:26:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a90253740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881b2b70 con 0x5624eb411270 2024-05-09T08:26:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a90253740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881b1700 con 0x5624eb406280 2024-05-09T08:26:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a86ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb4089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a86ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb4089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54452/0 (socket says 172.21.15.28:54452) 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.131+0000 7f5a86ffd640 1 -- 172.21.15.28:0/1942043633 learned_addr learned my addr 172.21.15.28:0/1942043633 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a86ffd640 1 -- 172.21.15.28:0/1942043633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb406280 msgr2=0x5624eb40e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/1942043633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb406280 0x5624eb40e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a86ffd640 1 -- 172.21.15.28:0/1942043633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb411270 msgr2=0x5624eb4292d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/1942043633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb411270 0x5624eb4292d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a86ffd640 1 -- 172.21.15.28:0/1942043633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624eb424040 con 0x5624eb406ba0 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/1942043633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb4089e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5a7802a940 tx=0x7f5a7802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a61298ff865211 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a6cff9640 1 -- 172.21.15.28:0/1942043633 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78057070 con 0x5624eb406ba0 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a6cff9640 1 -- 172.21.15.28:0/1942043633 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a78040070 con 0x5624eb406ba0 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a6cff9640 1 -- 172.21.15.28:0/1942043633 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78035030 con 0x5624eb406ba0 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a90253740 1 -- 172.21.15.28:0/1942043633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 msgr2=0x5624eb4089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a90253740 1 --2- 172.21.15.28:0/1942043633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb4089e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5a7802a940 tx=0x7f5a7802a420 comp rx=0 tx=0).stop 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.132+0000 7f5a90253740 1 -- 172.21.15.28:0/1942043633 shutdown_connections 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 --2- 172.21.15.28:0/1942043633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb411270 0x5624eb4292d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 --2- 172.21.15.28:0/1942043633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb406ba0 0x5624eb4089e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 --2- 172.21.15.28:0/1942043633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb406280 0x5624eb40e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 -- 172.21.15.28:0/1942043633 >> 172.21.15.28:0/1942043633 conn(0x5624eb4025a0 msgr2=0x5624eb400f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 -- 172.21.15.28:0/1942043633 shutdown_connections 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 -- 172.21.15.28:0/1942043633 wait complete. 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 -- start start 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 Processor -- start 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 0x5624eb4154d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb406ba0 0x5624eb415a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb411270 0x5624eb486b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881b24a0 con 0x5624eb411270 2024-05-09T08:26:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881b1dd0 con 0x5624eb406280 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881af010 con 0x5624eb406ba0 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a90253740 10 client.0 _ll_register_callbacks cb 0x7f5a78009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb411270 0x5624eb486b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb411270 0x5624eb486b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54458/0 (socket says 172.21.15.28:54458) 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.133+0000 7f5a867fc640 1 -- 172.21.15.28:0/1494000149 learned_addr learned my addr 172.21.15.28:0/1494000149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a867fc640 1 -- 172.21.15.28:0/1494000149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb406ba0 msgr2=0x5624eb415a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a867fc640 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb406ba0 0x5624eb415a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a867fc640 1 -- 172.21.15.28:0/1494000149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 msgr2=0x5624eb4154d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a867fc640 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 0x5624eb4154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a867fc640 1 -- 172.21.15.28:0/1494000149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624eb487080 con 0x5624eb411270 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a867fc640 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb411270 0x5624eb486b40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c012fd0 tx=0x7f5a7c00c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b51c7c4dedf4446 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a6aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a6aff5640 1 -- 172.21.15.28:0/1494000149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c030060 con 0x5624eb411270 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a6aff5640 1 -- 172.21.15.28:0/1494000149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f5a7c02b7b0 con 0x5624eb411270 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a90253740 1 -- 172.21.15.28:0/1494000149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a78040070 con 0x5624eb411270 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a6aff5640 1 -- 172.21.15.28:0/1494000149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5a7c018040 con 0x5624eb411270 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a90253740 10 client.15837 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a90253740 1 -- 172.21.15.28:0/1494000149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5624eb4882f0 con 0x5624eb411270 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a6aff5640 1 -- 172.21.15.28:0/1494000149 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a78040070 con 0x5624eb411270 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a6aff5640 1 -- 172.21.15.28:0/1494000149 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5a7c0108b0 con 0x5624eb411270 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a90253740 10 client.15837 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a90253740 10 client.15837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a90253740 20 client.15837 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a90253740 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5624eb488740 0x5624eb48ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a90253740 4 client.15837 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.134+0000 7f5a90253740 1 -- 172.21.15.28:0/1494000149 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5624eb373780 con 0x5624eb488740 2024-05-09T08:26:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.135+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5624eb488740 0x5624eb48ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.135+0000 7f5a6aff5640 10 client.15837 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.135+0000 7f5a86ffd640 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5624eb488740 0x5624eb48ab30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d98982af3fa304dc server_cookie=680e96558b26e84 in_seq=0 out_seq=0 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.135+0000 7f5a6aff5640 1 -- 172.21.15.28:0/1494000149 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5624eb373780 con 0x5624eb488740 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.135+0000 7f5a6aff5640 10 client.15837 handle_command_reply: tid=0 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.135+0000 7f5a90253740 1 client.15837 shutdown 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 -- 172.21.15.28:0/1494000149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb411270 msgr2=0x5624eb486b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb411270 0x5624eb486b40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c012fd0 tx=0x7f5a7c00c3d0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 -- 172.21.15.28:0/1494000149 shutdown_connections 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5624eb488740 0x5624eb48ab30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624eb411270 0x5624eb486b40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5624eb406ba0 0x5624eb415a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a6aff5640 0 client.15837 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 --2- 172.21.15.28:0/1494000149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624eb406280 0x5624eb4154d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 -- 172.21.15.28:0/1494000149 >> 172.21.15.28:0/1494000149 conn(0x5624eb4025a0 msgr2=0x5624eb4017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 -- 172.21.15.28:0/1494000149 shutdown_connections 2024-05-09T08:26:54.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 1 -- 172.21.15.28:0/1494000149 wait complete. 2024-05-09T08:26:54.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a90253740 20 client.15837 trim_cache size 0 max 16384 2024-05-09T08:26:54.145 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:54.145 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-09T08:26:54.145 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-09T08:26:54.145 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:54.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 -- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a700a2d90 msgr2=0x7f5a700a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a700a2d90 0x7f5a700a5250 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a800cca10 tx=0x7f5a800667a0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 -- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 msgr2=0x7f5a8819eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 0x7f5a8819eb30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009fd0 tx=0x7f5a8000b260 comp rx=0 tx=0).stop 2024-05-09T08:26:54.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 -- 172.21.15.28:0/2365904484 shutdown_connections 2024-05-09T08:26:54.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a88106710 0x7f5a8819eb30 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a700a2d90 0x7f5a700a5250 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a881010c0 0x7f5a8819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 --2- 172.21.15.28:0/2365904484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a88100700 0x7f5a8819a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 -- 172.21.15.28:0/2365904484 >> 172.21.15.28:0/2365904484 conn(0x7f5a88077830 msgr2=0x7f5a88104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 -- 172.21.15.28:0/2365904484 shutdown_connections 2024-05-09T08:26:54.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.136+0000 7f5a8d542640 1 -- 172.21.15.28:0/2365904484 wait complete. 2024-05-09T08:26:54.148 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:26:54.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.253+0000 7f273b7ee640 1 -- 172.21.15.28:0/504679125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27341021c0 msgr2=0x7f273410e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.253+0000 7f273b7ee640 1 --2- 172.21.15.28:0/504679125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27341021c0 0x7f273410e6d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f272c009930 tx=0x7f272c02a420 comp rx=0 tx=0).stop 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.253+0000 7f273b7ee640 1 -- 172.21.15.28:0/504679125 shutdown_connections 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.253+0000 7f273b7ee640 1 --2- 172.21.15.28:0/504679125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27341021c0 0x7f273410e6d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.253+0000 7f273b7ee640 1 --2- 172.21.15.28:0/504679125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2734101800 0x7f2734101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.253+0000 7f273b7ee640 1 --2- 172.21.15.28:0/504679125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2734107810 0x7f2734107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.253+0000 7f273b7ee640 1 -- 172.21.15.28:0/504679125 >> 172.21.15.28:0/504679125 conn(0x7f2734078bf0 msgr2=0x7f2734079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.253+0000 7f273b7ee640 1 -- 172.21.15.28:0/504679125 shutdown_connections 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.253+0000 7f273b7ee640 1 -- 172.21.15.28:0/504679125 wait complete. 2024-05-09T08:26:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273b7ee640 1 Processor -- start 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273b7ee640 1 -- start start 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273b7ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2734101800 0x7f27341980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273b7ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27341021c0 0x7f27341985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273b7ee640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2734107810 0x7f273419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273b7ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2734111c70 con 0x7f2734101800 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273b7ee640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2734111af0 con 0x7f2734107810 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273b7ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2734111df0 con 0x7f27341021c0 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273958a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2734101800 0x7f27341980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273958a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2734101800 0x7f27341980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54486/0 (socket says 172.21.15.28:54486) 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273958a640 1 -- 172.21.15.28:0/2303618251 learned_addr learned my addr 172.21.15.28:0/2303618251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f2739d8b640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2734107810 0x7f273419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273958a640 1 -- 172.21.15.28:0/2303618251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27341021c0 msgr2=0x7f27341985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273958a640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27341021c0 0x7f27341985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273958a640 1 -- 172.21.15.28:0/2303618251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2734107810 msgr2=0x7f273419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273958a640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2734107810 0x7f273419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273958a640 1 -- 172.21.15.28:0/2303618251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273419d0a0 con 0x7f2734101800 2024-05-09T08:26:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.254+0000 7f273958a640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2734101800 0x7f27341980b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f273000eab0 tx=0x7f273000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.255+0000 7f272a7fc640 1 -- 172.21.15.28:0/2303618251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273000ce40 con 0x7f2734101800 2024-05-09T08:26:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.255+0000 7f272a7fc640 1 -- 172.21.15.28:0/2303618251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2730011950 con 0x7f2734101800 2024-05-09T08:26:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.255+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f273419d390 con 0x7f2734101800 2024-05-09T08:26:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.255+0000 7f272a7fc640 1 -- 172.21.15.28:0/2303618251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730010ae0 con 0x7f2734101800 2024-05-09T08:26:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.255+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27341a4c20 con 0x7f2734101800 2024-05-09T08:26:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.256+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2734068aa0 con 0x7f2734101800 2024-05-09T08:26:54.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.257+0000 7f272a7fc640 1 -- 172.21.15.28:0/2303618251 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f2730011b00 con 0x7f2734101800 2024-05-09T08:26:54.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.258+0000 7f272a7fc640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27200a2d90 0x7f27200a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.258+0000 7f2738d89640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27200a2d90 0x7f27200a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.258+0000 7f272a7fc640 1 -- 172.21.15.28:0/2303618251 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f27300d00e0 con 0x7f2734101800 2024-05-09T08:26:54.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.258+0000 7f2738d89640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27200a2d90 0x7f27200a5250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2734103990 tx=0x7f2724005bf0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.261+0000 7f272a7fc640 1 -- 172.21.15.28:0/2303618251 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2730016030 con 0x7f2734101800 2024-05-09T08:26:54.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.485+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2734199400 con 0x7f2734101800 2024-05-09T08:26:54.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.486+0000 7f272a7fc640 1 -- 172.21.15.28:0/2303618251 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11932 (secure 0 0 0) 0x7f273009c760 con 0x7f2734101800 2024-05-09T08:26:54.486 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:26:54.487 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:26:16.129765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24463,24512,24476,24481,24499]},"id":1}]} 2024-05-09T08:26:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-05-09T08:26:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27200a2d90 msgr2=0x7f27200a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27200a2d90 0x7f27200a5250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2734103990 tx=0x7f2724005bf0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2734101800 msgr2=0x7f27341980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2734101800 0x7f27341980b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f273000eab0 tx=0x7f273000ef70 comp rx=0 tx=0).stop 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 shutdown_connections 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2734107810 0x7f273419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27200a2d90 0x7f27200a5250 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27341021c0 0x7f27341985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 --2- 172.21.15.28:0/2303618251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2734101800 0x7f27341980b0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 >> 172.21.15.28:0/2303618251 conn(0x7f2734078bf0 msgr2=0x7f2734104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 shutdown_connections 2024-05-09T08:26:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.488+0000 7f273b7ee640 1 -- 172.21.15.28:0/2303618251 wait complete. 2024-05-09T08:26:54.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:26:54.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.607+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/858191861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94100d80 msgr2=0x7f2f941064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.607+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/858191861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94100d80 0x7f2f941064b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009900 tx=0x7f2f7c02a410 comp rx=0 tx=0).stop 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.607+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/858191861 shutdown_connections 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.607+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/858191861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f94106b90 0x7f2f9410e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.607+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/858191861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94100d80 0x7f2f941064b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.607+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/858191861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f941003d0 0x7f2f941007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.607+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/858191861 >> 172.21.15.28:0/858191861 conn(0x7f2f940759c0 msgr2=0x7f2f94075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.607+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/858191861 shutdown_connections 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.607+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/858191861 wait complete. 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f9ad3b640 1 Processor -- start 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f9ad3b640 1 -- start start 2024-05-09T08:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f9ad3b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f941003d0 0x7f2f94198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f9ad3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94106b90 0x7f2f94198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f9ad3b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f9419ca70 0x7f2f9419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f9ad3b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f94111c80 con 0x7f2f94106b90 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f9ad3b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2f94111b00 con 0x7f2f941003d0 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f9ad3b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f94111e00 con 0x7f2f9419ca70 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94106b90 0x7f2f94198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94106b90 0x7f2f94198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51646/0 (socket says 172.21.15.28:51646) 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f93fff640 1 -- 172.21.15.28:0/2844130837 learned_addr learned my addr 172.21.15.28:0/2844130837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f941003d0 0x7f2f94198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f992d8640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f9419ca70 0x7f2f9419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f93fff640 1 -- 172.21.15.28:0/2844130837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f9419ca70 msgr2=0x7f2f9419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f93fff640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f9419ca70 0x7f2f9419cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f93fff640 1 -- 172.21.15.28:0/2844130837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f941003d0 msgr2=0x7f2f94198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f93fff640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f941003d0 0x7f2f94198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f93fff640 1 -- 172.21.15.28:0/2844130837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9419d560 con 0x7f2f94106b90 2024-05-09T08:26:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.608+0000 7f2f93fff640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94106b90 0x7f2f94198740 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c005940 tx=0x7f2f7c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.609+0000 7f2f91ffb640 1 -- 172.21.15.28:0/2844130837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c03f030 con 0x7f2f94106b90 2024-05-09T08:26:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.609+0000 7f2f91ffb640 1 -- 172.21.15.28:0/2844130837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f7c003ce0 con 0x7f2f94106b90 2024-05-09T08:26:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.609+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/2844130837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9419d7f0 con 0x7f2f94106b90 2024-05-09T08:26:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.609+0000 7f2f91ffb640 1 -- 172.21.15.28:0/2844130837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c035070 con 0x7f2f94106b90 2024-05-09T08:26:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.609+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/2844130837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f941a4fb0 con 0x7f2f94106b90 2024-05-09T08:26:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.611+0000 7f2f91ffb640 1 -- 172.21.15.28:0/2844130837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f2f7c042030 con 0x7f2f94106b90 2024-05-09T08:26:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.611+0000 7f2f9da49740 1 Processor -- start 2024-05-09T08:26:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f9da49740 1 -- start start 2024-05-09T08:26:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48029330 0x560d48029710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d48033de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48034320 0x560d48036710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f9da49740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d47f8ec40 con 0x560d48034320 2024-05-09T08:26:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f9da49740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560d47f36580 con 0x560d48029330 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f9da49740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560d47f8ac00 con 0x560d48029c50 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48034320 0x560d48036710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f992d8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48029330 0x560d48029710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48034320 0x560d48036710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51648/0 (socket says 172.21.15.28:51648) 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f992d8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48029330 0x560d48029710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33520/0 (socket says 172.21.15.28:33520) 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f93fff640 1 -- 172.21.15.28:0/2931844997 learned_addr learned my addr 172.21.15.28:0/2931844997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/2931844997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d48033de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f91ffb640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f800a2d90 0x7f2f800a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f992d8640 1 -- 172.21.15.28:0/2931844997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 msgr2=0x560d48033de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f992d8640 1 --2- 172.21.15.28:0/2931844997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d48033de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f992d8640 1 -- 172.21.15.28:0/2931844997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48034320 msgr2=0x560d48036710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f992d8640 1 --2- 172.21.15.28:0/2931844997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48034320 0x560d48036710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f992d8640 1 -- 172.21.15.28:0/2931844997 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d47ca23f0 con 0x560d48029330 2024-05-09T08:26:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f93fff640 1 --2- 172.21.15.28:0/2931844997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48034320 0x560d48036710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f992d8640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f800a2d90 0x7f2f800a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.612+0000 7f2f91ffb640 1 -- 172.21.15.28:0/2844130837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f2f7c0eff90 con 0x7f2f94106b90 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.613+0000 7f2f992d8640 1 --2- 172.21.15.28:0/2931844997 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48029330 0x560d48029710 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009e20 tx=0x7f2f8c00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9080958111ae9a0e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.613+0000 7f2f992d8640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f800a2d90 0x7f2f800a5250 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c020000 tx=0x7f2f8c01d000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.613+0000 7f2f8affd640 1 -- 172.21.15.28:0/2931844997 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c025440 con 0x560d48029330 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.613+0000 7f2f8affd640 1 -- 172.21.15.28:0/2931844997 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f8c01a8f0 con 0x560d48029330 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.613+0000 7f2f8affd640 1 -- 172.21.15.28:0/2931844997 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c0105f0 con 0x560d48029330 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.613+0000 7f2f9da49740 1 -- 172.21.15.28:0/2931844997 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48029330 msgr2=0x560d48029710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.613+0000 7f2f9da49740 1 --2- 172.21.15.28:0/2931844997 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48029330 0x560d48029710 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009e20 tx=0x7f2f8c00f730 comp rx=0 tx=0).stop 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 -- 172.21.15.28:0/2931844997 shutdown_connections 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 --2- 172.21.15.28:0/2931844997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48034320 0x560d48036710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 --2- 172.21.15.28:0/2931844997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d48033de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 --2- 172.21.15.28:0/2931844997 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48029330 0x560d48029710 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 -- 172.21.15.28:0/2931844997 >> 172.21.15.28:0/2931844997 conn(0x560d48025650 msgr2=0x560d48025a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 -- 172.21.15.28:0/2931844997 shutdown_connections 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 -- 172.21.15.28:0/2931844997 wait complete. 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 -- start start 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 Processor -- start 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d480312e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d48031820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d48031d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d48037110 con 0x560d48029330 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560d47f49c40 con 0x560d48034320 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560d48037290 con 0x560d48029c50 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f9da49740 10 client.0 _ll_register_callbacks cb 0x560d47ca1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.614+0000 7f2f98ad7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d480312e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f98ad7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d480312e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51660/0 (socket says 172.21.15.28:51660) 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f98ad7640 1 -- 172.21.15.28:0/1158006180 learned_addr learned my addr 172.21.15.28:0/1158006180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f992d8640 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d48031d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f93fff640 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d48031820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f98ad7640 1 -- 172.21.15.28:0/1158006180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 msgr2=0x560d48031820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d48031820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f98ad7640 1 -- 172.21.15.28:0/1158006180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 msgr2=0x560d48031d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d48031d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f98ad7640 1 -- 172.21.15.28:0/1158006180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d47d6c7d0 con 0x560d48029330 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f93fff640 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d48031820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d480312e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400ba40 tx=0x7f2f8400bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a00e167f79bc993 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f73fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f73fff640 1 -- 172.21.15.28:0/1158006180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84003e80 con 0x560d48029330 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f73fff640 1 -- 172.21.15.28:0/1158006180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f2f84011440 con 0x560d48029330 2024-05-09T08:26:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f9da49740 1 -- 172.21.15.28:0/1158006180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f8c01a8f0 con 0x560d48029330 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f73fff640 1 -- 172.21.15.28:0/1158006180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2f84011640 con 0x560d48029330 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f9da49740 10 client.15855 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.615+0000 7f2f9da49740 1 -- 172.21.15.28:0/1158006180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560d48043430 con 0x560d48029330 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.616+0000 7f2f73fff640 1 -- 172.21.15.28:0/1158006180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f8c01a8f0 con 0x560d48029330 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.616+0000 7f2f73fff640 1 -- 172.21.15.28:0/1158006180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2f8401ba00 con 0x560d48029330 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.616+0000 7f2f9da49740 10 client.15855 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.616+0000 7f2f9da49740 10 client.15855 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.616+0000 7f2f9da49740 20 client.15855 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.616+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x560d48044990 0x560d48046e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.616+0000 7f2f9da49740 4 client.15855 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.616+0000 7f2f9da49740 1 -- 172.21.15.28:0/1158006180 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d47f9c370 con 0x560d48044990 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.616+0000 7f2f93fff640 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x560d48044990 0x560d48046e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f93fff640 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x560d48044990 0x560d48046e00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4781e3a8a30c5827 server_cookie=b593ba949530b3de in_seq=0 out_seq=0 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f73fff640 10 client.15855 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f73fff640 1 -- 172.21.15.28:0/1158006180 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560d47f9c370 con 0x560d48044990 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f73fff640 10 client.15855 handle_command_reply: tid=0 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f9da49740 1 client.15855 shutdown 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f9da49740 1 -- 172.21.15.28:0/1158006180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 msgr2=0x560d480312e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d480312e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400ba40 tx=0x7f2f8400bf00 comp rx=0 tx=0).stop 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f9da49740 1 -- 172.21.15.28:0/1158006180 shutdown_connections 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d48031d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x560d48044990 0x560d48046e00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d48031820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1158006180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d480312e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f73fff640 0 client.15855 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:26:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.617+0000 7f2f9da49740 1 -- 172.21.15.28:0/1158006180 >> 172.21.15.28:0/1158006180 conn(0x560d48025650 msgr2=0x560d4802b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.618+0000 7f2f9da49740 1 -- 172.21.15.28:0/1158006180 shutdown_connections 2024-05-09T08:26:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.618+0000 7f2f9da49740 1 -- 172.21.15.28:0/1158006180 wait complete. 2024-05-09T08:26:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.618+0000 7f2f9da49740 20 client.15855 trim_cache size 0 max 16384 2024-05-09T08:26:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f9da49740 1 Processor -- start 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f9da49740 1 -- start start 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d48040000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d4802b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d4804c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f9da49740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f941aed50 con 0x560d48029330 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f9da49740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2f941b0890 con 0x560d48034320 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f9da49740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f941af420 con 0x560d48029c50 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f98ad7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d48040000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f98ad7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d48040000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51670/0 (socket says 172.21.15.28:51670) 2024-05-09T08:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.631+0000 7f2f98ad7640 1 -- 172.21.15.28:0/2644688387 learned_addr learned my addr 172.21.15.28:0/2644688387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f93fff640 1 --2- 172.21.15.28:0/2644688387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d4802b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f98ad7640 1 -- 172.21.15.28:0/2644688387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 msgr2=0x560d4802b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/2644688387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d4802b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f98ad7640 1 -- 172.21.15.28:0/2644688387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 msgr2=0x560d4804c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/2644688387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d4804c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f98ad7640 1 -- 172.21.15.28:0/2644688387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d48016d20 con 0x560d48029330 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/2644688387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d48040000 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2f8401bec0 tx=0x7f2f84003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce3178aceb3be1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f8a7fc640 1 -- 172.21.15.28:0/2644688387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8400b400 con 0x560d48029330 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f8a7fc640 1 -- 172.21.15.28:0/2644688387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f84005520 con 0x560d48029330 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f8a7fc640 1 -- 172.21.15.28:0/2644688387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84004680 con 0x560d48029330 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f9da49740 1 -- 172.21.15.28:0/2644688387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 msgr2=0x560d48040000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f9da49740 1 --2- 172.21.15.28:0/2644688387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d48040000 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2f8401bec0 tx=0x7f2f84003b00 comp rx=0 tx=0).stop 2024-05-09T08:26:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f9da49740 1 -- 172.21.15.28:0/2644688387 shutdown_connections 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f9da49740 1 --2- 172.21.15.28:0/2644688387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d4804c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f9da49740 1 --2- 172.21.15.28:0/2644688387 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029c50 0x560d4802b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f9da49740 1 --2- 172.21.15.28:0/2644688387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029330 0x560d48040000 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.632+0000 7f2f9da49740 1 -- 172.21.15.28:0/2644688387 >> 172.21.15.28:0/2644688387 conn(0x560d48025650 msgr2=0x560d4802ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 -- 172.21.15.28:0/2644688387 shutdown_connections 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 -- 172.21.15.28:0/2644688387 wait complete. 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 -- start start 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 Processor -- start 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029330 0x560d4803ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029c50 0x560d48038de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d48039520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f941b01c0 con 0x560d48029c50 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2f941afaf0 con 0x560d48034320 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f941acd30 con 0x560d48029330 2024-05-09T08:26:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f9da49740 10 client.0 _ll_register_callbacks cb 0x7f2f840042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029c50 0x560d48038de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029c50 0x560d48038de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51684/0 (socket says 172.21.15.28:51684) 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.633+0000 7f2f93fff640 1 -- 172.21.15.28:0/1068978201 learned_addr learned my addr 172.21.15.28:0/1068978201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029330 0x560d4803ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f992d8640 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d48039520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f93fff640 1 -- 172.21.15.28:0/1068978201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029330 msgr2=0x560d4803ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f93fff640 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029330 0x560d4803ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f93fff640 1 -- 172.21.15.28:0/1068978201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 msgr2=0x560d48039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f93fff640 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d48039520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f93fff640 1 -- 172.21.15.28:0/1068978201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d480aaf50 con 0x560d48029c50 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f992d8640 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d48039520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f93fff640 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029c50 0x560d48038de0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0b88e0 tx=0x7f2f7c0f23e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5004c4a99238799 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f73fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f73fff640 1 -- 172.21.15.28:0/1068978201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c049040 con 0x560d48029c50 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f73fff640 1 -- 172.21.15.28:0/1068978201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f2f7c0b8580 con 0x560d48029c50 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f9da49740 1 -- 172.21.15.28:0/1068978201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f84005520 con 0x560d48029c50 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f73fff640 1 -- 172.21.15.28:0/1068978201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2f7c0bb870 con 0x560d48029c50 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f9da49740 10 client.15861 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f9da49740 1 -- 172.21.15.28:0/1068978201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560d480ac0b0 con 0x560d48029c50 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f73fff640 1 -- 172.21.15.28:0/1068978201 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f84005520 con 0x560d48029c50 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.634+0000 7f2f73fff640 1 -- 172.21.15.28:0/1068978201 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2f7c03bce0 con 0x560d48029c50 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f9da49740 10 client.15861 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f9da49740 10 client.15861 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f9da49740 20 client.15861 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x560d480ac7e0 0x560d480aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f9da49740 4 client.15861 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f9da49740 1 -- 172.21.15.28:0/1068978201 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d47f966c0 con 0x560d480ac7e0 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x560d480ac7e0 0x560d480aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f98ad7640 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x560d480ac7e0 0x560d480aec00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c24af7b78671d954 server_cookie=7ce7b9e6af220fa in_seq=0 out_seq=0 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f73fff640 10 client.15861 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f73fff640 1 -- 172.21.15.28:0/1068978201 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d47f966c0 con 0x560d480ac7e0 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f73fff640 10 client.15861 handle_command_reply: tid=0 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.635+0000 7f2f9da49740 1 client.15861 shutdown 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 -- 172.21.15.28:0/1068978201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029c50 msgr2=0x560d48038de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029c50 0x560d48038de0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0b88e0 tx=0x7f2f7c0f23e0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 -- 172.21.15.28:0/1068978201 shutdown_connections 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560d48034320 0x560d48039520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d48029c50 0x560d48038de0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x560d480ac7e0 0x560d480aec00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 --2- 172.21.15.28:0/1068978201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d48029330 0x560d4803ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f73fff640 0 client.15861 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 -- 172.21.15.28:0/1068978201 >> 172.21.15.28:0/1068978201 conn(0x560d48025650 msgr2=0x560d48031b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 -- 172.21.15.28:0/1068978201 shutdown_connections 2024-05-09T08:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 1 -- 172.21.15.28:0/1068978201 wait complete. 2024-05-09T08:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9da49740 20 client.15861 trim_cache size 0 max 16384 2024-05-09T08:26:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:26:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f800a2d90 msgr2=0x7f2f800a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f800a2d90 0x7f2f800a5250 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c020000 tx=0x7f2f8c01d000 comp rx=0 tx=0).stop 2024-05-09T08:26:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94106b90 msgr2=0x7f2f94198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94106b90 0x7f2f94198740 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c005940 tx=0x7f2f7c0042f0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/2844130837 shutdown_connections 2024-05-09T08:26:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f800a2d90 0x7f2f800a5250 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f9419ca70 0x7f2f9419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f94106b90 0x7f2f94198740 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 --2- 172.21.15.28:0/2844130837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f941003d0 0x7f2f94198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.636+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/2844130837 >> 172.21.15.28:0/2844130837 conn(0x7f2f940759c0 msgr2=0x7f2f940ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.637+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/2844130837 shutdown_connections 2024-05-09T08:26:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.637+0000 7f2f9ad3b640 1 -- 172.21.15.28:0/2844130837 wait complete. 2024-05-09T08:26:54.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:26:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 -- 172.21.15.28:0/1825420609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 msgr2=0x7f1844106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 --2- 172.21.15.28:0/1825420609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 0x7f1844106af0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1834009930 tx=0x7f183402a420 comp rx=0 tx=0).stop 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 -- 172.21.15.28:0/1825420609 shutdown_connections 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 --2- 172.21.15.28:0/1825420609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18441010c0 0x7f184410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 --2- 172.21.15.28:0/1825420609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1844100700 0x7f1844100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 --2- 172.21.15.28:0/1825420609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 0x7f1844106af0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 -- 172.21.15.28:0/1825420609 >> 172.21.15.28:0/1825420609 conn(0x7f1844077830 msgr2=0x7f1844077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 -- 172.21.15.28:0/1825420609 shutdown_connections 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 -- 172.21.15.28:0/1825420609 wait complete. 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 Processor -- start 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.755+0000 7f1849b4f640 1 -- start start 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1849b4f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1844100700 0x7f184419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1849b4f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18441010c0 0x7f184419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1849b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 0x7f184419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1849b4f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1844111c20 con 0x7f1844106710 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1849b4f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1844111aa0 con 0x7f1844100700 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1849b4f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1844111da0 con 0x7f18441010c0 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 0x7f184419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f18437fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1844100700 0x7f184419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 0x7f184419ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51698/0 (socket says 172.21.15.28:51698) 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1843fff640 1 -- 172.21.15.28:0/152082524 learned_addr learned my addr 172.21.15.28:0/152082524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1842ffd640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18441010c0 0x7f184419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f18437fe640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1844100700 0x7f184419a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33570/0 (socket says 172.21.15.28:33570) 2024-05-09T08:26:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1843fff640 1 -- 172.21.15.28:0/152082524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18441010c0 msgr2=0x7f184419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1843fff640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18441010c0 0x7f184419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1843fff640 1 -- 172.21.15.28:0/152082524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1844100700 msgr2=0x7f184419a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1843fff640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1844100700 0x7f184419a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1843fff640 1 -- 172.21.15.28:0/152082524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184419f2f0 con 0x7f1844106710 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1843fff640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 0x7f184419ebc0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f183c00cad0 tx=0x7f183c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.756+0000 7f1840ff9640 1 -- 172.21.15.28:0/152082524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c00ae20 con 0x7f1844106710 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.757+0000 7f1840ff9640 1 -- 172.21.15.28:0/152082524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f183c005020 con 0x7f1844106710 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.757+0000 7f1849b4f640 1 -- 172.21.15.28:0/152082524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184419f5e0 con 0x7f1844106710 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.757+0000 7f1840ff9640 1 -- 172.21.15.28:0/152082524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c00f6f0 con 0x7f1844106710 2024-05-09T08:26:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.757+0000 7f1849b4f640 1 -- 172.21.15.28:0/152082524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18441a6e70 con 0x7f1844106710 2024-05-09T08:26:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f1840ff9640 1 -- 172.21.15.28:0/152082524 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f183c0051d0 con 0x7f1844106710 2024-05-09T08:26:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f184c859740 1 Processor -- start 2024-05-09T08:26:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f184c859740 1 -- start start 2024-05-09T08:26:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f184c859740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 0x561ac3d47710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f184c859740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47c50 0x561ac3d51de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f184c859740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d54710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f184c859740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ac3cacc40 con 0x561ac3d52320 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f184c859740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x561ac3c54580 con 0x561ac3d47c50 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f184c859740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561ac3ca8c00 con 0x561ac3d47330 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f1840ff9640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f182c0a2d90 0x7f182c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.759+0000 7f1842ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d54710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1842ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d54710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51706/0 (socket says 172.21.15.28:51706) 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1842ffd640 1 -- 172.21.15.28:0/587919810 learned_addr learned my addr 172.21.15.28:0/587919810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1843fff640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f182c0a2d90 0x7f182c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f18437fe640 1 --2- 172.21.15.28:0/587919810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47c50 0x561ac3d51de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1842ffd640 1 --2- 172.21.15.28:0/587919810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 0x561ac3d47710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1840ff9640 1 -- 172.21.15.28:0/152082524 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f183c0cfbf0 con 0x7f1844106710 2024-05-09T08:26:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1842ffd640 1 -- 172.21.15.28:0/587919810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 msgr2=0x561ac3d47710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1842ffd640 1 --2- 172.21.15.28:0/587919810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 0x561ac3d47710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1842ffd640 1 -- 172.21.15.28:0/587919810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47c50 msgr2=0x561ac3d51de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1842ffd640 1 --2- 172.21.15.28:0/587919810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47c50 0x561ac3d51de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1842ffd640 1 -- 172.21.15.28:0/587919810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ac39c03f0 con 0x561ac3d52320 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1843fff640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f182c0a2d90 0x7f182c0a5250 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f183c0ccf10 tx=0x7f183c000f40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1842ffd640 1 --2- 172.21.15.28:0/587919810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d54710 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1838012540 tx=0x7f183800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead759fc954ffb83 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1829ffb640 1 -- 172.21.15.28:0/587919810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183801e640 con 0x561ac3d52320 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1829ffb640 1 -- 172.21.15.28:0/587919810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f183801ecc0 con 0x561ac3d52320 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.760+0000 7f1829ffb640 1 -- 172.21.15.28:0/587919810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838010d20 con 0x561ac3d52320 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 -- 172.21.15.28:0/587919810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 msgr2=0x561ac3d54710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 --2- 172.21.15.28:0/587919810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d54710 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1838012540 tx=0x7f183800b870 comp rx=0 tx=0).stop 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 -- 172.21.15.28:0/587919810 shutdown_connections 2024-05-09T08:26:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 --2- 172.21.15.28:0/587919810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d54710 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 --2- 172.21.15.28:0/587919810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47c50 0x561ac3d51de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 --2- 172.21.15.28:0/587919810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 0x561ac3d47710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 -- 172.21.15.28:0/587919810 >> 172.21.15.28:0/587919810 conn(0x561ac3d43650 msgr2=0x561ac3d43a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 -- 172.21.15.28:0/587919810 shutdown_connections 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 -- 172.21.15.28:0/587919810 wait complete. 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.761+0000 7f184c859740 1 -- start start 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f184c859740 1 Processor -- start 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f184c859740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47330 0x561ac3d4f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f184c859740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47c50 0x561ac3d4f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f184c859740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d4fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f184c859740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ac3d55110 con 0x561ac3d52320 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f184c859740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x561ac3c67c40 con 0x561ac3d47330 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f184c859740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561ac3d55290 con 0x561ac3d47c50 2024-05-09T08:26:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f184c859740 10 client.0 _ll_register_callbacks cb 0x561ac39bfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d4fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d4fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51718/0 (socket says 172.21.15.28:51718) 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 -- 172.21.15.28:0/4184762902 learned_addr learned my addr 172.21.15.28:0/4184762902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47330 0x561ac3d4f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f18437fe640 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47c50 0x561ac3d4f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 -- 172.21.15.28:0/4184762902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47c50 msgr2=0x561ac3d4f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47c50 0x561ac3d4f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 -- 172.21.15.28:0/4184762902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47330 msgr2=0x561ac3d4f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47330 0x561ac3d4f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 -- 172.21.15.28:0/4184762902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ac3a8a7d0 con 0x561ac3d52320 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1842ffd640 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d4fd20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f1838001eb0 tx=0x7f18380109c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25bb4b7dbffdddd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1822ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1822ffd640 1 -- 172.21.15.28:0/4184762902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838005030 con 0x561ac3d52320 2024-05-09T08:26:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.762+0000 7f1822ffd640 1 -- 172.21.15.28:0/4184762902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f18380051d0 con 0x561ac3d52320 2024-05-09T08:26:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f184c859740 1 -- 172.21.15.28:0/4184762902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f183801ecc0 con 0x561ac3d52320 2024-05-09T08:26:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f1822ffd640 1 -- 172.21.15.28:0/4184762902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1838034050 con 0x561ac3d52320 2024-05-09T08:26:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f184c859740 10 client.15873 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f184c859740 1 -- 172.21.15.28:0/4184762902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ac3d61320 con 0x561ac3d52320 2024-05-09T08:26:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f1822ffd640 1 -- 172.21.15.28:0/4184762902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f183801ecc0 con 0x561ac3d52320 2024-05-09T08:26:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f1822ffd640 1 -- 172.21.15.28:0/4184762902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f183801c020 con 0x561ac3d52320 2024-05-09T08:26:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f184c859740 10 client.15873 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f184c859740 10 client.15873 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f184c859740 20 client.15873 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f184c859740 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561ac3d62670 0x561ac3d64a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f184c859740 4 client.15873 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f184c859740 1 -- 172.21.15.28:0/4184762902 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ac3cba370 con 0x561ac3d62670 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.763+0000 7f18437fe640 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561ac3d62670 0x561ac3d64a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.764+0000 7f1822ffd640 10 client.15873 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.764+0000 7f18437fe640 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561ac3d62670 0x561ac3d64a60 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=78ab4244846bf952 server_cookie=5c2c90b6f2f0df0b in_seq=0 out_seq=0 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.764+0000 7f1822ffd640 1 -- 172.21.15.28:0/4184762902 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ac3cba370 con 0x561ac3d62670 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.764+0000 7f1822ffd640 10 client.15873 handle_command_reply: tid=0 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.764+0000 7f184c859740 1 client.15873 shutdown 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 -- 172.21.15.28:0/4184762902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 msgr2=0x561ac3d4fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d4fd20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f1838001eb0 tx=0x7f18380109c0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 -- 172.21.15.28:0/4184762902 shutdown_connections 2024-05-09T08:26:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561ac3d62670 0x561ac3d64a60 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d52320 0x561ac3d4fd20 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47c50 0x561ac3d4f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 --2- 172.21.15.28:0/4184762902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47330 0x561ac3d4f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f1822ffd640 0 client.15873 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:26:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 -- 172.21.15.28:0/4184762902 >> 172.21.15.28:0/4184762902 conn(0x561ac3d43650 msgr2=0x561ac3d49450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 -- 172.21.15.28:0/4184762902 shutdown_connections 2024-05-09T08:26:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 1 -- 172.21.15.28:0/4184762902 wait complete. 2024-05-09T08:26:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.765+0000 7f184c859740 20 client.15873 trim_cache size 0 max 16384 2024-05-09T08:26:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.778+0000 7f184c859740 1 Processor -- start 2024-05-09T08:26:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.778+0000 7f184c859740 1 -- start start 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.778+0000 7f184c859740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47330 0x561ac3d5e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f184c859740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d4e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f184c859740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d52320 0x561ac3d49a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f184c859740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18441b1020 con 0x561ac3d47c50 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f184c859740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f18441b2b60 con 0x561ac3d47330 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f184c859740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f18441b16f0 con 0x561ac3d52320 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d4e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d4e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51724/0 (socket says 172.21.15.28:51724) 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18437fe640 1 -- 172.21.15.28:0/3732860048 learned_addr learned my addr 172.21.15.28:0/3732860048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18437fe640 1 -- 172.21.15.28:0/3732860048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d52320 msgr2=0x561ac3d49a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18437fe640 1 --2- 172.21.15.28:0/3732860048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d52320 0x561ac3d49a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18437fe640 1 -- 172.21.15.28:0/3732860048 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47330 msgr2=0x561ac3d5e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18437fe640 1 --2- 172.21.15.28:0/3732860048 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47330 0x561ac3d5e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18437fe640 1 -- 172.21.15.28:0/3732860048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ac3d47a80 con 0x561ac3d47c50 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18437fe640 1 --2- 172.21.15.28:0/3732860048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d4e420 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f183402a940 tx=0x7f18340058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ae3dc8e3fc7070 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18297fa640 1 -- 172.21.15.28:0/3732860048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834049070 con 0x561ac3d47c50 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18297fa640 1 -- 172.21.15.28:0/3732860048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1834045dc0 con 0x561ac3d47c50 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.779+0000 7f18297fa640 1 -- 172.21.15.28:0/3732860048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834035040 con 0x561ac3d47c50 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 -- 172.21.15.28:0/3732860048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 msgr2=0x561ac3d4e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 --2- 172.21.15.28:0/3732860048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d4e420 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f183402a940 tx=0x7f18340058d0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 -- 172.21.15.28:0/3732860048 shutdown_connections 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 --2- 172.21.15.28:0/3732860048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d52320 0x561ac3d49a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 --2- 172.21.15.28:0/3732860048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d4e420 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 --2- 172.21.15.28:0/3732860048 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d47330 0x561ac3d5e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 -- 172.21.15.28:0/3732860048 >> 172.21.15.28:0/3732860048 conn(0x561ac3d43650 msgr2=0x561ac3d3e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 -- 172.21.15.28:0/3732860048 shutdown_connections 2024-05-09T08:26:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 -- 172.21.15.28:0/3732860048 wait complete. 2024-05-09T08:26:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 -- start start 2024-05-09T08:26:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.780+0000 7f184c859740 1 Processor -- start 2024-05-09T08:26:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 0x561ac3d5a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d5aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d52320 0x561ac3d4baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18441b2490 con 0x561ac3d47c50 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f18441b1dc0 con 0x561ac3d52320 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f18441af000 con 0x561ac3d47330 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 10 client.0 _ll_register_callbacks cb 0x7f18380123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f18437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d5aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f18437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d5aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51738/0 (socket says 172.21.15.28:51738) 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f18437fe640 1 -- 172.21.15.28:0/452567863 learned_addr learned my addr 172.21.15.28:0/452567863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f1842ffd640 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 0x561ac3d5a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f18437fe640 1 -- 172.21.15.28:0/452567863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 msgr2=0x561ac3d5a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f18437fe640 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 0x561ac3d5a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f18437fe640 1 -- 172.21.15.28:0/452567863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d52320 msgr2=0x561ac3d4baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f18437fe640 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d52320 0x561ac3d4baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f18437fe640 1 -- 172.21.15.28:0/452567863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ac3d4c2f0 con 0x561ac3d47c50 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f18437fe640 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d5aa50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f18340096c0 tx=0x7f1834031aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0339851d605fbe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f1822ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f1822ffd640 1 -- 172.21.15.28:0/452567863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834052020 con 0x561ac3d47c50 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f1822ffd640 1 -- 172.21.15.28:0/452567863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f18340457a0 con 0x561ac3d47c50 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 1 -- 172.21.15.28:0/452567863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1834045dc0 con 0x561ac3d47c50 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f1822ffd640 1 -- 172.21.15.28:0/452567863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f183405d050 con 0x561ac3d47c50 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 10 client.15879 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.781+0000 7f184c859740 1 -- 172.21.15.28:0/452567863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ac3dc8600 con 0x561ac3d47c50 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f1822ffd640 1 -- 172.21.15.28:0/452567863 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1834045dc0 con 0x561ac3d47c50 2024-05-09T08:26:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f1822ffd640 1 -- 172.21.15.28:0/452567863 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1834004770 con 0x561ac3d47c50 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f184c859740 10 client.15879 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f184c859740 10 client.15879 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f184c859740 20 client.15879 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f184c859740 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561ac3dc8d70 0x561ac3dcb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f184c859740 4 client.15879 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f184c859740 1 -- 172.21.15.28:0/452567863 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ac3cb46c0 con 0x561ac3dc8d70 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f1842ffd640 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561ac3dc8d70 0x561ac3dcb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f1822ffd640 10 client.15879 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.782+0000 7f1842ffd640 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561ac3dc8d70 0x561ac3dcb190 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bb4fc885f8af560f server_cookie=b29a37040abdb78c in_seq=0 out_seq=0 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1822ffd640 1 -- 172.21.15.28:0/452567863 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ac3cb46c0 con 0x561ac3dc8d70 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1822ffd640 10 client.15879 handle_command_reply: tid=0 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 client.15879 shutdown 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 -- 172.21.15.28:0/452567863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 msgr2=0x561ac3d5aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d5aa50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f18340096c0 tx=0x7f1834031aa0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 -- 172.21.15.28:0/452567863 shutdown_connections 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561ac3dc8d70 0x561ac3dcb190 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ac3d52320 0x561ac3d4baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ac3d47c50 0x561ac3d5aa50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1822ffd640 0 client.15879 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 --2- 172.21.15.28:0/452567863 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ac3d47330 0x561ac3d5a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 -- 172.21.15.28:0/452567863 >> 172.21.15.28:0/452567863 conn(0x561ac3d43650 msgr2=0x561ac3d4e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 -- 172.21.15.28:0/452567863 shutdown_connections 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 1 -- 172.21.15.28:0/452567863 wait complete. 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f184c859740 20 client.15879 trim_cache size 0 max 16384 2024-05-09T08:26:54.794 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:26:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1849b4f640 1 -- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f182c0a2d90 msgr2=0x7f182c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1849b4f640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f182c0a2d90 0x7f182c0a5250 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f183c0ccf10 tx=0x7f183c000f40 comp rx=0 tx=0).stop 2024-05-09T08:26:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1849b4f640 1 -- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 msgr2=0x7f184419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1849b4f640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 0x7f184419ebc0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f183c00cad0 tx=0x7f183c00cf90 comp rx=0 tx=0).stop 2024-05-09T08:26:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1849b4f640 1 -- 172.21.15.28:0/152082524 shutdown_connections 2024-05-09T08:26:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1849b4f640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1844106710 0x7f184419ebc0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1849b4f640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f182c0a2d90 0x7f182c0a5250 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1849b4f640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18441010c0 0x7f184419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.783+0000 7f1849b4f640 1 --2- 172.21.15.28:0/152082524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1844100700 0x7f184419a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.784+0000 7f1849b4f640 1 -- 172.21.15.28:0/152082524 >> 172.21.15.28:0/152082524 conn(0x7f1844077830 msgr2=0x7f184410beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.784+0000 7f1849b4f640 1 -- 172.21.15.28:0/152082524 shutdown_connections 2024-05-09T08:26:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.784+0000 7f1849b4f640 1 -- 172.21.15.28:0/152082524 wait complete. 2024-05-09T08:26:54.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:26:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 -- 172.21.15.28:0/935112138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 msgr2=0x7efd78107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 --2- 172.21.15.28:0/935112138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 0x7efd78107bf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7efd68009930 tx=0x7efd6802a420 comp rx=0 tx=0).stop 2024-05-09T08:26:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 -- 172.21.15.28:0/935112138 shutdown_connections 2024-05-09T08:26:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 --2- 172.21.15.28:0/935112138 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd781021c0 0x7efd7810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 --2- 172.21.15.28:0/935112138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd78101800 0x7efd78101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 --2- 172.21.15.28:0/935112138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 0x7efd78107bf0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 -- 172.21.15.28:0/935112138 >> 172.21.15.28:0/935112138 conn(0x7efd780fd270 msgr2=0x7efd780ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 -- 172.21.15.28:0/935112138 shutdown_connections 2024-05-09T08:26:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 -- 172.21.15.28:0/935112138 wait complete. 2024-05-09T08:26:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 Processor -- start 2024-05-09T08:26:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.905+0000 7efd7e963640 1 -- start start 2024-05-09T08:26:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7e963640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd78101800 0x7efd78198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7e963640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd781021c0 0x7efd78198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7e963640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 0x7efd7819c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7e963640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd78111c70 con 0x7efd78107810 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7e963640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd78111af0 con 0x7efd781021c0 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7e963640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd78111df0 con 0x7efd78101800 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7cf00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 0x7efd7819c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7cf00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 0x7efd7819c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51768/0 (socket says 172.21.15.28:51768) 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7cf00640 1 -- 172.21.15.28:0/2027328388 learned_addr learned my addr 172.21.15.28:0/2027328388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd777fe640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd781021c0 0x7efd78198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd77fff640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd78101800 0x7efd78198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7cf00640 1 -- 172.21.15.28:0/2027328388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd78101800 msgr2=0x7efd78198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7cf00640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd78101800 0x7efd78198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7cf00640 1 -- 172.21.15.28:0/2027328388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd781021c0 msgr2=0x7efd78198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7cf00640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd781021c0 0x7efd78198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7cf00640 1 -- 172.21.15.28:0/2027328388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7819d100 con 0x7efd78107810 2024-05-09T08:26:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.906+0000 7efd7cf00640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 0x7efd7819c9d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7efd7000eaa0 tx=0x7efd7000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.907+0000 7efd757fa640 1 -- 172.21.15.28:0/2027328388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7000cdb0 con 0x7efd78107810 2024-05-09T08:26:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.907+0000 7efd757fa640 1 -- 172.21.15.28:0/2027328388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd70005020 con 0x7efd78107810 2024-05-09T08:26:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.907+0000 7efd7e963640 1 -- 172.21.15.28:0/2027328388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd7819d3f0 con 0x7efd78107810 2024-05-09T08:26:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.907+0000 7efd757fa640 1 -- 172.21.15.28:0/2027328388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd700116b0 con 0x7efd78107810 2024-05-09T08:26:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.907+0000 7efd7e963640 1 -- 172.21.15.28:0/2027328388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd781a4c80 con 0x7efd78107810 2024-05-09T08:26:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.909+0000 7efd757fa640 1 -- 172.21.15.28:0/2027328388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7efd70005560 con 0x7efd78107810 2024-05-09T08:26:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.909+0000 7efd81679740 1 Processor -- start 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.909+0000 7efd81679740 1 -- start start 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.909+0000 7efd81679740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 0x561ce9f53440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.909+0000 7efd81679740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f621f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.909+0000 7efd81679740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f64b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.909+0000 7efd81679740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ce9eb8c40 con 0x561ce9f53a10 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.909+0000 7efd81679740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x561ce9e60580 con 0x561ce9f62730 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.909+0000 7efd81679740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561ce9eb4c00 con 0x561ce9f53060 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f621f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f621f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51776/0 (socket says 172.21.15.28:51776) 2024-05-09T08:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd77fff640 1 -- 172.21.15.28:0/15939757 learned_addr learned my addr 172.21.15.28:0/15939757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd777fe640 1 --2- 172.21.15.28:0/15939757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f64b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd777fe640 1 --2- 172.21.15.28:0/15939757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 0x561ce9f53440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd77fff640 1 -- 172.21.15.28:0/15939757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 msgr2=0x561ce9f53440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd77fff640 1 --2- 172.21.15.28:0/15939757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 0x561ce9f53440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd77fff640 1 -- 172.21.15.28:0/15939757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 msgr2=0x561ce9f64b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd77fff640 1 --2- 172.21.15.28:0/15939757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f64b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd77fff640 1 -- 172.21.15.28:0/15939757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ce9bcc3f0 con 0x561ce9f53a10 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd757fa640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd600a2d90 0x7efd600a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd777fe640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd600a2d90 0x7efd600a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd77fff640 1 --2- 172.21.15.28:0/15939757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f621f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efd68009930 tx=0x7efd680040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab4a54698cb11fe3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd757fa640 1 -- 172.21.15.28:0/2027328388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7efd700cfc00 con 0x7efd78107810 2024-05-09T08:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd5e7fc640 1 -- 172.21.15.28:0/15939757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68035030 con 0x561ce9f53a10 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd5e7fc640 1 -- 172.21.15.28:0/15939757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd68005520 con 0x561ce9f53a10 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.910+0000 7efd5e7fc640 1 -- 172.21.15.28:0/15939757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68003850 con 0x561ce9f53a10 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 -- 172.21.15.28:0/15939757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 msgr2=0x561ce9f621f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 --2- 172.21.15.28:0/15939757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f621f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efd68009930 tx=0x7efd680040f0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd777fe640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd600a2d90 0x7efd600a5250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x561ce9f507e0 tx=0x7efd6c00a640 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 -- 172.21.15.28:0/15939757 shutdown_connections 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 --2- 172.21.15.28:0/15939757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f64b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 --2- 172.21.15.28:0/15939757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f621f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 --2- 172.21.15.28:0/15939757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 0x561ce9f53440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 -- 172.21.15.28:0/15939757 >> 172.21.15.28:0/15939757 conn(0x561ce9f4f650 msgr2=0x561ce9f4fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 -- 172.21.15.28:0/15939757 shutdown_connections 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 -- 172.21.15.28:0/15939757 wait complete. 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.911+0000 7efd81679740 1 -- start start 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd81679740 1 Processor -- start 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd81679740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 0x561ce9f5f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd81679740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f5fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd81679740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd81679740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ce9f65490 con 0x561ce9f53a10 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd81679740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x561ce9e73c40 con 0x561ce9f62730 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd81679740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561ce9f65610 con 0x561ce9f53060 2024-05-09T08:26:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd81679740 10 client.0 _ll_register_callbacks cb 0x561ce9bcbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f5fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f5fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51778/0 (socket says 172.21.15.28:51778) 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd777fe640 1 -- 172.21.15.28:0/1530049664 learned_addr learned my addr 172.21.15.28:0/1530049664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd77fff640 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 0x561ce9f5f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd7cf00640 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd777fe640 1 -- 172.21.15.28:0/1530049664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 msgr2=0x561ce9f5f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd777fe640 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 0x561ce9f5f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd777fe640 1 -- 172.21.15.28:0/1530049664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 msgr2=0x561ce9f600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd777fe640 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.912+0000 7efd777fe640 1 -- 172.21.15.28:0/1530049664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ce9c967d0 con 0x561ce9f53a10 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd777fe640 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f5fbb0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efd6c01e8b0 tx=0x7efd6c00a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84878a4bab2cd73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd577fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd577fe640 1 -- 172.21.15.28:0/1530049664 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c004020 con 0x561ce9f53a10 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd577fe640 1 -- 172.21.15.28:0/1530049664 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7efd6c0041c0 con 0x561ce9f53a10 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd81679740 1 -- 172.21.15.28:0/1530049664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd68005520 con 0x561ce9f53a10 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd577fe640 1 -- 172.21.15.28:0/1530049664 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efd6c0043c0 con 0x561ce9f53a10 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd81679740 10 client.15891 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd81679740 1 -- 172.21.15.28:0/1530049664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ce9f71720 con 0x561ce9f53a10 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd577fe640 1 -- 172.21.15.28:0/1530049664 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd68005520 con 0x561ce9f53a10 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd577fe640 1 -- 172.21.15.28:0/1530049664 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7efd6c0039f0 con 0x561ce9f53a10 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd81679740 10 client.15891 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd81679740 10 client.15891 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd81679740 20 client.15891 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd81679740 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x561ce9f72cd0 0x561ce9f750f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd81679740 4 client.15891 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.913+0000 7efd81679740 1 -- 172.21.15.28:0/1530049664 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ce9ec6370 con 0x561ce9f72cd0 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.914+0000 7efd77fff640 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x561ce9f72cd0 0x561ce9f750f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.914+0000 7efd577fe640 10 client.15891 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.914+0000 7efd77fff640 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x561ce9f72cd0 0x561ce9f750f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=13f38f38e6b1e798 server_cookie=592bfe9349fe7334 in_seq=0 out_seq=0 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd577fe640 1 -- 172.21.15.28:0/1530049664 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ce9ec6370 con 0x561ce9f72cd0 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd577fe640 10 client.15891 handle_command_reply: tid=0 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 client.15891 shutdown 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 -- 172.21.15.28:0/1530049664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 msgr2=0x561ce9f5fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f5fbb0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efd6c01e8b0 tx=0x7efd6c00a2d0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 -- 172.21.15.28:0/1530049664 shutdown_connections 2024-05-09T08:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x561ce9f72cd0 0x561ce9f750f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53a10 0x561ce9f5fbb0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd577fe640 0 client.15891 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:26:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 --2- 172.21.15.28:0/1530049664 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53060 0x561ce9f5f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 -- 172.21.15.28:0/1530049664 >> 172.21.15.28:0/1530049664 conn(0x561ce9f4f650 msgr2=0x561ce9f554a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.915+0000 7efd81679740 1 -- 172.21.15.28:0/1530049664 shutdown_connections 2024-05-09T08:26:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.916+0000 7efd81679740 1 -- 172.21.15.28:0/1530049664 wait complete. 2024-05-09T08:26:54.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.916+0000 7efd81679740 20 client.15891 trim_cache size 0 max 16384 2024-05-09T08:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd81679740 1 Processor -- start 2024-05-09T08:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd81679740 1 -- start start 2024-05-09T08:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd81679740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd81679740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53a10 0x561ce9f55690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd81679740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f5e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd81679740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd781acc60 con 0x561ce9f53060 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd81679740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd781ae7a0 con 0x561ce9f62730 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd81679740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd781ad330 con 0x561ce9f53a10 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51792/0 (socket says 172.21.15.28:51792) 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd77fff640 1 -- 172.21.15.28:0/1685923082 learned_addr learned my addr 172.21.15.28:0/1685923082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd777fe640 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53a10 0x561ce9f55690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd77fff640 1 -- 172.21.15.28:0/1685923082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53a10 msgr2=0x561ce9f55690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd7cf00640 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f5e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd77fff640 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53a10 0x561ce9f55690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd77fff640 1 -- 172.21.15.28:0/1685923082 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 msgr2=0x561ce9f5e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd77fff640 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f5e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd77fff640 1 -- 172.21.15.28:0/1685923082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ce9f537b0 con 0x561ce9f53060 2024-05-09T08:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd777fe640 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53a10 0x561ce9f55690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.929+0000 7efd7cf00640 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f5e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.930+0000 7efd77fff640 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6e3c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x561ce9f55c60 tx=0x7efd68004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58abf811a1a84cc2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.930+0000 7efd5dffb640 1 -- 172.21.15.28:0/1685923082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68055070 con 0x561ce9f53060 2024-05-09T08:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.930+0000 7efd5dffb640 1 -- 172.21.15.28:0/1685923082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd68049040 con 0x561ce9f53060 2024-05-09T08:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.930+0000 7efd5dffb640 1 -- 172.21.15.28:0/1685923082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68048050 con 0x561ce9f53060 2024-05-09T08:26:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 -- 172.21.15.28:0/1685923082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 msgr2=0x561ce9f6e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6e3c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x561ce9f55c60 tx=0x7efd68004c60 comp rx=0 tx=0).stop 2024-05-09T08:26:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 -- 172.21.15.28:0/1685923082 shutdown_connections 2024-05-09T08:26:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f62730 0x561ce9f5e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f53a10 0x561ce9f55690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 --2- 172.21.15.28:0/1685923082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6e3c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 -- 172.21.15.28:0/1685923082 >> 172.21.15.28:0/1685923082 conn(0x561ce9f4f650 msgr2=0x561ce9f4df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 -- 172.21.15.28:0/1685923082 shutdown_connections 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 -- 172.21.15.28:0/1685923082 wait complete. 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.931+0000 7efd81679740 1 -- start start 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd81679740 1 Processor -- start 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd81679740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd81679740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f53a10 0x561ce9f65e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd81679740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f62730 0x561ce9f663d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd81679740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd781ae0d0 con 0x561ce9f53060 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd81679740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd781ada00 con 0x561ce9f53a10 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd81679740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd781aac40 con 0x561ce9f62730 2024-05-09T08:26:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd81679740 10 client.0 _ll_register_callbacks cb 0x7efd6c01e560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51808/0 (socket says 172.21.15.28:51808) 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd77fff640 1 -- 172.21.15.28:0/604336629 learned_addr learned my addr 172.21.15.28:0/604336629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd7cf00640 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f62730 0x561ce9f663d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd777fe640 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f53a10 0x561ce9f65e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd77fff640 1 -- 172.21.15.28:0/604336629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f62730 msgr2=0x561ce9f663d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd77fff640 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f62730 0x561ce9f663d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd77fff640 1 -- 172.21.15.28:0/604336629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f53a10 msgr2=0x561ce9f65e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd77fff640 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f53a10 0x561ce9f65e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd77fff640 1 -- 172.21.15.28:0/604336629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ce9f66940 con 0x561ce9f53060 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd7cf00640 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f62730 0x561ce9f663d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd777fe640 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f53a10 0x561ce9f65e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd77fff640 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6aa20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7efd68053040 tx=0x7efd68000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7d357ce493ce5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.932+0000 7efd577fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd577fe640 1 -- 172.21.15.28:0/604336629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6805e060 con 0x561ce9f53060 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd81679740 1 -- 172.21.15.28:0/604336629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd68049040 con 0x561ce9f53060 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd577fe640 1 -- 172.21.15.28:0/604336629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7efd680042f0 con 0x561ce9f53060 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd577fe640 1 -- 172.21.15.28:0/604336629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efd68065050 con 0x561ce9f53060 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd81679740 10 client.15897 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd81679740 1 -- 172.21.15.28:0/604336629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ce9fd8f20 con 0x561ce9f53060 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd577fe640 1 -- 172.21.15.28:0/604336629 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd68049040 con 0x561ce9f53060 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd577fe640 1 -- 172.21.15.28:0/604336629 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7efd68058070 con 0x561ce9f53060 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd81679740 10 client.15897 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd81679740 10 client.15897 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd81679740 20 client.15897 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd81679740 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x561ce9fd91a0 0x561ce9fdb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd81679740 4 client.15897 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd81679740 1 -- 172.21.15.28:0/604336629 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ce9ec06c0 con 0x561ce9fd91a0 2024-05-09T08:26:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.933+0000 7efd777fe640 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x561ce9fd91a0 0x561ce9fdb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd777fe640 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x561ce9fd91a0 0x561ce9fdb590 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f59852f1f089283 server_cookie=b596b2f51b25e146 in_seq=0 out_seq=0 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd577fe640 10 client.15897 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd577fe640 1 -- 172.21.15.28:0/604336629 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ce9ec06c0 con 0x561ce9fd91a0 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd577fe640 10 client.15897 handle_command_reply: tid=0 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 client.15897 shutdown 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 -- 172.21.15.28:0/604336629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 msgr2=0x561ce9f6aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6aa20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7efd68053040 tx=0x7efd68000bc0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 -- 172.21.15.28:0/604336629 shutdown_connections 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x561ce9fd91a0 0x561ce9fdb590 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561ce9f62730 0x561ce9f663d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561ce9f53a10 0x561ce9f65e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd577fe640 0 client.15897 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 --2- 172.21.15.28:0/604336629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ce9f53060 0x561ce9f6aa20 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 -- 172.21.15.28:0/604336629 >> 172.21.15.28:0/604336629 conn(0x561ce9f4f650 msgr2=0x561ce9f4e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 -- 172.21.15.28:0/604336629 shutdown_connections 2024-05-09T08:26:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 1 -- 172.21.15.28:0/604336629 wait complete. 2024-05-09T08:26:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.934+0000 7efd81679740 20 client.15897 trim_cache size 0 max 16384 2024-05-09T08:26:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 -- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd600a2d90 msgr2=0x7efd600a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd600a2d90 0x7efd600a5250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x561ce9f507e0 tx=0x7efd6c00a640 comp rx=0 tx=0).stop 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 -- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 msgr2=0x7efd7819c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 0x7efd7819c9d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7efd7000eaa0 tx=0x7efd7000ef60 comp rx=0 tx=0).stop 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 -- 172.21.15.28:0/2027328388 shutdown_connections 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd78107810 0x7efd7819c9d0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd781021c0 0x7efd78198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd600a2d90 0x7efd600a5250 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 --2- 172.21.15.28:0/2027328388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd78101800 0x7efd78198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 -- 172.21.15.28:0/2027328388 >> 172.21.15.28:0/2027328388 conn(0x7efd780fd270 msgr2=0x7efd780fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 -- 172.21.15.28:0/2027328388 shutdown_connections 2024-05-09T08:26:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:54.935+0000 7efd7e963640 1 -- 172.21.15.28:0/2027328388 wait complete. 2024-05-09T08:26:54.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:26:55.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.061+0000 7f3b89e22640 1 -- 172.21.15.28:0/2149745374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8406a7f0 msgr2=0x7f3b8410e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.061+0000 7f3b89e22640 1 --2- 172.21.15.28:0/2149745374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8406a7f0 0x7f3b8410e710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009ca0 tx=0x7f3b7c02a950 comp rx=0 tx=0).stop 2024-05-09T08:26:55.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.061+0000 7f3b89e22640 1 -- 172.21.15.28:0/2149745374 shutdown_connections 2024-05-09T08:26:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.061+0000 7f3b89e22640 1 --2- 172.21.15.28:0/2149745374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8406a7f0 0x7f3b8410e710 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.061+0000 7f3b89e22640 1 --2- 172.21.15.28:0/2149745374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b84069e50 0x7f3b8406a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.061+0000 7f3b89e22640 1 --2- 172.21.15.28:0/2149745374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b84107930 0x7f3b84107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.061+0000 7f3b89e22640 1 -- 172.21.15.28:0/2149745374 >> 172.21.15.28:0/2149745374 conn(0x7f3b84077710 msgr2=0x7f3b84077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.061+0000 7f3b89e22640 1 -- 172.21.15.28:0/2149745374 shutdown_connections 2024-05-09T08:26:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.061+0000 7f3b89e22640 1 -- 172.21.15.28:0/2149745374 wait complete. 2024-05-09T08:26:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b89e22640 1 Processor -- start 2024-05-09T08:26:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b89e22640 1 -- start start 2024-05-09T08:26:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b89e22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b84069e50 0x7f3b841980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b89e22640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b8406a7f0 0x7f3b841985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b89e22640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b84107930 0x7f3b8419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b89e22640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b84111be0 con 0x7f3b84069e50 2024-05-09T08:26:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b89e22640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3b84111a60 con 0x7f3b84107930 2024-05-09T08:26:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b89e22640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b84111d60 con 0x7f3b8406a7f0 2024-05-09T08:26:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b84069e50 0x7f3b841980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b84069e50 0x7f3b841980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51826/0 (socket says 172.21.15.28:51826) 2024-05-09T08:26:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.062+0000 7f3b837fe640 1 -- 172.21.15.28:0/3632975924 learned_addr learned my addr 172.21.15.28:0/3632975924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b83fff640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b84107930 0x7f3b8419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b8406a7f0 0x7f3b841985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b837fe640 1 -- 172.21.15.28:0/3632975924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b8406a7f0 msgr2=0x7f3b841985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b837fe640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b8406a7f0 0x7f3b841985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b837fe640 1 -- 172.21.15.28:0/3632975924 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b84107930 msgr2=0x7f3b8419c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b837fe640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b84107930 0x7f3b8419c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b837fe640 1 -- 172.21.15.28:0/3632975924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8419d090 con 0x7f3b84069e50 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b8406a7f0 0x7f3b841985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b837fe640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b84069e50 0x7f3b841980a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400d950 tx=0x7f3b7400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b80ff9640 1 -- 172.21.15.28:0/3632975924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74017070 con 0x7f3b84069e50 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b80ff9640 1 -- 172.21.15.28:0/3632975924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b74009c30 con 0x7f3b84069e50 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b89e22640 1 -- 172.21.15.28:0/3632975924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8419d380 con 0x7f3b84069e50 2024-05-09T08:26:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b80ff9640 1 -- 172.21.15.28:0/3632975924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7400f6b0 con 0x7f3b84069e50 2024-05-09T08:26:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.063+0000 7f3b89e22640 1 -- 172.21.15.28:0/3632975924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b841a4c10 con 0x7f3b84069e50 2024-05-09T08:26:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.065+0000 7f3b80ff9640 1 -- 172.21.15.28:0/3632975924 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f3b740109d0 con 0x7f3b84069e50 2024-05-09T08:26:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b8cc05740 1 Processor -- start 2024-05-09T08:26:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b8cc05740 1 -- start start 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8b330 0x55562ea8b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8bc50 0x55562ea95de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea98710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55562e9f0c40 con 0x55562ea96320 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55562e998580 con 0x55562ea8bc50 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55562e9ecc00 con 0x55562ea8b330 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea98710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea98710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51834/0 (socket says 172.21.15.28:51834) 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b82ffd640 1 -- 172.21.15.28:0/2031430203 learned_addr learned my addr 172.21.15.28:0/2031430203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b80ff9640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3b6c0a2d90 0x7f3b6c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b83fff640 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8b330 0x55562ea8b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.066+0000 7f3b837fe640 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8bc50 0x55562ea95de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b82ffd640 1 -- 172.21.15.28:0/2031430203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8b330 msgr2=0x55562ea8b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b80ff9640 1 -- 172.21.15.28:0/3632975924 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3b740cf670 con 0x7f3b84069e50 2024-05-09T08:26:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8b330 0x55562ea8b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b83fff640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3b6c0a2d90 0x7f3b6c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b82ffd640 1 -- 172.21.15.28:0/2031430203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8bc50 msgr2=0x55562ea95de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8bc50 0x55562ea95de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b82ffd640 1 -- 172.21.15.28:0/2031430203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55562e7043f0 con 0x55562ea96320 2024-05-09T08:26:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b837fe640 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8bc50 0x55562ea95de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea98710 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800bb40 tx=0x7f3b7800d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f77df49d0192be server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b69ffb640 1 -- 172.21.15.28:0/2031430203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b78017420 con 0x55562ea96320 2024-05-09T08:26:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b83fff640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3b6c0a2d90 0x7f3b6c0a5250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c006010 tx=0x7f3b7c0095c0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b69ffb640 1 -- 172.21.15.28:0/2031430203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b78017aa0 con 0x55562ea96320 2024-05-09T08:26:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.067+0000 7f3b69ffb640 1 -- 172.21.15.28:0/2031430203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7800fb90 con 0x55562ea96320 2024-05-09T08:26:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 -- 172.21.15.28:0/2031430203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 msgr2=0x55562ea98710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea98710 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800bb40 tx=0x7f3b7800d340 comp rx=0 tx=0).stop 2024-05-09T08:26:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 -- 172.21.15.28:0/2031430203 shutdown_connections 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea98710 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8bc50 0x55562ea95de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/2031430203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8b330 0x55562ea8b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 -- 172.21.15.28:0/2031430203 >> 172.21.15.28:0/2031430203 conn(0x55562ea87650 msgr2=0x55562ea87a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 -- 172.21.15.28:0/2031430203 shutdown_connections 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 -- 172.21.15.28:0/2031430203 wait complete. 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 -- start start 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.068+0000 7f3b8cc05740 1 Processor -- start 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562ea93220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea93760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 0x55562ea93ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55562ea99110 con 0x55562ea8bc50 2024-05-09T08:26:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55562e9abc40 con 0x55562ea8b330 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55562ea99290 con 0x55562ea96320 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b8cc05740 10 client.0 _ll_register_callbacks cb 0x55562e703be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea93760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea93760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51838/0 (socket says 172.21.15.28:51838) 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b82ffd640 1 -- 172.21.15.28:0/832176773 learned_addr learned my addr 172.21.15.28:0/832176773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b837fe640 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562ea93220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b83fff640 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 0x55562ea93ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b82ffd640 1 -- 172.21.15.28:0/832176773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 msgr2=0x55562ea93ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 0x55562ea93ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b82ffd640 1 -- 172.21.15.28:0/832176773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 msgr2=0x55562ea93220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562ea93220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b82ffd640 1 -- 172.21.15.28:0/832176773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55562e7ce7d0 con 0x55562ea8bc50 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea93760 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3b78019040 tx=0x7f3b78017650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4affe36ea2c44e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b62ffd640 1 -- 172.21.15.28:0/832176773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b78026070 con 0x55562ea8bc50 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b62ffd640 1 -- 172.21.15.28:0/832176773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3b7800f910 con 0x55562ea8bc50 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.069+0000 7f3b8cc05740 1 -- 172.21.15.28:0/832176773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b78017aa0 con 0x55562ea8bc50 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b62ffd640 1 -- 172.21.15.28:0/832176773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b78032050 con 0x55562ea8bc50 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b8cc05740 10 client.15909 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b8cc05740 1 -- 172.21.15.28:0/832176773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55562eaa51f0 con 0x55562ea8bc50 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b62ffd640 1 -- 172.21.15.28:0/832176773 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b78017aa0 con 0x55562ea8bc50 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b62ffd640 1 -- 172.21.15.28:0/832176773 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3b7802b920 con 0x55562ea8bc50 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b8cc05740 10 client.15909 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b8cc05740 10 client.15909 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b8cc05740 20 client.15909 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x55562eaa6600 0x55562eaa89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b8cc05740 4 client.15909 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b8cc05740 1 -- 172.21.15.28:0/832176773 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55562e9fe370 con 0x55562eaa6600 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.070+0000 7f3b837fe640 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x55562eaa6600 0x55562eaa89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.071+0000 7f3b62ffd640 10 client.15909 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.071+0000 7f3b837fe640 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x55562eaa6600 0x55562eaa89f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=15b5ba8fc4fff58c server_cookie=ef35ca3dac10465 in_seq=0 out_seq=0 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b62ffd640 1 -- 172.21.15.28:0/832176773 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55562e9fe370 con 0x55562eaa6600 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b62ffd640 10 client.15909 handle_command_reply: tid=0 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 client.15909 shutdown 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 -- 172.21.15.28:0/832176773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 msgr2=0x55562ea93760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea93760 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3b78019040 tx=0x7f3b78017650 comp rx=0 tx=0).stop 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 -- 172.21.15.28:0/832176773 shutdown_connections 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x55562eaa6600 0x55562eaa89f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 0x55562ea93ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea93760 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b62ffd640 0 client.15909 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:26:55.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/832176773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562ea93220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 -- 172.21.15.28:0/832176773 >> 172.21.15.28:0/832176773 conn(0x55562ea87650 msgr2=0x55562ea8d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 -- 172.21.15.28:0/832176773 shutdown_connections 2024-05-09T08:26:55.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 1 -- 172.21.15.28:0/832176773 wait complete. 2024-05-09T08:26:55.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.072+0000 7f3b8cc05740 20 client.15909 trim_cache size 0 max 16384 2024-05-09T08:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b8cc05740 1 Processor -- start 2024-05-09T08:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b8cc05740 1 -- start start 2024-05-09T08:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562eaa1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea923c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 0x55562eaae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841aed70 con 0x55562ea8bc50 2024-05-09T08:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841b08b0 con 0x55562ea8b330 2024-05-09T08:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841af440 con 0x55562ea96320 2024-05-09T08:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea923c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea923c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51840/0 (socket says 172.21.15.28:51840) 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b82ffd640 1 -- 172.21.15.28:0/278818132 learned_addr learned my addr 172.21.15.28:0/278818132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b83fff640 1 --2- 172.21.15.28:0/278818132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 0x55562eaae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b82ffd640 1 -- 172.21.15.28:0/278818132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 msgr2=0x55562eaae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/278818132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 0x55562eaae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b82ffd640 1 -- 172.21.15.28:0/278818132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 msgr2=0x55562eaa1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/278818132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562eaa1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b82ffd640 1 -- 172.21.15.28:0/278818132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55562ea98ff0 con 0x55562ea8bc50 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.086+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/278818132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea923c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800ba60 tx=0x7f3b780176c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b243519ffa7ee3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b697fa640 1 -- 172.21.15.28:0/278818132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b78026040 con 0x55562ea8bc50 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b697fa640 1 -- 172.21.15.28:0/278818132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b7800b740 con 0x55562ea8bc50 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b697fa640 1 -- 172.21.15.28:0/278818132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7800f6f0 con 0x55562ea8bc50 2024-05-09T08:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b8cc05740 1 -- 172.21.15.28:0/278818132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 msgr2=0x55562ea923c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/278818132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea923c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800ba60 tx=0x7f3b780176c0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b8cc05740 1 -- 172.21.15.28:0/278818132 shutdown_connections 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/278818132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea96320 0x55562eaae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/278818132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea8bc50 0x55562ea923c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/278818132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562eaa1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b8cc05740 1 -- 172.21.15.28:0/278818132 >> 172.21.15.28:0/278818132 conn(0x55562ea87650 msgr2=0x55562ea87a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b8cc05740 1 -- 172.21.15.28:0/278818132 shutdown_connections 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.087+0000 7f3b8cc05740 1 -- 172.21.15.28:0/278818132 wait complete. 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b8cc05740 1 -- start start 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b8cc05740 1 Processor -- start 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562ea8ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8bc50 0x55562ea90440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b8cc05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea90980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841b01e0 con 0x55562ea96320 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841afb10 con 0x55562ea8b330 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b8cc05740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841acd50 con 0x55562ea8bc50 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b8cc05740 10 client.0 _ll_register_callbacks cb 0x7f3b7800b990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b83fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea90980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b83fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea90980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51856/0 (socket says 172.21.15.28:51856) 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b83fff640 1 -- 172.21.15.28:0/4096591524 learned_addr learned my addr 172.21.15.28:0/4096591524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b82ffd640 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8bc50 0x55562ea90440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b837fe640 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562ea8ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b83fff640 1 -- 172.21.15.28:0/4096591524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8bc50 msgr2=0x55562ea90440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b83fff640 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8bc50 0x55562ea90440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.088+0000 7f3b83fff640 1 -- 172.21.15.28:0/4096591524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 msgr2=0x55562ea8ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b83fff640 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562ea8ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b83fff640 1 -- 172.21.15.28:0/4096591524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55562eb0fd30 con 0x55562ea96320 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b83fff640 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea90980 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009ca0 tx=0x7f3b7c02a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fabe35a7d085ffb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b62ffd640 1 -- 172.21.15.28:0/4096591524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c042070 con 0x55562ea96320 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b62ffd640 1 -- 172.21.15.28:0/4096591524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3b7c004450 con 0x55562ea96320 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b8cc05740 1 -- 172.21.15.28:0/4096591524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55562e9f86c0 con 0x55562ea96320 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b62ffd640 1 -- 172.21.15.28:0/4096591524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b7c034040 con 0x55562ea96320 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b8cc05740 10 client.15915 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b8cc05740 1 -- 172.21.15.28:0/4096591524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55562eb11000 con 0x55562ea96320 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b62ffd640 1 -- 172.21.15.28:0/4096591524 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55562e9f86c0 con 0x55562ea96320 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b62ffd640 1 -- 172.21.15.28:0/4096591524 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3b7c051070 con 0x55562ea96320 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b8cc05740 10 client.15915 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b8cc05740 10 client.15915 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b8cc05740 20 client.15915 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:55.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x55562eb117b0 0x55562eb13bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b8cc05740 4 client.15915 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.089+0000 7f3b8cc05740 1 -- 172.21.15.28:0/4096591524 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55562e975820 con 0x55562eb117b0 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.090+0000 7f3b837fe640 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x55562eb117b0 0x55562eb13bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.090+0000 7f3b62ffd640 10 client.15915 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.090+0000 7f3b837fe640 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x55562eb117b0 0x55562eb13bd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=68fcb850ab602cb4 server_cookie=e36b8703c32f8cec in_seq=0 out_seq=0 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.090+0000 7f3b62ffd640 1 -- 172.21.15.28:0/4096591524 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55562e975820 con 0x55562eb117b0 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.090+0000 7f3b62ffd640 10 client.15915 handle_command_reply: tid=0 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.090+0000 7f3b8cc05740 1 client.15915 shutdown 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 -- 172.21.15.28:0/4096591524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 msgr2=0x55562ea90980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea90980 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009ca0 tx=0x7f3b7c02a950 comp rx=0 tx=0).stop 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 -- 172.21.15.28:0/4096591524 shutdown_connections 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x55562eb117b0 0x55562eb13bd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55562ea96320 0x55562ea90980 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55562ea8bc50 0x55562ea90440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 --2- 172.21.15.28:0/4096591524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562ea8b330 0x55562ea8ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b62ffd640 0 client.15915 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:26:55.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 -- 172.21.15.28:0/4096591524 >> 172.21.15.28:0/4096591524 conn(0x55562ea87650 msgr2=0x55562ea85f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 -- 172.21.15.28:0/4096591524 shutdown_connections 2024-05-09T08:26:55.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 1 -- 172.21.15.28:0/4096591524 wait complete. 2024-05-09T08:26:55.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b8cc05740 20 client.15915 trim_cache size 0 max 16384 2024-05-09T08:26:55.098 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 -- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3b6c0a2d90 msgr2=0x7f3b6c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3b6c0a2d90 0x7f3b6c0a5250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c006010 tx=0x7f3b7c0095c0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 -- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b84069e50 msgr2=0x7f3b841980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b84069e50 0x7f3b841980a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400d950 tx=0x7f3b7400de10 comp rx=0 tx=0).stop 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 -- 172.21.15.28:0/3632975924 shutdown_connections 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b84107930 0x7f3b8419c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3b6c0a2d90 0x7f3b6c0a5250 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b8406a7f0 0x7f3b841985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 --2- 172.21.15.28:0/3632975924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b84069e50 0x7f3b841980a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.091+0000 7f3b89e22640 1 -- 172.21.15.28:0/3632975924 >> 172.21.15.28:0/3632975924 conn(0x7f3b84077710 msgr2=0x7f3b840ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.092+0000 7f3b89e22640 1 -- 172.21.15.28:0/3632975924 shutdown_connections 2024-05-09T08:26:55.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.092+0000 7f3b89e22640 1 -- 172.21.15.28:0/3632975924 wait complete. 2024-05-09T08:26:55.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:26:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.214+0000 7fa07702a640 1 -- 172.21.15.28:0/3923497576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070107820 msgr2=0x7fa070107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.214+0000 7fa07702a640 1 --2- 172.21.15.28:0/3923497576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070107820 0x7fa070107c00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa060009cd0 tx=0x7fa06002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.214+0000 7fa07702a640 1 -- 172.21.15.28:0/3923497576 shutdown_connections 2024-05-09T08:26:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.214+0000 7fa07702a640 1 --2- 172.21.15.28:0/3923497576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa07006a870 0x7fa07010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.214+0000 7fa07702a640 1 --2- 172.21.15.28:0/3923497576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa070069eb0 0x7fa07006a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.214+0000 7fa07702a640 1 --2- 172.21.15.28:0/3923497576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070107820 0x7fa070107c00 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.214+0000 7fa07702a640 1 -- 172.21.15.28:0/3923497576 >> 172.21.15.28:0/3923497576 conn(0x7fa070077790 msgr2=0x7fa070077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.214+0000 7fa07702a640 1 -- 172.21.15.28:0/3923497576 shutdown_connections 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.214+0000 7fa07702a640 1 -- 172.21.15.28:0/3923497576 wait complete. 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa07702a640 1 Processor -- start 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa07702a640 1 -- start start 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa07702a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070069eb0 0x7fa070198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa07702a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa07006a870 0x7fa0701985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa07702a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa070107820 0x7fa07019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa07702a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa070111c30 con 0x7fa070069eb0 2024-05-09T08:26:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa07702a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa070111ab0 con 0x7fa070107820 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa07702a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa070111db0 con 0x7fa07006a870 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa074dc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070069eb0 0x7fa070198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa074dc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070069eb0 0x7fa070198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51882/0 (socket says 172.21.15.28:51882) 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa074dc6640 1 -- 172.21.15.28:0/1499851634 learned_addr learned my addr 172.21.15.28:0/1499851634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa06ffff640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa07006a870 0x7fa0701985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa0755c7640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa070107820 0x7fa07019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa074dc6640 1 -- 172.21.15.28:0/1499851634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa07006a870 msgr2=0x7fa0701985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa074dc6640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa07006a870 0x7fa0701985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa074dc6640 1 -- 172.21.15.28:0/1499851634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa070107820 msgr2=0x7fa07019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa074dc6640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa070107820 0x7fa07019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa074dc6640 1 -- 172.21.15.28:0/1499851634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07019d080 con 0x7fa070069eb0 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa06ffff640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa07006a870 0x7fa0701985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.215+0000 7fa074dc6640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070069eb0 0x7fa070198090 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa06002ae00 tx=0x7fa060003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.216+0000 7fa06dffb640 1 -- 172.21.15.28:0/1499851634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa060004040 con 0x7fa070069eb0 2024-05-09T08:26:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.216+0000 7fa06dffb640 1 -- 172.21.15.28:0/1499851634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0600046c0 con 0x7fa070069eb0 2024-05-09T08:26:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.216+0000 7fa06dffb640 1 -- 172.21.15.28:0/1499851634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa060046480 con 0x7fa070069eb0 2024-05-09T08:26:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.216+0000 7fa07702a640 1 -- 172.21.15.28:0/1499851634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa07019d310 con 0x7fa070069eb0 2024-05-09T08:26:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.216+0000 7fa07702a640 1 -- 172.21.15.28:0/1499851634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0701a4c00 con 0x7fa070069eb0 2024-05-09T08:26:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.218+0000 7fa06dffb640 1 -- 172.21.15.28:0/1499851634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7fa0600305a0 con 0x7fa070069eb0 2024-05-09T08:26:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.218+0000 7fa079e11740 1 Processor -- start 2024-05-09T08:26:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.218+0000 7fa079e11740 1 -- start start 2024-05-09T08:26:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa079e11740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77be210 0x5564d77be5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa079e11740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77beb30 0x5564d77cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa079e11740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa079e11740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564d7723c40 con 0x5564d77cd730 2024-05-09T08:26:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa079e11740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5564d76cb580 con 0x5564d77be210 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa079e11740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5564d771fc00 con 0x5564d77beb30 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06dffb640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa05c0a2e10 0x7fa05c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51896/0 (socket says 172.21.15.28:51896) 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06ffff640 1 -- 172.21.15.28:0/3169256746 learned_addr learned my addr 172.21.15.28:0/3169256746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa0755c7640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa05c0a2e10 0x7fa05c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06dffb640 1 -- 172.21.15.28:0/1499851634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa06002e030 con 0x7fa070069eb0 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06ffff640 1 -- 172.21.15.28:0/3169256746 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77beb30 msgr2=0x5564d77cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa0755c7640 1 --2- 172.21.15.28:0/3169256746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77be210 0x5564d77be5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06ffff640 1 --2- 172.21.15.28:0/3169256746 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77beb30 0x5564d77cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06ffff640 1 -- 172.21.15.28:0/3169256746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77be210 msgr2=0x5564d77be5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06ffff640 1 --2- 172.21.15.28:0/3169256746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77be210 0x5564d77be5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06ffff640 1 -- 172.21.15.28:0/3169256746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d74373f0 con 0x5564d77cd730 2024-05-09T08:26:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa0755c7640 1 --2- 172.21.15.28:0/3169256746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77be210 0x5564d77be5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.219+0000 7fa06ffff640 1 --2- 172.21.15.28:0/3169256746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cfb20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa05800bcb0 tx=0x7fa05800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d0a0cb9299f5a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa0755c7640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa05c0a2e10 0x7fa05c0a52d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa068005d50 tx=0x7fa068005dd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa066ffd640 1 -- 172.21.15.28:0/3169256746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058011440 con 0x5564d77cd730 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa066ffd640 1 -- 172.21.15.28:0/3169256746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa058011ac0 con 0x5564d77cd730 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa066ffd640 1 -- 172.21.15.28:0/3169256746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05801bb10 con 0x5564d77cd730 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa079e11740 1 -- 172.21.15.28:0/3169256746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 msgr2=0x5564d77cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa079e11740 1 --2- 172.21.15.28:0/3169256746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cfb20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa05800bcb0 tx=0x7fa05800f730 comp rx=0 tx=0).stop 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa079e11740 1 -- 172.21.15.28:0/3169256746 shutdown_connections 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa079e11740 1 --2- 172.21.15.28:0/3169256746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cfb20 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa079e11740 1 --2- 172.21.15.28:0/3169256746 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77beb30 0x5564d77cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa079e11740 1 --2- 172.21.15.28:0/3169256746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77be210 0x5564d77be5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa079e11740 1 -- 172.21.15.28:0/3169256746 >> 172.21.15.28:0/3169256746 conn(0x5564d77ba650 msgr2=0x5564d77baa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa079e11740 1 -- 172.21.15.28:0/3169256746 shutdown_connections 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.220+0000 7fa079e11740 1 -- 172.21.15.28:0/3169256746 wait complete. 2024-05-09T08:26:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa079e11740 1 -- start start 2024-05-09T08:26:55.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa079e11740 1 Processor -- start 2024-05-09T08:26:55.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa079e11740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77be210 0x5564d77ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa079e11740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa079e11740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa079e11740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564d77d0570 con 0x5564d77cd730 2024-05-09T08:26:55.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa079e11740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5564d76dec40 con 0x5564d77beb30 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa079e11740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5564d77d06f0 con 0x5564d77be210 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa079e11740 10 client.0 _ll_register_callbacks cb 0x5564d7436be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa0755c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa0755c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51906/0 (socket says 172.21.15.28:51906) 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.221+0000 7fa0755c7640 1 -- 172.21.15.28:0/791249026 learned_addr learned my addr 172.21.15.28:0/791249026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa06ffff640 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa074dc6640 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77be210 0x5564d77ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa0755c7640 1 -- 172.21.15.28:0/791249026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77be210 msgr2=0x5564d77ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa0755c7640 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77be210 0x5564d77ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa0755c7640 1 -- 172.21.15.28:0/791249026 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 msgr2=0x5564d77cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa0755c7640 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa0755c7640 1 -- 172.21.15.28:0/791249026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d75017d0 con 0x5564d77cd730 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa0755c7640 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cb170 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa068011e70 tx=0x7fa0680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24d6b0aa82a149e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa04ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa04ffff640 1 -- 172.21.15.28:0/791249026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068021070 con 0x5564d77cd730 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa04ffff640 1 -- 172.21.15.28:0/791249026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa06801d620 con 0x5564d77cd730 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa079e11740 1 -- 172.21.15.28:0/791249026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa058011ac0 con 0x5564d77cd730 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa04ffff640 1 -- 172.21.15.28:0/791249026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa06802d050 con 0x5564d77cd730 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa079e11740 10 client.15927 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa079e11740 1 -- 172.21.15.28:0/791249026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5564d77dc840 con 0x5564d77cd730 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.222+0000 7fa04ffff640 1 -- 172.21.15.28:0/791249026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa058011ac0 con 0x5564d77cd730 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa04ffff640 1 -- 172.21.15.28:0/791249026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa068025780 con 0x5564d77cd730 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa079e11740 10 client.15927 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa079e11740 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa079e11740 20 client.15927 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa079e11740 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d77ddcf0 0x5564d77e0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa079e11740 4 client.15927 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa079e11740 1 -- 172.21.15.28:0/791249026 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564d7731370 con 0x5564d77ddcf0 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa074dc6640 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d77ddcf0 0x5564d77e0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa04ffff640 10 client.15927 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.223+0000 7fa074dc6640 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d77ddcf0 0x5564d77e0120 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2347f1f0039a66dd server_cookie=c8e8d2b13721df6 in_seq=0 out_seq=0 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa04ffff640 1 -- 172.21.15.28:0/791249026 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564d7731370 con 0x5564d77ddcf0 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa04ffff640 10 client.15927 handle_command_reply: tid=0 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa079e11740 1 client.15927 shutdown 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa079e11740 1 -- 172.21.15.28:0/791249026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 msgr2=0x5564d77cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa079e11740 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cb170 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa068011e70 tx=0x7fa0680046c0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa079e11740 1 -- 172.21.15.28:0/791249026 shutdown_connections 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa079e11740 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d77ddcf0 0x5564d77e0120 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa079e11740 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77cd730 0x5564d77cb170 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa04ffff640 0 client.15927 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa079e11740 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa079e11740 1 --2- 172.21.15.28:0/791249026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77be210 0x5564d77ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.224+0000 7fa079e11740 1 -- 172.21.15.28:0/791249026 >> 172.21.15.28:0/791249026 conn(0x5564d77ba650 msgr2=0x5564d77c0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.225+0000 7fa079e11740 1 -- 172.21.15.28:0/791249026 shutdown_connections 2024-05-09T08:26:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.225+0000 7fa079e11740 1 -- 172.21.15.28:0/791249026 wait complete. 2024-05-09T08:26:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.225+0000 7fa079e11740 20 client.15927 trim_cache size 0 max 16384 2024-05-09T08:26:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa079e11740 1 Processor -- start 2024-05-09T08:26:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa079e11740 1 -- start start 2024-05-09T08:26:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa079e11740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77cce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa079e11740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77d94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa079e11740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77cd730 0x5564d77b9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa079e11740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0701acaf0 con 0x5564d77be210 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa079e11740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa0701ae630 con 0x5564d77beb30 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa079e11740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa0701ad1c0 con 0x5564d77cd730 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa074dc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa074dc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51914/0 (socket says 172.21.15.28:51914) 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.238+0000 7fa074dc6640 1 -- 172.21.15.28:0/4209970448 learned_addr learned my addr 172.21.15.28:0/4209970448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa074dc6640 1 -- 172.21.15.28:0/4209970448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77cd730 msgr2=0x5564d77b9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa074dc6640 1 --2- 172.21.15.28:0/4209970448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77cd730 0x5564d77b9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa074dc6640 1 -- 172.21.15.28:0/4209970448 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 msgr2=0x5564d77d94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa074dc6640 1 --2- 172.21.15.28:0/4209970448 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77d94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa074dc6640 1 -- 172.21.15.28:0/4209970448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d77abd20 con 0x5564d77be210 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa074dc6640 1 --2- 172.21.15.28:0/4209970448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77cce20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa0600bced0 tx=0x7fa0600c0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436bfe21c043e5f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa0667fc640 1 -- 172.21.15.28:0/4209970448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa060030890 con 0x5564d77be210 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa0667fc640 1 -- 172.21.15.28:0/4209970448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa060036040 con 0x5564d77be210 2024-05-09T08:26:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa0667fc640 1 -- 172.21.15.28:0/4209970448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0600bb4d0 con 0x5564d77be210 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa079e11740 1 -- 172.21.15.28:0/4209970448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 msgr2=0x5564d77cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa079e11740 1 --2- 172.21.15.28:0/4209970448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77cce20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa0600bced0 tx=0x7fa0600c0040 comp rx=0 tx=0).stop 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.239+0000 7fa079e11740 1 -- 172.21.15.28:0/4209970448 shutdown_connections 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 --2- 172.21.15.28:0/4209970448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77cd730 0x5564d77b9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 --2- 172.21.15.28:0/4209970448 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77d94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 --2- 172.21.15.28:0/4209970448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77cce20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 -- 172.21.15.28:0/4209970448 >> 172.21.15.28:0/4209970448 conn(0x5564d77ba650 msgr2=0x5564d77b8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 -- 172.21.15.28:0/4209970448 shutdown_connections 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 -- 172.21.15.28:0/4209970448 wait complete. 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 -- start start 2024-05-09T08:26:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 Processor -- start 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77d1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77d1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77cd730 0x5564d7843260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0701adf60 con 0x5564d77be210 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa0701ad890 con 0x5564d77beb30 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa0701aaad0 con 0x5564d77cd730 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa079e11740 10 client.0 _ll_register_callbacks cb 0x7fa06800e8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa074dc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77d1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.240+0000 7fa074dc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77d1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51916/0 (socket says 172.21.15.28:51916) 2024-05-09T08:26:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa074dc6640 1 -- 172.21.15.28:0/2703631344 learned_addr learned my addr 172.21.15.28:0/2703631344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa06ffff640 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77d1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa074dc6640 1 -- 172.21.15.28:0/2703631344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77cd730 msgr2=0x5564d7843260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa074dc6640 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77cd730 0x5564d7843260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa074dc6640 1 -- 172.21.15.28:0/2703631344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 msgr2=0x5564d77d1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa074dc6640 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77d1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa074dc6640 1 -- 172.21.15.28:0/2703631344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d77d2530 con 0x5564d77be210 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa074dc6640 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77d1720 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa060003fb0 tx=0x7fa0600f11c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8861029c90632e2c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa04ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa04ffff640 1 -- 172.21.15.28:0/2703631344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0600bd8d0 con 0x5564d77be210 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa04ffff640 1 -- 172.21.15.28:0/2703631344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa0600d60b0 con 0x5564d77be210 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa079e11740 1 -- 172.21.15.28:0/2703631344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa060036040 con 0x5564d77be210 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa04ffff640 1 -- 172.21.15.28:0/2703631344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa0600c5e40 con 0x5564d77be210 2024-05-09T08:26:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa079e11740 10 client.15933 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa079e11740 1 -- 172.21.15.28:0/2703631344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5564d78448a0 con 0x5564d77be210 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa04ffff640 1 -- 172.21.15.28:0/2703631344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa060036040 con 0x5564d77be210 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.241+0000 7fa04ffff640 1 -- 172.21.15.28:0/2703631344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa06004e070 con 0x5564d77be210 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa079e11740 10 client.15933 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa079e11740 10 client.15933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa079e11740 20 client.15933 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa079e11740 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d7844eb0 0x5564d78472a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa079e11740 4 client.15933 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa079e11740 1 -- 172.21.15.28:0/2703631344 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564d772b6c0 con 0x5564d7844eb0 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa06ffff640 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d7844eb0 0x5564d78472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa06ffff640 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d7844eb0 0x5564d78472a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca6a27dad517b4c6 server_cookie=af44527d9ff8d1bf in_seq=0 out_seq=0 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa04ffff640 10 client.15933 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa04ffff640 1 -- 172.21.15.28:0/2703631344 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564d772b6c0 con 0x5564d7844eb0 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa04ffff640 10 client.15933 handle_command_reply: tid=0 2024-05-09T08:26:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.242+0000 7fa079e11740 1 client.15933 shutdown 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 -- 172.21.15.28:0/2703631344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 msgr2=0x5564d77d1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77d1720 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa060003fb0 tx=0x7fa0600f11c0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 -- 172.21.15.28:0/2703631344 shutdown_connections 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d7844eb0 0x5564d78472a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d77cd730 0x5564d7843260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d77beb30 0x5564d77d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 --2- 172.21.15.28:0/2703631344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d77be210 0x5564d77d1720 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 -- 172.21.15.28:0/2703631344 >> 172.21.15.28:0/2703631344 conn(0x5564d77ba650 msgr2=0x5564d77b97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa04ffff640 0 client.15933 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 -- 172.21.15.28:0/2703631344 shutdown_connections 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 1 -- 172.21.15.28:0/2703631344 wait complete. 2024-05-09T08:26:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa079e11740 20 client.15933 trim_cache size 0 max 16384 2024-05-09T08:26:55.250 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:26:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 -- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa05c0a2e10 msgr2=0x7fa05c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa05c0a2e10 0x7fa05c0a52d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa068005d50 tx=0x7fa068005dd0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 -- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070069eb0 msgr2=0x7fa070198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070069eb0 0x7fa070198090 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa06002ae00 tx=0x7fa060003970 comp rx=0 tx=0).stop 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 -- 172.21.15.28:0/1499851634 shutdown_connections 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa070107820 0x7fa07019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa05c0a2e10 0x7fa05c0a52d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa07006a870 0x7fa0701985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 --2- 172.21.15.28:0/1499851634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa070069eb0 0x7fa070198090 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.243+0000 7fa07702a640 1 -- 172.21.15.28:0/1499851634 >> 172.21.15.28:0/1499851634 conn(0x7fa070077790 msgr2=0x7fa0700fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.244+0000 7fa07702a640 1 -- 172.21.15.28:0/1499851634 shutdown_connections 2024-05-09T08:26:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:55.244+0000 7fa07702a640 1 -- 172.21.15.28:0/1499851634 wait complete. 2024-05-09T08:26:55.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:55 smithi028 ceph-mon[30119]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T08:26:55.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:55 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2303618251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:55.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:55 smithi165 ceph-mon[34540]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T08:26:55.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:55 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2303618251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:55.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:55 smithi184 ceph-mon[34509]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-09T08:26:55.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:55 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2303618251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:26:56.256 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:26:56.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:26:56.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:26:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.371+0000 7fd94719d640 1 -- 172.21.15.28:0/4229475410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd94010bdf0 msgr2=0x7fd94010e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.371+0000 7fd94719d640 1 --2- 172.21.15.28:0/4229475410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd94010bdf0 0x7fd94010e290 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd938009ca0 tx=0x7fd93802a9e0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.371+0000 7fd94719d640 1 -- 172.21.15.28:0/4229475410 shutdown_connections 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.371+0000 7fd94719d640 1 --2- 172.21.15.28:0/4229475410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd94010bdf0 0x7fd94010e290 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.371+0000 7fd94719d640 1 --2- 172.21.15.28:0/4229475410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9401033d0 0x7fd94010b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.371+0000 7fd94719d640 1 --2- 172.21.15.28:0/4229475410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd940102ab0 0x7fd940102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.371+0000 7fd94719d640 1 -- 172.21.15.28:0/4229475410 >> 172.21.15.28:0/4229475410 conn(0x7fd940077830 msgr2=0x7fd940077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.371+0000 7fd94719d640 1 -- 172.21.15.28:0/4229475410 shutdown_connections 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.371+0000 7fd94719d640 1 -- 172.21.15.28:0/4229475410 wait complete. 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd94719d640 1 Processor -- start 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd94719d640 1 -- start start 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd94719d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940102ab0 0x7fd940100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd94719d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9401033d0 0x7fd940100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd94719d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94010bdf0 0x7fd940101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd94719d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd940111bf0 con 0x7fd9401033d0 2024-05-09T08:26:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd94719d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd940111a70 con 0x7fd940102ab0 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd94719d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd940111d70 con 0x7fd94010bdf0 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd93ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9401033d0 0x7fd940100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd93ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9401033d0 0x7fd940100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51942/0 (socket says 172.21.15.28:51942) 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.372+0000 7fd93ffff640 1 -- 172.21.15.28:0/1626359812 learned_addr learned my addr 172.21.15.28:0/1626359812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd944f39640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940102ab0 0x7fd940100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd94573a640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94010bdf0 0x7fd940101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd93ffff640 1 -- 172.21.15.28:0/1626359812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94010bdf0 msgr2=0x7fd940101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd93ffff640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94010bdf0 0x7fd940101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd93ffff640 1 -- 172.21.15.28:0/1626359812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940102ab0 msgr2=0x7fd940100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd93ffff640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940102ab0 0x7fd940100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd93ffff640 1 -- 172.21.15.28:0/1626359812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9401a6570 con 0x7fd9401033d0 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd94573a640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94010bdf0 0x7fd940101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd944f39640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940102ab0 0x7fd940100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd93ffff640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9401033d0 0x7fd940100c40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00e9f0 tx=0x7fd92c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd93dffb640 1 -- 172.21.15.28:0/1626359812 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c00cd80 con 0x7fd9401033d0 2024-05-09T08:26:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd93dffb640 1 -- 172.21.15.28:0/1626359812 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd92c010910 con 0x7fd9401033d0 2024-05-09T08:26:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd94719d640 1 -- 172.21.15.28:0/1626359812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9401a6860 con 0x7fd9401033d0 2024-05-09T08:26:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd93dffb640 1 -- 172.21.15.28:0/1626359812 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c018990 con 0x7fd9401033d0 2024-05-09T08:26:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.373+0000 7fd94719d640 1 -- 172.21.15.28:0/1626359812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9401a6e00 con 0x7fd9401033d0 2024-05-09T08:26:56.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.376+0000 7fd93dffb640 1 -- 172.21.15.28:0/1626359812 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7fd92c010ac0 con 0x7fd9401033d0 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.376+0000 7fd949f82740 1 Processor -- start 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.376+0000 7fd949f82740 1 -- start start 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd949f82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f6343f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd949f82740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6349c0 0x55e01f634e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd949f82740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f63f740 0x55e01f641b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd949f82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e01f599990 con 0x55e01f6360b0 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd949f82740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e01f5415b0 con 0x55e01f63f740 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd949f82740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e01f595950 con 0x55e01f6349c0 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f6343f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f6343f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51948/0 (socket says 172.21.15.28:51948) 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 -- 172.21.15.28:0/4288914576 learned_addr learned my addr 172.21.15.28:0/4288914576 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd93dffb640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd9300a2e10 0x7fd9300a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd93ffff640 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f63f740 0x55e01f641b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd944f39640 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6349c0 0x55e01f634e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd9300a2e10 0x7fd9300a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 -- 172.21.15.28:0/4288914576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6349c0 msgr2=0x55e01f634e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6349c0 0x55e01f634e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 -- 172.21.15.28:0/4288914576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f63f740 msgr2=0x55e01f641b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f63f740 0x55e01f641b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 -- 172.21.15.28:0/4288914576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01f3757e0 con 0x55e01f6360b0 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd93dffb640 1 -- 172.21.15.28:0/1626359812 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fd92c0cfcf0 con 0x7fd9401033d0 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd944f39640 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6349c0 0x55e01f634e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.377+0000 7fd94573a640 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f6343f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd93802eed0 tx=0x7fd93802ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3192d50479dc54 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.378+0000 7fd94573a640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd9300a2e10 0x7fd9300a52d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd93804c000 tx=0x7fd93804a000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.380+0000 7fd92affd640 1 -- 172.21.15.28:0/4288914576 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938041050 con 0x55e01f6360b0 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.380+0000 7fd92affd640 1 -- 172.21.15.28:0/4288914576 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd938042040 con 0x55e01f6360b0 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.380+0000 7fd92affd640 1 -- 172.21.15.28:0/4288914576 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93804b020 con 0x55e01f6360b0 2024-05-09T08:26:56.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 -- 172.21.15.28:0/4288914576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 msgr2=0x55e01f6343f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f6343f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd93802eed0 tx=0x7fd93802ef00 comp rx=0 tx=0).stop 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 -- 172.21.15.28:0/4288914576 shutdown_connections 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f63f740 0x55e01f641b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6349c0 0x55e01f634e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 --2- 172.21.15.28:0/4288914576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f6343f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 -- 172.21.15.28:0/4288914576 >> 172.21.15.28:0/4288914576 conn(0x55e01f5fd0a0 msgr2=0x55e01f630f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 -- 172.21.15.28:0/4288914576 shutdown_connections 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 -- 172.21.15.28:0/4288914576 wait complete. 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 -- start start 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.381+0000 7fd949f82740 1 Processor -- start 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd949f82740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f63c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd949f82740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 0x55e01f63cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd949f82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f63d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd949f82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e01f6425d0 con 0x55e01f63f740 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd949f82740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e01f554b40 con 0x55e01f6349c0 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd949f82740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e01f642750 con 0x55e01f6360b0 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd949f82740 10 client.0 _ll_register_callbacks cb 0x55e01f5490f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd94573a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f63d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd94573a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f63d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51962/0 (socket says 172.21.15.28:51962) 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd94573a640 1 -- 172.21.15.28:0/92412739 learned_addr learned my addr 172.21.15.28:0/92412739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd93ffff640 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 0x55e01f63cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd944f39640 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f63c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd94573a640 1 -- 172.21.15.28:0/92412739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 msgr2=0x55e01f63cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd94573a640 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 0x55e01f63cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd94573a640 1 -- 172.21.15.28:0/92412739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 msgr2=0x55e01f63c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd94573a640 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f63c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.382+0000 7fd94573a640 1 -- 172.21.15.28:0/92412739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01f3d5f70 con 0x55e01f63f740 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd94573a640 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f63d110 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd9380093e0 tx=0x7fd9380099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0d4b870782036d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd91ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd91ffff640 1 -- 172.21.15.28:0/92412739 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93804b030 con 0x55e01f63f740 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd949f82740 1 -- 172.21.15.28:0/92412739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd938042040 con 0x55e01f63f740 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd91ffff640 1 -- 172.21.15.28:0/92412739 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fd93802fb10 con 0x55e01f63f740 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd91ffff640 1 -- 172.21.15.28:0/92412739 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd938061050 con 0x55e01f63f740 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd949f82740 10 client.15945 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd949f82740 1 -- 172.21.15.28:0/92412739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e01f26d170 con 0x55e01f63f740 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd91ffff640 1 -- 172.21.15.28:0/92412739 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd938042040 con 0x55e01f63f740 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd91ffff640 1 -- 172.21.15.28:0/92412739 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd9380369d0 con 0x55e01f63f740 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd949f82740 10 client.15945 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd949f82740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd949f82740 20 client.15945 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd949f82740 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e01f64f6f0 0x55e01f651ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd949f82740 4 client.15945 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.383+0000 7fd949f82740 1 -- 172.21.15.28:0/92412739 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e01f5a70c0 con 0x55e01f64f6f0 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.384+0000 7fd944f39640 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e01f64f6f0 0x55e01f651ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.384+0000 7fd91ffff640 10 client.15945 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.384+0000 7fd944f39640 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e01f64f6f0 0x55e01f651ae0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cefbe07a38b0b611 server_cookie=875a2ffe46a32fcd in_seq=0 out_seq=0 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd91ffff640 1 -- 172.21.15.28:0/92412739 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e01f5a70c0 con 0x55e01f64f6f0 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd91ffff640 10 client.15945 handle_command_reply: tid=0 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 client.15945 shutdown 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 -- 172.21.15.28:0/92412739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 msgr2=0x55e01f63d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f63d110 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd9380093e0 tx=0x7fd9380099c0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 -- 172.21.15.28:0/92412739 shutdown_connections 2024-05-09T08:26:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e01f64f6f0 0x55e01f651ae0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f63d110 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 0x55e01f63cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd91ffff640 0 client.15945 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 --2- 172.21.15.28:0/92412739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f63c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 -- 172.21.15.28:0/92412739 >> 172.21.15.28:0/92412739 conn(0x55e01f5fd0a0 msgr2=0x55e01f636d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 -- 172.21.15.28:0/92412739 shutdown_connections 2024-05-09T08:26:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 1 -- 172.21.15.28:0/92412739 wait complete. 2024-05-09T08:26:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.385+0000 7fd949f82740 20 client.15945 trim_cache size 0 max 16384 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd949f82740 1 Processor -- start 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd949f82740 1 -- start start 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd949f82740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f62ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd949f82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f654b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd949f82740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f63f740 0x55e01f655040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd949f82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9401b1030 con 0x55e01f6360b0 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd949f82740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd9401b2b70 con 0x55e01f6349c0 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd949f82740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9401b1700 con 0x55e01f63f740 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd93ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f654b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.404+0000 7fd93ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f654b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51974/0 (socket says 172.21.15.28:51974) 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd93ffff640 1 -- 172.21.15.28:0/4114170371 learned_addr learned my addr 172.21.15.28:0/4114170371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd944f39640 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f62ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd93ffff640 1 -- 172.21.15.28:0/4114170371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f63f740 msgr2=0x55e01f655040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd94573a640 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f63f740 0x55e01f655040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd93ffff640 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f63f740 0x55e01f655040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd93ffff640 1 -- 172.21.15.28:0/4114170371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 msgr2=0x55e01f62ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd93ffff640 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f62ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd93ffff640 1 -- 172.21.15.28:0/4114170371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01f62f470 con 0x55e01f6360b0 2024-05-09T08:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd94573a640 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f63f740 0x55e01f655040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd944f39640 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f62ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd93ffff640 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f654b00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0cd040 tx=0x7fd92c024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70559cb8481cfd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd92a7fc640 1 -- 172.21.15.28:0/4114170371 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0a0080 con 0x55e01f6360b0 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd92a7fc640 1 -- 172.21.15.28:0/4114170371 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd92c0a4940 con 0x55e01f6360b0 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.405+0000 7fd92a7fc640 1 -- 172.21.15.28:0/4114170371 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0a4710 con 0x55e01f6360b0 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 -- 172.21.15.28:0/4114170371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 msgr2=0x55e01f654b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f654b00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0cd040 tx=0x7fd92c024000 comp rx=0 tx=0).stop 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 -- 172.21.15.28:0/4114170371 shutdown_connections 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f63f740 0x55e01f655040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f6360b0 0x55e01f654b00 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 --2- 172.21.15.28:0/4114170371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f62ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 -- 172.21.15.28:0/4114170371 >> 172.21.15.28:0/4114170371 conn(0x55e01f5fd0a0 msgr2=0x55e01f62f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 -- 172.21.15.28:0/4114170371 shutdown_connections 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 -- 172.21.15.28:0/4114170371 wait complete. 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 -- start start 2024-05-09T08:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.406+0000 7fd949f82740 1 Processor -- start 2024-05-09T08:26:56.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd949f82740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f6530a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd949f82740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 0x55e01f6535e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd949f82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f654560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd949f82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9401b24a0 con 0x55e01f63f740 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd949f82740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd9401b1dd0 con 0x55e01f6349c0 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd949f82740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9401af010 con 0x55e01f6360b0 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd949f82740 10 client.0 _ll_register_callbacks cb 0x7fd92c0cce40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd94573a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f654560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd94573a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f654560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51978/0 (socket says 172.21.15.28:51978) 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd94573a640 1 -- 172.21.15.28:0/3650446919 learned_addr learned my addr 172.21.15.28:0/3650446919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd944f39640 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f6530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd93ffff640 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 0x55e01f6535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd94573a640 1 -- 172.21.15.28:0/3650446919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 msgr2=0x55e01f6535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd94573a640 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 0x55e01f6535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd94573a640 1 -- 172.21.15.28:0/3650446919 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 msgr2=0x55e01f6530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd94573a640 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f6530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd94573a640 1 -- 172.21.15.28:0/3650446919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01f654c50 con 0x55e01f63f740 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.407+0000 7fd944f39640 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f6530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd94573a640 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f654560 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd938009620 tx=0x7fd93802ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71f3b554a6764c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd91ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd91ffff640 1 -- 172.21.15.28:0/3650446919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938041030 con 0x55e01f63f740 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd949f82740 1 -- 172.21.15.28:0/3650446919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd92c0a4940 con 0x55e01f63f740 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd91ffff640 1 -- 172.21.15.28:0/3650446919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fd93802fb20 con 0x55e01f63f740 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd91ffff640 1 -- 172.21.15.28:0/3650446919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd93806d050 con 0x55e01f63f740 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd949f82740 10 client.15951 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd949f82740 1 -- 172.21.15.28:0/3650446919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e01f6b6470 con 0x55e01f63f740 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd91ffff640 1 -- 172.21.15.28:0/3650446919 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd92c0a4940 con 0x55e01f63f740 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd91ffff640 1 -- 172.21.15.28:0/3650446919 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd938004510 con 0x55e01f63f740 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd949f82740 10 client.15951 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd949f82740 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd949f82740 20 client.15951 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd949f82740 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e01f6b6b30 0x55e01f6b8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd949f82740 4 client.15951 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.408+0000 7fd949f82740 1 -- 172.21.15.28:0/3650446919 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e01f5a1410 con 0x55e01f6b6b30 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.409+0000 7fd944f39640 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e01f6b6b30 0x55e01f6b8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.409+0000 7fd91ffff640 10 client.15951 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.409+0000 7fd944f39640 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e01f6b6b30 0x55e01f6b8f20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5858b9eaf0f2e4a server_cookie=9b0c10f9fe2f40d9 in_seq=0 out_seq=0 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.410+0000 7fd91ffff640 1 -- 172.21.15.28:0/3650446919 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e01f5a1410 con 0x55e01f6b6b30 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.410+0000 7fd91ffff640 10 client.15951 handle_command_reply: tid=0 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.410+0000 7fd949f82740 1 client.15951 shutdown 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 -- 172.21.15.28:0/3650446919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 msgr2=0x55e01f654560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f654560 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd938009620 tx=0x7fd93802ff90 comp rx=0 tx=0).stop 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 -- 172.21.15.28:0/3650446919 shutdown_connections 2024-05-09T08:26:56.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e01f6b6b30 0x55e01f6b8f20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01f63f740 0x55e01f654560 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e01f6360b0 0x55e01f6535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd91ffff640 0 client.15951 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 --2- 172.21.15.28:0/3650446919 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e01f6349c0 0x55e01f6530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 -- 172.21.15.28:0/3650446919 >> 172.21.15.28:0/3650446919 conn(0x55e01f5fd0a0 msgr2=0x55e01f630080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 -- 172.21.15.28:0/3650446919 shutdown_connections 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 1 -- 172.21.15.28:0/3650446919 wait complete. 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.411+0000 7fd949f82740 20 client.15951 trim_cache size 0 max 16384 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:56.417 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-09T08:26:56.418 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "7bc347d5-66b3-4181-8e92-30f7f4d326ae", 2024-05-09T08:26:56.418 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-09T08:26:56.418 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:56.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 -- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd9300a2e10 msgr2=0x7fd9300a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd9300a2e10 0x7fd9300a52d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd93804c000 tx=0x7fd93804a000 comp rx=0 tx=0).stop 2024-05-09T08:26:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 -- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9401033d0 msgr2=0x7fd940100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9401033d0 0x7fd940100c40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00e9f0 tx=0x7fd92c00eeb0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 -- 172.21.15.28:0/1626359812 shutdown_connections 2024-05-09T08:26:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd9300a2e10 0x7fd9300a52d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94010bdf0 0x7fd940101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9401033d0 0x7fd940100c40 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 --2- 172.21.15.28:0/1626359812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940102ab0 0x7fd940100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 -- 172.21.15.28:0/1626359812 >> 172.21.15.28:0/1626359812 conn(0x7fd940077830 msgr2=0x7fd9400fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 -- 172.21.15.28:0/1626359812 shutdown_connections 2024-05-09T08:26:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.412+0000 7fd94719d640 1 -- 172.21.15.28:0/1626359812 wait complete. 2024-05-09T08:26:56.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:26:56.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:26:56.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.539+0000 7f401f134640 1 -- 172.21.15.28:0/2597945559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018106710 msgr2=0x7f4018106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.539+0000 7f401f134640 1 --2- 172.21.15.28:0/2597945559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018106710 0x7f4018106af0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4008009930 tx=0x7f400802a420 comp rx=0 tx=0).stop 2024-05-09T08:26:56.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 -- 172.21.15.28:0/2597945559 shutdown_connections 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 --2- 172.21.15.28:0/2597945559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40181010c0 0x7f401810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 --2- 172.21.15.28:0/2597945559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4018100700 0x7f4018100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 --2- 172.21.15.28:0/2597945559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4018106710 0x7f4018106af0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 -- 172.21.15.28:0/2597945559 >> 172.21.15.28:0/2597945559 conn(0x7f4018077830 msgr2=0x7f4018077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 -- 172.21.15.28:0/2597945559 shutdown_connections 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 -- 172.21.15.28:0/2597945559 wait complete. 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 Processor -- start 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 -- start start 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4018100700 0x7f4018102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40181010c0 0x7f4018103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4018106710 0x7f4018103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4018110ae0 con 0x7f40181010c0 2024-05-09T08:26:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4018110960 con 0x7f4018106710 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.540+0000 7f401f134640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4018110c60 con 0x7f4018100700 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4017fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40181010c0 0x7f4018103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4017fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40181010c0 0x7f4018103200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52008/0 (socket says 172.21.15.28:52008) 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4017fff640 1 -- 172.21.15.28:0/1855669504 learned_addr learned my addr 172.21.15.28:0/1855669504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f401ced0640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4018100700 0x7f4018102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f401d6d1640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4018106710 0x7f4018103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4017fff640 1 -- 172.21.15.28:0/1855669504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4018100700 msgr2=0x7f4018102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4017fff640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4018100700 0x7f4018102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4017fff640 1 -- 172.21.15.28:0/1855669504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4018106710 msgr2=0x7f4018103740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4017fff640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4018106710 0x7f4018103740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4017fff640 1 -- 172.21.15.28:0/1855669504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40181a6610 con 0x7f40181010c0 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4017fff640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40181010c0 0x7f4018103200 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f400000cce0 tx=0x7f400000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4015ffb640 1 -- 172.21.15.28:0/1855669504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400000f650 con 0x7f40181010c0 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4015ffb640 1 -- 172.21.15.28:0/1855669504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f400000fcd0 con 0x7f40181010c0 2024-05-09T08:26:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f401f134640 1 -- 172.21.15.28:0/1855669504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40181a6900 con 0x7f40181010c0 2024-05-09T08:26:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.541+0000 7f4015ffb640 1 -- 172.21.15.28:0/1855669504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4000018c70 con 0x7f40181010c0 2024-05-09T08:26:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.542+0000 7f401f134640 1 -- 172.21.15.28:0/1855669504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40181a6e50 con 0x7f40181010c0 2024-05-09T08:26:56.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.544+0000 7f4015ffb640 1 -- 172.21.15.28:0/1855669504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f400000f7f0 con 0x7f40181010c0 2024-05-09T08:26:56.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.544+0000 7f4021e42740 1 Processor -- start 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.544+0000 7f4021e42740 1 -- start start 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f4021e42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f4a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f4021e42740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f4aba0 0x558df4f54d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f4021e42740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f55270 0x558df4f57660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401d6d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f4a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401d6d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f4a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52016/0 (socket says 172.21.15.28:52016) 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401d6d1640 1 -- 172.21.15.28:0/2471711070 learned_addr learned my addr 172.21.15.28:0/2471711070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f4021e42740 1 -- 172.21.15.28:0/2471711070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558df4eafd00 con 0x558df4f4a280 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f4021e42740 1 -- 172.21.15.28:0/2471711070 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558df4e574c0 con 0x558df4f4aba0 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f4021e42740 1 -- 172.21.15.28:0/2471711070 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558df4eabcc0 con 0x558df4f55270 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401ced0640 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f4aba0 0x558df4f54d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f4017fff640 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f55270 0x558df4f57660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401ced0640 1 -- 172.21.15.28:0/2471711070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f55270 msgr2=0x558df4f57660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401ced0640 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f55270 0x558df4f57660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401ced0640 1 -- 172.21.15.28:0/2471711070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 msgr2=0x558df4f4a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401ced0640 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f4a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f4015ffb640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f40040a2d90 0x7f40040a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401ced0640 1 -- 172.21.15.28:0/2471711070 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558df4c8d7d0 con 0x558df4f4aba0 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.545+0000 7f401d6d1640 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f4a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.546+0000 7f401d6d1640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f40040a2d90 0x7f40040a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.546+0000 7f4015ffb640 1 -- 172.21.15.28:0/1855669504 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f40000cfe70 con 0x7f40181010c0 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.546+0000 7f401ced0640 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f4aba0 0x558df4f54d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4008005940 tx=0x7f40080316e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8d324d6d8e62a75 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.546+0000 7f401d6d1640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f40040a2d90 0x7f40040a5250 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4010005fd0 tx=0x7f401000c400 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.546+0000 7f400effd640 1 -- 172.21.15.28:0/2471711070 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008040070 con 0x558df4f4aba0 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.546+0000 7f400effd640 1 -- 172.21.15.28:0/2471711070 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4008005020 con 0x558df4f4aba0 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.546+0000 7f400effd640 1 -- 172.21.15.28:0/2471711070 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008035040 con 0x558df4f4aba0 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 -- 172.21.15.28:0/2471711070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f4aba0 msgr2=0x558df4f54d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f4aba0 0x558df4f54d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4008005940 tx=0x7f40080316e0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 -- 172.21.15.28:0/2471711070 shutdown_connections 2024-05-09T08:26:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f55270 0x558df4f57660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f4aba0 0x558df4f54d30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 --2- 172.21.15.28:0/2471711070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f4a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 -- 172.21.15.28:0/2471711070 >> 172.21.15.28:0/2471711070 conn(0x558df4f465a0 msgr2=0x558df4f46980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 -- 172.21.15.28:0/2471711070 shutdown_connections 2024-05-09T08:26:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 -- 172.21.15.28:0/2471711070 wait complete. 2024-05-09T08:26:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 -- start start 2024-05-09T08:26:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.547+0000 7f4021e42740 1 Processor -- start 2024-05-09T08:26:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f4021e42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f52110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f4021e42740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f52650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f4021e42740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f52b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f4021e42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558df4f58000 con 0x558df4f4a280 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f4021e42740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558df4e6ab90 con 0x558df4f55270 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f4021e42740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558df4f58180 con 0x558df4f4aba0 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f4021e42740 10 client.0 _ll_register_callbacks cb 0x558df4ea6da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401ced0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f52110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401ced0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f52110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52032/0 (socket says 172.21.15.28:52032) 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401ced0640 1 -- 172.21.15.28:0/233478252 learned_addr learned my addr 172.21.15.28:0/233478252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401d6d1640 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f52b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f4017fff640 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f52650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401ced0640 1 -- 172.21.15.28:0/233478252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 msgr2=0x558df4f52650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401ced0640 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f52650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401ced0640 1 -- 172.21.15.28:0/233478252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 msgr2=0x558df4f52b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401ced0640 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f52b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401ced0640 1 -- 172.21.15.28:0/233478252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558df4bc33f0 con 0x558df4f4a280 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f401ced0640 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f52110 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4008009a50 tx=0x7f4008004810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcb5972df917c59 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f3ff7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.548+0000 7f3ff7fff640 1 -- 172.21.15.28:0/233478252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008040030 con 0x558df4f4a280 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f3ff7fff640 1 -- 172.21.15.28:0/233478252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f4008004240 con 0x558df4f4a280 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f4021e42740 1 -- 172.21.15.28:0/233478252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4008005020 con 0x558df4f4a280 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f3ff7fff640 1 -- 172.21.15.28:0/233478252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4008057050 con 0x558df4f4a280 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f4021e42740 10 client.15963 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f4021e42740 1 -- 172.21.15.28:0/233478252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558df4b830f0 con 0x558df4f4a280 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f3ff7fff640 1 -- 172.21.15.28:0/233478252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4008005020 con 0x558df4f4a280 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f3ff7fff640 1 -- 172.21.15.28:0/233478252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f40080035f0 con 0x558df4f4a280 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f4021e42740 10 client.15963 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f4021e42740 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f4021e42740 20 client.15963 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f4021e42740 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558df4f65340 0x558df4f67730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f4021e42740 4 client.15963 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.549+0000 7f4021e42740 1 -- 172.21.15.28:0/233478252 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558df4ebd430 con 0x558df4f65340 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.550+0000 7f4017fff640 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558df4f65340 0x558df4f67730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.550+0000 7f3ff7fff640 10 client.15963 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.550+0000 7f4017fff640 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558df4f65340 0x558df4f67730 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd31a02b3551b065 server_cookie=7302c23ca832d934 in_seq=0 out_seq=0 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f3ff7fff640 1 -- 172.21.15.28:0/233478252 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558df4ebd430 con 0x558df4f65340 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f3ff7fff640 10 client.15963 handle_command_reply: tid=0 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 client.15963 shutdown 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 -- 172.21.15.28:0/233478252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 msgr2=0x558df4f52110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f52110 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4008009a50 tx=0x7f4008004810 comp rx=0 tx=0).stop 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 -- 172.21.15.28:0/233478252 shutdown_connections 2024-05-09T08:26:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558df4f65340 0x558df4f67730 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f52b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f3ff7fff640 0 client.15963 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f52650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 --2- 172.21.15.28:0/233478252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f52110 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 -- 172.21.15.28:0/233478252 >> 172.21.15.28:0/233478252 conn(0x558df4f465a0 msgr2=0x558df4f457e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 -- 172.21.15.28:0/233478252 shutdown_connections 2024-05-09T08:26:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 1 -- 172.21.15.28:0/233478252 wait complete. 2024-05-09T08:26:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.551+0000 7f4021e42740 20 client.15963 trim_cache size 0 max 16384 2024-05-09T08:26:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.569+0000 7f4021e42740 1 Processor -- start 2024-05-09T08:26:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.569+0000 7f4021e42740 1 -- start start 2024-05-09T08:26:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f4021e42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f60ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f4021e42740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f44f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f4021e42740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f6cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f4021e42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f40181b0f80 con 0x558df4f4a280 2024-05-09T08:26:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f4021e42740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f40181b2ac0 con 0x558df4f55270 2024-05-09T08:26:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f4021e42740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f40181b1650 con 0x558df4f4aba0 2024-05-09T08:26:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401ced0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f60ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401ced0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f60ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52048/0 (socket says 172.21.15.28:52048) 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401ced0640 1 -- 172.21.15.28:0/2827768932 learned_addr learned my addr 172.21.15.28:0/2827768932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401d6d1640 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f6cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f4017fff640 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f44f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401ced0640 1 -- 172.21.15.28:0/2827768932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 msgr2=0x558df4f44f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401ced0640 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f44f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401ced0640 1 -- 172.21.15.28:0/2827768932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 msgr2=0x558df4f6cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401ced0640 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f6cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401ced0640 1 -- 172.21.15.28:0/2827768932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558df4f53070 con 0x558df4f4a280 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401d6d1640 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f6cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f401ced0640 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f60ec0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f40080092a0 tx=0x7f40080316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c62d61a88223466 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.570+0000 7f400e7fc640 1 -- 172.21.15.28:0/2827768932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008040030 con 0x558df4f4a280 2024-05-09T08:26:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f400e7fc640 1 -- 172.21.15.28:0/2827768932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4008004240 con 0x558df4f4a280 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f400e7fc640 1 -- 172.21.15.28:0/2827768932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008035070 con 0x558df4f4a280 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f4021e42740 1 -- 172.21.15.28:0/2827768932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 msgr2=0x558df4f60ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f4021e42740 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f60ec0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f40080092a0 tx=0x7f40080316e0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f4021e42740 1 -- 172.21.15.28:0/2827768932 shutdown_connections 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f4021e42740 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f6cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f4021e42740 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f44f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f4021e42740 1 --2- 172.21.15.28:0/2827768932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f60ec0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f4021e42740 1 -- 172.21.15.28:0/2827768932 >> 172.21.15.28:0/2827768932 conn(0x558df4f465a0 msgr2=0x558df4f52960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f4021e42740 1 -- 172.21.15.28:0/2827768932 shutdown_connections 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.571+0000 7f4021e42740 1 -- 172.21.15.28:0/2827768932 wait complete. 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4021e42740 1 -- start start 2024-05-09T08:26:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4021e42740 1 Processor -- start 2024-05-09T08:26:56.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4021e42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f5d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4021e42740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f5d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4021e42740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f5a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4021e42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f40181b23f0 con 0x558df4f4a280 2024-05-09T08:26:56.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4021e42740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f40181b1d20 con 0x558df4f55270 2024-05-09T08:26:56.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4021e42740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f40181aef60 con 0x558df4f4aba0 2024-05-09T08:26:56.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4021e42740 10 client.0 _ll_register_callbacks cb 0x7f401000c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:26:56.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f401ced0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f5d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f401ced0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f5d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52058/0 (socket says 172.21.15.28:52058) 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f401ced0640 1 -- 172.21.15.28:0/3533268726 learned_addr learned my addr 172.21.15.28:0/3533268726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.572+0000 7f4017fff640 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f5d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f401ced0640 1 -- 172.21.15.28:0/3533268726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 msgr2=0x558df4f5d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f401ced0640 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f5d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f401ced0640 1 -- 172.21.15.28:0/3533268726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 msgr2=0x558df4f5a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f401ced0640 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f5a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f401ced0640 1 -- 172.21.15.28:0/3533268726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558df4f5abd0 con 0x558df4f4a280 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f4017fff640 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f5d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f401ced0640 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f5d3d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4008009a50 tx=0x7f4008003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902f5c28e9dbfe4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f3ff7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f3ff7fff640 1 -- 172.21.15.28:0/3533268726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008035040 con 0x558df4f4a280 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f3ff7fff640 1 -- 172.21.15.28:0/3533268726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f400803c440 con 0x558df4f4a280 2024-05-09T08:26:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f4021e42740 1 -- 172.21.15.28:0/3533268726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4008004240 con 0x558df4f4a280 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f3ff7fff640 1 -- 172.21.15.28:0/3533268726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4008069050 con 0x558df4f4a280 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f4021e42740 10 client.15969 fetch_fsmap learned FSMap version 28 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f4021e42740 1 -- 172.21.15.28:0/3533268726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558df4fcc0a0 con 0x558df4f4a280 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f3ff7fff640 1 -- 172.21.15.28:0/3533268726 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4008004240 con 0x558df4f4a280 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f3ff7fff640 1 -- 172.21.15.28:0/3533268726 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f400805b690 con 0x558df4f4a280 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f4021e42740 10 client.15969 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f4021e42740 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f4021e42740 20 client.15969 populate_metadata read hostname 'smithi028' 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.573+0000 7f4021e42740 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558df4f65340 0x558df4fce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f4021e42740 4 client.15969 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f4021e42740 1 -- 172.21.15.28:0/3533268726 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558df4eb7780 con 0x558df4f65340 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f4017fff640 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558df4f65340 0x558df4fce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f4017fff640 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558df4f65340 0x558df4fce750 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e40cb2d7eb33b4d9 server_cookie=29c91655fa8f2505 in_seq=0 out_seq=0 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f3ff7fff640 10 client.15969 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f3ff7fff640 1 -- 172.21.15.28:0/3533268726 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558df4eb7780 con 0x558df4f65340 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f3ff7fff640 10 client.15969 handle_command_reply: tid=0 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f4021e42740 1 client.15969 shutdown 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f4021e42740 1 -- 172.21.15.28:0/3533268726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 msgr2=0x558df4f5d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.574+0000 7f4021e42740 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f5d3d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4008009a50 tx=0x7f4008003820 comp rx=0 tx=0).stop 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f4021e42740 1 -- 172.21.15.28:0/3533268726 shutdown_connections 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f4021e42740 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558df4f65340 0x558df4fce750 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f4021e42740 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558df4f55270 0x558df4f5a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f3ff7fff640 0 client.15969 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f4021e42740 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558df4f4aba0 0x558df4f5d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f4021e42740 1 --2- 172.21.15.28:0/3533268726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558df4f4a280 0x558df4f5d3d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f4021e42740 1 -- 172.21.15.28:0/3533268726 >> 172.21.15.28:0/3533268726 conn(0x558df4f465a0 msgr2=0x558df4f506a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f4021e42740 1 -- 172.21.15.28:0/3533268726 shutdown_connections 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f4021e42740 1 -- 172.21.15.28:0/3533268726 wait complete. 2024-05-09T08:26:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.575+0000 7f4021e42740 20 client.15969 trim_cache size 0 max 16384 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout: "7bc347d5-66b3-4181-8e92-30f7f4d326ae": { 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "7bc347d5-66b3-4181-8e92-30f7f4d326ae", 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:26:56.582 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:26:56.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 -- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f40040a2d90 msgr2=0x7f40040a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f40040a2d90 0x7f40040a5250 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4010005fd0 tx=0x7f401000c400 comp rx=0 tx=0).stop 2024-05-09T08:26:56.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 -- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40181010c0 msgr2=0x7f4018103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:56.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40181010c0 0x7f4018103200 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f400000cce0 tx=0x7f400000a620 comp rx=0 tx=0).stop 2024-05-09T08:26:56.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 -- 172.21.15.28:0/1855669504 shutdown_connections 2024-05-09T08:26:56.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4018106710 0x7f4018103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40181010c0 0x7f4018103200 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f40040a2d90 0x7f40040a5250 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 --2- 172.21.15.28:0/1855669504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4018100700 0x7f4018102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:56.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 -- 172.21.15.28:0/1855669504 >> 172.21.15.28:0/1855669504 conn(0x7f4018077830 msgr2=0x7f4018068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:56.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 -- 172.21.15.28:0/1855669504 shutdown_connections 2024-05-09T08:26:56.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:26:56.576+0000 7f401f134640 1 -- 172.21.15.28:0/1855669504 wait complete. 2024-05-09T08:26:56.590 INFO:tasks.cephfs.filesystem:scrub status for tag:7bc347d5-66b3-4181-8e92-30f7f4d326ae - {'path': '/', 'tag': '7bc347d5-66b3-4181-8e92-30f7f4d326ae', 'options': 'recursive,force'} 2024-05-09T08:26:57.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:57 smithi028 ceph-mon[30119]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:26:57.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:57 smithi028 ceph-mon[30119]: scrub queued for path: / 2024-05-09T08:26:57.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:57 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:57.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:57 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:26:57.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:57 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:57.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:57 smithi165 ceph-mon[34540]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:26:57.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:57 smithi165 ceph-mon[34540]: scrub queued for path: / 2024-05-09T08:26:57.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:57 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:57.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:57 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:26:57.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:57 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:57.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:57 smithi184 ceph-mon[34509]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:26:57.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:57 smithi184 ceph-mon[34509]: scrub queued for path: / 2024-05-09T08:26:57.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:57 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:57.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:57 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:26:57.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:57 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:26:59.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:26:59 smithi028 ceph-mon[30119]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:26:59.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:26:59 smithi165 ceph-mon[34540]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:26:59.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:26:59 smithi184 ceph-mon[34509]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:27:00.833 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:27:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:27:00.515+0000 7fc73f57b700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:27:00.833 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:27:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:27:00.515+0000 7fc73f57b700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:27:00.833 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:27:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:27:00.515+0000 7fc73f57b700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:27:00.833 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:27:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:27:00.515+0000 7fc73f57b700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:27:00.833 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:27:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:27:00.515+0000 7fc73f57b700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:27:01.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:01 smithi028 ceph-mon[30119]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:27:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:01 smithi028 ceph-mon[30119]: scrub summary: idle 2024-05-09T08:27:01.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:01 smithi165 ceph-mon[34540]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:27:01.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:01 smithi165 ceph-mon[34540]: scrub summary: idle 2024-05-09T08:27:01.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:01 smithi184 ceph-mon[34509]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:27:01.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:01 smithi184 ceph-mon[34509]: scrub summary: idle 2024-05-09T08:27:03.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:03 smithi165 ceph-mon[34540]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:03.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:03 smithi165 ceph-mon[34540]: mgrmap e24: x(active, since 44s), standbys: y, z 2024-05-09T08:27:03.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:03 smithi184 ceph-mon[34509]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:03.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:03 smithi184 ceph-mon[34509]: mgrmap e24: x(active, since 44s), standbys: y, z 2024-05-09T08:27:03.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:03 smithi028 ceph-mon[30119]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:03.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:03 smithi028 ceph-mon[30119]: mgrmap e24: x(active, since 44s), standbys: y, z 2024-05-09T08:27:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.701+0000 7f77fe888640 1 -- 172.21.15.28:0/325319462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f8102ac0 msgr2=0x7f77f8102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.701+0000 7f77fe888640 1 --2- 172.21.15.28:0/325319462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f8102ac0 0x7f77f8102ea0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009900 tx=0x7f77e802a510 comp rx=0 tx=0).stop 2024-05-09T08:27:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.701+0000 7f77fe888640 1 -- 172.21.15.28:0/325319462 shutdown_connections 2024-05-09T08:27:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.701+0000 7f77fe888640 1 --2- 172.21.15.28:0/325319462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77f810be00 0x7f77f810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.701+0000 7f77fe888640 1 --2- 172.21.15.28:0/325319462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f81033e0 0x7f77f810b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.701+0000 7f77fe888640 1 --2- 172.21.15.28:0/325319462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f8102ac0 0x7f77f8102ea0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.701+0000 7f77fe888640 1 -- 172.21.15.28:0/325319462 >> 172.21.15.28:0/325319462 conn(0x7f77f80778b0 msgr2=0x7f77f8077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.701+0000 7f77fe888640 1 -- 172.21.15.28:0/325319462 shutdown_connections 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.701+0000 7f77fe888640 1 -- 172.21.15.28:0/325319462 wait complete. 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.702+0000 7f77fe888640 1 Processor -- start 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.702+0000 7f77fe888640 1 -- start start 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.702+0000 7f77fe888640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f8102ac0 0x7f77f8198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.702+0000 7f77fe888640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77f81033e0 0x7f77f81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.702+0000 7f77fe888640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f810be00 0x7f77f819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.702+0000 7f77fe888640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77f8111c40 con 0x7f77f810be00 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.702+0000 7f77fe888640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f77f8111ac0 con 0x7f77f8102ac0 2024-05-09T08:27:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.702+0000 7f77fe888640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f77f8111dc0 con 0x7f77f81033e0 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fce25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f810be00 0x7f77f819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fce25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f810be00 0x7f77f819c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48388/0 (socket says 172.21.15.28:48388) 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fce25640 1 -- 172.21.15.28:0/167408554 learned_addr learned my addr 172.21.15.28:0/167408554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77f7fff640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f8102ac0 0x7f77f8198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77f77fe640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77f81033e0 0x7f77f81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fce25640 1 -- 172.21.15.28:0/167408554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77f81033e0 msgr2=0x7f77f81986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fce25640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77f81033e0 0x7f77f81986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fce25640 1 -- 172.21.15.28:0/167408554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f8102ac0 msgr2=0x7f77f8198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fce25640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f8102ac0 0x7f77f8198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fce25640 1 -- 172.21.15.28:0/167408554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f819d120 con 0x7f77f810be00 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77f77fe640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77f81033e0 0x7f77f81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fce25640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f810be00 0x7f77f819c9f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f77f000eaa0 tx=0x7f77f000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77f57fa640 1 -- 172.21.15.28:0/167408554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f000cdb0 con 0x7f77f810be00 2024-05-09T08:27:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77fe888640 1 -- 172.21.15.28:0/167408554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f819d410 con 0x7f77f810be00 2024-05-09T08:27:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77f57fa640 1 -- 172.21.15.28:0/167408554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f0005020 con 0x7f77f810be00 2024-05-09T08:27:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.703+0000 7f77f57fa640 1 -- 172.21.15.28:0/167408554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f00116b0 con 0x7f77f810be00 2024-05-09T08:27:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.704+0000 7f77fe888640 1 -- 172.21.15.28:0/167408554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f81a4ca0 con 0x7f77f810be00 2024-05-09T08:27:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.706+0000 7f77f57fa640 1 -- 172.21.15.28:0/167408554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f77f0005560 con 0x7f77f810be00 2024-05-09T08:27:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.706+0000 7f77f57fa640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77e00a2ec0 0x7f77e00a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.707+0000 7f77f7fff640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77e00a2ec0 0x7f77e00a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.707+0000 7f77f57fa640 1 -- 172.21.15.28:0/167408554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f77f00cfde0 con 0x7f77f810be00 2024-05-09T08:27:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.707+0000 7f77f7fff640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77e00a2ec0 0x7f77e00a5380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f77e8005ab0 tx=0x7f77e8005a20 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.707+0000 7f7801592740 1 Processor -- start 2024-05-09T08:27:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.707+0000 7f7801592740 1 -- start start 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f7801592740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923450 0x55a721923830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f7801592740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923d70 0x55a72192e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f7801592740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721930a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f7801592740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a721888340 con 0x55a72192e630 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f7801592740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a721830290 con 0x55a721923d70 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f7801592740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a721884300 con 0x55a721923450 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77fce25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721930a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77fce25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721930a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48402/0 (socket says 172.21.15.28:48402) 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77fce25640 1 -- 172.21.15.28:0/3470062046 learned_addr learned my addr 172.21.15.28:0/3470062046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77f77fe640 1 --2- 172.21.15.28:0/3470062046 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923d70 0x55a72192e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77fce25640 1 -- 172.21.15.28:0/3470062046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923450 msgr2=0x55a721923830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77fce25640 1 --2- 172.21.15.28:0/3470062046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923450 0x55a721923830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77fce25640 1 -- 172.21.15.28:0/3470062046 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923d70 msgr2=0x55a72192e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77fce25640 1 --2- 172.21.15.28:0/3470062046 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923d70 0x55a72192e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77fce25640 1 -- 172.21.15.28:0/3470062046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7218548e0 con 0x55a72192e630 2024-05-09T08:27:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77fce25640 1 --2- 172.21.15.28:0/3470062046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721930a20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f77f00cd0f0 tx=0x7f77f009bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a20c91614612148 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77de7fc640 1 -- 172.21.15.28:0/3470062046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f001f070 con 0x55a72192e630 2024-05-09T08:27:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77de7fc640 1 -- 172.21.15.28:0/3470062046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f00051d0 con 0x55a72192e630 2024-05-09T08:27:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.708+0000 7f77de7fc640 1 -- 172.21.15.28:0/3470062046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f009b330 con 0x55a72192e630 2024-05-09T08:27:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.709+0000 7f7801592740 1 -- 172.21.15.28:0/3470062046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 msgr2=0x55a721930a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.709+0000 7f7801592740 1 --2- 172.21.15.28:0/3470062046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721930a20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f77f00cd0f0 tx=0x7f77f009bda0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.709+0000 7f7801592740 1 -- 172.21.15.28:0/3470062046 shutdown_connections 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.709+0000 7f7801592740 1 --2- 172.21.15.28:0/3470062046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721930a20 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.709+0000 7f7801592740 1 --2- 172.21.15.28:0/3470062046 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923d70 0x55a72192e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.709+0000 7f7801592740 1 --2- 172.21.15.28:0/3470062046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923450 0x55a721923830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.709+0000 7f7801592740 1 -- 172.21.15.28:0/3470062046 >> 172.21.15.28:0/3470062046 conn(0x55a72191f960 msgr2=0x55a72191fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.709+0000 7f7801592740 1 -- 172.21.15.28:0/3470062046 shutdown_connections 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.709+0000 7f7801592740 1 -- 172.21.15.28:0/3470062046 wait complete. 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f7801592740 1 -- start start 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f7801592740 1 Processor -- start 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f7801592740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923450 0x55a72192b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f7801592740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 0x55a72192bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f7801592740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a72192e630 0x55a72192c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f7801592740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a721931260 con 0x55a721923450 2024-05-09T08:27:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f7801592740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a721843790 con 0x55a72192e630 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f7801592740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a7219313e0 con 0x55a721923d70 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f7801592740 10 client.0 _ll_register_callbacks cb 0x55a72187ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f77f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923450 0x55a72192b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f77f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923450 0x55a72192b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48410/0 (socket says 172.21.15.28:48410) 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f77f7fff640 1 -- 172.21.15.28:0/2132295934 learned_addr learned my addr 172.21.15.28:0/2132295934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.710+0000 7f77f77fe640 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 0x55a72192bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77fce25640 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a72192e630 0x55a72192c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77f7fff640 1 -- 172.21.15.28:0/2132295934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 msgr2=0x55a72192bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77f7fff640 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 0x55a72192bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77f7fff640 1 -- 172.21.15.28:0/2132295934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a72192e630 msgr2=0x55a72192c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77f7fff640 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a72192e630 0x55a72192c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77f7fff640 1 -- 172.21.15.28:0/2132295934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a72159c210 con 0x55a721923450 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77f7fff640 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923450 0x55a72192b620 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f77e8033000 tx=0x7f77e80056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf30b5e6312c596f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77d77fe640 1 -- 172.21.15.28:0/2132295934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e804b070 con 0x55a721923450 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77d77fe640 1 -- 172.21.15.28:0/2132295934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f77e8003800 con 0x55a721923450 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f7801592740 1 -- 172.21.15.28:0/2132295934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77f00051d0 con 0x55a721923450 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f77d77fe640 1 -- 172.21.15.28:0/2132295934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f77e8057050 con 0x55a721923450 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f7801592740 10 client.15987 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.711+0000 7f7801592740 1 -- 172.21.15.28:0/2132295934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a721668620 con 0x55a721923450 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.712+0000 7f77d77fe640 1 -- 172.21.15.28:0/2132295934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f00051d0 con 0x55a721923450 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.712+0000 7f77d77fe640 1 -- 172.21.15.28:0/2132295934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f77e8047b60 con 0x55a721923450 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.712+0000 7f7801592740 10 client.15987 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.712+0000 7f7801592740 10 client.15987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.712+0000 7f7801592740 20 client.15987 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.712+0000 7f7801592740 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a72193e3d0 0x55a7219407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.712+0000 7f7801592740 4 client.15987 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.712+0000 7f7801592740 1 -- 172.21.15.28:0/2132295934 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a721895a70 con 0x55a72193e3d0 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.712+0000 7f77f77fe640 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a72193e3d0 0x55a7219407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.713+0000 7f77d77fe640 10 client.15987 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.713+0000 7f77f77fe640 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a72193e3d0 0x55a7219407c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1aefb6c0f3cff412 server_cookie=a811e9f03fe628f6 in_seq=0 out_seq=0 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.713+0000 7f77d77fe640 1 -- 172.21.15.28:0/2132295934 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a721895a70 con 0x55a72193e3d0 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.713+0000 7f77d77fe640 10 client.15987 handle_command_reply: tid=0 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.713+0000 7f7801592740 1 client.15987 shutdown 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.713+0000 7f7801592740 1 -- 172.21.15.28:0/2132295934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923450 msgr2=0x55a72192b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.713+0000 7f7801592740 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923450 0x55a72192b620 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f77e8033000 tx=0x7f77e80056d0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f7801592740 1 -- 172.21.15.28:0/2132295934 shutdown_connections 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f7801592740 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a72193e3d0 0x55a7219407c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f7801592740 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a72192e630 0x55a72192c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f7801592740 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 0x55a72192bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f77d77fe640 0 client.15987 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f7801592740 1 --2- 172.21.15.28:0/2132295934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923450 0x55a72192b620 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f7801592740 1 -- 172.21.15.28:0/2132295934 >> 172.21.15.28:0/2132295934 conn(0x55a72191f960 msgr2=0x55a721925690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f7801592740 1 -- 172.21.15.28:0/2132295934 shutdown_connections 2024-05-09T08:27:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f7801592740 1 -- 172.21.15.28:0/2132295934 wait complete. 2024-05-09T08:27:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.714+0000 7f7801592740 20 client.15987 trim_cache size 0 max 16384 2024-05-09T08:27:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f7801592740 1 Processor -- start 2024-05-09T08:27:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f7801592740 1 -- start start 2024-05-09T08:27:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f7801592740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 0x55a72193a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f7801592740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923d70 0x55a72192bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f7801592740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a72192e630 0x55a721946090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f7801592740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77f81aeed0 con 0x55a721923d70 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f7801592740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f77f81b0a10 con 0x55a721923450 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f7801592740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f77f81af5a0 con 0x55a72192e630 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923d70 0x55a72192bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923d70 0x55a72192bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48420/0 (socket says 172.21.15.28:48420) 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f77fe640 1 -- 172.21.15.28:0/3632677709 learned_addr learned my addr 172.21.15.28:0/3632677709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f7fff640 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 0x55a72193a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f77fe640 1 -- 172.21.15.28:0/3632677709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a72192e630 msgr2=0x55a721946090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77fce25640 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a72192e630 0x55a721946090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f77fe640 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a72192e630 0x55a721946090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f77fe640 1 -- 172.21.15.28:0/3632677709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 msgr2=0x55a72193a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f77fe640 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 0x55a72193a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f77fe640 1 -- 172.21.15.28:0/3632677709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7219206f0 con 0x55a721923d70 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77fce25640 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a72192e630 0x55a721946090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.729+0000 7f77f77fe640 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923d70 0x55a72192bec0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00ba90 tx=0x7f77ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a4e33ae1f2a0f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f77ddffb640 1 -- 172.21.15.28:0/3632677709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec020070 con 0x55a721923d70 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f77ddffb640 1 -- 172.21.15.28:0/3632677709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77ec01cc10 con 0x55a721923d70 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f77ddffb640 1 -- 172.21.15.28:0/3632677709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0143b0 con 0x55a721923d70 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f7801592740 1 -- 172.21.15.28:0/3632677709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923d70 msgr2=0x55a72192bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f7801592740 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923d70 0x55a72192bec0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00ba90 tx=0x7f77ec008040 comp rx=0 tx=0).stop 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f7801592740 1 -- 172.21.15.28:0/3632677709 shutdown_connections 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f7801592740 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a72192e630 0x55a721946090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f7801592740 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a721923d70 0x55a72192bec0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f7801592740 1 --2- 172.21.15.28:0/3632677709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 0x55a72193a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.730+0000 7f7801592740 1 -- 172.21.15.28:0/3632677709 >> 172.21.15.28:0/3632677709 conn(0x55a72191f960 msgr2=0x55a72191ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 -- 172.21.15.28:0/3632677709 shutdown_connections 2024-05-09T08:27:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 -- 172.21.15.28:0/3632677709 wait complete. 2024-05-09T08:27:04.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 -- start start 2024-05-09T08:27:04.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 Processor -- start 2024-05-09T08:27:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 0x55a721936630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 0x55a721936b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721933660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77f81b0340 con 0x55a72192e630 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f77f81afc70 con 0x55a721923450 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f77f81aceb0 con 0x55a721923d70 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f7801592740 10 client.0 _ll_register_callbacks cb 0x7f77e803f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.731+0000 7f77fce25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721933660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77fce25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721933660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48428/0 (socket says 172.21.15.28:48428) 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77fce25640 1 -- 172.21.15.28:0/870977120 learned_addr learned my addr 172.21.15.28:0/870977120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77f7fff640 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 0x55a721936630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77f77fe640 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 0x55a721936b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77fce25640 1 -- 172.21.15.28:0/870977120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 msgr2=0x55a721936b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77fce25640 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 0x55a721936b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77fce25640 1 -- 172.21.15.28:0/870977120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 msgr2=0x55a721936630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77fce25640 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 0x55a721936630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77fce25640 1 -- 172.21.15.28:0/870977120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a721933e90 con 0x55a72192e630 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77fce25640 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721933660 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f77f0099f90 tx=0x7f77f00a4ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae03427bf8151d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77d77fe640 1 -- 172.21.15.28:0/870977120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f001f040 con 0x55a72192e630 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77d77fe640 1 -- 172.21.15.28:0/870977120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f77f00cd4d0 con 0x55a72192e630 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f7801592740 1 -- 172.21.15.28:0/870977120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77ec01cc10 con 0x55a72192e630 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f77d77fe640 1 -- 172.21.15.28:0/870977120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f77f009c3e0 con 0x55a72192e630 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f7801592740 10 client.15993 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.732+0000 7f7801592740 1 -- 172.21.15.28:0/870977120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a7219a9820 con 0x55a72192e630 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f77d77fe640 1 -- 172.21.15.28:0/870977120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77ec01cc10 con 0x55a72192e630 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f77d77fe640 1 -- 172.21.15.28:0/870977120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f77f000f040 con 0x55a72192e630 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f7801592740 10 client.15993 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f7801592740 10 client.15993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f7801592740 20 client.15993 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:04.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f7801592740 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a7219a9b70 0x55a7219abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f7801592740 4 client.15993 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:27:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f7801592740 1 -- 172.21.15.28:0/870977120 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a72188fdc0 con 0x55a7219a9b70 2024-05-09T08:27:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f77f7fff640 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a7219a9b70 0x55a7219abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f77d77fe640 10 client.15993 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:04.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:04.733+0000 7f77f7fff640 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a7219a9b70 0x55a7219abf90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f19d2123bf64ca61 server_cookie=fcc9dbd5dddbe34b in_seq=0 out_seq=0 2024-05-09T08:27:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f77d77fe640 1 -- 172.21.15.28:0/870977120 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a72188fdc0 con 0x55a7219a9b70 2024-05-09T08:27:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f77d77fe640 10 client.15993 handle_command_reply: tid=0 2024-05-09T08:27:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 client.15993 shutdown 2024-05-09T08:27:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 -- 172.21.15.28:0/870977120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 msgr2=0x55a721933660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721933660 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f77f0099f90 tx=0x7f77f00a4ad0 comp rx=0 tx=0).stop 2024-05-09T08:27:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 -- 172.21.15.28:0/870977120 shutdown_connections 2024-05-09T08:27:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a7219a9b70 0x55a7219abf90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a72192e630 0x55a721933660 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a721923d70 0x55a721936b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 --2- 172.21.15.28:0/870977120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a721923450 0x55a721936630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f77d77fe640 0 client.15993 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 -- 172.21.15.28:0/870977120 >> 172.21.15.28:0/870977120 conn(0x55a72191f960 msgr2=0x55a721924e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 -- 172.21.15.28:0/870977120 shutdown_connections 2024-05-09T08:27:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 1 -- 172.21.15.28:0/870977120 wait complete. 2024-05-09T08:27:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.409+0000 7f7801592740 20 client.15993 trim_cache size 0 max 16384 2024-05-09T08:27:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:27:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:27:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:27:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:27:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 -- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77e00a2ec0 msgr2=0x7f77e00a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77e00a2ec0 0x7f77e00a5380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f77e8005ab0 tx=0x7f77e8005a20 comp rx=0 tx=0).stop 2024-05-09T08:27:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 -- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f810be00 msgr2=0x7f77f819c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f810be00 0x7f77f819c9f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f77f000eaa0 tx=0x7f77f000ef60 comp rx=0 tx=0).stop 2024-05-09T08:27:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 -- 172.21.15.28:0/167408554 shutdown_connections 2024-05-09T08:27:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77f810be00 0x7f77f819c9f0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77e00a2ec0 0x7f77e00a5380 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77f81033e0 0x7f77f81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 --2- 172.21.15.28:0/167408554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f8102ac0 0x7f77f8198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 -- 172.21.15.28:0/167408554 >> 172.21.15.28:0/167408554 conn(0x7f77f80778b0 msgr2=0x7f77f80fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 -- 172.21.15.28:0/167408554 shutdown_connections 2024-05-09T08:27:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:05.410+0000 7f77fe888640 1 -- 172.21.15.28:0/167408554 wait complete. 2024-05-09T08:27:05.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:05 smithi028 ceph-mon[30119]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 26 op/s 2024-05-09T08:27:05.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:05 smithi165 ceph-mon[34540]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 26 op/s 2024-05-09T08:27:05.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:05 smithi184 ceph-mon[34509]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 26 op/s 2024-05-09T08:27:07.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:07 smithi028 ceph-mon[30119]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-09T08:27:07.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:07 smithi165 ceph-mon[34540]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-09T08:27:07.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:07 smithi184 ceph-mon[34509]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-09T08:27:09.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:09 smithi028 ceph-mon[30119]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:27:09.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:09 smithi165 ceph-mon[34540]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:27:09.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:09 smithi184 ceph-mon[34509]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:27:11.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:11 smithi028 ceph-mon[30119]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-05-09T08:27:11.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:11 smithi165 ceph-mon[34540]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-05-09T08:27:11.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:11 smithi184 ceph-mon[34509]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-05-09T08:27:13.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:13 smithi028 ceph-mon[30119]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-05-09T08:27:13.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:13 smithi165 ceph-mon[34540]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-05-09T08:27:13.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:13 smithi184 ceph-mon[34509]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-05-09T08:27:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:15 smithi028 ceph-mon[30119]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-05-09T08:27:15.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:15 smithi165 ceph-mon[34540]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-05-09T08:27:15.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:15 smithi184 ceph-mon[34509]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-05-09T08:27:17.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:17 smithi028 ceph-mon[30119]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-05-09T08:27:17.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:17 smithi165 ceph-mon[34540]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-05-09T08:27:17.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:17 smithi184 ceph-mon[34509]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-05-09T08:27:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.549+0000 7f0ef9e58640 1 -- 172.21.15.28:0/4059727569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef4107820 msgr2=0x7f0ef4107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.549+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/4059727569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef4107820 0x7f0ef4107c00 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009930 tx=0x7f0ee402a420 comp rx=0 tx=0).stop 2024-05-09T08:27:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.549+0000 7f0ef9e58640 1 -- 172.21.15.28:0/4059727569 shutdown_connections 2024-05-09T08:27:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.549+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/4059727569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41021d0 0x7f0ef410e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.549+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/4059727569 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef4101810 0x7f0ef4101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.549+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/4059727569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef4107820 0x7f0ef4107c00 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.549+0000 7f0ef9e58640 1 -- 172.21.15.28:0/4059727569 >> 172.21.15.28:0/4059727569 conn(0x7f0ef40fd2e0 msgr2=0x7f0ef40ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.549+0000 7f0ef9e58640 1 -- 172.21.15.28:0/4059727569 shutdown_connections 2024-05-09T08:27:18.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.549+0000 7f0ef9e58640 1 -- 172.21.15.28:0/4059727569 wait complete. 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.550+0000 7f0ef9e58640 1 Processor -- start 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.550+0000 7f0ef9e58640 1 -- start start 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef9e58640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef4101810 0x7f0ef4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef9e58640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41021d0 0x7f0ef4074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef9e58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef4107820 0x7f0ef40745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef9e58640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4111bf0 con 0x7f0ef4107820 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef9e58640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4111a70 con 0x7f0ef41021d0 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef4101810 0x7f0ef4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef4101810 0x7f0ef4073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:49560/0 (socket says 172.21.15.28:49560) 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef37fe640 1 -- 172.21.15.28:0/1141361848 learned_addr learned my addr 172.21.15.28:0/1141361848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef9e58640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4111d70 con 0x7f0ef4101810 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef3fff640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef4107820 0x7f0ef40745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.551+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41021d0 0x7f0ef4074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef37fe640 1 -- 172.21.15.28:0/1141361848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41021d0 msgr2=0x7f0ef4074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41021d0 0x7f0ef4074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef37fe640 1 -- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef4107820 msgr2=0x7f0ef40745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef4107820 0x7f0ef40745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef37fe640 1 -- 172.21.15.28:0/1141361848 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef41a6590 con 0x7f0ef4101810 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef3fff640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef4107820 0x7f0ef40745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41021d0 0x7f0ef4074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef4101810 0x7f0ef4073b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009230 tx=0x7f0ee4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef0ff9640 1 -- 172.21.15.28:0/1141361848 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee403f030 con 0x7f0ef4101810 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef9e58640 1 -- 172.21.15.28:0/1141361848 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef41a6820 con 0x7f0ef4101810 2024-05-09T08:27:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef9e58640 1 -- 172.21.15.28:0/1141361848 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef41a6d60 con 0x7f0ef4101810 2024-05-09T08:27:18.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef0ff9640 1 -- 172.21.15.28:0/1141361848 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee4003ad0 con 0x7f0ef4101810 2024-05-09T08:27:18.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.552+0000 7f0ef0ff9640 1 -- 172.21.15.28:0/1141361848 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4035070 con 0x7f0ef4101810 2024-05-09T08:27:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.556+0000 7f0ef0ff9640 1 -- 172.21.15.28:0/1141361848 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f0ee4042030 con 0x7f0ef4101810 2024-05-09T08:27:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.556+0000 7f0efcb65740 1 Processor -- start 2024-05-09T08:27:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.556+0000 7f0efcb65740 1 -- start start 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076621830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621d70 0x55c07662c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c07662c630 0x55c07662ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0efcb65740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c076586340 con 0x55c07662c630 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0efcb65740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55c07652e290 con 0x55c076621d70 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0efcb65740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c076582300 con 0x55c076621450 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076621830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076621830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:49576/0 (socket says 172.21.15.28:49576) 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef3fff640 1 -- 172.21.15.28:0/602041458 learned_addr learned my addr 172.21.15.28:0/602041458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621d70 0x55c07662c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c07662c630 0x55c07662ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef0ff9640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0edc0a2f40 0x7f0edc0a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef37fe640 1 -- 172.21.15.28:0/602041458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 msgr2=0x55c076621830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076621830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef37fe640 1 -- 172.21.15.28:0/602041458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c07662c630 msgr2=0x55c07662ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c07662c630 0x55c07662ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef37fe640 1 -- 172.21.15.28:0/602041458 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0765528e0 con 0x55c076621d70 2024-05-09T08:27:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.557+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c07662c630 0x55c07662ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.558+0000 7f0ef3fff640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0edc0a2f40 0x7f0edc0a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.558+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621d70 0x55c07662c0f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40ed150 tx=0x7f0ee4086c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4f172d3fb9107c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.558+0000 7f0ef0ff9640 1 -- 172.21.15.28:0/1141361848 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0ee40eff50 con 0x7f0ef4101810 2024-05-09T08:27:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.558+0000 7f0ed9ffb640 1 -- 172.21.15.28:0/602041458 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee403b460 con 0x55c076621d70 2024-05-09T08:27:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.558+0000 7f0ed9ffb640 1 -- 172.21.15.28:0/602041458 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee408ad10 con 0x55c076621d70 2024-05-09T08:27:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0ed9ffb640 1 -- 172.21.15.28:0/602041458 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40bbfd0 con 0x55c076621d70 2024-05-09T08:27:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 -- 172.21.15.28:0/602041458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621d70 msgr2=0x55c07662c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0ef3fff640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0edc0a2f40 0x7f0edc0a5400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009fd0 tx=0x7f0eec004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621d70 0x55c07662c0f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40ed150 tx=0x7f0ee4086c00 comp rx=0 tx=0).stop 2024-05-09T08:27:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 -- 172.21.15.28:0/602041458 shutdown_connections 2024-05-09T08:27:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c07662c630 0x55c07662ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621d70 0x55c07662c0f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 --2- 172.21.15.28:0/602041458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076621830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 -- 172.21.15.28:0/602041458 >> 172.21.15.28:0/602041458 conn(0x55c07661d960 msgr2=0x55c07661dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 -- 172.21.15.28:0/602041458 shutdown_connections 2024-05-09T08:27:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 -- 172.21.15.28:0/602041458 wait complete. 2024-05-09T08:27:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.559+0000 7f0efcb65740 1 -- start start 2024-05-09T08:27:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0efcb65740 1 Processor -- start 2024-05-09T08:27:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621450 0x55c076629510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076629a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c07662c630 0x55c076629f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0efcb65740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c07662f020 con 0x55c076621d70 2024-05-09T08:27:18.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0efcb65740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55c076541790 con 0x55c076621450 2024-05-09T08:27:18.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0efcb65740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c07662f1a0 con 0x55c07662c630 2024-05-09T08:27:18.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0efcb65740 10 client.0 _ll_register_callbacks cb 0x55c07657ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:18.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0ef2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076629a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0ef2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076629a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45194/0 (socket says 172.21.15.28:45194) 2024-05-09T08:27:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0ef2ffd640 1 -- 172.21.15.28:0/720796285 learned_addr learned my addr 172.21.15.28:0/720796285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.560+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621450 0x55c076629510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ef37fe640 1 -- 172.21.15.28:0/720796285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c07662c630 msgr2=0x55c076629f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ef3fff640 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c07662c630 0x55c076629f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c07662c630 0x55c076629f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ef37fe640 1 -- 172.21.15.28:0/720796285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 msgr2=0x55c076629a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076629a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ef37fe640 1 -- 172.21.15.28:0/720796285 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c07629a210 con 0x55c076621450 2024-05-09T08:27:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076629a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ef3fff640 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c07662c630 0x55c076629f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:27:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621450 0x55c076629510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40ed210 tx=0x7f0ee40bb9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90a3e093ab4cc50b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ed2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:27:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/720796285 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4049070 con 0x55c076621450 2024-05-09T08:27:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/720796285 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0ee40ba2d0 con 0x55c076621450 2024-05-09T08:27:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0efcb65740 1 -- 172.21.15.28:0/720796285 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ee408ad10 con 0x55c076621450 2024-05-09T08:27:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/720796285 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0ee403b460 con 0x55c076621450 2024-05-09T08:27:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0efcb65740 10 client.25496 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.561+0000 7f0efcb65740 1 -- 172.21.15.28:0/720796285 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c076366620 con 0x55c076621450 2024-05-09T08:27:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.562+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/720796285 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee408ad10 con 0x55c076621450 2024-05-09T08:27:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.562+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/720796285 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0ee408f1c0 con 0x55c076621450 2024-05-09T08:27:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.562+0000 7f0efcb65740 10 client.25496 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.562+0000 7f0efcb65740 10 client.25496 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.562+0000 7f0efcb65740 20 client.25496 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.562+0000 7f0efcb65740 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c07663be50 0x55c07663e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.562+0000 7f0efcb65740 4 client.25496 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.562+0000 7f0efcb65740 1 -- 172.21.15.28:0/720796285 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c076593a70 con 0x55c07663be50 2024-05-09T08:27:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.562+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c07663be50 0x55c07663e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.563+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c07663be50 0x55c07663e240 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b4264d75acdd798 server_cookie=5fbd79e6a227da94 in_seq=0 out_seq=0 2024-05-09T08:27:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.563+0000 7f0ed2ffd640 10 client.25496 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.563+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/720796285 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c076593a70 con 0x55c07663be50 2024-05-09T08:27:18.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.563+0000 7f0ed2ffd640 10 client.25496 handle_command_reply: tid=0 2024-05-09T08:27:18.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 client.25496 shutdown 2024-05-09T08:27:18.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 -- 172.21.15.28:0/720796285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621450 msgr2=0x55c076629510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:18.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621450 0x55c076629510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40ed210 tx=0x7f0ee40bb9f0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 -- 172.21.15.28:0/720796285 shutdown_connections 2024-05-09T08:27:18.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c07663be50 0x55c07663e240 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0ed2ffd640 0 client.25496 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c07662c630 0x55c076629f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076629a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 --2- 172.21.15.28:0/720796285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c076621450 0x55c076629510 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 -- 172.21.15.28:0/720796285 >> 172.21.15.28:0/720796285 conn(0x55c07661d960 msgr2=0x55c0766236e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 -- 172.21.15.28:0/720796285 shutdown_connections 2024-05-09T08:27:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 1 -- 172.21.15.28:0/720796285 wait complete. 2024-05-09T08:27:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.564+0000 7f0efcb65740 20 client.25496 trim_cache size 0 max 16384 2024-05-09T08:27:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.594+0000 7f0efcb65740 1 Processor -- start 2024-05-09T08:27:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.594+0000 7f0efcb65740 1 -- start start 2024-05-09T08:27:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.594+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076637dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.594+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076623e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.594+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 0x55c076643ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.594+0000 7f0efcb65740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b0ef0 con 0x55c076621d70 2024-05-09T08:27:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.594+0000 7f0efcb65740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b2a30 con 0x55c07662c630 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.594+0000 7f0efcb65740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b15c0 con 0x55c076621450 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076623e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076623e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45204/0 (socket says 172.21.15.28:45204) 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef2ffd640 1 -- 172.21.15.28:0/2880932653 learned_addr learned my addr 172.21.15.28:0/2880932653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef3fff640 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 0x55c076643ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076637dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef2ffd640 1 -- 172.21.15.28:0/2880932653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 msgr2=0x55c076637dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076637dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef2ffd640 1 -- 172.21.15.28:0/2880932653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 msgr2=0x55c076643ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 0x55c076643ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef2ffd640 1 -- 172.21.15.28:0/2880932653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c07663e820 con 0x55c076621d70 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076637dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:18.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.595+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076623e10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8011ab0 tx=0x7f0ee8011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e21fa0478e5c33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:18.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.596+0000 7f0ed97fa640 1 -- 172.21.15.28:0/2880932653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8010b70 con 0x55c076621d70 2024-05-09T08:27:18.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.596+0000 7f0ed97fa640 1 -- 172.21.15.28:0/2880932653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee8005500 con 0x55c076621d70 2024-05-09T08:27:18.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.596+0000 7f0ed97fa640 1 -- 172.21.15.28:0/2880932653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8004e00 con 0x55c076621d70 2024-05-09T08:27:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.597+0000 7f0efcb65740 1 -- 172.21.15.28:0/2880932653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 msgr2=0x55c076623e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.597+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076623e10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8011ab0 tx=0x7f0ee8011f70 comp rx=0 tx=0).stop 2024-05-09T08:27:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.597+0000 7f0efcb65740 1 -- 172.21.15.28:0/2880932653 shutdown_connections 2024-05-09T08:27:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.597+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 0x55c076643ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.597+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076623e10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.597+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2880932653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076637dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.597+0000 7f0efcb65740 1 -- 172.21.15.28:0/2880932653 >> 172.21.15.28:0/2880932653 conn(0x55c07661d960 msgr2=0x55c07661c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.597+0000 7f0efcb65740 1 -- 172.21.15.28:0/2880932653 shutdown_connections 2024-05-09T08:27:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.597+0000 7f0efcb65740 1 -- 172.21.15.28:0/2880932653 wait complete. 2024-05-09T08:27:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.598+0000 7f0efcb65740 1 -- start start 2024-05-09T08:27:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.598+0000 7f0efcb65740 1 Processor -- start 2024-05-09T08:27:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.598+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076634850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.598+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076625fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0efcb65740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 0x55c0766266a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0ef2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076625fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0ef2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076625fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45220/0 (socket says 172.21.15.28:45220) 2024-05-09T08:27:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0ef2ffd640 1 -- 172.21.15.28:0/2631614287 learned_addr learned my addr 172.21.15.28:0/2631614287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076634850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b2360 con 0x55c076621d70 2024-05-09T08:27:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0ef3fff640 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 0x55c0766266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b1c90 con 0x55c07662c630 2024-05-09T08:27:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41aeed0 con 0x55c076621450 2024-05-09T08:27:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0efcb65740 10 client.0 _ll_register_callbacks cb 0x7f0ee800bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0ef2ffd640 1 -- 172.21.15.28:0/2631614287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 msgr2=0x55c076634850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076634850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0ef2ffd640 1 -- 172.21.15.28:0/2631614287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 msgr2=0x55c0766266a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.599+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 0x55c0766266a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0ef2ffd640 1 -- 172.21.15.28:0/2631614287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0766a1330 con 0x55c076621d70 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ee8005500 con 0x55c076621d70 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0ef2ffd640 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076625fb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8011a80 tx=0x7f0ee80056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd67be30d653e33b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0ed2ffd640 10 client.16011 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/2631614287 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8004d80 con 0x55c076621d70 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/2631614287 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0ee8005020 con 0x55c076621d70 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/2631614287 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0ee8005260 con 0x55c076621d70 2024-05-09T08:27:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0efcb65740 10 client.16011 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55c0766a2390 con 0x55c076621d70 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.600+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/2631614287 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0ee8010970 con 0x55c076621d70 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.601+0000 7f0efcb65740 10 client.16011 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.601+0000 7f0efcb65740 10 client.16011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.601+0000 7f0efcb65740 20 client.16011 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.601+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c07663be50 0x55c0766a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.601+0000 7f0efcb65740 4 client.16011 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.601+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c07658ddc0 con 0x55c07663be50 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.601+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c07663be50 0x55c0766a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.601+0000 7f0ed2ffd640 10 client.16011 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.601+0000 7f0ef37fe640 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c07663be50 0x55c0766a4a30 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88c2102e56628f1e server_cookie=436b601b6f38e48e in_seq=0 out_seq=0 2024-05-09T08:27:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0ed2ffd640 1 -- 172.21.15.28:0/2631614287 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c07658ddc0 con 0x55c07663be50 2024-05-09T08:27:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0ed2ffd640 10 client.16011 handle_command_reply: tid=0 2024-05-09T08:27:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 client.16011 shutdown 2024-05-09T08:27:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 msgr2=0x55c076625fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076625fb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8011a80 tx=0x7f0ee80056b0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 shutdown_connections 2024-05-09T08:27:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c07663be50 0x55c0766a4a30 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c07662c630 0x55c0766266a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0ed2ffd640 0 client.16011 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c076621d70 0x55c076625fb0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 --2- 172.21.15.28:0/2631614287 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c076621450 0x55c076634850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 >> 172.21.15.28:0/2631614287 conn(0x55c07661d960 msgr2=0x55c0766294d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 shutdown_connections 2024-05-09T08:27:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 1 -- 172.21.15.28:0/2631614287 wait complete. 2024-05-09T08:27:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.605+0000 7f0efcb65740 20 client.16011 trim_cache size 0 max 16384 2024-05-09T08:27:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:27:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:27:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:27:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 -- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0edc0a2f40 msgr2=0x7f0edc0a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0edc0a2f40 0x7f0edc0a5400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009fd0 tx=0x7f0eec004ab0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 -- 172.21.15.28:0/1141361848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef4101810 msgr2=0x7f0ef4073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef4101810 0x7f0ef4073b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009230 tx=0x7f0ee4004270 comp rx=0 tx=0).stop 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 -- 172.21.15.28:0/1141361848 shutdown_connections 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef4107820 0x7f0ef40745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41021d0 0x7f0ef4074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0edc0a2f40 0x7f0edc0a5400 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 --2- 172.21.15.28:0/1141361848 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ef4101810 0x7f0ef4073b30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 -- 172.21.15.28:0/1141361848 >> 172.21.15.28:0/1141361848 conn(0x7f0ef40fd2e0 msgr2=0x7f0ef40fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.606+0000 7f0ef9e58640 1 -- 172.21.15.28:0/1141361848 shutdown_connections 2024-05-09T08:27:18.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:18.607+0000 7f0ef9e58640 1 -- 172.21.15.28:0/1141361848 wait complete. 2024-05-09T08:27:19.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:19 smithi028 ceph-mon[30119]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 10 op/s 2024-05-09T08:27:19.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:19 smithi165 ceph-mon[34540]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 10 op/s 2024-05-09T08:27:19.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:19 smithi184 ceph-mon[34509]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 10 op/s 2024-05-09T08:27:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:21 smithi028 ceph-mon[30119]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-09T08:27:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:21 smithi165 ceph-mon[34540]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-09T08:27:21.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:21 smithi184 ceph-mon[34509]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-09T08:27:23.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:23 smithi028 ceph-mon[30119]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:27:23.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:23 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:23.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:23 smithi165 ceph-mon[34540]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:27:23.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:23 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:23.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:23 smithi184 ceph-mon[34509]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:27:23.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:23 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:24.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:27:24.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:24.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:24.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:24 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:24 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:27:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:24.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:24 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:27:26.591 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:27:26.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:27:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.703+0000 7fd6809b7640 1 -- 172.21.15.28:0/1995682850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c10bdf0 msgr2=0x7fd67c10e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.703+0000 7fd6809b7640 1 --2- 172.21.15.28:0/1995682850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c10bdf0 0x7fd67c10e290 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd674009ca0 tx=0x7fd67402a9e0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.703+0000 7fd6809b7640 1 -- 172.21.15.28:0/1995682850 shutdown_connections 2024-05-09T08:27:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.703+0000 7fd6809b7640 1 --2- 172.21.15.28:0/1995682850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c10bdf0 0x7fd67c10e290 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.703+0000 7fd6809b7640 1 --2- 172.21.15.28:0/1995682850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd67c1033d0 0x7fd67c10b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.703+0000 7fd6809b7640 1 --2- 172.21.15.28:0/1995682850 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd67c102ab0 0x7fd67c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.703+0000 7fd6809b7640 1 -- 172.21.15.28:0/1995682850 >> 172.21.15.28:0/1995682850 conn(0x7fd67c077830 msgr2=0x7fd67c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:26.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.703+0000 7fd6809b7640 1 -- 172.21.15.28:0/1995682850 shutdown_connections 2024-05-09T08:27:26.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.703+0000 7fd6809b7640 1 -- 172.21.15.28:0/1995682850 wait complete. 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.704+0000 7fd6809b7640 1 Processor -- start 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.704+0000 7fd6809b7640 1 -- start start 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.704+0000 7fd6809b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c102ab0 0x7fd67c198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd6809b7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd67c1033d0 0x7fd67c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd6809b7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd67c10bdf0 0x7fd67c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd6809b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c111bf0 con 0x7fd67c102ab0 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd6809b7640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c111a70 con 0x7fd67c10bdf0 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd6809b7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c111d70 con 0x7fd67c1033d0 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c102ab0 0x7fd67c198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c102ab0 0x7fd67c198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34928/0 (socket says 172.21.15.28:34928) 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67a59c640 1 -- 172.21.15.28:0/896944055 learned_addr learned my addr 172.21.15.28:0/896944055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd67c10bdf0 0x7fd67c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67a59c640 1 -- 172.21.15.28:0/896944055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd67c1033d0 msgr2=0x7fd67c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67a59c640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd67c1033d0 0x7fd67c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67a59c640 1 -- 172.21.15.28:0/896944055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd67c10bdf0 msgr2=0x7fd67c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67a59c640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd67c10bdf0 0x7fd67c19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67a59c640 1 -- 172.21.15.28:0/896944055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c19d0b0 con 0x7fd67c102ab0 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd67c10bdf0 0x7fd67c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.705+0000 7fd67a59c640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c102ab0 0x7fd67c198110 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00e9f0 tx=0x7fd66c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.706+0000 7fd66b7fe640 1 -- 172.21.15.28:0/896944055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c00cd80 con 0x7fd67c102ab0 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.706+0000 7fd6809b7640 1 -- 172.21.15.28:0/896944055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67c19d3a0 con 0x7fd67c102ab0 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.706+0000 7fd66b7fe640 1 -- 172.21.15.28:0/896944055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c005500 con 0x7fd67c102ab0 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.706+0000 7fd66b7fe640 1 -- 172.21.15.28:0/896944055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c010690 con 0x7fd67c102ab0 2024-05-09T08:27:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.706+0000 7fd6809b7640 1 -- 172.21.15.28:0/896944055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67c1a4c30 con 0x7fd67c102ab0 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.709+0000 7fd6837a6740 1 Processor -- start 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.709+0000 7fd6837a6740 1 -- start start 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.709+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d4f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.709+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d59d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.709+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d5c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.709+0000 7fd6837a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dad7cb4d00 con 0x55dad7d4fba0 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.709+0000 7fd6837a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55dad7c5c4c0 con 0x55dad7d5a270 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.709+0000 7fd6837a6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dad7cb0cc0 con 0x55dad7d4f280 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd66b7fe640 1 -- 172.21.15.28:0/896944055 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fd66c0046c0 con 0x7fd67c102ab0 2024-05-09T08:27:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd67ad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d4f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd679d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d5c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd67ad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d4f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42514/0 (socket says 172.21.15.28:42514) 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd67ad9d640 1 -- 172.21.15.28:0/1713285602 learned_addr learned my addr 172.21.15.28:0/1713285602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd67a59c640 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d59d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd679d9b640 1 -- 172.21.15.28:0/1713285602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 msgr2=0x55dad7d4f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd679d9b640 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d4f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd679d9b640 1 -- 172.21.15.28:0/1713285602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 msgr2=0x55dad7d59d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd679d9b640 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d59d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd679d9b640 1 -- 172.21.15.28:0/1713285602 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dad7a927d0 con 0x55dad7d5a270 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d4f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.710+0000 7fd67a59c640 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d59d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.711+0000 7fd66b7fe640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6640a2e20 0x7fd6640a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.711+0000 7fd679d9b640 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d5c660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd670010c40 tx=0x7fd670009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6aeae5479b3b9315 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.711+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6640a2e20 0x7fd6640a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.711+0000 7fd66b7fe640 1 -- 172.21.15.28:0/896944055 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fd66c0d0030 con 0x7fd67c102ab0 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.711+0000 7fd668ff9640 1 -- 172.21.15.28:0/1713285602 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67001e070 con 0x55dad7d5a270 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.711+0000 7fd668ff9640 1 -- 172.21.15.28:0/1713285602 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd670010df0 con 0x55dad7d5a270 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.711+0000 7fd668ff9640 1 -- 172.21.15.28:0/1713285602 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6700196b0 con 0x55dad7d5a270 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 -- 172.21.15.28:0/1713285602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 msgr2=0x55dad7d5c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6640a2e20 0x7fd6640a52e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd674036550 tx=0x7fd67403a040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d5c660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd670010c40 tx=0x7fd670009900 comp rx=0 tx=0).stop 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 -- 172.21.15.28:0/1713285602 shutdown_connections 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d5c660 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d59d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1713285602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d4f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 -- 172.21.15.28:0/1713285602 >> 172.21.15.28:0/1713285602 conn(0x55dad7d4b5a0 msgr2=0x55dad7d4b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 -- 172.21.15.28:0/1713285602 shutdown_connections 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 -- 172.21.15.28:0/1713285602 wait complete. 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.712+0000 7fd6837a6740 1 -- start start 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd6837a6740 1 Processor -- start 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 0x55dad7d57700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d57c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd6837a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dad7d5d060 con 0x55dad7d5a270 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd6837a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55dad7c6fb90 con 0x55dad7d4fba0 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd6837a6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dad7d5d1e0 con 0x55dad7d4f280 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd6837a6740 10 client.0 _ll_register_callbacks cb 0x55dad7cabda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd67a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd67ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d57c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd67a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42520/0 (socket says 172.21.15.28:42520) 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd67a59c640 1 -- 172.21.15.28:0/1573776859 learned_addr learned my addr 172.21.15.28:0/1573776859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd679d9b640 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 0x55dad7d57700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd67a59c640 1 -- 172.21.15.28:0/1573776859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 msgr2=0x55dad7d57700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd67a59c640 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 0x55dad7d57700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.713+0000 7fd67a59c640 1 -- 172.21.15.28:0/1573776859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 msgr2=0x55dad7d57c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd67a59c640 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d57c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd67a59c640 1 -- 172.21.15.28:0/1573776859 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dad79c83f0 con 0x55dad7d4f280 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d57c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd67a59c640 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d571c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c6a0 tx=0x7fd66c0981d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9586e18e3e6b839d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd661ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd661ffb640 1 -- 172.21.15.28:0/1573776859 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c09b890 con 0x55dad7d4f280 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd6837a6740 1 -- 172.21.15.28:0/1573776859 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd670010df0 con 0x55dad7d4f280 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd661ffb640 1 -- 172.21.15.28:0/1573776859 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fd66c0d22d0 con 0x55dad7d4f280 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd661ffb640 1 -- 172.21.15.28:0/1573776859 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd66c09c630 con 0x55dad7d4f280 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd6837a6740 10 client.25450 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.714+0000 7fd6837a6740 1 -- 172.21.15.28:0/1573776859 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dad79880f0 con 0x55dad7d4f280 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.715+0000 7fd661ffb640 1 -- 172.21.15.28:0/1573776859 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c0b7030 con 0x55dad7d4f280 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.715+0000 7fd661ffb640 1 -- 172.21.15.28:0/1573776859 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd66c0cd510 con 0x55dad7d4f280 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.715+0000 7fd6837a6740 10 client.25450 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.715+0000 7fd6837a6740 10 client.25450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.715+0000 7fd6837a6740 20 client.25450 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.715+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dad7d6aa10 0x55dad7d6ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.715+0000 7fd6837a6740 4 client.25450 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.715+0000 7fd6837a6740 1 -- 172.21.15.28:0/1573776859 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dad7cc2430 con 0x55dad7d6aa10 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.715+0000 7fd679d9b640 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dad7d6aa10 0x55dad7d6ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.716+0000 7fd661ffb640 10 client.25450 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.716+0000 7fd679d9b640 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dad7d6aa10 0x55dad7d6ce60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=819380f6efa3bb35 server_cookie=43a6246592a7c1d4 in_seq=0 out_seq=0 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.716+0000 7fd661ffb640 1 -- 172.21.15.28:0/1573776859 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dad7cc2430 con 0x55dad7d6aa10 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.716+0000 7fd661ffb640 10 client.25450 handle_command_reply: tid=0 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.716+0000 7fd6837a6740 1 client.25450 shutdown 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 -- 172.21.15.28:0/1573776859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 msgr2=0x55dad7d571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d571c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c6a0 tx=0x7fd66c0981d0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 -- 172.21.15.28:0/1573776859 shutdown_connections 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dad7d6aa10 0x55dad7d6ce60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d57c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd661ffb640 0 client.25450 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 0x55dad7d57700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1573776859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d571c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 -- 172.21.15.28:0/1573776859 >> 172.21.15.28:0/1573776859 conn(0x55dad7d4b5a0 msgr2=0x55dad7d513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 -- 172.21.15.28:0/1573776859 shutdown_connections 2024-05-09T08:27:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 1 -- 172.21.15.28:0/1573776859 wait complete. 2024-05-09T08:27:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.717+0000 7fd6837a6740 20 client.25450 trim_cache size 0 max 16384 2024-05-09T08:27:26.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.752+0000 7fd6837a6740 1 Processor -- start 2024-05-09T08:27:26.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.752+0000 7fd6837a6740 1 -- start start 2024-05-09T08:27:26.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d65f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d51a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d72740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd6837a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1accf0 con 0x55dad7d4fba0 2024-05-09T08:27:26.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd6837a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1ae830 con 0x55dad7d5a270 2024-05-09T08:27:26.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd6837a6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1ad3c0 con 0x55dad7d4f280 2024-05-09T08:27:26.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd679d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d51a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd679d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d51a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34962/0 (socket says 172.21.15.28:34962) 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd679d9b640 1 -- 172.21.15.28:0/1314670930 learned_addr learned my addr 172.21.15.28:0/1314670930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d72740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd679d9b640 1 -- 172.21.15.28:0/1314670930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 msgr2=0x55dad7d65f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd67a59c640 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d65f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd679d9b640 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d65f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd679d9b640 1 -- 172.21.15.28:0/1314670930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 msgr2=0x55dad7d72740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd679d9b640 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d72740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.753+0000 7fd679d9b640 1 -- 172.21.15.28:0/1314670930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dad7d55730 con 0x55dad7d4fba0 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.754+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d72740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.754+0000 7fd679d9b640 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d51a60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd670013010 tx=0x7fd670003d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914cdbe5d4c0a312 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.754+0000 7fd663fff640 1 -- 172.21.15.28:0/1314670930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67001e040 con 0x55dad7d4fba0 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.754+0000 7fd663fff640 1 -- 172.21.15.28:0/1314670930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd67001f040 con 0x55dad7d4fba0 2024-05-09T08:27:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.754+0000 7fd663fff640 1 -- 172.21.15.28:0/1314670930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670004060 con 0x55dad7d4fba0 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.755+0000 7fd6837a6740 1 -- 172.21.15.28:0/1314670930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 msgr2=0x55dad7d51a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.755+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d51a60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd670013010 tx=0x7fd670003d90 comp rx=0 tx=0).stop 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.755+0000 7fd6837a6740 1 -- 172.21.15.28:0/1314670930 shutdown_connections 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.755+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d5a270 0x55dad7d72740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.755+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d4fba0 0x55dad7d51a60 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.755+0000 7fd6837a6740 1 --2- 172.21.15.28:0/1314670930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d65f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.755+0000 7fd6837a6740 1 -- 172.21.15.28:0/1314670930 >> 172.21.15.28:0/1314670930 conn(0x55dad7d4b5a0 msgr2=0x55dad7d57a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.755+0000 7fd6837a6740 1 -- 172.21.15.28:0/1314670930 shutdown_connections 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.755+0000 7fd6837a6740 1 -- 172.21.15.28:0/1314670930 wait complete. 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.756+0000 7fd6837a6740 1 -- start start 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.756+0000 7fd6837a6740 1 Processor -- start 2024-05-09T08:27:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d62700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 0x55dad7d53a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd6837a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d53fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd6837a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1ae160 con 0x55dad7d5a270 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd6837a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1ada90 con 0x55dad7d4fba0 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd6837a6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1aacd0 con 0x55dad7d4f280 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd67ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d53fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd67ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d53fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34974/0 (socket says 172.21.15.28:34974) 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd679d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 0x55dad7d53a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd67ad9d640 1 -- 172.21.15.28:0/711752179 learned_addr learned my addr 172.21.15.28:0/711752179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd67a59c640 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d62700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd67ad9d640 1 -- 172.21.15.28:0/711752179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 msgr2=0x55dad7d62700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d62700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd67ad9d640 1 -- 172.21.15.28:0/711752179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 msgr2=0x55dad7d53a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:26.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 0x55dad7d53a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd67ad9d640 1 -- 172.21.15.28:0/711752179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dad7d54710 con 0x55dad7d5a270 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd6837a6740 10 client.0 _ll_register_callbacks cb 0x7fd66c004ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.757+0000 7fd6837a6740 1 -- 172.21.15.28:0/711752179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd67001f040 con 0x55dad7d5a270 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd67ad9d640 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d53fd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd674004af0 tx=0x7fd67402a9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d839e589a135662d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd661ffb640 10 client.16029 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd661ffb640 1 -- 172.21.15.28:0/711752179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674056020 con 0x55dad7d5a270 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd661ffb640 1 -- 172.21.15.28:0/711752179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fd6740365d0 con 0x55dad7d5a270 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd661ffb640 1 -- 172.21.15.28:0/711752179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd674033040 con 0x55dad7d5a270 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd6837a6740 10 client.16029 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd6837a6740 1 -- 172.21.15.28:0/711752179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55dad7dd0dc0 con 0x55dad7d5a270 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd661ffb640 1 -- 172.21.15.28:0/711752179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd674063020 con 0x55dad7d5a270 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd6837a6740 10 client.16029 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd6837a6740 10 client.16029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.758+0000 7fd6837a6740 20 client.16029 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.759+0000 7fd6837a6740 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dad7d6aa10 0x55dad7dd34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.759+0000 7fd6837a6740 4 client.16029 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.759+0000 7fd6837a6740 1 -- 172.21.15.28:0/711752179 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dad7cbc780 con 0x55dad7d6aa10 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.759+0000 7fd67a59c640 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dad7d6aa10 0x55dad7dd34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.759+0000 7fd661ffb640 10 client.16029 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.759+0000 7fd67a59c640 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dad7d6aa10 0x55dad7dd34c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f3d06af2210d376 server_cookie=16493eb5ec44602 in_seq=0 out_seq=0 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd661ffb640 1 -- 172.21.15.28:0/711752179 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dad7cbc780 con 0x55dad7d6aa10 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd661ffb640 10 client.16029 handle_command_reply: tid=0 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 client.16029 shutdown 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 -- 172.21.15.28:0/711752179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 msgr2=0x55dad7d53fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d53fd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd674004af0 tx=0x7fd67402a9e0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 -- 172.21.15.28:0/711752179 shutdown_connections 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dad7d6aa10 0x55dad7dd34c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd661ffb640 0 client.16029 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dad7d5a270 0x55dad7d53fd0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dad7d4fba0 0x55dad7d53a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 --2- 172.21.15.28:0/711752179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dad7d4f280 0x55dad7d62700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 -- 172.21.15.28:0/711752179 >> 172.21.15.28:0/711752179 conn(0x55dad7d4b5a0 msgr2=0x55dad7d49f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 -- 172.21.15.28:0/711752179 shutdown_connections 2024-05-09T08:27:26.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 1 -- 172.21.15.28:0/711752179 wait complete. 2024-05-09T08:27:26.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6837a6740 20 client.16029 trim_cache size 0 max 16384 2024-05-09T08:27:26.765 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:27:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-09T08:27:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-09T08:27:26.765 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:27:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6809b7640 1 -- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6640a2e20 msgr2=0x7fd6640a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.760+0000 7fd6809b7640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6640a2e20 0x7fd6640a52e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd674036550 tx=0x7fd67403a040 comp rx=0 tx=0).stop 2024-05-09T08:27:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 -- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c102ab0 msgr2=0x7fd67c198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c102ab0 0x7fd67c198110 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00e9f0 tx=0x7fd66c00edf0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 -- 172.21.15.28:0/896944055 shutdown_connections 2024-05-09T08:27:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd67c10bdf0 0x7fd67c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6640a2e20 0x7fd6640a52e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd67c1033d0 0x7fd67c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 --2- 172.21.15.28:0/896944055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c102ab0 0x7fd67c198110 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 -- 172.21.15.28:0/896944055 >> 172.21.15.28:0/896944055 conn(0x7fd67c077830 msgr2=0x7fd67c0fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:26.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 -- 172.21.15.28:0/896944055 shutdown_connections 2024-05-09T08:27:26.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.761+0000 7fd6809b7640 1 -- 172.21.15.28:0/896944055 wait complete. 2024-05-09T08:27:26.773 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:27:26.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:27:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.882+0000 7f1ddf293640 1 -- 172.21.15.28:0/1306016300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd8102ac0 msgr2=0x7f1dd8102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.882+0000 7f1ddf293640 1 --2- 172.21.15.28:0/1306016300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd8102ac0 0x7f1dd8102ea0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009900 tx=0x7f1dd402a510 comp rx=0 tx=0).stop 2024-05-09T08:27:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.882+0000 7f1ddf293640 1 -- 172.21.15.28:0/1306016300 shutdown_connections 2024-05-09T08:27:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.882+0000 7f1ddf293640 1 --2- 172.21.15.28:0/1306016300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dd810be00 0x7f1dd810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.882+0000 7f1ddf293640 1 --2- 172.21.15.28:0/1306016300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dd81033e0 0x7f1dd810b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.882+0000 7f1ddf293640 1 --2- 172.21.15.28:0/1306016300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd8102ac0 0x7f1dd8102ea0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.882+0000 7f1ddf293640 1 -- 172.21.15.28:0/1306016300 >> 172.21.15.28:0/1306016300 conn(0x7f1dd8077830 msgr2=0x7f1dd8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.882+0000 7f1ddf293640 1 -- 172.21.15.28:0/1306016300 shutdown_connections 2024-05-09T08:27:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.882+0000 7f1ddf293640 1 -- 172.21.15.28:0/1306016300 wait complete. 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddf293640 1 Processor -- start 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddf293640 1 -- start start 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddf293640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dd81033e0 0x7f1dd8100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddf293640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dd810be00 0x7f1dd8100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddf293640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd81023b0 0x7f1dd8101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddf293640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd8111c00 con 0x7f1dd81023b0 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddf293640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd8111a80 con 0x7f1dd81033e0 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddf293640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd8111d80 con 0x7f1dd810be00 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddd830640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd81023b0 0x7f1dd8101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddd830640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd81023b0 0x7f1dd8101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34984/0 (socket says 172.21.15.28:34984) 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddd830640 1 -- 172.21.15.28:0/3430416059 learned_addr learned my addr 172.21.15.28:0/3430416059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddc82e640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dd810be00 0x7f1dd8100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.883+0000 7f1ddd02f640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dd81033e0 0x7f1dd8100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1ddd830640 1 -- 172.21.15.28:0/3430416059 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dd810be00 msgr2=0x7f1dd8100c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1ddd830640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dd810be00 0x7f1dd8100c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1ddd830640 1 -- 172.21.15.28:0/3430416059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dd81033e0 msgr2=0x7f1dd8100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1ddd830640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dd81033e0 0x7f1dd8100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1ddd830640 1 -- 172.21.15.28:0/3430416059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd81ab060 con 0x7f1dd81023b0 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1ddd02f640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dd81033e0 0x7f1dd8100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1ddd830640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd81023b0 0x7f1dd8101190 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000efd0 tx=0x7f1dd000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1dce7fc640 1 -- 172.21.15.28:0/3430416059 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0003b70 con 0x7f1dd81023b0 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd81ab350 con 0x7f1dd81023b0 2024-05-09T08:27:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1dce7fc640 1 -- 172.21.15.28:0/3430416059 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dd0003d10 con 0x7f1dd81023b0 2024-05-09T08:27:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1dce7fc640 1 -- 172.21.15.28:0/3430416059 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd00109f0 con 0x7f1dd81023b0 2024-05-09T08:27:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.884+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd81ab8f0 con 0x7f1dd81023b0 2024-05-09T08:27:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.885+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dd80691c0 con 0x7f1dd81023b0 2024-05-09T08:27:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.887+0000 7f1dce7fc640 1 -- 172.21.15.28:0/3430416059 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f1dd0010b90 con 0x7f1dd81023b0 2024-05-09T08:27:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.887+0000 7f1dce7fc640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1dc40a2ea0 0x7f1dc40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.888+0000 7f1dce7fc640 1 -- 172.21.15.28:0/3430416059 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f1dd00cfa60 con 0x7f1dd81023b0 2024-05-09T08:27:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.888+0000 7f1ddd02f640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1dc40a2ea0 0x7f1dc40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.888+0000 7f1ddd02f640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1dc40a2ea0 0x7f1dc40a5360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009230 tx=0x7f1dd4005a20 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:26.892+0000 7f1dce7fc640 1 -- 172.21.15.28:0/3430416059 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1dd009c060 con 0x7f1dd81023b0 2024-05-09T08:27:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.135+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1dd8106f80 con 0x7f1dd81023b0 2024-05-09T08:27:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.136+0000 7f1dce7fc640 1 -- 172.21.15.28:0/3430416059 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11932 (secure 0 0 0) 0x7f1dd009b960 con 0x7f1dd81023b0 2024-05-09T08:27:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:27:27.137 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:26:16.129765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24463,24512,24476,24481,24499]},"id":1}]} 2024-05-09T08:27:27.137 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-05-09T08:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1dc40a2ea0 msgr2=0x7f1dc40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1dc40a2ea0 0x7f1dc40a5360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009230 tx=0x7f1dd4005a20 comp rx=0 tx=0).stop 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd81023b0 msgr2=0x7f1dd8101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd81023b0 0x7f1dd8101190 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000efd0 tx=0x7f1dd000c6a0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 shutdown_connections 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dd81023b0 0x7f1dd8101190 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1dc40a2ea0 0x7f1dc40a5360 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dd810be00 0x7f1dd8100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 --2- 172.21.15.28:0/3430416059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dd81033e0 0x7f1dd8100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 >> 172.21.15.28:0/3430416059 conn(0x7f1dd8077830 msgr2=0x7f1dd80fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 shutdown_connections 2024-05-09T08:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.138+0000 7f1ddf293640 1 -- 172.21.15.28:0/3430416059 wait complete. 2024-05-09T08:27:27.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:27:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.263+0000 7ff34a17d640 1 -- 172.21.15.28:0/1655350229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3441010c0 msgr2=0x7ff34410d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.263+0000 7ff34a17d640 1 --2- 172.21.15.28:0/1655350229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3441010c0 0x7ff34410d5d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff33c009cd0 tx=0x7ff33c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.263+0000 7ff34a17d640 1 -- 172.21.15.28:0/1655350229 shutdown_connections 2024-05-09T08:27:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.263+0000 7ff34a17d640 1 --2- 172.21.15.28:0/1655350229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3441010c0 0x7ff34410d5d0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.263+0000 7ff34a17d640 1 --2- 172.21.15.28:0/1655350229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff344100700 0x7ff344100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.263+0000 7ff34a17d640 1 --2- 172.21.15.28:0/1655350229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff344106710 0x7ff344106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.263+0000 7ff34a17d640 1 -- 172.21.15.28:0/1655350229 >> 172.21.15.28:0/1655350229 conn(0x7ff344077830 msgr2=0x7ff344077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.263+0000 7ff34a17d640 1 -- 172.21.15.28:0/1655350229 shutdown_connections 2024-05-09T08:27:27.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.263+0000 7ff34a17d640 1 -- 172.21.15.28:0/1655350229 wait complete. 2024-05-09T08:27:27.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.264+0000 7ff34a17d640 1 Processor -- start 2024-05-09T08:27:27.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.264+0000 7ff34a17d640 1 -- start start 2024-05-09T08:27:27.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.264+0000 7ff34a17d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff344100700 0x7ff34419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.264+0000 7ff34a17d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3441010c0 0x7ff34419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.264+0000 7ff34a17d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff344106710 0x7ff34419e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.264+0000 7ff34a17d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff34410fa60 con 0x7ff344106710 2024-05-09T08:27:27.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.264+0000 7ff34a17d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff34410f8e0 con 0x7ff344100700 2024-05-09T08:27:27.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.264+0000 7ff34a17d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff34410fbe0 con 0x7ff3441010c0 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff343fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff344106710 0x7ff34419e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff343fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff344106710 0x7ff34419e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35014/0 (socket says 172.21.15.28:35014) 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff343fff640 1 -- 172.21.15.28:0/3378236192 learned_addr learned my addr 172.21.15.28:0/3378236192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff3437fe640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff344100700 0x7ff34419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff343fff640 1 -- 172.21.15.28:0/3378236192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3441010c0 msgr2=0x7ff34419a610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff343fff640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3441010c0 0x7ff34419a610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff343fff640 1 -- 172.21.15.28:0/3378236192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff344100700 msgr2=0x7ff34419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff343fff640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff344100700 0x7ff34419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff343fff640 1 -- 172.21.15.28:0/3378236192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34419f0c0 con 0x7ff344106710 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.265+0000 7ff343fff640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff344106710 0x7ff34419e990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff33c005e30 tx=0x7ff33c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.266+0000 7ff340ff9640 1 -- 172.21.15.28:0/3378236192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c004240 con 0x7ff344106710 2024-05-09T08:27:27.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.266+0000 7ff340ff9640 1 -- 172.21.15.28:0/3378236192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff33c0305a0 con 0x7ff344106710 2024-05-09T08:27:27.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.266+0000 7ff340ff9640 1 -- 172.21.15.28:0/3378236192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c03a680 con 0x7ff344106710 2024-05-09T08:27:27.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.266+0000 7ff34a17d640 1 -- 172.21.15.28:0/3378236192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff34419f350 con 0x7ff344106710 2024-05-09T08:27:27.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.267+0000 7ff34a17d640 1 -- 172.21.15.28:0/3378236192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3441a6c40 con 0x7ff344106710 2024-05-09T08:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.270+0000 7ff34ce85740 1 Processor -- start 2024-05-09T08:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.270+0000 7ff340ff9640 1 -- 172.21.15.28:0/3378236192 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7ff33c0043e0 con 0x7ff344106710 2024-05-09T08:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff34ce85740 1 -- start start 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde067cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 0x55fde06875f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde0689f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff34ce85740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fde05d9c40 con 0x55fde067cb40 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff34ce85740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fde0581580 con 0x55fde067d460 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff34ce85740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fde05d5c00 con 0x55fde0687b30 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff342ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde067cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff342ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde067cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35024/0 (socket says 172.21.15.28:35024) 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff342ffd640 1 -- 172.21.15.28:0/2977570227 learned_addr learned my addr 172.21.15.28:0/2977570227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff3437fe640 1 --2- 172.21.15.28:0/2977570227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 0x55fde06875f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff342ffd640 1 --2- 172.21.15.28:0/2977570227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde0689f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff342ffd640 1 -- 172.21.15.28:0/2977570227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 msgr2=0x55fde0689f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff342ffd640 1 --2- 172.21.15.28:0/2977570227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde0689f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff342ffd640 1 -- 172.21.15.28:0/2977570227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 msgr2=0x55fde06875f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff342ffd640 1 --2- 172.21.15.28:0/2977570227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 0x55fde06875f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.271+0000 7ff342ffd640 1 -- 172.21.15.28:0/2977570227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde02ed3f0 con 0x55fde067cb40 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.272+0000 7ff342ffd640 1 --2- 172.21.15.28:0/2977570227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde067cf20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff338009f30 tx=0x7ff338009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bd52229fc32576 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.272+0000 7ff329ffb640 1 -- 172.21.15.28:0/2977570227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338018030 con 0x55fde067cb40 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.272+0000 7ff329ffb640 1 -- 172.21.15.28:0/2977570227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff338005500 con 0x55fde067cb40 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.272+0000 7ff329ffb640 1 -- 172.21.15.28:0/2977570227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338012450 con 0x55fde067cb40 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.272+0000 7ff340ff9640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff32c0a2d70 0x7ff32c0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.272+0000 7ff3437fe640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff32c0a2d70 0x7ff32c0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 -- 172.21.15.28:0/2977570227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 msgr2=0x55fde067cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 --2- 172.21.15.28:0/2977570227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde067cf20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff338009f30 tx=0x7ff338009760 comp rx=0 tx=0).stop 2024-05-09T08:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff340ff9640 1 -- 172.21.15.28:0/3378236192 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7ff33c0f0780 con 0x7ff344106710 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 -- 172.21.15.28:0/2977570227 shutdown_connections 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 --2- 172.21.15.28:0/2977570227 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde0689f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 --2- 172.21.15.28:0/2977570227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 0x55fde06875f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 --2- 172.21.15.28:0/2977570227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde067cf20 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 -- 172.21.15.28:0/2977570227 >> 172.21.15.28:0/2977570227 conn(0x55fde0678e60 msgr2=0x55fde0679240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff3437fe640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff32c0a2d70 0x7ff32c0a5230 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55fde067e100 tx=0x7ff334005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 -- 172.21.15.28:0/2977570227 shutdown_connections 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 -- 172.21.15.28:0/2977570227 wait complete. 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.273+0000 7ff34ce85740 1 -- start start 2024-05-09T08:27:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff34ce85740 1 Processor -- start 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 0x55fde0684990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde067d460 0x55fde0684ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde0687b30 0x55fde0685410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff34ce85740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fde068a970 con 0x55fde0687b30 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff34ce85740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fde0594c40 con 0x55fde067cb40 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff34ce85740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fde068aaf0 con 0x55fde067d460 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff34ce85740 10 client.0 _ll_register_callbacks cb 0x55fde02ecbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff3437fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 0x55fde0684990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff342ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde067d460 0x55fde0684ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff3437fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 0x55fde0684990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35576/0 (socket says 172.21.15.28:35576) 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff3437fe640 1 -- 172.21.15.28:0/996125890 learned_addr learned my addr 172.21.15.28:0/996125890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.274+0000 7ff343fff640 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde0687b30 0x55fde0685410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff342ffd640 1 -- 172.21.15.28:0/996125890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 msgr2=0x55fde0684990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff342ffd640 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 0x55fde0684990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff342ffd640 1 -- 172.21.15.28:0/996125890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde0687b30 msgr2=0x55fde0685410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff342ffd640 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde0687b30 0x55fde0685410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff342ffd640 1 -- 172.21.15.28:0/996125890 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde03b77d0 con 0x55fde067d460 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff3437fe640 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 0x55fde0684990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff343fff640 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde0687b30 0x55fde0685410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff342ffd640 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde067d460 0x55fde0684ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff338009f00 tx=0x7ff338005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe49594109e42868 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff322ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff322ffd640 1 -- 172.21.15.28:0/996125890 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338018070 con 0x55fde067d460 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff34ce85740 1 -- 172.21.15.28:0/996125890 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff338005500 con 0x55fde067d460 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.275+0000 7ff322ffd640 1 -- 172.21.15.28:0/996125890 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7ff338004050 con 0x55fde067d460 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff322ffd640 1 -- 172.21.15.28:0/996125890 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff338031050 con 0x55fde067d460 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff34ce85740 10 client.25468 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff34ce85740 1 -- 172.21.15.28:0/996125890 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fde0696aa0 con 0x55fde067d460 2024-05-09T08:27:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff322ffd640 1 -- 172.21.15.28:0/996125890 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff338005500 con 0x55fde067d460 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff322ffd640 1 -- 172.21.15.28:0/996125890 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff338003ad0 con 0x55fde067d460 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff34ce85740 10 client.25468 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff34ce85740 10 client.25468 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff34ce85740 20 client.25468 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff34ce85740 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55fde0697eb0 0x55fde069a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff34ce85740 4 client.25468 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff34ce85740 1 -- 172.21.15.28:0/996125890 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fde05e7370 con 0x55fde0697eb0 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.276+0000 7ff3437fe640 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55fde0697eb0 0x55fde069a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.277+0000 7ff322ffd640 10 client.25468 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.277+0000 7ff3437fe640 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55fde0697eb0 0x55fde069a2a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e3054d0b55f8a12e server_cookie=df4c9f3ac1a4ad98 in_seq=0 out_seq=0 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.277+0000 7ff322ffd640 1 -- 172.21.15.28:0/996125890 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fde05e7370 con 0x55fde0697eb0 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.277+0000 7ff322ffd640 10 client.25468 handle_command_reply: tid=0 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.277+0000 7ff34ce85740 1 client.25468 shutdown 2024-05-09T08:27:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 -- 172.21.15.28:0/996125890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde067d460 msgr2=0x55fde0684ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde067d460 0x55fde0684ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff338009f00 tx=0x7ff338005d30 comp rx=0 tx=0).stop 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 -- 172.21.15.28:0/996125890 shutdown_connections 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde0687b30 0x55fde0685410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55fde0697eb0 0x55fde069a2a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde067d460 0x55fde0684ed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff322ffd640 0 client.25468 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 --2- 172.21.15.28:0/996125890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 0x55fde0684990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 -- 172.21.15.28:0/996125890 >> 172.21.15.28:0/996125890 conn(0x55fde0678e60 msgr2=0x55fde066f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 -- 172.21.15.28:0/996125890 shutdown_connections 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 1 -- 172.21.15.28:0/996125890 wait complete. 2024-05-09T08:27:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.278+0000 7ff34ce85740 20 client.25468 trim_cache size 0 max 16384 2024-05-09T08:27:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff34ce85740 1 Processor -- start 2024-05-09T08:27:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff34ce85740 1 -- start start 2024-05-09T08:27:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde0693890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 0x55fde069f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde069fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff34ce85740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3441ac740 con 0x55fde067cb40 2024-05-09T08:27:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff34ce85740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff3441ae280 con 0x55fde067d460 2024-05-09T08:27:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff34ce85740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff3441ace10 con 0x55fde0687b30 2024-05-09T08:27:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff3437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde0693890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff3437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde0693890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35048/0 (socket says 172.21.15.28:35048) 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff3437fe640 1 -- 172.21.15.28:0/153109238 learned_addr learned my addr 172.21.15.28:0/153109238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.305+0000 7ff342ffd640 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 0x55fde069f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff3437fe640 1 -- 172.21.15.28:0/153109238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 msgr2=0x55fde069fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff343fff640 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde069fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff3437fe640 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde069fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff3437fe640 1 -- 172.21.15.28:0/153109238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 msgr2=0x55fde069f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff3437fe640 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 0x55fde069f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff3437fe640 1 -- 172.21.15.28:0/153109238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde0661d20 con 0x55fde067cb40 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff342ffd640 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 0x55fde069f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff3437fe640 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde0693890 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff334003640 tx=0x7ff33401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d853d5cf71d597 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff3297fa640 1 -- 172.21.15.28:0/153109238 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33402b530 con 0x55fde067cb40 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff3297fa640 1 -- 172.21.15.28:0/153109238 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff334005500 con 0x55fde067cb40 2024-05-09T08:27:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.306+0000 7ff3297fa640 1 -- 172.21.15.28:0/153109238 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff334004c40 con 0x55fde067cb40 2024-05-09T08:27:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.307+0000 7ff34ce85740 1 -- 172.21.15.28:0/153109238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 msgr2=0x55fde0693890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.307+0000 7ff34ce85740 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde0693890 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff334003640 tx=0x7ff33401b000 comp rx=0 tx=0).stop 2024-05-09T08:27:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.307+0000 7ff34ce85740 1 -- 172.21.15.28:0/153109238 shutdown_connections 2024-05-09T08:27:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.307+0000 7ff34ce85740 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde069fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.307+0000 7ff34ce85740 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067d460 0x55fde069f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.307+0000 7ff34ce85740 1 --2- 172.21.15.28:0/153109238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067cb40 0x55fde0693890 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.307+0000 7ff34ce85740 1 -- 172.21.15.28:0/153109238 >> 172.21.15.28:0/153109238 conn(0x55fde0678e60 msgr2=0x55fde0683280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.307+0000 7ff34ce85740 1 -- 172.21.15.28:0/153109238 shutdown_connections 2024-05-09T08:27:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.307+0000 7ff34ce85740 1 -- 172.21.15.28:0/153109238 wait complete. 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff34ce85740 1 -- start start 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff34ce85740 1 Processor -- start 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 0x55fde068fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067d460 0x55fde06902e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff34ce85740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde06810c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff34ce85740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3441adbb0 con 0x55fde067d460 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff34ce85740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff3441ad4e0 con 0x55fde067cb40 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff34ce85740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff3441aa760 con 0x55fde0687b30 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff34ce85740 10 client.0 _ll_register_callbacks cb 0x7ff334009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff342ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067d460 0x55fde06902e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.308+0000 7ff342ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067d460 0x55fde06902e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35052/0 (socket says 172.21.15.28:35052) 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff342ffd640 1 -- 172.21.15.28:0/1666331764 learned_addr learned my addr 172.21.15.28:0/1666331764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff343fff640 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde06810c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff342ffd640 1 -- 172.21.15.28:0/1666331764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 msgr2=0x55fde06810c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff342ffd640 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde06810c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff342ffd640 1 -- 172.21.15.28:0/1666331764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 msgr2=0x55fde068fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff342ffd640 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 0x55fde068fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff342ffd640 1 -- 172.21.15.28:0/1666331764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde06818f0 con 0x55fde067d460 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff342ffd640 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067d460 0x55fde06902e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x55fde0685f60 tx=0x7ff338004b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40d5e7c033a1d6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff322ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff322ffd640 1 -- 172.21.15.28:0/1666331764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338018040 con 0x55fde067d460 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff322ffd640 1 -- 172.21.15.28:0/1666331764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7ff338004000 con 0x55fde067d460 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.309+0000 7ff34ce85740 1 -- 172.21.15.28:0/1666331764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff334005500 con 0x55fde067d460 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff322ffd640 1 -- 172.21.15.28:0/1666331764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff33803f050 con 0x55fde067d460 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff34ce85740 10 client.16053 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff34ce85740 1 -- 172.21.15.28:0/1666331764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fde06fdd60 con 0x55fde067d460 2024-05-09T08:27:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff322ffd640 1 -- 172.21.15.28:0/1666331764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff334005500 con 0x55fde067d460 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff322ffd640 1 -- 172.21.15.28:0/1666331764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff338034020 con 0x55fde067d460 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff34ce85740 10 client.16053 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff34ce85740 10 client.16053 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff34ce85740 20 client.16053 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff34ce85740 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55fde0697eb0 0x55fde0700400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff34ce85740 4 client.16053 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.310+0000 7ff34ce85740 1 -- 172.21.15.28:0/1666331764 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fde05e16c0 con 0x55fde0697eb0 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.311+0000 7ff3437fe640 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55fde0697eb0 0x55fde0700400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.311+0000 7ff3437fe640 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55fde0697eb0 0x55fde0700400 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d6f5449124ffb38b server_cookie=426be1c467b7d962 in_seq=0 out_seq=0 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.311+0000 7ff322ffd640 10 client.16053 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.311+0000 7ff322ffd640 1 -- 172.21.15.28:0/1666331764 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fde05e16c0 con 0x55fde0697eb0 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.311+0000 7ff322ffd640 10 client.16053 handle_command_reply: tid=0 2024-05-09T08:27:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.311+0000 7ff34ce85740 1 client.16053 shutdown 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.311+0000 7ff34ce85740 1 -- 172.21.15.28:0/1666331764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067d460 msgr2=0x55fde06902e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.311+0000 7ff34ce85740 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067d460 0x55fde06902e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x55fde0685f60 tx=0x7ff338004b80 comp rx=0 tx=0).stop 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34ce85740 1 -- 172.21.15.28:0/1666331764 shutdown_connections 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34ce85740 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55fde0697eb0 0x55fde0700400 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34ce85740 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fde0687b30 0x55fde06810c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff322ffd640 0 client.16053 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34ce85740 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fde067d460 0x55fde06902e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34ce85740 1 --2- 172.21.15.28:0/1666331764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fde067cb40 0x55fde068fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34ce85740 1 -- 172.21.15.28:0/1666331764 >> 172.21.15.28:0/1666331764 conn(0x55fde0678e60 msgr2=0x55fde066ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34ce85740 1 -- 172.21.15.28:0/1666331764 shutdown_connections 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34ce85740 1 -- 172.21.15.28:0/1666331764 wait complete. 2024-05-09T08:27:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34ce85740 20 client.16053 trim_cache size 0 max 16384 2024-05-09T08:27:27.318 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:27:27.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34a17d640 1 -- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff32c0a2d70 msgr2=0x7ff32c0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34a17d640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff32c0a2d70 0x7ff32c0a5230 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55fde067e100 tx=0x7ff334005f90 comp rx=0 tx=0).stop 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34a17d640 1 -- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff344106710 msgr2=0x7ff34419e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34a17d640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff344106710 0x7ff34419e990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff33c005e30 tx=0x7ff33c003f60 comp rx=0 tx=0).stop 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34a17d640 1 -- 172.21.15.28:0/3378236192 shutdown_connections 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34a17d640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff344106710 0x7ff34419e990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34a17d640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff32c0a2d70 0x7ff32c0a5230 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.312+0000 7ff34a17d640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3441010c0 0x7ff34419a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.313+0000 7ff34a17d640 1 --2- 172.21.15.28:0/3378236192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff344100700 0x7ff34419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.313+0000 7ff34a17d640 1 -- 172.21.15.28:0/3378236192 >> 172.21.15.28:0/3378236192 conn(0x7ff344077830 msgr2=0x7ff344102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.313+0000 7ff34a17d640 1 -- 172.21.15.28:0/3378236192 shutdown_connections 2024-05-09T08:27:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.313+0000 7ff34a17d640 1 -- 172.21.15.28:0/3378236192 wait complete. 2024-05-09T08:27:27.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:27 smithi028 ceph-mon[30119]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:27.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:27:27.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:27 smithi165 ceph-mon[34540]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:27.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:27 smithi184 ceph-mon[34509]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.438+0000 7fa33c7dc640 1 -- 172.21.15.28:0/2731426396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa334100630 msgr2=0x7fa334100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.438+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/2731426396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa334100630 0x7fa334100a10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa330009900 tx=0x7fa33002a410 comp rx=0 tx=0).stop 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.439+0000 7fa33c7dc640 1 -- 172.21.15.28:0/2731426396 shutdown_connections 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.439+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/2731426396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa33410e1e0 0x7fa3341105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.439+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/2731426396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa334100f50 0x7fa33410dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.439+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/2731426396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa334100630 0x7fa334100a10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.439+0000 7fa33c7dc640 1 -- 172.21.15.28:0/2731426396 >> 172.21.15.28:0/2731426396 conn(0x7fa334077830 msgr2=0x7fa334077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.439+0000 7fa33c7dc640 1 -- 172.21.15.28:0/2731426396 shutdown_connections 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.439+0000 7fa33c7dc640 1 -- 172.21.15.28:0/2731426396 wait complete. 2024-05-09T08:27:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.439+0000 7fa33c7dc640 1 Processor -- start 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33c7dc640 1 -- start start 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33c7dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa334100630 0x7fa33419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33c7dc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa334100f50 0x7fa33419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33c7dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33410e1e0 0x7fa33419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33ad79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33410e1e0 0x7fa33419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33ad79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33410e1e0 0x7fa33419ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35068/0 (socket says 172.21.15.28:35068) 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33ad79640 1 -- 172.21.15.28:0/1529783950 learned_addr learned my addr 172.21.15.28:0/1529783950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa334113dd0 con 0x7fa33410e1e0 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa334113c50 con 0x7fa334100f50 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa339d77640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa334100f50 0x7fa33419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa334113f50 con 0x7fa334100630 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33a578640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa334100630 0x7fa33419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33ad79640 1 -- 172.21.15.28:0/1529783950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa334100630 msgr2=0x7fa33419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33ad79640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa334100630 0x7fa33419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.440+0000 7fa33ad79640 1 -- 172.21.15.28:0/1529783950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa334100f50 msgr2=0x7fa33419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.441+0000 7fa33ad79640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa334100f50 0x7fa33419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.441+0000 7fa33ad79640 1 -- 172.21.15.28:0/1529783950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33419f2e0 con 0x7fa33410e1e0 2024-05-09T08:27:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.441+0000 7fa33ad79640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33410e1e0 0x7fa33419ebb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00eb40 tx=0x7fa32c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.441+0000 7fa32b7fe640 1 -- 172.21.15.28:0/1529783950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c004b20 con 0x7fa33410e1e0 2024-05-09T08:27:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.441+0000 7fa32b7fe640 1 -- 172.21.15.28:0/1529783950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa32c004cc0 con 0x7fa33410e1e0 2024-05-09T08:27:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.441+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33419f5d0 con 0x7fa33410e1e0 2024-05-09T08:27:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.442+0000 7fa32b7fe640 1 -- 172.21.15.28:0/1529783950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c010850 con 0x7fa33410e1e0 2024-05-09T08:27:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.442+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3341a6e60 con 0x7fa33410e1e0 2024-05-09T08:27:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.445+0000 7fa32b7fe640 1 -- 172.21.15.28:0/1529783950 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fa32c0109f0 con 0x7fa33410e1e0 2024-05-09T08:27:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.445+0000 7fa33f5ca740 1 Processor -- start 2024-05-09T08:27:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa33f5ca740 1 -- start start 2024-05-09T08:27:27.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa32b7fe640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa3200a2ea0 0x7fa3200a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b482330 0x56487b482710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482c50 0x56487b48cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56487b3e7c40 con 0x56487b482c50 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56487b38f580 con 0x56487b48d320 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56487b3e3c00 con 0x56487b482330 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa339d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482c50 0x56487b48cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa339d77640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa3200a2ea0 0x7fa3200a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa339d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482c50 0x56487b48cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35080/0 (socket says 172.21.15.28:35080) 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa339d77640 1 -- 172.21.15.28:0/646598446 learned_addr learned my addr 172.21.15.28:0/646598446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa33ad79640 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.446+0000 7fa33a578640 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b482330 0x56487b482710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa339d77640 1 -- 172.21.15.28:0/646598446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b482330 msgr2=0x56487b482710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa339d77640 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b482330 0x56487b482710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa339d77640 1 -- 172.21.15.28:0/646598446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 msgr2=0x56487b48f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa339d77640 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa339d77640 1 -- 172.21.15.28:0/646598446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56487b0fb3f0 con 0x56487b482c50 2024-05-09T08:27:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa33ad79640 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa32b7fe640 1 -- 172.21.15.28:0/1529783950 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa32c0d0140 con 0x7fa33410e1e0 2024-05-09T08:27:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa339d77640 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482c50 0x56487b48cde0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa3240127c0 tx=0x7fa324012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c011b9c71733de07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa328ff9640 1 -- 172.21.15.28:0/646598446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324022070 con 0x56487b482c50 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa328ff9640 1 -- 172.21.15.28:0/646598446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa32400fb40 con 0x56487b482c50 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.447+0000 7fa328ff9640 1 -- 172.21.15.28:0/646598446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32400d5b0 con 0x56487b482c50 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa33f5ca740 1 -- 172.21.15.28:0/646598446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482c50 msgr2=0x56487b48cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482c50 0x56487b48cde0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa3240127c0 tx=0x7fa324012bc0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa339d77640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa3200a2ea0 0x7fa3200a5360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa324012fd0 tx=0x7fa32401df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa33f5ca740 1 -- 172.21.15.28:0/646598446 shutdown_connections 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482c50 0x56487b48cde0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/646598446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b482330 0x56487b482710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa33f5ca740 1 -- 172.21.15.28:0/646598446 >> 172.21.15.28:0/646598446 conn(0x56487b47e650 msgr2=0x56487b47ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa33f5ca740 1 -- 172.21.15.28:0/646598446 shutdown_connections 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.448+0000 7fa33f5ca740 1 -- 172.21.15.28:0/646598446 wait complete. 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33f5ca740 1 -- start start 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33f5ca740 1 Processor -- start 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482330 0x56487b48a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 0x56487b4995e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56487b490110 con 0x56487b482330 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56487b3a2c40 con 0x56487b48d320 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56487b490290 con 0x56487b48adf0 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33f5ca740 10 client.0 _ll_register_callbacks cb 0x56487b0fabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33a578640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482330 0x56487b48a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33a578640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482330 0x56487b48a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35088/0 (socket says 172.21.15.28:35088) 2024-05-09T08:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33a578640 1 -- 172.21.15.28:0/3456218491 learned_addr learned my addr 172.21.15.28:0/3456218491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33ad79640 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 0x56487b4995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33a578640 1 -- 172.21.15.28:0/3456218491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 msgr2=0x56487b4995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33a578640 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 0x56487b4995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.449+0000 7fa33a578640 1 -- 172.21.15.28:0/3456218491 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 msgr2=0x56487b48a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa339d77640 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa33a578640 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa33a578640 1 -- 172.21.15.28:0/3456218491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56487b1c57d0 con 0x56487b482330 2024-05-09T08:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa339d77640 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa33a578640 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482330 0x56487b48a370 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa330009900 tx=0x7fa3300317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c524da36d93224 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa31dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa31dffb640 1 -- 172.21.15.28:0/3456218491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa330035040 con 0x56487b482330 2024-05-09T08:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa31dffb640 1 -- 172.21.15.28:0/3456218491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa330004240 con 0x56487b482330 2024-05-09T08:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa33f5ca740 1 -- 172.21.15.28:0/3456218491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa32400fb40 con 0x56487b482330 2024-05-09T08:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa31dffb640 1 -- 172.21.15.28:0/3456218491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa3300319b0 con 0x56487b482330 2024-05-09T08:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa33f5ca740 10 client.16065 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa33f5ca740 1 -- 172.21.15.28:0/3456218491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56487b49c8b0 con 0x56487b482330 2024-05-09T08:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.450+0000 7fa31dffb640 1 -- 172.21.15.28:0/3456218491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa32400fb40 con 0x56487b482330 2024-05-09T08:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.451+0000 7fa31dffb640 1 -- 172.21.15.28:0/3456218491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa330004a50 con 0x56487b482330 2024-05-09T08:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.451+0000 7fa33f5ca740 10 client.16065 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.451+0000 7fa33f5ca740 10 client.16065 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-05-09T08:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.451+0000 7fa33f5ca740 20 client.16065 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.451+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x56487b49dd80 0x56487b4a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.451+0000 7fa33f5ca740 4 client.16065 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.451+0000 7fa33f5ca740 1 -- 172.21.15.28:0/3456218491 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56487b3f5370 con 0x56487b49dd80 2024-05-09T08:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.451+0000 7fa339d77640 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x56487b49dd80 0x56487b4a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.452+0000 7fa31dffb640 10 client.16065 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.452+0000 7fa339d77640 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x56487b49dd80 0x56487b4a0240 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1b3132720909487f server_cookie=df5df8b0a76febc6 in_seq=0 out_seq=0 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.452+0000 7fa31dffb640 1 -- 172.21.15.28:0/3456218491 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56487b3f5370 con 0x56487b49dd80 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.452+0000 7fa31dffb640 10 client.16065 handle_command_reply: tid=0 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.452+0000 7fa33f5ca740 1 client.16065 shutdown 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 -- 172.21.15.28:0/3456218491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482330 msgr2=0x56487b48a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482330 0x56487b48a370 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa330009900 tx=0x7fa3300317c0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 -- 172.21.15.28:0/3456218491 shutdown_connections 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x56487b49dd80 0x56487b4a0240 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 0x56487b4995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b48a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa31dffb640 0 client.16065 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/3456218491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b482330 0x56487b48a370 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 -- 172.21.15.28:0/3456218491 >> 172.21.15.28:0/3456218491 conn(0x56487b47e650 msgr2=0x56487b484350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 -- 172.21.15.28:0/3456218491 shutdown_connections 2024-05-09T08:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 1 -- 172.21.15.28:0/3456218491 wait complete. 2024-05-09T08:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.453+0000 7fa33f5ca740 20 client.16065 trim_cache size 0 max 16384 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33f5ca740 1 Processor -- start 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33f5ca740 1 -- start start 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48adf0 0x56487b4a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b48d320 0x56487b4a6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b499030 0x56487b499470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa31800ffe0 con 0x56487b48d320 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa318011b20 con 0x56487b48adf0 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa3180106b0 con 0x56487b499030 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa339d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b48d320 0x56487b4a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33ad79640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b499030 0x56487b499470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33ad79640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b499030 0x56487b499470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42652/0 (socket says 172.21.15.28:42652) 2024-05-09T08:27:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33ad79640 1 -- 172.21.15.28:0/384934927 learned_addr learned my addr 172.21.15.28:0/384934927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33a578640 1 --2- 172.21.15.28:0/384934927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48adf0 0x56487b4a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33ad79640 1 -- 172.21.15.28:0/384934927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48adf0 msgr2=0x56487b4a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33ad79640 1 --2- 172.21.15.28:0/384934927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48adf0 0x56487b4a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33ad79640 1 -- 172.21.15.28:0/384934927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b48d320 msgr2=0x56487b4a6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33ad79640 1 --2- 172.21.15.28:0/384934927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b48d320 0x56487b4a6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.467+0000 7fa33ad79640 1 -- 172.21.15.28:0/384934927 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56487b46fd20 con 0x56487b499030 2024-05-09T08:27:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.468+0000 7fa33ad79640 1 --2- 172.21.15.28:0/384934927 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b499030 0x56487b499470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa32c09bd40 tx=0x7fa32c09bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7896493bc31462d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.468+0000 7fa31ffff640 1 -- 172.21.15.28:0/384934927 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c09c040 con 0x56487b499030 2024-05-09T08:27:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.468+0000 7fa31ffff640 1 -- 172.21.15.28:0/384934927 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa32c0a03f0 con 0x56487b499030 2024-05-09T08:27:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.468+0000 7fa31ffff640 1 -- 172.21.15.28:0/384934927 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c096bb0 con 0x56487b499030 2024-05-09T08:27:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 -- 172.21.15.28:0/384934927 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b499030 msgr2=0x56487b499470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/384934927 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b499030 0x56487b499470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa32c09bd40 tx=0x7fa32c09bc40 comp rx=0 tx=0).stop 2024-05-09T08:27:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 -- 172.21.15.28:0/384934927 shutdown_connections 2024-05-09T08:27:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/384934927 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b499030 0x56487b499470 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/384934927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b48d320 0x56487b4a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/384934927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48adf0 0x56487b4a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 -- 172.21.15.28:0/384934927 >> 172.21.15.28:0/384934927 conn(0x56487b47e650 msgr2=0x56487b4796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 -- 172.21.15.28:0/384934927 shutdown_connections 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 -- 172.21.15.28:0/384934927 wait complete. 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 -- start start 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.469+0000 7fa33f5ca740 1 Processor -- start 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 0x56487b495540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b495a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33f5ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b499030 0x56487b492510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33f5ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa318011450 con 0x56487b499030 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33ad79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b499030 0x56487b492510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33ad79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b499030 0x56487b492510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35104/0 (socket says 172.21.15.28:35104) 2024-05-09T08:27:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33ad79640 1 -- 172.21.15.28:0/2209215112 learned_addr learned my addr 172.21.15.28:0/2209215112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa318010d80 con 0x56487b48d320 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa31800dfc0 con 0x56487b48adf0 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33a578640 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 0x56487b495540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33f5ca740 10 client.0 _ll_register_callbacks cb 0x7fa330009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa339d77640 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b495a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33ad79640 1 -- 172.21.15.28:0/2209215112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 msgr2=0x56487b495540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33ad79640 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 0x56487b495540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33ad79640 1 -- 172.21.15.28:0/2209215112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 msgr2=0x56487b495a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33ad79640 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b495a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.470+0000 7fa33ad79640 1 -- 172.21.15.28:0/2209215112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56487b492ad0 con 0x56487b499030 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.471+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa32c0a03f0 con 0x56487b499030 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.471+0000 7fa33ad79640 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b499030 0x56487b492510 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0bc010 tx=0x7fa32c0cd2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11cba44c5a197c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.471+0000 7fa31dffb640 10 client.16071 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.471+0000 7fa31dffb640 1 -- 172.21.15.28:0/2209215112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c07ae60 con 0x56487b499030 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.471+0000 7fa31dffb640 1 -- 172.21.15.28:0/2209215112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa32c096bb0 con 0x56487b499030 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.471+0000 7fa31dffb640 1 -- 172.21.15.28:0/2209215112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa32c096db0 con 0x56487b499030 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.471+0000 7fa33f5ca740 10 client.16071 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.471+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=28}) -- 0x56487b504b40 con 0x56487b499030 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa31dffb640 1 -- 172.21.15.28:0/2209215112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa32c010ce0 con 0x56487b499030 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa33f5ca740 10 client.16071 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa33f5ca740 10 client.16071 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa33f5ca740 20 client.16071 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x56487b505200 0x56487b507620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa33f5ca740 4 client.16071 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56487b3ef6c0 con 0x56487b505200 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa33a578640 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x56487b505200 0x56487b507620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa31dffb640 10 client.16071 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.472+0000 7fa33a578640 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x56487b505200 0x56487b507620 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=964e59990a8196f2 server_cookie=45914c1d75f4a8b in_seq=0 out_seq=0 2024-05-09T08:27:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa31dffb640 1 -- 172.21.15.28:0/2209215112 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56487b3ef6c0 con 0x56487b505200 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa31dffb640 10 client.16071 handle_command_reply: tid=0 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 client.16071 shutdown 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b499030 msgr2=0x56487b492510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b499030 0x56487b492510 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0bc010 tx=0x7fa32c0cd2e0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 shutdown_connections 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x56487b505200 0x56487b507620 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56487b499030 0x56487b492510 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa31dffb640 0 client.16071 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56487b48d320 0x56487b495a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 --2- 172.21.15.28:0/2209215112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56487b48adf0 0x56487b495540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 >> 172.21.15.28:0/2209215112 conn(0x56487b47e650 msgr2=0x56487b488c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 shutdown_connections 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 1 -- 172.21.15.28:0/2209215112 wait complete. 2024-05-09T08:27:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.473+0000 7fa33f5ca740 20 client.16071 trim_cache size 0 max 16384 2024-05-09T08:27:27.479 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:27:27.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa3200a2ea0 msgr2=0x7fa3200a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa3200a2ea0 0x7fa3200a5360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa324012fd0 tx=0x7fa32401df90 comp rx=0 tx=0).stop 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33410e1e0 msgr2=0x7fa33419ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33410e1e0 0x7fa33419ebb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00eb40 tx=0x7fa32c00c2d0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 shutdown_connections 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33410e1e0 0x7fa33419ebb0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa334100f50 0x7fa33419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa3200a2ea0 0x7fa3200a5360 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 --2- 172.21.15.28:0/1529783950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa334100630 0x7fa33419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 >> 172.21.15.28:0/1529783950 conn(0x7fa334077830 msgr2=0x7fa33410f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 shutdown_connections 2024-05-09T08:27:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.474+0000 7fa33c7dc640 1 -- 172.21.15.28:0/1529783950 wait complete. 2024-05-09T08:27:27.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.602+0000 7f0c46f63640 1 -- 172.21.15.28:0/3992049101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 msgr2=0x7f0c40107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.602+0000 7f0c46f63640 1 --2- 172.21.15.28:0/3992049101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 0x7f0c40107c20 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009930 tx=0x7f0c3402a420 comp rx=0 tx=0).stop 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.602+0000 7f0c46f63640 1 -- 172.21.15.28:0/3992049101 shutdown_connections 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.602+0000 7f0c46f63640 1 --2- 172.21.15.28:0/3992049101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c401021f0 0x7f0c4010e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.602+0000 7f0c46f63640 1 --2- 172.21.15.28:0/3992049101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c40101830 0x7f0c40101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.602+0000 7f0c46f63640 1 --2- 172.21.15.28:0/3992049101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 0x7f0c40107c20 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.602+0000 7f0c46f63640 1 -- 172.21.15.28:0/3992049101 >> 172.21.15.28:0/3992049101 conn(0x7f0c40078cc0 msgr2=0x7f0c400790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.602+0000 7f0c46f63640 1 -- 172.21.15.28:0/3992049101 shutdown_connections 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.602+0000 7f0c46f63640 1 -- 172.21.15.28:0/3992049101 wait complete. 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.603+0000 7f0c46f63640 1 Processor -- start 2024-05-09T08:27:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.603+0000 7f0c46f63640 1 -- start start 2024-05-09T08:27:27.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c46f63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c40101830 0x7f0c40198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c46f63640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c401021f0 0x7f0c401986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c46f63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 0x7f0c4019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c46f63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c40111c50 con 0x7f0c40107840 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c46f63640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c40111ad0 con 0x7f0c401021f0 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c46f63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0c40111dd0 con 0x7f0c40101830 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c45500640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 0x7f0c4019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c45500640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 0x7f0c4019ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35122/0 (socket says 172.21.15.28:35122) 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c45500640 1 -- 172.21.15.28:0/910818774 learned_addr learned my addr 172.21.15.28:0/910818774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c401021f0 0x7f0c401986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c44cff640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c40101830 0x7f0c40198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c45500640 1 -- 172.21.15.28:0/910818774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c40101830 msgr2=0x7f0c40198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c45500640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c40101830 0x7f0c40198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c45500640 1 -- 172.21.15.28:0/910818774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c401021f0 msgr2=0x7f0c401986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c45500640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c401021f0 0x7f0c401986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c45500640 1 -- 172.21.15.28:0/910818774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4019d130 con 0x7f0c40107840 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.604+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c401021f0 0x7f0c401986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.605+0000 7f0c45500640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 0x7f0c4019ca00 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800e960 tx=0x7f0c3800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.605+0000 7f0c3dffb640 1 -- 172.21.15.28:0/910818774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c38004d20 con 0x7f0c40107840 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.605+0000 7f0c46f63640 1 -- 172.21.15.28:0/910818774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c4019d420 con 0x7f0c40107840 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.605+0000 7f0c3dffb640 1 -- 172.21.15.28:0/910818774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c38005020 con 0x7f0c40107840 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.605+0000 7f0c3dffb640 1 -- 172.21.15.28:0/910818774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c380116d0 con 0x7f0c40107840 2024-05-09T08:27:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.605+0000 7f0c46f63640 1 -- 172.21.15.28:0/910818774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c401a4cb0 con 0x7f0c40107840 2024-05-09T08:27:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.608+0000 7f0c3dffb640 1 -- 172.21.15.28:0/910818774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f0c38011870 con 0x7f0c40107840 2024-05-09T08:27:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c3dffb640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0c300a2f20 0x7f0c300a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c49c77740 1 Processor -- start 2024-05-09T08:27:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c49c77740 1 -- start start 2024-05-09T08:27:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c44cff640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0c300a2f20 0x7f0c300a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c3dffb640 1 -- 172.21.15.28:0/910818774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0c380cfb80 con 0x7f0c40107840 2024-05-09T08:27:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 0x55e78df49710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49c50 0x55e78df53de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 0x55e78df56710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c49c77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e78deaec40 con 0x55e78df54320 2024-05-09T08:27:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c49c77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e78de56580 con 0x55e78df49c50 2024-05-09T08:27:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c49c77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e78deaac00 con 0x55e78df49330 2024-05-09T08:27:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.609+0000 7f0c44cff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 0x55e78df49710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c3ffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49c50 0x55e78df53de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c3ffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49c50 0x55e78df53de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35692/0 (socket says 172.21.15.28:35692) 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c3ffff640 1 -- 172.21.15.28:0/2246826902 learned_addr learned my addr 172.21.15.28:0/2246826902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c45500640 1 --2- 172.21.15.28:0/2246826902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 0x55e78df56710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c44cff640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0c300a2f20 0x7f0c300a53e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009900 tx=0x7f0c340058d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c3ffff640 1 -- 172.21.15.28:0/2246826902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 msgr2=0x55e78df49710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/2246826902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 0x55e78df49710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c3ffff640 1 -- 172.21.15.28:0/2246826902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 msgr2=0x55e78df56710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/2246826902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 0x55e78df56710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c3ffff640 1 -- 172.21.15.28:0/2246826902 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e78dbc23f0 con 0x55e78df49c50 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/2246826902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49c50 0x55e78df53de0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c010450 tx=0x7f0c2c010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1a316d72dd75223 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c2affd640 1 -- 172.21.15.28:0/2246826902 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c011760 con 0x55e78df49c50 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c2affd640 1 -- 172.21.15.28:0/2246826902 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c2c011de0 con 0x55e78df49c50 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.610+0000 7f0c2affd640 1 -- 172.21.15.28:0/2246826902 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c01a380 con 0x55e78df49c50 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.611+0000 7f0c49c77740 1 -- 172.21.15.28:0/2246826902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49c50 msgr2=0x55e78df53de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.611+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2246826902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49c50 0x55e78df53de0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c010450 tx=0x7f0c2c010910 comp rx=0 tx=0).stop 2024-05-09T08:27:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.611+0000 7f0c49c77740 1 -- 172.21.15.28:0/2246826902 shutdown_connections 2024-05-09T08:27:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.611+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2246826902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 0x55e78df56710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.611+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2246826902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49c50 0x55e78df53de0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.611+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2246826902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 0x55e78df49710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.611+0000 7f0c49c77740 1 -- 172.21.15.28:0/2246826902 >> 172.21.15.28:0/2246826902 conn(0x55e78df45650 msgr2=0x55e78df45a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.611+0000 7f0c49c77740 1 -- 172.21.15.28:0/2246826902 shutdown_connections 2024-05-09T08:27:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.611+0000 7f0c49c77740 1 -- 172.21.15.28:0/2246826902 wait complete. 2024-05-09T08:27:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.612+0000 7f0c49c77740 1 -- start start 2024-05-09T08:27:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.612+0000 7f0c49c77740 1 Processor -- start 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.612+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df51240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.612+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df51780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.612+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df54320 0x55e78df51cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.612+0000 7f0c49c77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e78df57110 con 0x55e78df49c50 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.612+0000 7f0c49c77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e78de69c40 con 0x55e78df49330 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.612+0000 7f0c49c77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e78df57290 con 0x55e78df54320 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c49c77740 10 client.0 _ll_register_callbacks cb 0x55e78dbc1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df51780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c44cff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df51240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df51780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35132/0 (socket says 172.21.15.28:35132) 2024-05-09T08:27:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c3ffff640 1 -- 172.21.15.28:0/2008675416 learned_addr learned my addr 172.21.15.28:0/2008675416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c45500640 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df54320 0x55e78df51cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c44cff640 1 -- 172.21.15.28:0/2008675416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df54320 msgr2=0x55e78df51cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c44cff640 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df54320 0x55e78df51cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c44cff640 1 -- 172.21.15.28:0/2008675416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 msgr2=0x55e78df51780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c44cff640 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df51780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c44cff640 1 -- 172.21.15.28:0/2008675416 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e78dc8c7d0 con 0x55e78df49330 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df51780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.613+0000 7f0c44cff640 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df51240 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0c34005940 tx=0x7f0c34009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8ddb12641be6163 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2008675416 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34035040 con 0x55e78df49330 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2008675416 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0c34003b80 con 0x55e78df49330 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c49c77740 1 -- 172.21.15.28:0/2008675416 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c2c011de0 con 0x55e78df49330 2024-05-09T08:27:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2008675416 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0c3405a050 con 0x55e78df49330 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c49c77740 10 client.25568 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c49c77740 1 -- 172.21.15.28:0/2008675416 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e78df676a0 con 0x55e78df49330 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2008675416 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c2c011de0 con 0x55e78df49330 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2008675416 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0c3404f020 con 0x55e78df49330 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c49c77740 10 client.25568 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c49c77740 10 client.25568 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c49c77740 20 client.25568 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55e78df68b60 0x55e78df6af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c49c77740 4 client.25568 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.615+0000 7f0c49c77740 1 -- 172.21.15.28:0/2008675416 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e78debc370 con 0x55e78df68b60 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.616+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55e78df68b60 0x55e78df6af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.616+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55e78df68b60 0x55e78df6af50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f2ca097c1144fd05 server_cookie=863199f2ac1fecca in_seq=0 out_seq=0 2024-05-09T08:27:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.616+0000 7f0c1ffff640 10 client.25568 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:27:27.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2008675416 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e78debc370 con 0x55e78df68b60 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c1ffff640 10 client.25568 handle_command_reply: tid=0 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 client.25568 shutdown 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 -- 172.21.15.28:0/2008675416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 msgr2=0x55e78df51240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df51240 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0c34005940 tx=0x7f0c34009260 comp rx=0 tx=0).stop 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 -- 172.21.15.28:0/2008675416 shutdown_connections 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55e78df68b60 0x55e78df6af50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df54320 0x55e78df51cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df51780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2008675416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df51240 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 -- 172.21.15.28:0/2008675416 >> 172.21.15.28:0/2008675416 conn(0x55e78df45650 msgr2=0x55e78df4b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c1ffff640 0 client.25568 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:27:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 -- 172.21.15.28:0/2008675416 shutdown_connections 2024-05-09T08:27:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 1 -- 172.21.15.28:0/2008675416 wait complete. 2024-05-09T08:27:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.617+0000 7f0c49c77740 20 client.25568 trim_cache size 0 max 16384 2024-05-09T08:27:27.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.631+0000 7f0c49c77740 1 Processor -- start 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.631+0000 7f0c49c77740 1 -- start start 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.631+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 0x55e78df5ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.631+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df4bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.631+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df54320 0x55e78df503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.631+0000 7f0c49c77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c401acca0 con 0x55e78df49c50 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.631+0000 7f0c49c77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c401ae7e0 con 0x55e78df54320 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.631+0000 7f0c49c77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0c401ad370 con 0x55e78df49330 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df4bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df4bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35140/0 (socket says 172.21.15.28:35140) 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c3ffff640 1 -- 172.21.15.28:0/3489889504 learned_addr learned my addr 172.21.15.28:0/3489889504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c44cff640 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 0x55e78df5ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c45500640 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df54320 0x55e78df503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c3ffff640 1 -- 172.21.15.28:0/3489889504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 msgr2=0x55e78df5ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 0x55e78df5ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c3ffff640 1 -- 172.21.15.28:0/3489889504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df54320 msgr2=0x55e78df503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df54320 0x55e78df503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c3ffff640 1 -- 172.21.15.28:0/3489889504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e78df36d20 con 0x55e78df49c50 2024-05-09T08:27:27.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c45500640 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df54320 0x55e78df503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:27.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.632+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df4bad0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c01cb90 tx=0x7f0c2c01c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d221e6ae16db2054 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.633+0000 7f0c2a7fc640 1 -- 172.21.15.28:0/3489889504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c02f070 con 0x55e78df49c50 2024-05-09T08:27:27.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.633+0000 7f0c2a7fc640 1 -- 172.21.15.28:0/3489889504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c2c00ba40 con 0x55e78df49c50 2024-05-09T08:27:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.633+0000 7f0c2a7fc640 1 -- 172.21.15.28:0/3489889504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c02b5f0 con 0x55e78df49c50 2024-05-09T08:27:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.633+0000 7f0c49c77740 1 -- 172.21.15.28:0/3489889504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 msgr2=0x55e78df4bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.633+0000 7f0c49c77740 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df4bad0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c01cb90 tx=0x7f0c2c01c660 comp rx=0 tx=0).stop 2024-05-09T08:27:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 -- 172.21.15.28:0/3489889504 shutdown_connections 2024-05-09T08:27:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df54320 0x55e78df503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df49c50 0x55e78df4bad0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 --2- 172.21.15.28:0/3489889504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78df49330 0x55e78df5ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 -- 172.21.15.28:0/3489889504 >> 172.21.15.28:0/3489889504 conn(0x55e78df45650 msgr2=0x55e78df43f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 -- 172.21.15.28:0/3489889504 shutdown_connections 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 -- 172.21.15.28:0/3489889504 wait complete. 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 -- start start 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 Processor -- start 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df576f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.634+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 0x55e78df57c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c49c77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78dfc7070 0x55e78dfc74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c49c77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c401ae110 con 0x55e78df54320 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c49c77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c401ada40 con 0x55e78df49330 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c49c77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0c401aac80 con 0x55e78dfc7070 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c49c77740 10 client.0 _ll_register_callbacks cb 0x7f0c34009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 0x55e78df57c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 0x55e78df57c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35152/0 (socket says 172.21.15.28:35152) 2024-05-09T08:27:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c3ffff640 1 -- 172.21.15.28:0/2207013699 learned_addr learned my addr 172.21.15.28:0/2207013699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c44cff640 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c45500640 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78dfc7070 0x55e78dfc74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c44cff640 1 -- 172.21.15.28:0/2207013699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78dfc7070 msgr2=0x55e78dfc74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c44cff640 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78dfc7070 0x55e78dfc74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c44cff640 1 -- 172.21.15.28:0/2207013699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 msgr2=0x55e78df57c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c44cff640 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 0x55e78df57c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c44cff640 1 -- 172.21.15.28:0/2207013699 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e78dfc7ab0 con 0x55e78df49330 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c44cff640 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df576f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009230 tx=0x7f0c34003fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f1fda3a4be3732d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.635+0000 7f0c1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2207013699 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34047e10 con 0x55e78df49330 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2207013699 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0c3403c440 con 0x55e78df49330 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c49c77740 1 -- 172.21.15.28:0/2207013699 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c2c00ba40 con 0x55e78df49330 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2207013699 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0c34031740 con 0x55e78df49330 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c49c77740 10 client.25574 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c49c77740 1 -- 172.21.15.28:0/2207013699 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e78dfcee10 con 0x55e78df49330 2024-05-09T08:27:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2207013699 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c34042070 con 0x55e78df49330 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2207013699 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0c3405d070 con 0x55e78df49330 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c49c77740 10 client.25574 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c49c77740 10 client.25574 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.636+0000 7f0c49c77740 20 client.25574 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.637+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55e78dfcf5b0 0x55e78dfd19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.637+0000 7f0c49c77740 4 client.25574 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.637+0000 7f0c49c77740 1 -- 172.21.15.28:0/2207013699 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0c2c00ba40 con 0x55e78dfcf5b0 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.637+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55e78dfcf5b0 0x55e78dfd19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.637+0000 7f0c1ffff640 10 client.25574 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:27:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.637+0000 7f0c3ffff640 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55e78dfcf5b0 0x55e78dfd19d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7325550ce351fb1d server_cookie=f1d3821022b592d0 in_seq=0 out_seq=0 2024-05-09T08:27:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.637+0000 7f0c1ffff640 1 -- 172.21.15.28:0/2207013699 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0c2c00ba40 con 0x55e78dfcf5b0 2024-05-09T08:27:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.637+0000 7f0c1ffff640 10 client.25574 handle_command_reply: tid=0 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.637+0000 7f0c49c77740 1 client.25574 shutdown 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 -- 172.21.15.28:0/2207013699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 msgr2=0x55e78df576f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df576f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009230 tx=0x7f0c34003fd0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 -- 172.21.15.28:0/2207013699 shutdown_connections 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x55e78dfcf5b0 0x55e78dfd19d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e78dfc7070 0x55e78dfc74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e78df54320 0x55e78df57c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 --2- 172.21.15.28:0/2207013699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e78df49330 0x55e78df576f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 -- 172.21.15.28:0/2207013699 >> 172.21.15.28:0/2207013699 conn(0x55e78df45650 msgr2=0x55e78df43f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c1ffff640 0 client.25574 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 -- 172.21.15.28:0/2207013699 shutdown_connections 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 1 -- 172.21.15.28:0/2207013699 wait complete. 2024-05-09T08:27:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.638+0000 7f0c49c77740 20 client.25574 trim_cache size 0 max 16384 2024-05-09T08:27:27.644 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:27:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 -- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0c300a2f20 msgr2=0x7f0c300a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0c300a2f20 0x7f0c300a53e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009900 tx=0x7f0c340058d0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 -- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 msgr2=0x7f0c4019ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 0x7f0c4019ca00 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800e960 tx=0x7f0c3800ee20 comp rx=0 tx=0).stop 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 -- 172.21.15.28:0/910818774 shutdown_connections 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c40107840 0x7f0c4019ca00 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c401021f0 0x7f0c401986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0c300a2f20 0x7f0c300a53e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 --2- 172.21.15.28:0/910818774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0c40101830 0x7f0c40198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 -- 172.21.15.28:0/910818774 >> 172.21.15.28:0/910818774 conn(0x7f0c40078cc0 msgr2=0x7f0c400ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 -- 172.21.15.28:0/910818774 shutdown_connections 2024-05-09T08:27:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.639+0000 7f0c46f63640 1 -- 172.21.15.28:0/910818774 wait complete. 2024-05-09T08:27:27.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:27:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.768+0000 7f14457dc640 1 -- 172.21.15.28:0/3452286812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440106710 msgr2=0x7f1440106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.768+0000 7f14457dc640 1 --2- 172.21.15.28:0/3452286812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440106710 0x7f1440106af0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1430009930 tx=0x7f143002a420 comp rx=0 tx=0).stop 2024-05-09T08:27:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.768+0000 7f14457dc640 1 -- 172.21.15.28:0/3452286812 shutdown_connections 2024-05-09T08:27:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.768+0000 7f14457dc640 1 --2- 172.21.15.28:0/3452286812 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14401010c0 0x7f144010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.768+0000 7f14457dc640 1 --2- 172.21.15.28:0/3452286812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1440100700 0x7f1440100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.768+0000 7f14457dc640 1 --2- 172.21.15.28:0/3452286812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440106710 0x7f1440106af0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.768+0000 7f14457dc640 1 -- 172.21.15.28:0/3452286812 >> 172.21.15.28:0/3452286812 conn(0x7f1440077830 msgr2=0x7f1440077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.768+0000 7f14457dc640 1 -- 172.21.15.28:0/3452286812 shutdown_connections 2024-05-09T08:27:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.768+0000 7f14457dc640 1 -- 172.21.15.28:0/3452286812 wait complete. 2024-05-09T08:27:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.769+0000 7f14457dc640 1 Processor -- start 2024-05-09T08:27:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.769+0000 7f14457dc640 1 -- start start 2024-05-09T08:27:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.769+0000 7f14457dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440100700 0x7f1440196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.769+0000 7f14457dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14401010c0 0x7f1440197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.769+0000 7f14457dc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1440106710 0x7f14401977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.769+0000 7f14457dc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1440110b20 con 0x7f1440100700 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.769+0000 7f14457dc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f14401109a0 con 0x7f1440106710 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.769+0000 7f14457dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1440110ca0 con 0x7f14401010c0 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440100700 0x7f1440196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440100700 0x7f1440196d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35166/0 (socket says 172.21.15.28:35166) 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143effd640 1 -- 172.21.15.28:0/1865095600 learned_addr learned my addr 172.21.15.28:0/1865095600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143e7fc640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14401010c0 0x7f1440197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143effd640 1 -- 172.21.15.28:0/1865095600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14401010c0 msgr2=0x7f1440197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143effd640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14401010c0 0x7f1440197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143f7fe640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1440106710 0x7f14401977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143effd640 1 -- 172.21.15.28:0/1865095600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1440106710 msgr2=0x7f14401977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143effd640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1440106710 0x7f14401977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143effd640 1 -- 172.21.15.28:0/1865095600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14401a65a0 con 0x7f1440100700 2024-05-09T08:27:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143e7fc640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14401010c0 0x7f1440197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143f7fe640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1440106710 0x7f14401977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:27:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f143effd640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440100700 0x7f1440196d30 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1430009230 tx=0x7f1430004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f1427fff640 1 -- 172.21.15.28:0/1865095600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143003f030 con 0x7f1440100700 2024-05-09T08:27:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.770+0000 7f14457dc640 1 -- 172.21.15.28:0/1865095600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14401a6830 con 0x7f1440100700 2024-05-09T08:27:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.771+0000 7f1427fff640 1 -- 172.21.15.28:0/1865095600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1430004440 con 0x7f1440100700 2024-05-09T08:27:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.771+0000 7f1427fff640 1 -- 172.21.15.28:0/1865095600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1430035070 con 0x7f1440100700 2024-05-09T08:27:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.771+0000 7f14457dc640 1 -- 172.21.15.28:0/1865095600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14401a6d70 con 0x7f1440100700 2024-05-09T08:27:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.774+0000 7f1427fff640 1 -- 172.21.15.28:0/1865095600 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f1430042030 con 0x7f1440100700 2024-05-09T08:27:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.774+0000 7f14485ca740 1 Processor -- start 2024-05-09T08:27:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.774+0000 7f14485ca740 1 -- start start 2024-05-09T08:27:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.774+0000 7f14485ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34e330 0x5597ed34e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f14485ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34ec50 0x5597ed358de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34e330 0x5597ed34e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34e330 0x5597ed34e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35174/0 (socket says 172.21.15.28:35174) 2024-05-09T08:27:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f14485ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed359320 0x5597ed35b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f14485ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5597ed2b3c40 con 0x5597ed34e330 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f14485ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5597ed25b580 con 0x5597ed34ec50 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f14485ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5597ed2afc00 con 0x5597ed359320 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143f7fe640 1 -- 172.21.15.28:0/975393959 learned_addr learned my addr 172.21.15.28:0/975393959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143effd640 1 --2- 172.21.15.28:0/975393959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34ec50 0x5597ed358de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143e7fc640 1 --2- 172.21.15.28:0/975393959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed359320 0x5597ed35b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143f7fe640 1 -- 172.21.15.28:0/975393959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed359320 msgr2=0x5597ed35b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143f7fe640 1 --2- 172.21.15.28:0/975393959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed359320 0x5597ed35b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143f7fe640 1 -- 172.21.15.28:0/975393959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34ec50 msgr2=0x5597ed358de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143f7fe640 1 --2- 172.21.15.28:0/975393959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34ec50 0x5597ed358de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f143f7fe640 1 -- 172.21.15.28:0/975393959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597ecfc73f0 con 0x5597ed34e330 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.775+0000 7f1427fff640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14280a2e20 0x7f14280a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.776+0000 7f1427fff640 1 -- 172.21.15.28:0/1865095600 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f14300ef780 con 0x7f1440100700 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.776+0000 7f143effd640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14280a2e20 0x7f14280a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.776+0000 7f143f7fe640 1 --2- 172.21.15.28:0/975393959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34e330 0x5597ed34e710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1438009ef0 tx=0x7f143800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a4967f4497183c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.776+0000 7f14257fa640 1 -- 172.21.15.28:0/975393959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438003b70 con 0x5597ed34e330 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.776+0000 7f14257fa640 1 -- 172.21.15.28:0/975393959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1438003d10 con 0x5597ed34e330 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.776+0000 7f14257fa640 1 -- 172.21.15.28:0/975393959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438011a30 con 0x5597ed34e330 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f143effd640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14280a2e20 0x7f14280a52e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14300ec9e0 tx=0x7f14300045f0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f14485ca740 1 -- 172.21.15.28:0/975393959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34e330 msgr2=0x5597ed34e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f14485ca740 1 --2- 172.21.15.28:0/975393959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34e330 0x5597ed34e710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1438009ef0 tx=0x7f143800f730 comp rx=0 tx=0).stop 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f14485ca740 1 -- 172.21.15.28:0/975393959 shutdown_connections 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f14485ca740 1 --2- 172.21.15.28:0/975393959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed359320 0x5597ed35b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f14485ca740 1 --2- 172.21.15.28:0/975393959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34ec50 0x5597ed358de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f14485ca740 1 --2- 172.21.15.28:0/975393959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34e330 0x5597ed34e710 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f14485ca740 1 -- 172.21.15.28:0/975393959 >> 172.21.15.28:0/975393959 conn(0x5597ed34a650 msgr2=0x5597ed34aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f14485ca740 1 -- 172.21.15.28:0/975393959 shutdown_connections 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.777+0000 7f14485ca740 1 -- 172.21.15.28:0/975393959 wait complete. 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.778+0000 7f14485ca740 1 -- start start 2024-05-09T08:27:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.778+0000 7f14485ca740 1 Processor -- start 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.778+0000 7f14485ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 0x5597ed3562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.778+0000 7f14485ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34ec50 0x5597ed356830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.778+0000 7f14485ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed356d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.778+0000 7f143e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34ec50 0x5597ed356830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34ec50 0x5597ed356830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35176/0 (socket says 172.21.15.28:35176) 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143e7fc640 1 -- 172.21.15.28:0/2527192096 learned_addr learned my addr 172.21.15.28:0/2527192096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.778+0000 7f14485ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5597ed35c110 con 0x5597ed34ec50 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5597ed26ec40 con 0x5597ed359320 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5597ed35c290 con 0x5597ed34e330 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143f7fe640 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 0x5597ed3562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143e7fc640 1 -- 172.21.15.28:0/2527192096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 msgr2=0x5597ed3562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143e7fc640 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 0x5597ed3562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143e7fc640 1 -- 172.21.15.28:0/2527192096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 msgr2=0x5597ed356d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143f7fe640 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed356d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143e7fc640 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed356d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143e7fc640 1 -- 172.21.15.28:0/2527192096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597ed0917d0 con 0x5597ed34ec50 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143f7fe640 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed356d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f143e7fc640 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34ec50 0x5597ed356830 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f14340137c0 tx=0x7f1434013c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34a530c9bfed055 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f14485ca740 10 client.0 _ll_register_callbacks cb 0x5597ecfc6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f1422ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f1422ff5640 1 -- 172.21.15.28:0/2527192096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1434020060 con 0x5597ed34ec50 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f1422ff5640 1 -- 172.21.15.28:0/2527192096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f1434011dd0 con 0x5597ed34ec50 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.779+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1438003d10 con 0x5597ed34ec50 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f1422ff5640 1 -- 172.21.15.28:0/2527192096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f143400f450 con 0x5597ed34ec50 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f14485ca740 10 client.16101 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5597ed367fa0 con 0x5597ed34ec50 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f1422ff5640 1 -- 172.21.15.28:0/2527192096 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f143401a030 con 0x5597ed34ec50 2024-05-09T08:27:27.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f14485ca740 10 client.16101 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f14485ca740 10 client.16101 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:27:27.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f14485ca740 20 client.16101 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f14485ca740 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5597ed369390 0x5597ed36b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f14485ca740 4 client.16101 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:27.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.780+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597ed2c1370 con 0x5597ed369390 2024-05-09T08:27:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.781+0000 7f143f7fe640 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5597ed369390 0x5597ed36b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.781+0000 7f143f7fe640 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5597ed369390 0x5597ed36b820 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed3d813b8cec932f server_cookie=18257ebd2ab1f5b3 in_seq=0 out_seq=0 2024-05-09T08:27:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.781+0000 7f1422ff5640 10 client.16101 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:27:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f1422ff5640 1 -- 172.21.15.28:0/2527192096 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5597ed2c1370 con 0x5597ed369390 2024-05-09T08:27:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f1422ff5640 10 client.16101 handle_command_reply: tid=0 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 client.16101 shutdown 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34ec50 msgr2=0x5597ed356830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34ec50 0x5597ed356830 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f14340137c0 tx=0x7f1434013c80 comp rx=0 tx=0).stop 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 shutdown_connections 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5597ed369390 0x5597ed36b820 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed356d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f1422ff5640 0 client.16101 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed34ec50 0x5597ed356830 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 --2- 172.21.15.28:0/2527192096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 0x5597ed3562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 >> 172.21.15.28:0/2527192096 conn(0x5597ed34a650 msgr2=0x5597ed350450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 shutdown_connections 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 1 -- 172.21.15.28:0/2527192096 wait complete. 2024-05-09T08:27:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.782+0000 7f14485ca740 20 client.16101 trim_cache size 0 max 16384 2024-05-09T08:27:27.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f14485ca740 1 Processor -- start 2024-05-09T08:27:27.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f14485ca740 1 -- start start 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f14485ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 0x5597ed364fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f14485ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed350a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f14485ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed355020 0x5597ed349630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f14485ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14401b1010 con 0x5597ed355020 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f14485ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f14401b2b50 con 0x5597ed359320 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f14485ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14401b16e0 con 0x5597ed34e330 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f143f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed355020 0x5597ed349630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f143f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed355020 0x5597ed349630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35182/0 (socket says 172.21.15.28:35182) 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f143f7fe640 1 -- 172.21.15.28:0/1315852374 learned_addr learned my addr 172.21.15.28:0/1315852374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f143f7fe640 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 0x5597ed364fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.796+0000 7f143e7fc640 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed350a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f143f7fe640 1 -- 172.21.15.28:0/1315852374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 msgr2=0x5597ed364fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f143f7fe640 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 0x5597ed364fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f143f7fe640 1 -- 172.21.15.28:0/1315852374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 msgr2=0x5597ed350a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f143f7fe640 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed350a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f143f7fe640 1 -- 172.21.15.28:0/1315852374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597ed34ea80 con 0x5597ed355020 2024-05-09T08:27:27.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f143e7fc640 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed350a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:27:27.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f143f7fe640 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed355020 0x5597ed349630 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f14380115e0 tx=0x7f1438011610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89aed033c114e228 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f1424ff9640 1 -- 172.21.15.28:0/1315852374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143802a5b0 con 0x5597ed355020 2024-05-09T08:27:27.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f1424ff9640 1 -- 172.21.15.28:0/1315852374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f143802a750 con 0x5597ed355020 2024-05-09T08:27:27.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.797+0000 7f1424ff9640 1 -- 172.21.15.28:0/1315852374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14380118e0 con 0x5597ed355020 2024-05-09T08:27:27.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 -- 172.21.15.28:0/1315852374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed355020 msgr2=0x5597ed349630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed355020 0x5597ed349630 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f14380115e0 tx=0x7f1438011610 comp rx=0 tx=0).stop 2024-05-09T08:27:27.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 -- 172.21.15.28:0/1315852374 shutdown_connections 2024-05-09T08:27:27.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed355020 0x5597ed349630 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed359320 0x5597ed350a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 --2- 172.21.15.28:0/1315852374 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed34e330 0x5597ed364fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 -- 172.21.15.28:0/1315852374 >> 172.21.15.28:0/1315852374 conn(0x5597ed34a650 msgr2=0x5597ed348f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 -- 172.21.15.28:0/1315852374 shutdown_connections 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 -- 172.21.15.28:0/1315852374 wait complete. 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 -- start start 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.798+0000 7f14485ca740 1 Processor -- start 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f14485ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34e330 0x5597ed3611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f14485ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed355020 0x5597ed3616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f14485ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed359320 0x5597ed35d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f14485ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14401b2480 con 0x5597ed359320 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f14485ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f14401b1db0 con 0x5597ed34e330 2024-05-09T08:27:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f14485ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14401aeff0 con 0x5597ed355020 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f14485ca740 10 client.0 _ll_register_callbacks cb 0x7f14340096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed359320 0x5597ed35d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed359320 0x5597ed35d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35188/0 (socket says 172.21.15.28:35188) 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 -- 172.21.15.28:0/3759284795 learned_addr learned my addr 172.21.15.28:0/3759284795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143e7fc640 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed355020 0x5597ed3616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34e330 0x5597ed3611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 -- 172.21.15.28:0/3759284795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed355020 msgr2=0x5597ed3616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed355020 0x5597ed3616f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 -- 172.21.15.28:0/3759284795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34e330 msgr2=0x5597ed3611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34e330 0x5597ed3611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 -- 172.21.15.28:0/3759284795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597ed35dd40 con 0x5597ed359320 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f143f7fe640 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed359320 0x5597ed35d630 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1438012000 tx=0x7f1438009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1511e0543d2412a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.799+0000 7f1422ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f1422ff5640 1 -- 172.21.15.28:0/3759284795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143802a900 con 0x5597ed359320 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f1422ff5640 1 -- 172.21.15.28:0/3759284795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f1438011890 con 0x5597ed359320 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f14485ca740 1 -- 172.21.15.28:0/3759284795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f143802a750 con 0x5597ed359320 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f1422ff5640 1 -- 172.21.15.28:0/3759284795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f143803f050 con 0x5597ed359320 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f14485ca740 10 client.16107 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f14485ca740 1 -- 172.21.15.28:0/3759284795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5597ed3d0320 con 0x5597ed359320 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f1422ff5640 1 -- 172.21.15.28:0/3759284795 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f143802a750 con 0x5597ed359320 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f1422ff5640 1 -- 172.21.15.28:0/3759284795 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f143801b4e0 con 0x5597ed359320 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f14485ca740 10 client.16107 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f14485ca740 10 client.16107 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f14485ca740 20 client.16107 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f14485ca740 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5597ed3d0790 0x5597ed3d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f14485ca740 4 client.16107 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:27:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.800+0000 7f14485ca740 1 -- 172.21.15.28:0/3759284795 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5597ed2bb6c0 con 0x5597ed3d0790 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.801+0000 7f143e7fc640 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5597ed3d0790 0x5597ed3d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.801+0000 7f1422ff5640 10 client.16107 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.801+0000 7f143e7fc640 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5597ed3d0790 0x5597ed3d2b80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e77a5f71883a6ae2 server_cookie=5500dc5082fe3e8f in_seq=0 out_seq=0 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.801+0000 7f1422ff5640 1 -- 172.21.15.28:0/3759284795 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5597ed2bb6c0 con 0x5597ed3d0790 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.801+0000 7f1422ff5640 10 client.16107 handle_command_reply: tid=0 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.801+0000 7f14485ca740 1 client.16107 shutdown 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 -- 172.21.15.28:0/3759284795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed359320 msgr2=0x5597ed35d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed359320 0x5597ed35d630 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1438012000 tx=0x7f1438009480 comp rx=0 tx=0).stop 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 -- 172.21.15.28:0/3759284795 shutdown_connections 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5597ed3d0790 0x5597ed3d2b80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f1422ff5640 0 client.16107 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597ed359320 0x5597ed35d630 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597ed355020 0x5597ed3616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 --2- 172.21.15.28:0/3759284795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597ed34e330 0x5597ed3611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 -- 172.21.15.28:0/3759284795 >> 172.21.15.28:0/3759284795 conn(0x5597ed34a650 msgr2=0x5597ed354790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 -- 172.21.15.28:0/3759284795 shutdown_connections 2024-05-09T08:27:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 1 -- 172.21.15.28:0/3759284795 wait complete. 2024-05-09T08:27:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.802+0000 7f14485ca740 20 client.16107 trim_cache size 0 max 16384 2024-05-09T08:27:27.808 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:27:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 -- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14280a2e20 msgr2=0x7f14280a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14280a2e20 0x7f14280a52e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14300ec9e0 tx=0x7f14300045f0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 -- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440100700 msgr2=0x7f1440196d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440100700 0x7f1440196d30 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1430009230 tx=0x7f1430004040 comp rx=0 tx=0).stop 2024-05-09T08:27:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 -- 172.21.15.28:0/1865095600 shutdown_connections 2024-05-09T08:27:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1440106710 0x7f14401977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14280a2e20 0x7f14280a52e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14401010c0 0x7f1440197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 --2- 172.21.15.28:0/1865095600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1440100700 0x7f1440196d30 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 -- 172.21.15.28:0/1865095600 >> 172.21.15.28:0/1865095600 conn(0x7f1440077830 msgr2=0x7f1440102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 -- 172.21.15.28:0/1865095600 shutdown_connections 2024-05-09T08:27:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.803+0000 7f14457dc640 1 -- 172.21.15.28:0/1865095600 wait complete. 2024-05-09T08:27:27.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:27:27.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.925+0000 7fda8067d640 1 -- 172.21.15.28:0/1908451740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda781021d0 msgr2=0x7fda7810e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.925+0000 7fda8067d640 1 --2- 172.21.15.28:0/1908451740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda781021d0 0x7fda7810e6e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fda70009cd0 tx=0x7fda7002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.925+0000 7fda8067d640 1 -- 172.21.15.28:0/1908451740 shutdown_connections 2024-05-09T08:27:27.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.925+0000 7fda8067d640 1 --2- 172.21.15.28:0/1908451740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda781021d0 0x7fda7810e6e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.925+0000 7fda8067d640 1 --2- 172.21.15.28:0/1908451740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda78101810 0x7fda78101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.925+0000 7fda8067d640 1 --2- 172.21.15.28:0/1908451740 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78107820 0x7fda78107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.925+0000 7fda8067d640 1 -- 172.21.15.28:0/1908451740 >> 172.21.15.28:0/1908451740 conn(0x7fda78078c30 msgr2=0x7fda78079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.926+0000 7fda8067d640 1 -- 172.21.15.28:0/1908451740 shutdown_connections 2024-05-09T08:27:27.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.926+0000 7fda8067d640 1 -- 172.21.15.28:0/1908451740 wait complete. 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.926+0000 7fda8067d640 1 Processor -- start 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.926+0000 7fda8067d640 1 -- start start 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda8067d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78101810 0x7fda78198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda8067d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda781021d0 0x7fda78198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda8067d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda78107820 0x7fda7819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda8067d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda78111bf0 con 0x7fda78107820 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda8067d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fda78111a70 con 0x7fda781021d0 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda8067d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fda78111d70 con 0x7fda78101810 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7dc18640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda781021d0 0x7fda78198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7ec1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda78107820 0x7fda7819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7dc18640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda781021d0 0x7fda78198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35810/0 (socket says 172.21.15.28:35810) 2024-05-09T08:27:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7dc18640 1 -- 172.21.15.28:0/2552741670 learned_addr learned my addr 172.21.15.28:0/2552741670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7e419640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78101810 0x7fda78198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7dc18640 1 -- 172.21.15.28:0/2552741670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78101810 msgr2=0x7fda78198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7dc18640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78101810 0x7fda78198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7dc18640 1 -- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda78107820 msgr2=0x7fda7819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7dc18640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda78107820 0x7fda7819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.927+0000 7fda7dc18640 1 -- 172.21.15.28:0/2552741670 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7819d040 con 0x7fda781021d0 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.928+0000 7fda7ec1a640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda78107820 0x7fda7819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.928+0000 7fda7e419640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78101810 0x7fda78198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.928+0000 7fda7dc18640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda781021d0 0x7fda78198590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fda6800e9f0 tx=0x7fda6800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.928+0000 7fda6f7fe640 1 -- 172.21.15.28:0/2552741670 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6800ccc0 con 0x7fda781021d0 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.928+0000 7fda6f7fe640 1 -- 172.21.15.28:0/2552741670 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda68010910 con 0x7fda781021d0 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.928+0000 7fda6f7fe640 1 -- 172.21.15.28:0/2552741670 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda680188f0 con 0x7fda781021d0 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.928+0000 7fda8067d640 1 -- 172.21.15.28:0/2552741670 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda7819d330 con 0x7fda781021d0 2024-05-09T08:27:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.928+0000 7fda8067d640 1 -- 172.21.15.28:0/2552741670 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda78073a60 con 0x7fda781021d0 2024-05-09T08:27:27.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.931+0000 7fda8338f740 1 Processor -- start 2024-05-09T08:27:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda8338f740 1 -- start start 2024-05-09T08:27:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda8338f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe0b330 0x5653dbe0b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda8338f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0bc50 0x5653dbe15de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda8338f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe18710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda8338f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5653dbd70c40 con 0x5653dbe16320 2024-05-09T08:27:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda8338f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5653dbd18580 con 0x5653dbe0bc50 2024-05-09T08:27:27.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda8338f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5653dbd6cc00 con 0x5653dbe0b330 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda7e419640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0bc50 0x5653dbe15de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda7e419640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0bc50 0x5653dbe15de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35822/0 (socket says 172.21.15.28:35822) 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda7ec1a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe0b330 0x5653dbe0b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda7e419640 1 -- 172.21.15.28:0/1018540301 learned_addr learned my addr 172.21.15.28:0/1018540301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.932+0000 7fda6f7fe640 1 -- 172.21.15.28:0/2552741670 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fda68010ac0 con 0x7fda781021d0 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.933+0000 7fda7dc18640 1 --2- 172.21.15.28:0/1018540301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe18710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.933+0000 7fda7e419640 1 -- 172.21.15.28:0/1018540301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe0b330 msgr2=0x5653dbe0b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.933+0000 7fda7e419640 1 --2- 172.21.15.28:0/1018540301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe0b330 0x5653dbe0b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.933+0000 7fda7e419640 1 -- 172.21.15.28:0/1018540301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 msgr2=0x5653dbe18710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.933+0000 7fda7e419640 1 --2- 172.21.15.28:0/1018540301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe18710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.933+0000 7fda7e419640 1 -- 172.21.15.28:0/1018540301 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653dba843f0 con 0x5653dbe0bc50 2024-05-09T08:27:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.933+0000 7fda7dc18640 1 --2- 172.21.15.28:0/1018540301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe18710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:27:27.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.933+0000 7fda7e419640 1 --2- 172.21.15.28:0/1018540301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0bc50 0x5653dbe15de0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda7400f440 tx=0x7fda7400f900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ef7b24993d67d58 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.933+0000 7fda6cff9640 1 -- 172.21.15.28:0/1018540301 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74005220 con 0x5653dbe0bc50 2024-05-09T08:27:27.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda6cff9640 1 -- 172.21.15.28:0/1018540301 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda74004c80 con 0x5653dbe0bc50 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda6f7fe640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2e20 0x7fda640a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda6f7fe640 1 -- 172.21.15.28:0/2552741670 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fda680d0190 con 0x7fda781021d0 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda8338f740 1 -- 172.21.15.28:0/1018540301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0bc50 msgr2=0x5653dbe15de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda8338f740 1 --2- 172.21.15.28:0/1018540301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0bc50 0x5653dbe15de0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda7400f440 tx=0x7fda7400f900 comp rx=0 tx=0).stop 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda8338f740 1 -- 172.21.15.28:0/1018540301 shutdown_connections 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda8338f740 1 --2- 172.21.15.28:0/1018540301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe18710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda8338f740 1 --2- 172.21.15.28:0/1018540301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0bc50 0x5653dbe15de0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda8338f740 1 --2- 172.21.15.28:0/1018540301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe0b330 0x5653dbe0b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda8338f740 1 -- 172.21.15.28:0/1018540301 >> 172.21.15.28:0/1018540301 conn(0x5653dbe07650 msgr2=0x5653dbe07a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.934+0000 7fda6cff9640 1 -- 172.21.15.28:0/1018540301 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7fda74010cd0 con 0x5653dbe0bc50 2024-05-09T08:27:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.935+0000 7fda7ec1a640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2e20 0x7fda640a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.935+0000 7fda7ec1a640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2e20 0x7fda640a52e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fda70006010 tx=0x7fda70031800 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.935+0000 7fda8338f740 1 -- 172.21.15.28:0/1018540301 shutdown_connections 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.935+0000 7fda8338f740 1 -- 172.21.15.28:0/1018540301 wait complete. 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.935+0000 7fda8338f740 1 -- start start 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.935+0000 7fda8338f740 1 Processor -- start 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda8338f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe0b330 0x5653dbe15ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda8338f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe16320 0x5653dbe12db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda8338f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe13770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7e419640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe0b330 0x5653dbe15ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7e419640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe0b330 0x5653dbe15ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35216/0 (socket says 172.21.15.28:35216) 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7e419640 1 -- 172.21.15.28:0/4234088916 learned_addr learned my addr 172.21.15.28:0/4234088916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5653dbe19160 con 0x5653dbe0b330 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5653dbd2bc40 con 0x5653dbe16320 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5653dbe192e0 con 0x5653dbe132f0 2024-05-09T08:27:27.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda8338f740 10 client.0 _ll_register_callbacks cb 0x5653dba83be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7ec1a640 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe13770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7dc18640 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe16320 0x5653dbe12db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7e419640 1 -- 172.21.15.28:0/4234088916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 msgr2=0x5653dbe13770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7e419640 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe13770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7e419640 1 -- 172.21.15.28:0/4234088916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe16320 msgr2=0x5653dbe12db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7e419640 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe16320 0x5653dbe12db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.936+0000 7fda7e419640 1 -- 172.21.15.28:0/4234088916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653dbb4e7d0 con 0x5653dbe0b330 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.937+0000 7fda7e419640 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe0b330 0x5653dbe15ac0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fda74000c00 tx=0x7fda74010c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2066ff91128db773 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.937+0000 7fda61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.937+0000 7fda61ffb640 1 -- 172.21.15.28:0/4234088916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74026070 con 0x5653dbe0b330 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.937+0000 7fda61ffb640 1 -- 172.21.15.28:0/4234088916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fda74005240 con 0x5653dbe0b330 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.937+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda74004c80 con 0x5653dbe0b330 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.937+0000 7fda61ffb640 1 -- 172.21.15.28:0/4234088916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda74032050 con 0x5653dbe0b330 2024-05-09T08:27:27.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.937+0000 7fda8338f740 10 client.16116 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.937+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5653dbe25310 con 0x5653dbe0b330 2024-05-09T08:27:27.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.938+0000 7fda61ffb640 1 -- 172.21.15.28:0/4234088916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda74004c80 con 0x5653dbe0b330 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.938+0000 7fda61ffb640 1 -- 172.21.15.28:0/4234088916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fda74018920 con 0x5653dbe0b330 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.939+0000 7fda8338f740 10 client.16116 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.939+0000 7fda8338f740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.939+0000 7fda8338f740 20 client.16116 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.939+0000 7fda8338f740 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5653dbe26720 0x5653dbe28b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.939+0000 7fda8338f740 4 client.16116 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.939+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653dbd7e370 con 0x5653dbe26720 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.939+0000 7fda7dc18640 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5653dbe26720 0x5653dbe28b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.939+0000 7fda61ffb640 10 client.16116 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:27.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.939+0000 7fda7dc18640 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5653dbe26720 0x5653dbe28b10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95c67604d9c5119f server_cookie=b9a1ef268632a03b in_seq=0 out_seq=0 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda61ffb640 1 -- 172.21.15.28:0/4234088916 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653dbd7e370 con 0x5653dbe26720 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda61ffb640 10 client.16116 handle_command_reply: tid=0 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 client.16116 shutdown 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe0b330 msgr2=0x5653dbe15ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe0b330 0x5653dbe15ac0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fda74000c00 tx=0x7fda74010c40 comp rx=0 tx=0).stop 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 shutdown_connections 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5653dbe26720 0x5653dbe28b10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda61ffb640 0 client.16116 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe13770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe16320 0x5653dbe12db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 --2- 172.21.15.28:0/4234088916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe0b330 0x5653dbe15ac0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 >> 172.21.15.28:0/4234088916 conn(0x5653dbe07650 msgr2=0x5653dbe0cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.940+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 shutdown_connections 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.941+0000 7fda8338f740 1 -- 172.21.15.28:0/4234088916 wait complete. 2024-05-09T08:27:27.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.941+0000 7fda8338f740 20 client.16116 trim_cache size 0 max 16384 2024-05-09T08:27:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.967+0000 7fda8338f740 1 Processor -- start 2024-05-09T08:27:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.967+0000 7fda8338f740 1 -- start start 2024-05-09T08:27:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.967+0000 7fda8338f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 0x5653dbe220a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda8338f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe12690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda8338f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe2e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda8338f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b1060 con 0x5653dbe16320 2024-05-09T08:27:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda8338f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b2ba0 con 0x5653dbe0b330 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda8338f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b1730 con 0x5653dbe132f0 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7ec1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe2e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7ec1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe2e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35220/0 (socket says 172.21.15.28:35220) 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7ec1a640 1 -- 172.21.15.28:0/568955253 learned_addr learned my addr 172.21.15.28:0/568955253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7e419640 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 0x5653dbe220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7dc18640 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe12690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7ec1a640 1 -- 172.21.15.28:0/568955253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 msgr2=0x5653dbe12690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7ec1a640 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe12690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7ec1a640 1 -- 172.21.15.28:0/568955253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 msgr2=0x5653dbe220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7ec1a640 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 0x5653dbe220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7ec1a640 1 -- 172.21.15.28:0/568955253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653dbe13ae0 con 0x5653dbe16320 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7dc18640 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe12690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.968+0000 7fda7ec1a640 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe2e140 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fda70031df0 tx=0x7fda70040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677a1c22e5b3bd81 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.969+0000 7fda63fff640 1 -- 172.21.15.28:0/568955253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70056070 con 0x5653dbe16320 2024-05-09T08:27:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.969+0000 7fda63fff640 1 -- 172.21.15.28:0/568955253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda70036040 con 0x5653dbe16320 2024-05-09T08:27:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.969+0000 7fda63fff640 1 -- 172.21.15.28:0/568955253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70030ae0 con 0x5653dbe16320 2024-05-09T08:27:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 -- 172.21.15.28:0/568955253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 msgr2=0x5653dbe2e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe2e140 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fda70031df0 tx=0x7fda70040000 comp rx=0 tx=0).stop 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 -- 172.21.15.28:0/568955253 shutdown_connections 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe2e140 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe12690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 --2- 172.21.15.28:0/568955253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 0x5653dbe220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 -- 172.21.15.28:0/568955253 >> 172.21.15.28:0/568955253 conn(0x5653dbe07650 msgr2=0x5653dbe07a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 -- 172.21.15.28:0/568955253 shutdown_connections 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 -- 172.21.15.28:0/568955253 wait complete. 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 -- start start 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.970+0000 7fda8338f740 1 Processor -- start 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda8338f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 0x5653dbe19b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda8338f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe1a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda8338f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe1e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda8338f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b24d0 con 0x5653dbe16320 2024-05-09T08:27:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda8338f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b1e00 con 0x5653dbe0b330 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda7ec1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe1e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda7ec1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe1e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35228/0 (socket says 172.21.15.28:35228) 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda7ec1a640 1 -- 172.21.15.28:0/3177406154 learned_addr learned my addr 172.21.15.28:0/3177406154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda7e419640 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 0x5653dbe19b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda8338f740 1 -- 172.21.15.28:0/3177406154 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fda781af040 con 0x5653dbe132f0 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda7dc18640 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe1a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda8338f740 10 client.0 _ll_register_callbacks cb 0x7fda70009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.971+0000 7fda7dc18640 1 -- 172.21.15.28:0/3177406154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 msgr2=0x5653dbe19b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda7dc18640 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 0x5653dbe19b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda7dc18640 1 -- 172.21.15.28:0/3177406154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 msgr2=0x5653dbe1e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda7dc18640 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe1e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda7dc18640 1 -- 172.21.15.28:0/3177406154 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653dbe1e670 con 0x5653dbe132f0 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda7ec1a640 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe1e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda8338f740 1 -- 172.21.15.28:0/3177406154 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda70036040 con 0x5653dbe132f0 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda7dc18640 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe1a0d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda680cd390 tx=0x7fda680bc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c37a0f947744d76d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda61ffb640 10 client.25540 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda61ffb640 1 -- 172.21.15.28:0/3177406154 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda680a0110 con 0x5653dbe132f0 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda61ffb640 1 -- 172.21.15.28:0/3177406154 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fda68098700 con 0x5653dbe132f0 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda61ffb640 1 -- 172.21.15.28:0/3177406154 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda680cd670 con 0x5653dbe132f0 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda8338f740 10 client.25540 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.972+0000 7fda8338f740 1 -- 172.21.15.28:0/3177406154 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5653dbe8c460 con 0x5653dbe132f0 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda61ffb640 1 -- 172.21.15.28:0/3177406154 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fda6809b4e0 con 0x5653dbe132f0 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda8338f740 10 client.25540 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda8338f740 10 client.25540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda8338f740 20 client.25540 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda8338f740 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5653dbe26720 0x5653dbe8eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda8338f740 4 client.25540 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda8338f740 1 -- 172.21.15.28:0/3177406154 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653dbd786c0 con 0x5653dbe26720 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda7e419640 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5653dbe26720 0x5653dbe8eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda61ffb640 10 client.25540 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.973+0000 7fda7e419640 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5653dbe26720 0x5653dbe8eb10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d79746b2baf0976 server_cookie=9a999c92cb88e59e in_seq=0 out_seq=0 2024-05-09T08:27:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda61ffb640 1 -- 172.21.15.28:0/3177406154 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653dbd786c0 con 0x5653dbe26720 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda61ffb640 10 client.25540 handle_command_reply: tid=0 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 client.25540 shutdown 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 -- 172.21.15.28:0/3177406154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 msgr2=0x5653dbe1a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe1a0d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda680cd390 tx=0x7fda680bc040 comp rx=0 tx=0).stop 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 -- 172.21.15.28:0/3177406154 shutdown_connections 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5653dbe26720 0x5653dbe8eb10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653dbe16320 0x5653dbe1e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5653dbe132f0 0x5653dbe1a0d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 --2- 172.21.15.28:0/3177406154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653dbe0b330 0x5653dbe19b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 -- 172.21.15.28:0/3177406154 >> 172.21.15.28:0/3177406154 conn(0x5653dbe07650 msgr2=0x5653dbe0b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda61ffb640 0 client.25540 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 -- 172.21.15.28:0/3177406154 shutdown_connections 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 1 -- 172.21.15.28:0/3177406154 wait complete. 2024-05-09T08:27:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8338f740 20 client.25540 trim_cache size 0 max 16384 2024-05-09T08:27:27.979 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:27:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.974+0000 7fda8067d640 1 -- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2e20 msgr2=0x7fda640a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2e20 0x7fda640a52e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fda70006010 tx=0x7fda70031800 comp rx=0 tx=0).stop 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 -- 172.21.15.28:0/2552741670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda781021d0 msgr2=0x7fda78198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda781021d0 0x7fda78198590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fda6800e9f0 tx=0x7fda6800edf0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 -- 172.21.15.28:0/2552741670 shutdown_connections 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda78107820 0x7fda7819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda781021d0 0x7fda78198590 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2e20 0x7fda640a52e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 --2- 172.21.15.28:0/2552741670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78101810 0x7fda78198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 -- 172.21.15.28:0/2552741670 >> 172.21.15.28:0/2552741670 conn(0x7fda78078c30 msgr2=0x7fda78103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 -- 172.21.15.28:0/2552741670 shutdown_connections 2024-05-09T08:27:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:27.975+0000 7fda8067d640 1 -- 172.21.15.28:0/2552741670 wait complete. 2024-05-09T08:27:28.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:28 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3430416059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:27:28.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:28 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3430416059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:27:28.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:28 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3430416059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:27:28.987 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:27:28.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:27:28.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:27:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.110+0000 7fd306d1b640 1 -- 172.21.15.28:0/2165428998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd300102ab0 msgr2=0x7fd300102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.110+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2165428998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd300102ab0 0x7fd300102e90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009900 tx=0x7fd2f402a510 comp rx=0 tx=0).stop 2024-05-09T08:27:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.111+0000 7fd306d1b640 1 -- 172.21.15.28:0/2165428998 shutdown_connections 2024-05-09T08:27:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.111+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2165428998 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd30010bdf0 0x7fd30010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.111+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2165428998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3001033d0 0x7fd30010b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.111+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2165428998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd300102ab0 0x7fd300102e90 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.111+0000 7fd306d1b640 1 -- 172.21.15.28:0/2165428998 >> 172.21.15.28:0/2165428998 conn(0x7fd300077830 msgr2=0x7fd300077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.111+0000 7fd306d1b640 1 -- 172.21.15.28:0/2165428998 shutdown_connections 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.111+0000 7fd306d1b640 1 -- 172.21.15.28:0/2165428998 wait complete. 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.111+0000 7fd306d1b640 1 Processor -- start 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.111+0000 7fd306d1b640 1 -- start start 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd306d1b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300102ab0 0x7fd300198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd306d1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3001033d0 0x7fd300198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd306d1b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd30010bdf0 0x7fd30019c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd306d1b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd300111bf0 con 0x7fd3001033d0 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd306d1b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd300111a70 con 0x7fd300102ab0 2024-05-09T08:27:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd306d1b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd300111d70 con 0x7fd30010bdf0 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300102ab0 0x7fd300198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd2fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3001033d0 0x7fd300198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300102ab0 0x7fd300198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35854/0 (socket says 172.21.15.28:35854) 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd304ab7640 1 -- 172.21.15.28:0/2508700585 learned_addr learned my addr 172.21.15.28:0/2508700585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd3052b8640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd30010bdf0 0x7fd30019c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd304ab7640 1 -- 172.21.15.28:0/2508700585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd30010bdf0 msgr2=0x7fd30019c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd304ab7640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd30010bdf0 0x7fd30019c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd304ab7640 1 -- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3001033d0 msgr2=0x7fd300198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd304ab7640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3001033d0 0x7fd300198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd304ab7640 1 -- 172.21.15.28:0/2508700585 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30019d100 con 0x7fd300102ab0 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.112+0000 7fd2fffff640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3001033d0 0x7fd300198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.113+0000 7fd304ab7640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300102ab0 0x7fd300198110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4006010 tx=0x7fd2f40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.113+0000 7fd2fdffb640 1 -- 172.21.15.28:0/2508700585 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f403f030 con 0x7fd300102ab0 2024-05-09T08:27:29.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.113+0000 7fd2fdffb640 1 -- 172.21.15.28:0/2508700585 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2f4003ce0 con 0x7fd300102ab0 2024-05-09T08:27:29.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.113+0000 7fd306d1b640 1 -- 172.21.15.28:0/2508700585 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd30019d390 con 0x7fd300102ab0 2024-05-09T08:27:29.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.113+0000 7fd2fdffb640 1 -- 172.21.15.28:0/2508700585 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4035070 con 0x7fd300102ab0 2024-05-09T08:27:29.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.113+0000 7fd306d1b640 1 -- 172.21.15.28:0/2508700585 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3001a4c80 con 0x7fd300102ab0 2024-05-09T08:27:29.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.115+0000 7fd2fdffb640 1 -- 172.21.15.28:0/2508700585 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fd2f4042030 con 0x7fd300102ab0 2024-05-09T08:27:29.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.116+0000 7fd2fdffb640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2f00a2e20 0x7fd2f00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.116+0000 7fd2fdffb640 1 -- 172.21.15.28:0/2508700585 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fd2f40efe70 con 0x7fd300102ab0 2024-05-09T08:27:29.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.116+0000 7fd2fffff640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2f00a2e20 0x7fd2f00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.117+0000 7fd309a2c740 1 Processor -- start 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.117+0000 7fd309a2c740 1 -- start start 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.117+0000 7fd2fffff640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2f00a2e20 0x7fd2f00a52e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd30010dbe0 tx=0x7fd2ec00b390 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.117+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e203f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.117+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e20db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.117+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 0x557592e36680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.117+0000 7fd309a2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557592d85990 con 0x557592e20930 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.117+0000 7fd309a2c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557592d2d5b0 con 0x557592e34290 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.117+0000 7fd309a2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557592d81950 con 0x557592e21dc0 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e203f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e203f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42824/0 (socket says 172.21.15.28:42824) 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd304ab7640 1 -- 172.21.15.28:0/943080260 learned_addr learned my addr 172.21.15.28:0/943080260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd2fffff640 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e20db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd3052b8640 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 0x557592e36680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd304ab7640 1 -- 172.21.15.28:0/943080260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 msgr2=0x557592e36680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd304ab7640 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 0x557592e36680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd304ab7640 1 -- 172.21.15.28:0/943080260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 msgr2=0x557592e20db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd304ab7640 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e20db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd304ab7640 1 -- 172.21.15.28:0/943080260 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557592b617e0 con 0x557592e21dc0 2024-05-09T08:27:29.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd2fffff640 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e20db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:29.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.118+0000 7fd304ab7640 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e203f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd2f40ed7f0 tx=0x7fd2f4003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce538f5866851d36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.119+0000 7fd2eaffd640 1 -- 172.21.15.28:0/943080260 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4049040 con 0x557592e21dc0 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.119+0000 7fd2eaffd640 1 -- 172.21.15.28:0/943080260 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2f403b580 con 0x557592e21dc0 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.119+0000 7fd2eaffd640 1 -- 172.21.15.28:0/943080260 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40c46e0 con 0x557592e21dc0 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.119+0000 7fd309a2c740 1 -- 172.21.15.28:0/943080260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 msgr2=0x557592e203f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.119+0000 7fd309a2c740 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e203f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd2f40ed7f0 tx=0x7fd2f4003f90 comp rx=0 tx=0).stop 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.119+0000 7fd309a2c740 1 -- 172.21.15.28:0/943080260 shutdown_connections 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.119+0000 7fd309a2c740 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 0x557592e36680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.119+0000 7fd309a2c740 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e20db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.120+0000 7fd309a2c740 1 --2- 172.21.15.28:0/943080260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e203f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.120+0000 7fd309a2c740 1 -- 172.21.15.28:0/943080260 >> 172.21.15.28:0/943080260 conn(0x557592de90a0 msgr2=0x557592e1cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.120+0000 7fd309a2c740 1 -- 172.21.15.28:0/943080260 shutdown_connections 2024-05-09T08:27:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.120+0000 7fd309a2c740 1 -- 172.21.15.28:0/943080260 wait complete. 2024-05-09T08:27:29.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.120+0000 7fd309a2c740 1 -- start start 2024-05-09T08:27:29.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.121+0000 7fd309a2c740 1 Processor -- start 2024-05-09T08:27:29.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.121+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e2d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e2dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e2d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e2d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35258/0 (socket says 172.21.15.28:35258) 2024-05-09T08:27:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd304ab7640 1 -- 172.21.15.28:0/433501934 learned_addr learned my addr 172.21.15.28:0/433501934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd309a2c740 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 0x557592e336c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557592e27a60 con 0x557592e20930 2024-05-09T08:27:29.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557592d40b40 con 0x557592e21dc0 2024-05-09T08:27:29.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd2fffff640 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e2dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557592e27be0 con 0x557592e34290 2024-05-09T08:27:29.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.122+0000 7fd3052b8640 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 0x557592e336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd309a2c740 10 client.0 _ll_register_callbacks cb 0x557592d350f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2fffff640 1 -- 172.21.15.28:0/433501934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 msgr2=0x557592e336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2fffff640 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 0x557592e336c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2fffff640 1 -- 172.21.15.28:0/433501934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 msgr2=0x557592e2d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2fffff640 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e2d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2fffff640 1 -- 172.21.15.28:0/433501934 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557592bc1f70 con 0x557592e21dc0 2024-05-09T08:27:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd304ab7640 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e2d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd3052b8640 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 0x557592e336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557592d930c0 con 0x557592e21dc0 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2fffff640 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e2dee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x557592e23600 tx=0x7fd2ec004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72a4e9be074988c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2dffff640 10 client.25613 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2dffff640 1 -- 172.21.15.28:0/433501934 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec004020 con 0x557592e21dc0 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2dffff640 1 -- 172.21.15.28:0/433501934 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fd2ec0041c0 con 0x557592e21dc0 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd2dffff640 1 -- 172.21.15.28:0/433501934 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd2ec004400 con 0x557592e21dc0 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.123+0000 7fd309a2c740 10 client.25613 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.124+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({fsmap=28}) -- 0x557592a59170 con 0x557592e21dc0 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.124+0000 7fd2dffff640 1 -- 172.21.15.28:0/433501934 <== mon.2 v2:172.21.15.165:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd2ec026070 con 0x557592e21dc0 2024-05-09T08:27:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.124+0000 7fd309a2c740 10 client.25613 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.124+0000 7fd309a2c740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.124+0000 7fd309a2c740 20 client.25613 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.124+0000 7fd309a2c740 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557592e4c3b0 0x557592e4e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.124+0000 7fd309a2c740 4 client.25613 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.124+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557592d8d410 con 0x557592e4c3b0 2024-05-09T08:27:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.124+0000 7fd304ab7640 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557592e4c3b0 0x557592e4e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.125+0000 7fd2dffff640 10 client.25613 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:29.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.125+0000 7fd304ab7640 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557592e4c3b0 0x557592e4e7a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac9fa663184d7fe0 server_cookie=d61d41895d01ba5d in_seq=0 out_seq=0 2024-05-09T08:27:29.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.125+0000 7fd2dffff640 1 -- 172.21.15.28:0/433501934 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557592d8d410 con 0x557592e4c3b0 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.125+0000 7fd2dffff640 10 client.25613 handle_command_reply: tid=0 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.125+0000 7fd309a2c740 1 client.25613 shutdown 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.125+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 msgr2=0x557592e2dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e2dee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x557592e23600 tx=0x7fd2ec004730 comp rx=0 tx=0).stop 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 shutdown_connections 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557592e4c3b0 0x557592e4e7a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd2dffff640 0 client.25613 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 0x557592e336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e2dee0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 1 --2- 172.21.15.28:0/433501934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e2d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 >> 172.21.15.28:0/433501934 conn(0x557592de90a0 msgr2=0x557592e22d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 shutdown_connections 2024-05-09T08:27:29.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 1 -- 172.21.15.28:0/433501934 wait complete. 2024-05-09T08:27:29.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.126+0000 7fd309a2c740 20 client.25613 trim_cache size 0 max 16384 2024-05-09T08:27:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.161+0000 7fd309a2c740 1 Processor -- start 2024-05-09T08:27:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.161+0000 7fd309a2c740 1 -- start start 2024-05-09T08:27:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e33630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e22ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 0x557592e2d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd309a2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001acb60 con 0x557592e20930 2024-05-09T08:27:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd309a2c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001ae6a0 con 0x557592e34290 2024-05-09T08:27:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd309a2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001ad230 con 0x557592e21dc0 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e33630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e33630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35264/0 (socket says 172.21.15.28:35264) 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd304ab7640 1 -- 172.21.15.28:0/2005246187 learned_addr learned my addr 172.21.15.28:0/2005246187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd3052b8640 1 --2- 172.21.15.28:0/2005246187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 0x557592e2d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd2fffff640 1 --2- 172.21.15.28:0/2005246187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e22ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd304ab7640 1 -- 172.21.15.28:0/2005246187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 msgr2=0x557592e22ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd304ab7640 1 --2- 172.21.15.28:0/2005246187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e22ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd304ab7640 1 -- 172.21.15.28:0/2005246187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 msgr2=0x557592e2d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd304ab7640 1 --2- 172.21.15.28:0/2005246187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 0x557592e2d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd304ab7640 1 -- 172.21.15.28:0/2005246187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557592e21400 con 0x557592e20930 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.162+0000 7fd304ab7640 1 --2- 172.21.15.28:0/2005246187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e33630 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4000c00 tx=0x7fd2f4005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a3e1634bb78195 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd2ea7fc640 1 -- 172.21.15.28:0/2005246187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4049040 con 0x557592e20930 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd2ea7fc640 1 -- 172.21.15.28:0/2005246187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2f4004460 con 0x557592e20930 2024-05-09T08:27:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd2ea7fc640 1 -- 172.21.15.28:0/2005246187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40ed350 con 0x557592e20930 2024-05-09T08:27:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd309a2c740 1 -- 172.21.15.28:0/2005246187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 msgr2=0x557592e33630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd309a2c740 1 --2- 172.21.15.28:0/2005246187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e33630 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4000c00 tx=0x7fd2f4005890 comp rx=0 tx=0).stop 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd309a2c740 1 -- 172.21.15.28:0/2005246187 shutdown_connections 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd309a2c740 1 --2- 172.21.15.28:0/2005246187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e34290 0x557592e2d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd309a2c740 1 --2- 172.21.15.28:0/2005246187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e21dc0 0x557592e22ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd309a2c740 1 --2- 172.21.15.28:0/2005246187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e33630 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.163+0000 7fd309a2c740 1 -- 172.21.15.28:0/2005246187 >> 172.21.15.28:0/2005246187 conn(0x557592de90a0 msgr2=0x557592e22490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.164+0000 7fd309a2c740 1 -- 172.21.15.28:0/2005246187 shutdown_connections 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.164+0000 7fd309a2c740 1 -- 172.21.15.28:0/2005246187 wait complete. 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.164+0000 7fd309a2c740 1 -- start start 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.164+0000 7fd309a2c740 1 Processor -- start 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.164+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e5c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.164+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e5cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd309a2c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 0x557592eb2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd309a2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001adfd0 con 0x557592e20930 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd309a2c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001ad900 con 0x557592e21dc0 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd309a2c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001aab40 con 0x557592e34290 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd309a2c740 10 client.0 _ll_register_callbacks cb 0x7fd2ec011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e5c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd304ab7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e5c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35266/0 (socket says 172.21.15.28:35266) 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd304ab7640 1 -- 172.21.15.28:0/392286548 learned_addr learned my addr 172.21.15.28:0/392286548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd2fffff640 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e5cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd3052b8640 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 0x557592eb2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd304ab7640 1 -- 172.21.15.28:0/392286548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 msgr2=0x557592eb2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd304ab7640 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 0x557592eb2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd304ab7640 1 -- 172.21.15.28:0/392286548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 msgr2=0x557592e5cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd304ab7640 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e5cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd304ab7640 1 -- 172.21.15.28:0/392286548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557592eb27b0 con 0x557592e20930 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd2fffff640 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e5cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd304ab7640 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e5c690 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd2f404a000 tx=0x7fd2f4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9042039d7cea33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd2dffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.165+0000 7fd2dffff640 1 -- 172.21.15.28:0/392286548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40b5060 con 0x557592e20930 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.166+0000 7fd309a2c740 1 -- 172.21.15.28:0/392286548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2f4004460 con 0x557592e20930 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.166+0000 7fd2dffff640 1 -- 172.21.15.28:0/392286548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fd2f40ed350 con 0x557592e20930 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.166+0000 7fd2dffff640 1 -- 172.21.15.28:0/392286548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd2f4049040 con 0x557592e20930 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.166+0000 7fd309a2c740 10 client.16140 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.166+0000 7fd309a2c740 1 -- 172.21.15.28:0/392286548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557592eb3af0 con 0x557592e20930 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.166+0000 7fd2dffff640 1 -- 172.21.15.28:0/392286548 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2f4004460 con 0x557592e20930 2024-05-09T08:27:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.166+0000 7fd2dffff640 1 -- 172.21.15.28:0/392286548 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd2f40c8070 con 0x557592e20930 2024-05-09T08:27:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.167+0000 7fd309a2c740 10 client.16140 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.167+0000 7fd309a2c740 10 client.16140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.167+0000 7fd309a2c740 20 client.16140 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.167+0000 7fd309a2c740 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557592eb3f80 0x557592eb6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.167+0000 7fd309a2c740 4 client.16140 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:27:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.167+0000 7fd309a2c740 1 -- 172.21.15.28:0/392286548 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557592d0ada0 con 0x557592eb3f80 2024-05-09T08:27:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.167+0000 7fd2fffff640 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557592eb3f80 0x557592eb6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.167+0000 7fd2fffff640 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557592eb3f80 0x557592eb6370 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dde22874de2b88d server_cookie=157effebb3ac15ec in_seq=0 out_seq=0 2024-05-09T08:27:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.167+0000 7fd2dffff640 10 client.16140 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd2dffff640 1 -- 172.21.15.28:0/392286548 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557592d0ada0 con 0x557592eb3f80 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd2dffff640 10 client.16140 handle_command_reply: tid=0 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 client.16140 shutdown 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 -- 172.21.15.28:0/392286548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 msgr2=0x557592e5c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e5c690 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd2f404a000 tx=0x7fd2f4003f90 comp rx=0 tx=0).stop 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 -- 172.21.15.28:0/392286548 shutdown_connections 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557592eb3f80 0x557592eb6370 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557592e34290 0x557592eb2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557592e21dc0 0x557592e5cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd2dffff640 0 client.16140 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 --2- 172.21.15.28:0/392286548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557592e20930 0x557592e5c690 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 -- 172.21.15.28:0/392286548 >> 172.21.15.28:0/392286548 conn(0x557592de90a0 msgr2=0x557592e22cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.169+0000 7fd309a2c740 1 -- 172.21.15.28:0/392286548 shutdown_connections 2024-05-09T08:27:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd309a2c740 1 -- 172.21.15.28:0/392286548 wait complete. 2024-05-09T08:27:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd309a2c740 20 client.16140 trim_cache size 0 max 16384 2024-05-09T08:27:29.174 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:27:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-09T08:27:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "13db0616-1433-48c9-95ff-2eb2e1805eb9", 2024-05-09T08:27:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-09T08:27:29.174 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:27:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 -- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2f00a2e20 msgr2=0x7fd2f00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2f00a2e20 0x7fd2f00a52e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd30010dbe0 tx=0x7fd2ec00b390 comp rx=0 tx=0).stop 2024-05-09T08:27:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 -- 172.21.15.28:0/2508700585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300102ab0 msgr2=0x7fd300198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300102ab0 0x7fd300198110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4006010 tx=0x7fd2f40042f0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 -- 172.21.15.28:0/2508700585 shutdown_connections 2024-05-09T08:27:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2f00a2e20 0x7fd2f00a52e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd30010bdf0 0x7fd30019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3001033d0 0x7fd300198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 --2- 172.21.15.28:0/2508700585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300102ab0 0x7fd300198110 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.170+0000 7fd306d1b640 1 -- 172.21.15.28:0/2508700585 >> 172.21.15.28:0/2508700585 conn(0x7fd300077830 msgr2=0x7fd3000fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.171+0000 7fd306d1b640 1 -- 172.21.15.28:0/2508700585 shutdown_connections 2024-05-09T08:27:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.171+0000 7fd306d1b640 1 -- 172.21.15.28:0/2508700585 wait complete. 2024-05-09T08:27:29.186 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:27:29.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:27:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.296+0000 7f18bbfaf640 1 -- 172.21.15.28:0/4122859842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b410e1e0 msgr2=0x7f18b41105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.296+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/4122859842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b410e1e0 0x7f18b41105d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009ca0 tx=0x7f18ac02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.296+0000 7f18bbfaf640 1 -- 172.21.15.28:0/4122859842 shutdown_connections 2024-05-09T08:27:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.296+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/4122859842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b410e1e0 0x7f18b41105d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.296+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/4122859842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18b4100f50 0x7f18b410dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.296+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/4122859842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4100630 0x7f18b4100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.296+0000 7f18bbfaf640 1 -- 172.21.15.28:0/4122859842 >> 172.21.15.28:0/4122859842 conn(0x7f18b4077830 msgr2=0x7f18b4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.296+0000 7f18bbfaf640 1 -- 172.21.15.28:0/4122859842 shutdown_connections 2024-05-09T08:27:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.296+0000 7f18bbfaf640 1 -- 172.21.15.28:0/4122859842 wait complete. 2024-05-09T08:27:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18bbfaf640 1 Processor -- start 2024-05-09T08:27:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18bbfaf640 1 -- start start 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18bbfaf640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4100630 0x7f18b419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18bbfaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b4100f50 0x7f18b419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18bbfaf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18b410e1e0 0x7f18b419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18bbfaf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18b4113e60 con 0x7f18b4100f50 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18bbfaf640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f18b4113ce0 con 0x7f18b4100630 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18bbfaf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f18b4113fe0 con 0x7f18b410e1e0 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18b954a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b4100f50 0x7f18b419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18b954a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b4100f50 0x7f18b419a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35284/0 (socket says 172.21.15.28:35284) 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.297+0000 7f18b954a640 1 -- 172.21.15.28:0/2078152655 learned_addr learned my addr 172.21.15.28:0/2078152655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18ba54c640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18b410e1e0 0x7f18b419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18b954a640 1 -- 172.21.15.28:0/2078152655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18b410e1e0 msgr2=0x7f18b419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18b954a640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18b410e1e0 0x7f18b419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18b954a640 1 -- 172.21.15.28:0/2078152655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4100630 msgr2=0x7f18b419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18b954a640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4100630 0x7f18b419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18b954a640 1 -- 172.21.15.28:0/2078152655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b419f280 con 0x7f18b4100f50 2024-05-09T08:27:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18b954a640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b4100f50 0x7f18b419a820 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f18a400d950 tx=0x7f18a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18aaffd640 1 -- 172.21.15.28:0/2078152655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4017070 con 0x7f18b4100f50 2024-05-09T08:27:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18aaffd640 1 -- 172.21.15.28:0/2078152655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a4009c30 con 0x7f18b4100f50 2024-05-09T08:27:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18aaffd640 1 -- 172.21.15.28:0/2078152655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a400f6b0 con 0x7f18b4100f50 2024-05-09T08:27:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.298+0000 7f18bbfaf640 1 -- 172.21.15.28:0/2078152655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b419f570 con 0x7f18b4100f50 2024-05-09T08:27:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.299+0000 7f18bbfaf640 1 -- 172.21.15.28:0/2078152655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b41a6e00 con 0x7f18b4100f50 2024-05-09T08:27:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.301+0000 7f18aaffd640 1 -- 172.21.15.28:0/2078152655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f18a40109d0 con 0x7f18b4100f50 2024-05-09T08:27:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.301+0000 7f18bed95740 1 Processor -- start 2024-05-09T08:27:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18aaffd640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f18a00a2f20 0x7f18a00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18bed95740 1 -- start start 2024-05-09T08:27:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18aaffd640 1 -- 172.21.15.28:0/2078152655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f18a40cf660 con 0x7f18b4100f50 2024-05-09T08:27:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18ba54c640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f18a00a2f20 0x7f18a00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18bed95740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fb2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18bed95740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 0x556732fbd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18bed95740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fbda80 0x556732fbfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18bed95740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556732f0fd00 con 0x556732fb2a90 2024-05-09T08:27:29.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18bed95740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x556732eb74c0 con 0x556732fb33b0 2024-05-09T08:27:29.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.302+0000 7f18bed95740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556732f0bcc0 con 0x556732fbda80 2024-05-09T08:27:29.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b9d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fb2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b9d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fb2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35290/0 (socket says 172.21.15.28:35290) 2024-05-09T08:27:29.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b9d4b640 1 -- 172.21.15.28:0/3972091029 learned_addr learned my addr 172.21.15.28:0/3972091029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b954a640 1 --2- 172.21.15.28:0/3972091029 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 0x556732fbd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b9d4b640 1 -- 172.21.15.28:0/3972091029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fbda80 msgr2=0x556732fbfe70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:27:29.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/3972091029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fbda80 0x556732fbfe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b9d4b640 1 -- 172.21.15.28:0/3972091029 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 msgr2=0x556732fbd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/3972091029 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 0x556732fbd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b9d4b640 1 -- 172.21.15.28:0/3972091029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556732ced7d0 con 0x556732fb2a90 2024-05-09T08:27:29.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18ba54c640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f18a00a2f20 0x7f18a00a53e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18ac004af0 tx=0x7f18ac005d80 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/3972091029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fb2e70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f18b000bb10 tx=0x7f18b000d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51751a72788c5ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.303+0000 7f189ffff640 1 -- 172.21.15.28:0/3972091029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0016070 con 0x556732fb2a90 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.304+0000 7f189ffff640 1 -- 172.21.15.28:0/3972091029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b0005500 con 0x556732fb2a90 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.304+0000 7f189ffff640 1 -- 172.21.15.28:0/3972091029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0010460 con 0x556732fb2a90 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.304+0000 7f18bed95740 1 -- 172.21.15.28:0/3972091029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 msgr2=0x556732fb2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.304+0000 7f18bed95740 1 --2- 172.21.15.28:0/3972091029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fb2e70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f18b000bb10 tx=0x7f18b000d2e0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.304+0000 7f18bed95740 1 -- 172.21.15.28:0/3972091029 shutdown_connections 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.304+0000 7f18bed95740 1 --2- 172.21.15.28:0/3972091029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fbda80 0x556732fbfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.304+0000 7f18bed95740 1 --2- 172.21.15.28:0/3972091029 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 0x556732fbd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.304+0000 7f18bed95740 1 --2- 172.21.15.28:0/3972091029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fb2e70 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.304+0000 7f18bed95740 1 -- 172.21.15.28:0/3972091029 >> 172.21.15.28:0/3972091029 conn(0x556732faedb0 msgr2=0x556732faf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.305+0000 7f18bed95740 1 -- 172.21.15.28:0/3972091029 shutdown_connections 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.305+0000 7f18bed95740 1 -- 172.21.15.28:0/3972091029 wait complete. 2024-05-09T08:27:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.305+0000 7f18bed95740 1 -- start start 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.305+0000 7f18bed95740 1 Processor -- start 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.305+0000 7f18bed95740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fbaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.305+0000 7f18bed95740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 0x556732fbaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18bed95740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fbda80 0x556732fbb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18bed95740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556732fc0870 con 0x556732fb2a90 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18bed95740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x556732ecab90 con 0x556732fb33b0 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18bed95740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556732fc09f0 con 0x556732fbda80 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b9d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fbaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b9d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fbaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35294/0 (socket says 172.21.15.28:35294) 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b9d4b640 1 -- 172.21.15.28:0/2164347068 learned_addr learned my addr 172.21.15.28:0/2164347068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18bed95740 10 client.0 _ll_register_callbacks cb 0x556732f06da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b954a640 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 0x556732fbaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18ba54c640 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fbda80 0x556732fbb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b9d4b640 1 -- 172.21.15.28:0/2164347068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fbda80 msgr2=0x556732fbb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fbda80 0x556732fbb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b9d4b640 1 -- 172.21.15.28:0/2164347068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 msgr2=0x556732fbaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 0x556732fbaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b9d4b640 1 -- 172.21.15.28:0/2164347068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556732c233f0 con 0x556732fb2a90 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b954a640 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 0x556732fbaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fbaa20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005d00 tx=0x7f18b00056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2575c4821a581f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.306+0000 7f189d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f189d7fa640 1 -- 172.21.15.28:0/2164347068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0016030 con 0x556732fb2a90 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f189d7fa640 1 -- 172.21.15.28:0/2164347068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f18b0005200 con 0x556732fb2a90 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f18bed95740 1 -- 172.21.15.28:0/2164347068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18b0005500 con 0x556732fb2a90 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f189d7fa640 1 -- 172.21.15.28:0/2164347068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f18b002d050 con 0x556732fb2a90 2024-05-09T08:27:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f18bed95740 10 client.16152 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f18bed95740 1 -- 172.21.15.28:0/2164347068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556732be30f0 con 0x556732fb2a90 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f189d7fa640 1 -- 172.21.15.28:0/2164347068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b0005500 con 0x556732fb2a90 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f189d7fa640 1 -- 172.21.15.28:0/2164347068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f18b00035f0 con 0x556732fb2a90 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f18bed95740 10 client.16152 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f18bed95740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.307+0000 7f18bed95740 20 client.16152 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.308+0000 7f18bed95740 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556732fcde00 0x556732fd01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.308+0000 7f18bed95740 4 client.16152 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.308+0000 7f18bed95740 1 -- 172.21.15.28:0/2164347068 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556732f1d430 con 0x556732fcde00 2024-05-09T08:27:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.308+0000 7f18b954a640 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556732fcde00 0x556732fd01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.308+0000 7f18b954a640 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556732fcde00 0x556732fd01f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e63a98cb0923c9f9 server_cookie=d2856fd722d2ced in_seq=0 out_seq=0 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.308+0000 7f189d7fa640 10 client.16152 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f189d7fa640 1 -- 172.21.15.28:0/2164347068 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556732f1d430 con 0x556732fcde00 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f189d7fa640 10 client.16152 handle_command_reply: tid=0 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 client.16152 shutdown 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 -- 172.21.15.28:0/2164347068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 msgr2=0x556732fbaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fbaa20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005d00 tx=0x7f18b00056b0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 -- 172.21.15.28:0/2164347068 shutdown_connections 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556732fcde00 0x556732fd01f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fbda80 0x556732fbb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fb33b0 0x556732fbaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 --2- 172.21.15.28:0/2164347068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fbaa20 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 -- 172.21.15.28:0/2164347068 >> 172.21.15.28:0/2164347068 conn(0x556732faedb0 msgr2=0x556732fa57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f189d7fa640 0 client.16152 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 -- 172.21.15.28:0/2164347068 shutdown_connections 2024-05-09T08:27:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.309+0000 7f18bed95740 1 -- 172.21.15.28:0/2164347068 wait complete. 2024-05-09T08:27:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.310+0000 7f18bed95740 20 client.16152 trim_cache size 0 max 16384 2024-05-09T08:27:29.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.327+0000 7f18bed95740 1 Processor -- start 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.327+0000 7f18bed95740 1 -- start start 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.327+0000 7f18bed95740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.327+0000 7f18bed95740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fb33b0 0x556732fd5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18bed95740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fbda80 0x556732fd5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18bed95740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41accc0 con 0x556732fb2a90 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18bed95740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41ae800 con 0x556732fbda80 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18bed95740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41ad390 con 0x556732fb33b0 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18b9d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18b9d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35310/0 (socket says 172.21.15.28:35310) 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18b9d4b640 1 -- 172.21.15.28:0/2891279539 learned_addr learned my addr 172.21.15.28:0/2891279539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18ba54c640 1 --2- 172.21.15.28:0/2891279539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fbda80 0x556732fd5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18b9d4b640 1 -- 172.21.15.28:0/2891279539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fb33b0 msgr2=0x556732fd5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/2891279539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fb33b0 0x556732fd5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18b9d4b640 1 -- 172.21.15.28:0/2891279539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fbda80 msgr2=0x556732fd5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/2891279539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fbda80 0x556732fd5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18b9d4b640 1 -- 172.21.15.28:0/2891279539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556732faebe0 con 0x556732fb2a90 2024-05-09T08:27:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/2891279539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc9730 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f18b0006010 tx=0x7f18b000d860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6546cd5483b7c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f189f7fe640 1 -- 172.21.15.28:0/2891279539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0000e60 con 0x556732fb2a90 2024-05-09T08:27:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f189f7fe640 1 -- 172.21.15.28:0/2891279539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b001f070 con 0x556732fb2a90 2024-05-09T08:27:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.328+0000 7f189f7fe640 1 -- 172.21.15.28:0/2891279539 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0016070 con 0x556732fb2a90 2024-05-09T08:27:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.329+0000 7f18bed95740 1 -- 172.21.15.28:0/2891279539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 msgr2=0x556732fc9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.329+0000 7f18bed95740 1 --2- 172.21.15.28:0/2891279539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc9730 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f18b0006010 tx=0x7f18b000d860 comp rx=0 tx=0).stop 2024-05-09T08:27:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.329+0000 7f18bed95740 1 -- 172.21.15.28:0/2891279539 shutdown_connections 2024-05-09T08:27:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.329+0000 7f18bed95740 1 --2- 172.21.15.28:0/2891279539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fbda80 0x556732fd5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.329+0000 7f18bed95740 1 --2- 172.21.15.28:0/2891279539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fb33b0 0x556732fd5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.329+0000 7f18bed95740 1 --2- 172.21.15.28:0/2891279539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc9730 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.329+0000 7f18bed95740 1 -- 172.21.15.28:0/2891279539 >> 172.21.15.28:0/2891279539 conn(0x556732faedb0 msgr2=0x556732fb5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.329+0000 7f18bed95740 1 -- 172.21.15.28:0/2891279539 shutdown_connections 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.329+0000 7f18bed95740 1 -- 172.21.15.28:0/2891279539 wait complete. 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18bed95740 1 -- start start 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18bed95740 1 Processor -- start 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18bed95740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18bed95740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fb33b0 0x556732fc13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18bed95740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fbda80 0x55673302c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18b9d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18b9d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35316/0 (socket says 172.21.15.28:35316) 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18b9d4b640 1 -- 172.21.15.28:0/3290625672 learned_addr learned my addr 172.21.15.28:0/3290625672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41ae130 con 0x556732fb2a90 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41ada60 con 0x556732fbda80 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.330+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41aaca0 con 0x556732fb33b0 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18bed95740 10 client.0 _ll_register_callbacks cb 0x7f18a400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18b954a640 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fb33b0 0x556732fc13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18b9d4b640 1 -- 172.21.15.28:0/3290625672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fb33b0 msgr2=0x556732fc13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fb33b0 0x556732fc13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18b9d4b640 1 -- 172.21.15.28:0/3290625672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fbda80 msgr2=0x55673302c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fbda80 0x55673302c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18b9d4b640 1 -- 172.21.15.28:0/3290625672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55673302c690 con 0x556732fb2a90 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18b001f070 con 0x556732fb2a90 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18b9d4b640 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc0e90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f18b000bb10 tx=0x7f18b000b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29d70a6b22b8ea0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f189d7fa640 10 client.16158 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f189d7fa640 1 -- 172.21.15.28:0/3290625672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0016030 con 0x556732fb2a90 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f189d7fa640 1 -- 172.21.15.28:0/3290625672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f18b000dc60 con 0x556732fb2a90 2024-05-09T08:27:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f189d7fa640 1 -- 172.21.15.28:0/3290625672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f18b0005070 con 0x556732fb2a90 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18bed95740 10 client.16158 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.331+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=28}) -- 0x556733033980 con 0x556732fb2a90 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.332+0000 7f189d7fa640 1 -- 172.21.15.28:0/3290625672 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f18b0005210 con 0x556732fb2a90 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.332+0000 7f18bed95740 10 client.16158 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.332+0000 7f18bed95740 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.332+0000 7f18bed95740 20 client.16158 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.332+0000 7f18bed95740 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556733033e90 0x556733036280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.332+0000 7f18bed95740 4 client.16158 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.332+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556732f17780 con 0x556733033e90 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.332+0000 7f18b954a640 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556733033e90 0x556733036280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f189d7fa640 10 client.16158 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18b954a640 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556733033e90 0x556733036280 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1596ffee82d545a server_cookie=bb2787f7d40402da in_seq=0 out_seq=0 2024-05-09T08:27:29.337 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:29 smithi028 ceph-mon[30119]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f189d7fa640 1 -- 172.21.15.28:0/3290625672 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556732f17780 con 0x556733033e90 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f189d7fa640 10 client.16158 handle_command_reply: tid=0 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18bed95740 1 client.16158 shutdown 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 msgr2=0x556732fc0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18bed95740 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc0e90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f18b000bb10 tx=0x7f18b000b760 comp rx=0 tx=0).stop 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 shutdown_connections 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18bed95740 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556733033e90 0x556733036280 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f189d7fa640 0 client.16158 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18bed95740 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556732fbda80 0x55673302c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18bed95740 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556732fb33b0 0x556732fc13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18bed95740 1 --2- 172.21.15.28:0/3290625672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556732fb2a90 0x556732fc0e90 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.333+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 >> 172.21.15.28:0/3290625672 conn(0x556732faedb0 msgr2=0x556732fa4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 shutdown_connections 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bed95740 1 -- 172.21.15.28:0/3290625672 wait complete. 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bed95740 20 client.16158 trim_cache size 0 max 16384 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "13db0616-1433-48c9-95ff-2eb2e1805eb9": { 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-09T08:27:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "13db0616-1433-48c9-95ff-2eb2e1805eb9", 2024-05-09T08:27:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-09T08:27:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:27:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:27:29.340 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:27:29.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 -- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f18a00a2f20 msgr2=0x7f18a00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f18a00a2f20 0x7f18a00a53e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18ac004af0 tx=0x7f18ac005d80 comp rx=0 tx=0).stop 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 -- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b4100f50 msgr2=0x7f18b419a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b4100f50 0x7f18b419a820 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f18a400d950 tx=0x7f18a400de10 comp rx=0 tx=0).stop 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 -- 172.21.15.28:0/2078152655 shutdown_connections 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f18a00a2f20 0x7f18a00a53e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18b410e1e0 0x7f18b419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b4100f50 0x7f18b419a820 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 --2- 172.21.15.28:0/2078152655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4100630 0x7f18b419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 -- 172.21.15.28:0/2078152655 >> 172.21.15.28:0/2078152655 conn(0x7f18b4077830 msgr2=0x7f18b410f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.334+0000 7f18bbfaf640 1 -- 172.21.15.28:0/2078152655 shutdown_connections 2024-05-09T08:27:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:29.335+0000 7f18bbfaf640 1 -- 172.21.15.28:0/2078152655 wait complete. 2024-05-09T08:27:29.346 INFO:tasks.cephfs.filesystem:scrub status for tag:13db0616-1433-48c9-95ff-2eb2e1805eb9 - {'path': '/', 'tag': '13db0616-1433-48c9-95ff-2eb2e1805eb9', 'options': 'recursive,force'} 2024-05-09T08:27:29.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:29 smithi165 ceph-mon[34540]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:27:29.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:29 smithi184 ceph-mon[34509]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:27:30.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:30 smithi165 ceph-mon[34540]: scrub queued for path: / 2024-05-09T08:27:30.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:30 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:27:30.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:30 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:27:30.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:30 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:27:30.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:30 smithi184 ceph-mon[34509]: scrub queued for path: / 2024-05-09T08:27:30.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:30 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:27:30.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:30 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:27:30.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:30 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:27:30.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:30 smithi028 ceph-mon[30119]: scrub queued for path: / 2024-05-09T08:27:30.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:30 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:27:30.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:30 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:27:30.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:30 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:27:31.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:31 smithi165 ceph-mon[34540]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:27:31.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:31 smithi165 ceph-mon[34540]: scrub summary: idle 2024-05-09T08:27:31.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:31 smithi184 ceph-mon[34509]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:27:31.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:31 smithi184 ceph-mon[34509]: scrub summary: idle 2024-05-09T08:27:31.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:31 smithi028 ceph-mon[30119]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:27:31.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:31 smithi028 ceph-mon[30119]: scrub summary: idle 2024-05-09T08:27:31.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.702+0000 7f139d7b0640 1 -- 172.21.15.28:0/3055010770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 msgr2=0x7f1398102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:31.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.702+0000 7f139d7b0640 1 --2- 172.21.15.28:0/3055010770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 0x7f1398102ea0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1388009900 tx=0x7f138802a510 comp rx=0 tx=0).stop 2024-05-09T08:27:31.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.702+0000 7f139d7b0640 1 -- 172.21.15.28:0/3055010770 shutdown_connections 2024-05-09T08:27:31.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.702+0000 7f139d7b0640 1 --2- 172.21.15.28:0/3055010770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f139810be00 0x7f139810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.702+0000 7f139d7b0640 1 --2- 172.21.15.28:0/3055010770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13981033e0 0x7f139810b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.702+0000 7f139d7b0640 1 --2- 172.21.15.28:0/3055010770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 0x7f1398102ea0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.702+0000 7f139d7b0640 1 -- 172.21.15.28:0/3055010770 >> 172.21.15.28:0/3055010770 conn(0x7f13980778b0 msgr2=0x7f1398077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:31.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.702+0000 7f139d7b0640 1 -- 172.21.15.28:0/3055010770 shutdown_connections 2024-05-09T08:27:31.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.702+0000 7f139d7b0640 1 -- 172.21.15.28:0/3055010770 wait complete. 2024-05-09T08:27:31.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f139d7b0640 1 Processor -- start 2024-05-09T08:27:31.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f139d7b0640 1 -- start start 2024-05-09T08:27:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f139d7b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 0x7f1398100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f139d7b0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13981033e0 0x7f1398100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f139d7b0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f139810be00 0x7f1398101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f139d7b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1398111c00 con 0x7f1398102ac0 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f139d7b0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1398111a80 con 0x7f139810be00 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f139d7b0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1398111d80 con 0x7f13981033e0 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f1396ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 0x7f1398100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f1396ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 0x7f1398100710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35324/0 (socket says 172.21.15.28:35324) 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.703+0000 7f1396ffd640 1 -- 172.21.15.28:0/1806177035 learned_addr learned my addr 172.21.15.28:0/1806177035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f13977fe640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f139810be00 0x7f1398101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f13967fc640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13981033e0 0x7f1398100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f1396ffd640 1 -- 172.21.15.28:0/1806177035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13981033e0 msgr2=0x7f1398100c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f1396ffd640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13981033e0 0x7f1398100c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f1396ffd640 1 -- 172.21.15.28:0/1806177035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f139810be00 msgr2=0x7f1398101190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f1396ffd640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f139810be00 0x7f1398101190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f1396ffd640 1 -- 172.21.15.28:0/1806177035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13981a65a0 con 0x7f1398102ac0 2024-05-09T08:27:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f13977fe640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f139810be00 0x7f1398101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f1396ffd640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 0x7f1398100710 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1388006010 tx=0x7f13880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f137ffff640 1 -- 172.21.15.28:0/1806177035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138803f030 con 0x7f1398102ac0 2024-05-09T08:27:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f139d7b0640 1 -- 172.21.15.28:0/1806177035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13981a6830 con 0x7f1398102ac0 2024-05-09T08:27:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f137ffff640 1 -- 172.21.15.28:0/1806177035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1388003ce0 con 0x7f1398102ac0 2024-05-09T08:27:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f137ffff640 1 -- 172.21.15.28:0/1806177035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388035070 con 0x7f1398102ac0 2024-05-09T08:27:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.704+0000 7f139d7b0640 1 -- 172.21.15.28:0/1806177035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13981a6d70 con 0x7f1398102ac0 2024-05-09T08:27:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.707+0000 7f137ffff640 1 -- 172.21.15.28:0/1806177035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f1388042030 con 0x7f1398102ac0 2024-05-09T08:27:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f137ffff640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13800a2f40 0x7f13800a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f13a059e740 1 Processor -- start 2024-05-09T08:27:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f137ffff640 1 -- 172.21.15.28:0/1806177035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f13880effa0 con 0x7f1398102ac0 2024-05-09T08:27:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f13a059e740 1 -- start start 2024-05-09T08:27:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f13967fc640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13800a2f40 0x7f13800a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f13a059e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x564810713040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f13a059e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f13a059e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810720230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f13a059e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56481066f340 con 0x564810713580 2024-05-09T08:27:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f13a059e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x564810617290 con 0x564810712c60 2024-05-09T08:27:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.708+0000 7f13a059e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56481066b300 con 0x56481071de40 2024-05-09T08:27:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f1396ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x564810713040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13967fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f1396ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x564810713040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35972/0 (socket says 172.21.15.28:35972) 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f1396ffd640 1 -- 172.21.15.28:0/4105652830 learned_addr learned my addr 172.21.15.28:0/4105652830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13977fe640 1 --2- 172.21.15.28:0/4105652830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810720230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13967fc640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13800a2f40 0x7f13800a5400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f138c0095f0 tx=0x7f138c005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13977fe640 1 -- 172.21.15.28:0/4105652830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 msgr2=0x564810713040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13977fe640 1 --2- 172.21.15.28:0/4105652830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x564810713040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13977fe640 1 -- 172.21.15.28:0/4105652830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 msgr2=0x56481071d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13977fe640 1 --2- 172.21.15.28:0/4105652830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13977fe640 1 -- 172.21.15.28:0/4105652830 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56481063b8e0 con 0x56481071de40 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13967fc640 1 --2- 172.21.15.28:0/4105652830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f13977fe640 1 --2- 172.21.15.28:0/4105652830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810720230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1390009f00 tx=0x7f139000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75148959da3a825b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.709+0000 7f137d7fa640 1 -- 172.21.15.28:0/4105652830 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390015070 con 0x56481071de40 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.710+0000 7f137d7fa640 1 -- 172.21.15.28:0/4105652830 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f139001ab40 con 0x56481071de40 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.710+0000 7f137d7fa640 1 -- 172.21.15.28:0/4105652830 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390011b90 con 0x56481071de40 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.710+0000 7f13a059e740 1 -- 172.21.15.28:0/4105652830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 msgr2=0x564810720230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.710+0000 7f13a059e740 1 --2- 172.21.15.28:0/4105652830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810720230 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1390009f00 tx=0x7f139000f730 comp rx=0 tx=0).stop 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 -- 172.21.15.28:0/4105652830 shutdown_connections 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 --2- 172.21.15.28:0/4105652830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810720230 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 --2- 172.21.15.28:0/4105652830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 --2- 172.21.15.28:0/4105652830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x564810713040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 -- 172.21.15.28:0/4105652830 >> 172.21.15.28:0/4105652830 conn(0x56481070f170 msgr2=0x56481070f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 -- 172.21.15.28:0/4105652830 shutdown_connections 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 -- 172.21.15.28:0/4105652830 wait complete. 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 -- start start 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 Processor -- start 2024-05-09T08:27:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.711+0000 7f13a059e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x56481071ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f13a059e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f13a059e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x56481071b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f13967fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f1396ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x56481071ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f1396ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x56481071ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35978/0 (socket says 172.21.15.28:35978) 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f13977fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x56481071b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f13977fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x56481071b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42932/0 (socket says 172.21.15.28:42932) 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f13967fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35338/0 (socket says 172.21.15.28:35338) 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f1396ffd640 1 -- 172.21.15.28:0/2981260075 learned_addr learned my addr 172.21.15.28:0/2981260075 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564810720a70 con 0x564810713580 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56481062a790 con 0x564810712c60 2024-05-09T08:27:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.712+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564810720bf0 con 0x56481071de40 2024-05-09T08:27:31.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f1396ffd640 1 -- 172.21.15.28:0/2981260075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 msgr2=0x56481071b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:31.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f1396ffd640 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x56481071b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f1396ffd640 1 -- 172.21.15.28:0/2981260075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 msgr2=0x56481071b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f1396ffd640 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f1396ffd640 1 -- 172.21.15.28:0/2981260075 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564810383210 con 0x564810712c60 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f13967fc640 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f1396ffd640 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x56481071ae20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f13880bbd30 tx=0x7f1388003e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3747ecac48e76b5e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f137aff5640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f13a059e740 10 client.0 _ll_register_callbacks cb 0x564810665a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f137aff5640 1 -- 172.21.15.28:0/2981260075 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13880d6070 con 0x564810712c60 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.713+0000 7f137aff5640 1 -- 172.21.15.28:0/2981260075 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f13880c4810 con 0x564810712c60 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.714+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f139001ab40 con 0x564810712c60 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.714+0000 7f137aff5640 1 -- 172.21.15.28:0/2981260075 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f13880ac050 con 0x564810712c60 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.714+0000 7f13a059e740 10 client.25643 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.714+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({fsmap=28}) -- 0x56481044f620 con 0x564810712c60 2024-05-09T08:27:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.714+0000 7f137aff5640 1 -- 172.21.15.28:0/2981260075 <== mon.2 v2:172.21.15.165:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f13880ed480 con 0x564810712c60 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.714+0000 7f13a059e740 10 client.25643 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.714+0000 7f13a059e740 10 client.25643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.715+0000 7f13a059e740 20 client.25643 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.715+0000 7f13a059e740 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56481072d7e0 0x56481072fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.715+0000 7f13a059e740 4 client.25643 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.715+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56481067ca70 con 0x56481072d7e0 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.715+0000 7f13967fc640 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56481072d7e0 0x56481072fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13967fc640 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56481072d7e0 0x56481072fca0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2dacf0fab378da1 server_cookie=f66dc9d16a0659f8 in_seq=0 out_seq=0 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f137aff5640 10 client.25643 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f137aff5640 1 -- 172.21.15.28:0/2981260075 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56481067ca70 con 0x56481072d7e0 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f137aff5640 10 client.25643 handle_command_reply: tid=0 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13a059e740 1 client.25643 shutdown 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 msgr2=0x56481071ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13a059e740 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x56481071ae20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f13880bbd30 tx=0x7f1388003e90 comp rx=0 tx=0).stop 2024-05-09T08:27:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 shutdown_connections 2024-05-09T08:27:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13a059e740 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56481072d7e0 0x56481072fca0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f137aff5640 0 client.25643 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13a059e740 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x56481071b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13a059e740 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x56481071b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13a059e740 1 --2- 172.21.15.28:0/2981260075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x56481071ae20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.716+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 >> 172.21.15.28:0/2981260075 conn(0x56481070f170 msgr2=0x564810714ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.717+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 shutdown_connections 2024-05-09T08:27:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.717+0000 7f13a059e740 1 -- 172.21.15.28:0/2981260075 wait complete. 2024-05-09T08:27:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.717+0000 7f13a059e740 20 client.25643 trim_cache size 0 max 16384 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13a059e740 1 Processor -- start 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13a059e740 1 -- start start 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13a059e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564810712c60 0x56481071d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13a059e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810713580 0x564810715670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13a059e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56481071de40 0x56481071b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13a059e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13981acbb0 con 0x56481071de40 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56481071de40 0x56481071b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56481071de40 0x56481071b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35350/0 (socket says 172.21.15.28:35350) 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13977fe640 1 -- 172.21.15.28:0/2685960786 learned_addr learned my addr 172.21.15.28:0/2685960786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13967fc640 1 --2- 172.21.15.28:0/2685960786 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810713580 0x564810715670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f1396ffd640 1 --2- 172.21.15.28:0/2685960786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564810712c60 0x56481071d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13a059e740 1 -- 172.21.15.28:0/2685960786 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f13981ae6f0 con 0x564810713580 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.732+0000 7f13a059e740 1 -- 172.21.15.28:0/2685960786 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13981ad280 con 0x564810712c60 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.733+0000 7f13977fe640 1 -- 172.21.15.28:0/2685960786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564810712c60 msgr2=0x56481071d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.733+0000 7f13977fe640 1 --2- 172.21.15.28:0/2685960786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564810712c60 0x56481071d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.733+0000 7f13977fe640 1 -- 172.21.15.28:0/2685960786 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810713580 msgr2=0x564810715670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.733+0000 7f13977fe640 1 --2- 172.21.15.28:0/2685960786 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810713580 0x564810715670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.733+0000 7f13977fe640 1 -- 172.21.15.28:0/2685960786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564810719e10 con 0x56481071de40 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.733+0000 7f13977fe640 1 --2- 172.21.15.28:0/2685960786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56481071de40 0x56481071b6c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1390009e50 tx=0x7f1390000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc84337764a2aed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.733+0000 7f137cff9640 1 -- 172.21.15.28:0/2685960786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390029070 con 0x56481071de40 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.733+0000 7f137cff9640 1 -- 172.21.15.28:0/2685960786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f139001a940 con 0x56481071de40 2024-05-09T08:27:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.733+0000 7f137cff9640 1 -- 172.21.15.28:0/2685960786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390015040 con 0x56481071de40 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.734+0000 7f13a059e740 1 -- 172.21.15.28:0/2685960786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56481071de40 msgr2=0x56481071b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.734+0000 7f13a059e740 1 --2- 172.21.15.28:0/2685960786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56481071de40 0x56481071b6c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1390009e50 tx=0x7f1390000f40 comp rx=0 tx=0).stop 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.734+0000 7f13a059e740 1 -- 172.21.15.28:0/2685960786 shutdown_connections 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.734+0000 7f13a059e740 1 --2- 172.21.15.28:0/2685960786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56481071de40 0x56481071b6c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.734+0000 7f13a059e740 1 --2- 172.21.15.28:0/2685960786 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810713580 0x564810715670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.734+0000 7f13a059e740 1 --2- 172.21.15.28:0/2685960786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564810712c60 0x56481071d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.734+0000 7f13a059e740 1 -- 172.21.15.28:0/2685960786 >> 172.21.15.28:0/2685960786 conn(0x56481070f170 msgr2=0x564810705a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.734+0000 7f13a059e740 1 -- 172.21.15.28:0/2685960786 shutdown_connections 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.734+0000 7f13a059e740 1 -- 172.21.15.28:0/2685960786 wait complete. 2024-05-09T08:27:31.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.735+0000 7f13a059e740 1 -- start start 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.735+0000 7f13a059e740 1 Processor -- start 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.735+0000 7f13a059e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x564810718290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.735+0000 7f13a059e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x5648107187d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.735+0000 7f13967fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x5648107187d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.735+0000 7f13967fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x5648107187d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35356/0 (socket says 172.21.15.28:35356) 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.735+0000 7f13967fc640 1 -- 172.21.15.28:0/3627323600 learned_addr learned my addr 172.21.15.28:0/3627323600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13a059e740 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810792e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13981ae020 con 0x564810713580 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f13981ad950 con 0x564810712c60 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13981aab90 con 0x56481071de40 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f1396ffd640 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x564810718290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13977fe640 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810792e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13967fc640 1 -- 172.21.15.28:0/3627323600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 msgr2=0x564810792e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13967fc640 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810792e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13967fc640 1 -- 172.21.15.28:0/3627323600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 msgr2=0x564810718290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13967fc640 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x564810718290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13967fc640 1 -- 172.21.15.28:0/3627323600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564810793390 con 0x564810713580 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13977fe640 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810792e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13a059e740 10 client.0 _ll_register_callbacks cb 0x7f13880dc0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13967fc640 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x5648107187d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f138c009950 tx=0x7f138c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14a1724e94606dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f137aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f139001a940 con 0x564810713580 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.736+0000 7f137aff5640 1 -- 172.21.15.28:0/3627323600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c020d10 con 0x564810713580 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f137aff5640 1 -- 172.21.15.28:0/3627323600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f138c02b380 con 0x564810713580 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f137aff5640 1 -- 172.21.15.28:0/3627323600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f138c02b580 con 0x564810713580 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f13a059e740 10 client.16176 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5648107944a0 con 0x564810713580 2024-05-09T08:27:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f137aff5640 1 -- 172.21.15.28:0/3627323600 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f138c02b720 con 0x564810713580 2024-05-09T08:27:31.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f13a059e740 10 client.16176 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:31.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f13a059e740 10 client.16176 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:31.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f13a059e740 20 client.16176 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:31.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f13a059e740 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564810794c50 0x564810797110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:31.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f13a059e740 4 client.16176 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:27:31.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56481070efc0 con 0x564810794c50 2024-05-09T08:27:31.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.737+0000 7f1396ffd640 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564810794c50 0x564810797110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:31.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.738+0000 7f1396ffd640 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564810794c50 0x564810797110 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9f0e4202461b0e1 server_cookie=c991805738befd65 in_seq=0 out_seq=0 2024-05-09T08:27:31.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.738+0000 7f137aff5640 10 client.16176 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:31.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f137aff5640 1 -- 172.21.15.28:0/3627323600 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56481070efc0 con 0x564810794c50 2024-05-09T08:27:31.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f137aff5640 10 client.16176 handle_command_reply: tid=0 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f13a059e740 1 client.16176 shutdown 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 msgr2=0x5648107187d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f13a059e740 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x5648107187d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f138c009950 tx=0x7f138c004730 comp rx=0 tx=0).stop 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 shutdown_connections 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f13a059e740 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564810794c50 0x564810797110 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f137aff5640 0 client.16176 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f13a059e740 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56481071de40 0x564810792e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f13a059e740 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564810713580 0x5648107187d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f13a059e740 1 --2- 172.21.15.28:0/3627323600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564810712c60 0x564810718290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.741+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 >> 172.21.15.28:0/3627323600 conn(0x56481070f170 msgr2=0x56481071c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 shutdown_connections 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f13a059e740 1 -- 172.21.15.28:0/3627323600 wait complete. 2024-05-09T08:27:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f13a059e740 20 client.16176 trim_cache size 0 max 16384 2024-05-09T08:27:31.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:27:31.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:27:31.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:27:31.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 -- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13800a2f40 msgr2=0x7f13800a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13800a2f40 0x7f13800a5400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f138c0095f0 tx=0x7f138c005f90 comp rx=0 tx=0).stop 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 -- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 msgr2=0x7f1398100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 0x7f1398100710 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1388006010 tx=0x7f13880042f0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 -- 172.21.15.28:0/1806177035 shutdown_connections 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f139810be00 0x7f1398101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13800a2f40 0x7f13800a5400 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13981033e0 0x7f1398100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 --2- 172.21.15.28:0/1806177035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398102ac0 0x7f1398100710 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.742+0000 7f139d7b0640 1 -- 172.21.15.28:0/1806177035 >> 172.21.15.28:0/1806177035 conn(0x7f13980778b0 msgr2=0x7f13980fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.743+0000 7f139d7b0640 1 -- 172.21.15.28:0/1806177035 shutdown_connections 2024-05-09T08:27:31.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:31.743+0000 7f139d7b0640 1 -- 172.21.15.28:0/1806177035 wait complete. 2024-05-09T08:27:33.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:33 smithi165 ceph-mon[34540]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:27:33.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:33 smithi184 ceph-mon[34509]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:27:33.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:33 smithi028 ceph-mon[30119]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-09T08:27:35.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:35 smithi165 ceph-mon[34540]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:27:35.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:35 smithi184 ceph-mon[34509]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:27:35.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:35 smithi028 ceph-mon[30119]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-09T08:27:37.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:37 smithi165 ceph-mon[34540]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:27:37.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:37 smithi184 ceph-mon[34509]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:27:37.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:37 smithi028 ceph-mon[30119]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-09T08:27:37.682 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 97% (12242/12535) Updating files: 98% (12285/12535) Updating files: 99% (12410/12535) Updating files: 100% (12535/12535) Updating files: 100% (12535/12535), done. 2024-05-09T08:27:38.404 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to 'be85c202969d25af6e130678e7d73a20e9c4ca11'. 2024-05-09T08:27:38.404 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:38.404 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-09T08:27:38.404 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-09T08:27:38.404 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-05-09T08:27:38.404 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:38.404 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-09T08:27:38.404 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-09T08:27:38.404 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:38.405 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-05-09T08:27:38.405 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:38.405 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-05-09T08:27:38.405 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:38.405 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-05-09T08:27:38.405 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:38.405 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-09T08:27:38.405 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:38.405 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at be85c202969 Merge pull request #57242 from cbodley/wip-65767-squid 2024-05-09T08:27:38.408 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-09T08:27:38.481 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-09T08:27:38.483 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T08:27:38.483 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-09T08:27:39.156 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-09T08:27:39.210 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-09T08:27:39.252 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T08:27:39.254 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T08:27:39.254 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-09T08:27:39.294 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T08:27:39.298 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:27:39.298 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-09T08:27:39.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:39 smithi165 ceph-mon[34540]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:39.355 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-05-09T08:27:39.356 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-05-09T08:27:39.357 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=be85c202969d25af6e130678e7d73a20e9c4ca11 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-05-09T08:27:39.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:39 smithi184 ceph-mon[34509]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:39.418 INFO:tasks.workunit.client.0.smithi028.stdout:getting blogbench 2024-05-09T08:27:39.418 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'getting blogbench' 2024-05-09T08:27:39.418 INFO:tasks.workunit.client.0.smithi028.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-05-09T08:27:39.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:39 smithi028 ceph-mon[30119]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:39.439 INFO:tasks.workunit.client.0.smithi028.stderr:--2024-05-09 08:27:39-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-05-09T08:27:39.439 INFO:tasks.workunit.client.0.smithi028.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-05-09T08:27:39.476 INFO:tasks.workunit.client.0.smithi028.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-05-09T08:27:39.512 INFO:tasks.workunit.client.0.smithi028.stderr:HTTP request sent, awaiting response... 200 OK 2024-05-09T08:27:39.512 INFO:tasks.workunit.client.0.smithi028.stderr:Length: 103355 (101K) [application/octet-stream] 2024-05-09T08:27:39.513 INFO:tasks.workunit.client.0.smithi028.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-05-09T08:27:39.516 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:39.585 INFO:tasks.workunit.client.0.smithi028.stderr: 0K .......... .......... .......... .......... .......... 49% 713K 0s 2024-05-09T08:27:39.621 INFO:tasks.workunit.client.0.smithi028.stderr: 50K .......... .......... .......... .......... .......... 99% 1.38M 0s 2024-05-09T08:27:39.622 INFO:tasks.workunit.client.0.smithi028.stderr: 100K 100% 1.74T=0.1s 2024-05-09T08:27:39.622 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:39.622 INFO:tasks.workunit.client.0.smithi028.stderr:2024-05-09 08:27:39 (956 KB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-05-09T08:27:39.622 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-09T08:27:39.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-05-09T08:27:39.655 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/ 2024-05-09T08:27:39.656 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/m4/ 2024-05-09T08:27:39.657 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-05-09T08:27:39.658 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/man/ 2024-05-09T08:27:39.659 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/man/blogbench.8 2024-05-09T08:27:39.660 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/man/Makefile.am 2024-05-09T08:27:39.662 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/man/Makefile.in 2024-05-09T08:27:39.663 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/ 2024-05-09T08:27:39.664 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/mysnprintf.c 2024-05-09T08:27:39.665 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/mysnprintf.h 2024-05-09T08:27:39.666 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/blogbench.c 2024-05-09T08:27:39.667 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/blogbench.h 2024-05-09T08:27:39.667 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/fakesnprintf.c 2024-05-09T08:27:39.669 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/fakesnprintf.h 2024-05-09T08:27:39.670 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/blogbench_p.h 2024-05-09T08:27:39.671 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-05-09T08:27:39.672 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-05-09T08:27:39.673 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/Makefile.am 2024-05-09T08:27:39.674 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/Makefile.in 2024-05-09T08:27:39.676 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/process.c 2024-05-09T08:27:39.677 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/globals.h 2024-05-09T08:27:39.678 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/commenter.c 2024-05-09T08:27:39.679 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/rewriter.c 2024-05-09T08:27:39.680 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/reader.c 2024-05-09T08:27:39.681 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/writer.c 2024-05-09T08:27:39.682 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/src/helpers.c 2024-05-09T08:27:39.683 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/NEWS 2024-05-09T08:27:39.684 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/compile 2024-05-09T08:27:39.685 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/depcomp 2024-05-09T08:27:39.686 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/aclocal.m4 2024-05-09T08:27:39.686 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/README 2024-05-09T08:27:39.688 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/configure 2024-05-09T08:27:39.691 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/configure.ac 2024-05-09T08:27:39.693 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/config.guess 2024-05-09T08:27:39.694 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/install-sh 2024-05-09T08:27:39.695 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/config.sub 2024-05-09T08:27:39.696 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/missing 2024-05-09T08:27:39.697 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/mkinstalldirs 2024-05-09T08:27:39.698 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/Makefile.am 2024-05-09T08:27:39.699 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/Makefile.in 2024-05-09T08:27:39.700 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/config.h.in 2024-05-09T08:27:39.701 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/AUTHORS 2024-05-09T08:27:39.702 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/INSTALL 2024-05-09T08:27:39.703 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/ChangeLog 2024-05-09T08:27:39.704 INFO:tasks.workunit.client.0.smithi028.stdout:blogbench-1.0/COPYING 2024-05-09T08:27:39.705 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd blogbench-1.0/ 2024-05-09T08:27:39.705 INFO:tasks.workunit.client.0.smithi028.stdout:making blogbench 2024-05-09T08:27:39.706 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'making blogbench' 2024-05-09T08:27:39.706 INFO:tasks.workunit.client.0.smithi028.stderr:+ ./configure 2024-05-09T08:27:39.826 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-05-09T08:27:40.834 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether build environment is sane... yes 2024-05-09T08:27:40.845 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-05-09T08:27:40.862 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether make sets $(MAKE)... yes 2024-05-09T08:27:40.873 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-05-09T08:27:40.926 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-05-09T08:27:40.928 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-05-09T08:27:40.931 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-05-09T08:27:40.971 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-05-09T08:27:41.017 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-05-09T08:27:41.053 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-05-09T08:27:41.087 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-05-09T08:27:41.156 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ANSI C... none needed 2024-05-09T08:27:41.222 INFO:tasks.workunit.client.0.smithi028.stdout:checking for style of include used by make... GNU 2024-05-09T08:27:41.338 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... gcc3 2024-05-09T08:27:41.338 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-05-09T08:27:41.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:41 smithi165 ceph-mon[34540]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:27:41.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:41 smithi184 ceph-mon[34509]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:27:41.390 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-05-09T08:27:41.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:41 smithi028 ceph-mon[30119]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-09T08:27:41.434 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... grep -E 2024-05-09T08:27:41.443 INFO:tasks.workunit.client.0.smithi028.stdout:checking for AIX... no 2024-05-09T08:27:41.513 INFO:tasks.workunit.client.0.smithi028.stdout:checking for library containing strerror... none required 2024-05-09T08:27:41.682 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-05-09T08:27:41.758 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-05-09T08:27:41.808 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-05-09T08:27:41.864 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-05-09T08:27:41.920 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-05-09T08:27:41.977 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-05-09T08:27:42.030 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-05-09T08:27:42.083 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-05-09T08:27:42.135 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-05-09T08:27:42.187 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-05-09T08:27:42.241 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-05-09T08:27:42.254 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... (cached) yes 2024-05-09T08:27:42.259 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... (cached) yes 2024-05-09T08:27:42.264 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... (cached) yes 2024-05-09T08:27:42.321 INFO:tasks.workunit.client.0.smithi028.stdout:checking locale.h usability... yes 2024-05-09T08:27:42.343 INFO:tasks.workunit.client.0.smithi028.stdout:checking locale.h presence... yes 2024-05-09T08:27:42.343 INFO:tasks.workunit.client.0.smithi028.stdout:checking for locale.h... yes 2024-05-09T08:27:42.359 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... (cached) yes 2024-05-09T08:27:42.364 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... (cached) yes 2024-05-09T08:27:42.414 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/socket.h usability... yes 2024-05-09T08:27:42.437 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/socket.h presence... yes 2024-05-09T08:27:42.438 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/socket.h... yes 2024-05-09T08:27:42.495 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-05-09T08:27:42.516 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-05-09T08:27:42.516 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-05-09T08:27:42.573 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h usability... yes 2024-05-09T08:27:42.595 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h presence... yes 2024-05-09T08:27:42.596 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fcntl.h... yes 2024-05-09T08:27:42.658 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/fcntl.h usability... yes 2024-05-09T08:27:42.679 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/fcntl.h presence... yes 2024-05-09T08:27:42.679 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/fcntl.h... yes 2024-05-09T08:27:42.753 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/param.h usability... yes 2024-05-09T08:27:42.781 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/param.h presence... yes 2024-05-09T08:27:42.782 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/param.h... yes 2024-05-09T08:27:42.837 INFO:tasks.workunit.client.0.smithi028.stdout:checking ioctl.h usability... no 2024-05-09T08:27:42.859 INFO:tasks.workunit.client.0.smithi028.stdout:checking ioctl.h presence... no 2024-05-09T08:27:42.860 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ioctl.h... no 2024-05-09T08:27:42.920 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/ioctl.h usability... yes 2024-05-09T08:27:42.942 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/ioctl.h presence... yes 2024-05-09T08:27:42.943 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/ioctl.h... yes 2024-05-09T08:27:43.001 INFO:tasks.workunit.client.0.smithi028.stdout:checking return type of signal handlers... void 2024-05-09T08:27:43.047 INFO:tasks.workunit.client.0.smithi028.stdout:checking for size_t... yes 2024-05-09T08:27:43.087 INFO:tasks.workunit.client.0.smithi028.stdout:checking for dev_t... yes 2024-05-09T08:27:43.133 INFO:tasks.workunit.client.0.smithi028.stdout:checking for short... yes 2024-05-09T08:27:43.215 INFO:tasks.workunit.client.0.smithi028.stdout:checking size of short... 2 2024-05-09T08:27:43.264 INFO:tasks.workunit.client.0.smithi028.stdout:checking for int... yes 2024-05-09T08:27:43.338 INFO:tasks.workunit.client.0.smithi028.stdout:checking size of int... 4 2024-05-09T08:27:43.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:43 smithi165 ceph-mon[34540]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:43.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:43 smithi184 ceph-mon[34509]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:43.386 INFO:tasks.workunit.client.0.smithi028.stdout:checking for long... yes 2024-05-09T08:27:43.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:43 smithi028 ceph-mon[30119]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-09T08:27:43.464 INFO:tasks.workunit.client.0.smithi028.stdout:checking size of long... 8 2024-05-09T08:27:43.511 INFO:tasks.workunit.client.0.smithi028.stdout:checking for long long... yes 2024-05-09T08:27:43.592 INFO:tasks.workunit.client.0.smithi028.stdout:checking size of long long... 8 2024-05-09T08:27:43.642 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ssize_t... yes 2024-05-09T08:27:43.684 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sig_atomic_t... yes 2024-05-09T08:27:43.735 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc needs -traditional... no 2024-05-09T08:27:43.771 INFO:tasks.workunit.client.0.smithi028.stdout:checking for an ANSI C-conforming const... yes 2024-05-09T08:27:43.796 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inline... inline 2024-05-09T08:27:43.860 INFO:tasks.workunit.client.0.smithi028.stdout:checking for working alloca.h... yes 2024-05-09T08:27:43.909 INFO:tasks.workunit.client.0.smithi028.stdout:checking for alloca... yes 2024-05-09T08:27:43.981 INFO:tasks.workunit.client.0.smithi028.stdout:checking for working memcmp... yes 2024-05-09T08:27:44.039 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strftime... yes 2024-05-09T08:27:44.119 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-05-09T08:27:44.180 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether stat accepts an empty string... no 2024-05-09T08:27:44.238 INFO:tasks.workunit.client.0.smithi028.stdout:checking for vprintf... yes 2024-05-09T08:27:44.297 INFO:tasks.workunit.client.0.smithi028.stdout:checking for _doprnt... no 2024-05-09T08:27:44.319 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-05-09T08:27:44.331 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-05-09T08:27:44.389 INFO:tasks.workunit.client.0.smithi028.stdout:checking for the pthreads library -lpthreads... no 2024-05-09T08:27:44.444 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether pthreads work without any flags... yes 2024-05-09T08:27:44.491 INFO:tasks.workunit.client.0.smithi028.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-05-09T08:27:44.491 INFO:tasks.workunit.client.0.smithi028.stdout:checking if more special flags are required for pthreads... no 2024-05-09T08:27:44.491 INFO:tasks.workunit.client.0.smithi028.stdout:checking for cc_r... gcc 2024-05-09T08:27:44.625 INFO:tasks.workunit.client.0.smithi028.stdout:checking for setrlimit... yes 2024-05-09T08:27:44.705 INFO:tasks.workunit.client.0.smithi028.stdout:checking for setproctitle... no 2024-05-09T08:27:44.788 INFO:tasks.workunit.client.0.smithi028.stdout:checking for getopt_long... yes 2024-05-09T08:27:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.852+0000 7f0562709640 1 -- 172.21.15.28:0/2868088200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c107840 msgr2=0x7f055c107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.852+0000 7f0562709640 1 --2- 172.21.15.28:0/2868088200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c107840 0x7f055c107c20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f054c009930 tx=0x7f054c02a420 comp rx=0 tx=0).stop 2024-05-09T08:27:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.852+0000 7f0562709640 1 -- 172.21.15.28:0/2868088200 shutdown_connections 2024-05-09T08:27:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.852+0000 7f0562709640 1 --2- 172.21.15.28:0/2868088200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c1021f0 0x7f055c10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.852+0000 7f0562709640 1 --2- 172.21.15.28:0/2868088200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c101830 0x7f055c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.852+0000 7f0562709640 1 --2- 172.21.15.28:0/2868088200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c107840 0x7f055c107c20 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.852+0000 7f0562709640 1 -- 172.21.15.28:0/2868088200 >> 172.21.15.28:0/2868088200 conn(0x7f055c0fd320 msgr2=0x7f055c0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:44.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.852+0000 7f0562709640 1 -- 172.21.15.28:0/2868088200 shutdown_connections 2024-05-09T08:27:44.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.852+0000 7f0562709640 1 -- 172.21.15.28:0/2868088200 wait complete. 2024-05-09T08:27:44.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.853+0000 7f0562709640 1 Processor -- start 2024-05-09T08:27:44.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.853+0000 7f0562709640 1 -- start start 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f0562709640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c101830 0x7f055c198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f0562709640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c1021f0 0x7f055c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f0562709640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c107840 0x7f055c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f0562709640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f055c111c50 con 0x7f055c101830 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f0562709640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f055c111ad0 con 0x7f055c1021f0 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f0562709640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f055c111dd0 con 0x7f055c107840 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c101830 0x7f055c198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c101830 0x7f055c198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37496/0 (socket says 172.21.15.28:37496) 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055bfff640 1 -- 172.21.15.28:0/2481493897 learned_addr learned my addr 172.21.15.28:0/2481493897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c1021f0 0x7f055c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f0560ca6640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c107840 0x7f055c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055bfff640 1 -- 172.21.15.28:0/2481493897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c107840 msgr2=0x7f055c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055bfff640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c107840 0x7f055c19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055bfff640 1 -- 172.21.15.28:0/2481493897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c1021f0 msgr2=0x7f055c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055bfff640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c1021f0 0x7f055c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055bfff640 1 -- 172.21.15.28:0/2481493897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055c19d0b0 con 0x7f055c101830 2024-05-09T08:27:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.854+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c1021f0 0x7f055c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.855+0000 7f055bfff640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c101830 0x7f055c198110 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f054c009230 tx=0x7f054c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.855+0000 7f05597fa640 1 -- 172.21.15.28:0/2481493897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c03f030 con 0x7f055c101830 2024-05-09T08:27:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.855+0000 7f0562709640 1 -- 172.21.15.28:0/2481493897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f055c19d340 con 0x7f055c101830 2024-05-09T08:27:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.855+0000 7f05597fa640 1 -- 172.21.15.28:0/2481493897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f054c003ad0 con 0x7f055c101830 2024-05-09T08:27:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.855+0000 7f05597fa640 1 -- 172.21.15.28:0/2481493897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c035070 con 0x7f055c101830 2024-05-09T08:27:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.855+0000 7f0562709640 1 -- 172.21.15.28:0/2481493897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f055c1a4c30 con 0x7f055c101830 2024-05-09T08:27:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.857+0000 7f0565415740 1 Processor -- start 2024-05-09T08:27:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.857+0000 7f05597fa640 1 -- 172.21.15.28:0/2481493897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f054c042030 con 0x7f055c101830 2024-05-09T08:27:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.857+0000 7f0565415740 1 -- start start 2024-05-09T08:27:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0565415740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5450 0x5634fc2e5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0565415740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 0x5634fc2f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0565415740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 0x5634fc2f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0565415740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5634fc24a340 con 0x5634fc2e5450 2024-05-09T08:27:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0565415740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5634fc1f2290 con 0x5634fc2f0630 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0565415740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5634fc246300 con 0x5634fc2e5d70 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0560ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5450 0x5634fc2e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f055bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 0x5634fc2f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f055b7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 0x5634fc2f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0560ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5450 0x5634fc2e5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37506/0 (socket says 172.21.15.28:37506) 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0560ca6640 1 -- 172.21.15.28:0/2840435633 learned_addr learned my addr 172.21.15.28:0/2840435633 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f055b7fe640 1 -- 172.21.15.28:0/2840435633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 msgr2=0x5634fc2f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2840435633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 0x5634fc2f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f055b7fe640 1 -- 172.21.15.28:0/2840435633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5450 msgr2=0x5634fc2e5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2840435633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5450 0x5634fc2e5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f055b7fe640 1 -- 172.21.15.28:0/2840435633 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634fc2168e0 con 0x5634fc2f0630 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f0560ca6640 1 --2- 172.21.15.28:0/2840435633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5450 0x5634fc2e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.858+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2840435633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 0x5634fc2f2a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f055c106e30 tx=0x7f055000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cac6bbac4f99d2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.859+0000 7f05427fc640 1 -- 172.21.15.28:0/2840435633 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550004030 con 0x5634fc2f0630 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.859+0000 7f05427fc640 1 -- 172.21.15.28:0/2840435633 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0550005500 con 0x5634fc2f0630 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.859+0000 7f05427fc640 1 -- 172.21.15.28:0/2840435633 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550004c60 con 0x5634fc2f0630 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.859+0000 7f05597fa640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f05440a2ec0 0x7f05440a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.859+0000 7f0560ca6640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f05440a2ec0 0x7f05440a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.859+0000 7f05597fa640 1 -- 172.21.15.28:0/2481493897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f054c0effc0 con 0x7f055c101830 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.859+0000 7f0565415740 1 -- 172.21.15.28:0/2840435633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 msgr2=0x5634fc2f2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.859+0000 7f0560ca6640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f05440a2ec0 0x7f05440a5380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0554011490 tx=0x7f0554008630 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.859+0000 7f0565415740 1 --2- 172.21.15.28:0/2840435633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 0x5634fc2f2a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f055c106e30 tx=0x7f055000f730 comp rx=0 tx=0).stop 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 -- 172.21.15.28:0/2840435633 shutdown_connections 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 --2- 172.21.15.28:0/2840435633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 0x5634fc2f2a20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 --2- 172.21.15.28:0/2840435633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 0x5634fc2f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 --2- 172.21.15.28:0/2840435633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5450 0x5634fc2e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 -- 172.21.15.28:0/2840435633 >> 172.21.15.28:0/2840435633 conn(0x5634fc2e1960 msgr2=0x5634fc2e1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 -- 172.21.15.28:0/2840435633 shutdown_connections 2024-05-09T08:27:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 -- 172.21.15.28:0/2840435633 wait complete. 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 -- start start 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 Processor -- start 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5450 0x5634fc2ed490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ed9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 0x5634fc2edf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5634fc2f3260 con 0x5634fc2e5d70 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5634fc205790 con 0x5634fc2f0630 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5634fc2f33e0 con 0x5634fc2e5450 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.860+0000 7f0565415740 10 client.0 _ll_register_callbacks cb 0x5634fc240a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ed9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ed9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37512/0 (socket says 172.21.15.28:37512) 2024-05-09T08:27:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055b7fe640 1 -- 172.21.15.28:0/2964391136 learned_addr learned my addr 172.21.15.28:0/2964391136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f0560ca6640 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 0x5634fc2edf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055bfff640 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5450 0x5634fc2ed490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055b7fe640 1 -- 172.21.15.28:0/2964391136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5450 msgr2=0x5634fc2ed490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5450 0x5634fc2ed490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055b7fe640 1 -- 172.21.15.28:0/2964391136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 msgr2=0x5634fc2edf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 0x5634fc2edf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055b7fe640 1 -- 172.21.15.28:0/2964391136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634fbf5e210 con 0x5634fc2e5d70 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055bfff640 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5450 0x5634fc2ed490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ed9d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0550000c00 tx=0x7f0550005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1630efca74da4873 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f053b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f053b7fe640 1 -- 172.21.15.28:0/2964391136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055002b060 con 0x5634fc2e5d70 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f053b7fe640 1 -- 172.21.15.28:0/2964391136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0550004bf0 con 0x5634fc2e5d70 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f0565415740 1 -- 172.21.15.28:0/2964391136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0550005500 con 0x5634fc2e5d70 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f053b7fe640 1 -- 172.21.15.28:0/2964391136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0550032050 con 0x5634fc2e5d70 2024-05-09T08:27:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f0565415740 10 client.16188 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.861+0000 7f0565415740 1 -- 172.21.15.28:0/2964391136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5634fc02a620 con 0x5634fc2e5d70 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f053b7fe640 1 -- 172.21.15.28:0/2964391136 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0550005500 con 0x5634fc2e5d70 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f053b7fe640 1 -- 172.21.15.28:0/2964391136 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f055001b860 con 0x5634fc2e5d70 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f0565415740 10 client.16188 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f0565415740 10 client.16188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f0565415740 20 client.16188 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f0565415740 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5634fc300190 0x5634fc302580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f0565415740 4 client.16188 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f0565415740 1 -- 172.21.15.28:0/2964391136 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634fc257a70 con 0x5634fc300190 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f055bfff640 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5634fc300190 0x5634fc302580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f055bfff640 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5634fc300190 0x5634fc302580 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac3b615c2c0f600 server_cookie=b8a5fc73502cde7b in_seq=0 out_seq=0 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.862+0000 7f053b7fe640 10 client.16188 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f053b7fe640 1 -- 172.21.15.28:0/2964391136 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634fc257a70 con 0x5634fc300190 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f053b7fe640 10 client.16188 handle_command_reply: tid=0 2024-05-09T08:27:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f0565415740 1 client.16188 shutdown 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f0565415740 1 -- 172.21.15.28:0/2964391136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 msgr2=0x5634fc2ed9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f0565415740 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ed9d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0550000c00 tx=0x7f0550005720 comp rx=0 tx=0).stop 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f0565415740 1 -- 172.21.15.28:0/2964391136 shutdown_connections 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f0565415740 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5634fc300190 0x5634fc302580 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f0565415740 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2f0630 0x5634fc2edf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f0565415740 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ed9d0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f053b7fe640 0 client.16188 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f0565415740 1 --2- 172.21.15.28:0/2964391136 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5450 0x5634fc2ed490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.863+0000 7f0565415740 1 -- 172.21.15.28:0/2964391136 >> 172.21.15.28:0/2964391136 conn(0x5634fc2e1960 msgr2=0x5634fc2e76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.864+0000 7f0565415740 1 -- 172.21.15.28:0/2964391136 shutdown_connections 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.864+0000 7f0565415740 1 -- 172.21.15.28:0/2964391136 wait complete. 2024-05-09T08:27:44.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.864+0000 7f0565415740 20 client.16188 trim_cache size 0 max 16384 2024-05-09T08:27:44.873 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strtoull... yes 2024-05-09T08:27:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0565415740 1 Processor -- start 2024-05-09T08:27:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0565415740 1 -- start start 2024-05-09T08:27:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0565415740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0565415740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ec5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0565415740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2f0630 0x5634fc2e0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0565415740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1aee60 con 0x5634fc2e5d70 2024-05-09T08:27:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0565415740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1b09a0 con 0x5634fc2e5450 2024-05-09T08:27:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0565415740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1af530 con 0x5634fc2f0630 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ec5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ec5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37522/0 (socket says 172.21.15.28:37522) 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055b7fe640 1 -- 172.21.15.28:0/2925123332 learned_addr learned my addr 172.21.15.28:0/2925123332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0560ca6640 1 --2- 172.21.15.28:0/2925123332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2f0630 0x5634fc2e0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055b7fe640 1 -- 172.21.15.28:0/2925123332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2f0630 msgr2=0x5634fc2e0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2925123332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2f0630 0x5634fc2e0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055b7fe640 1 -- 172.21.15.28:0/2925123332 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 msgr2=0x5634fc2efbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2925123332 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2efbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055b7fe640 1 -- 172.21.15.28:0/2925123332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634fc2f3380 con 0x5634fc2e5d70 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055bfff640 1 --2- 172.21.15.28:0/2925123332 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2efbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f055b7fe640 1 --2- 172.21.15.28:0/2925123332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ec5f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f05500059f0 tx=0x7f055000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0c1940b86df959 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.879+0000 7f0541ffb640 1 -- 172.21.15.28:0/2925123332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055002b070 con 0x5634fc2e5d70 2024-05-09T08:27:44.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0541ffb640 1 -- 172.21.15.28:0/2925123332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0550020070 con 0x5634fc2e5d70 2024-05-09T08:27:44.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0541ffb640 1 -- 172.21.15.28:0/2925123332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05500118d0 con 0x5634fc2e5d70 2024-05-09T08:27:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0565415740 1 -- 172.21.15.28:0/2925123332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 msgr2=0x5634fc2ec5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0565415740 1 --2- 172.21.15.28:0/2925123332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ec5f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f05500059f0 tx=0x7f055000bab0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0565415740 1 -- 172.21.15.28:0/2925123332 shutdown_connections 2024-05-09T08:27:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0565415740 1 --2- 172.21.15.28:0/2925123332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2f0630 0x5634fc2e0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0565415740 1 --2- 172.21.15.28:0/2925123332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2e5d70 0x5634fc2ec5f0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0565415740 1 --2- 172.21.15.28:0/2925123332 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2efbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0565415740 1 -- 172.21.15.28:0/2925123332 >> 172.21.15.28:0/2925123332 conn(0x5634fc2e1960 msgr2=0x5634fc2e0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0565415740 1 -- 172.21.15.28:0/2925123332 shutdown_connections 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.880+0000 7f0565415740 1 -- 172.21.15.28:0/2925123332 wait complete. 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0565415740 1 -- start start 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0565415740 1 Processor -- start 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0565415740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0565415740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 0x5634fc2f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0565415740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2f0630 0x5634fc2f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0565415740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1b02d0 con 0x5634fc2f0630 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0565415740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1afc00 con 0x5634fc2e5450 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0565415740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1ace40 con 0x5634fc2e5d70 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0565415740 10 client.0 _ll_register_callbacks cb 0x7f0550015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0560ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2f0630 0x5634fc2f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0560ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2f0630 0x5634fc2f80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37526/0 (socket says 172.21.15.28:37526) 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0560ca6640 1 -- 172.21.15.28:0/1007713542 learned_addr learned my addr 172.21.15.28:0/1007713542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f055bfff640 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f055b7fe640 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 0x5634fc2f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0560ca6640 1 -- 172.21.15.28:0/1007713542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 msgr2=0x5634fc2f40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0560ca6640 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 0x5634fc2f40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0560ca6640 1 -- 172.21.15.28:0/1007713542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 msgr2=0x5634fc2f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0560ca6640 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f0560ca6640 1 -- 172.21.15.28:0/1007713542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634fc2f8720 con 0x5634fc2f0630 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.881+0000 7f055bfff640 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f0560ca6640 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2f0630 0x5634fc2f80b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f055400fee0 tx=0x7f0554004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384b13f130e9155a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f053b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f053b7fe640 1 -- 172.21.15.28:0/1007713542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055401dca0 con 0x5634fc2f0630 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f0565415740 1 -- 172.21.15.28:0/1007713542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0550020070 con 0x5634fc2f0630 2024-05-09T08:27:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f053b7fe640 1 -- 172.21.15.28:0/1007713542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f05540273f0 con 0x5634fc2f0630 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f053b7fe640 1 -- 172.21.15.28:0/1007713542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0554035050 con 0x5634fc2f0630 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f0565415740 10 client.16194 fetch_fsmap learned FSMap version 28 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f0565415740 1 -- 172.21.15.28:0/1007713542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5634fc366030 con 0x5634fc2f0630 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f053b7fe640 1 -- 172.21.15.28:0/1007713542 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0550020070 con 0x5634fc2f0630 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f053b7fe640 1 -- 172.21.15.28:0/1007713542 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f055401c420 con 0x5634fc2f0630 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.882+0000 7f0565415740 10 client.16194 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.883+0000 7f0565415740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.883+0000 7f0565415740 20 client.16194 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.883+0000 7f0565415740 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5634fc3667b0 0x5634fc368bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.883+0000 7f0565415740 4 client.16194 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:27:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.883+0000 7f0565415740 1 -- 172.21.15.28:0/1007713542 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5634fc251dc0 con 0x5634fc3667b0 2024-05-09T08:27:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.883+0000 7f055bfff640 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5634fc3667b0 0x5634fc368bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.883+0000 7f053b7fe640 10 client.16194 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.883+0000 7f055bfff640 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5634fc3667b0 0x5634fc368bd0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fbf02240b4fbeae server_cookie=1f453f4c7968d357 in_seq=0 out_seq=0 2024-05-09T08:27:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.974+0000 7f053b7fe640 1 -- 172.21.15.28:0/1007713542 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5634fc251dc0 con 0x5634fc3667b0 2024-05-09T08:27:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.974+0000 7f053b7fe640 10 client.16194 handle_command_reply: tid=0 2024-05-09T08:27:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.974+0000 7f0565415740 1 client.16194 shutdown 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 -- 172.21.15.28:0/1007713542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2f0630 msgr2=0x5634fc2f80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2f0630 0x5634fc2f80b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f055400fee0 tx=0x7f0554004a40 comp rx=0 tx=0).stop 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 -- 172.21.15.28:0/1007713542 shutdown_connections 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5634fc3667b0 0x5634fc368bd0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fc2f0630 0x5634fc2f80b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5634fc2e5d70 0x5634fc2f40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 --2- 172.21.15.28:0/1007713542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5634fc2e5450 0x5634fc2f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f053b7fe640 0 client.16194 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 -- 172.21.15.28:0/1007713542 >> 172.21.15.28:0/1007713542 conn(0x5634fc2e1960 msgr2=0x5634fc2e0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 -- 172.21.15.28:0/1007713542 shutdown_connections 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 1 -- 172.21.15.28:0/1007713542 wait complete. 2024-05-09T08:27:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0565415740 20 client.16194 trim_cache size 0 max 16384 2024-05-09T08:27:44.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:27:44.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:27:44.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:27:44.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0562709640 1 -- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f05440a2ec0 msgr2=0x7f05440a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0562709640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f05440a2ec0 0x7f05440a5380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0554011490 tx=0x7f0554008630 comp rx=0 tx=0).stop 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0562709640 1 -- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c101830 msgr2=0x7f055c198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.975+0000 7f0562709640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c101830 0x7f055c198110 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f054c009230 tx=0x7f054c004270 comp rx=0 tx=0).stop 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.976+0000 7f0562709640 1 -- 172.21.15.28:0/2481493897 shutdown_connections 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.976+0000 7f0562709640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f05440a2ec0 0x7f05440a5380 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.976+0000 7f0562709640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c107840 0x7f055c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.976+0000 7f0562709640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c1021f0 0x7f055c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.976+0000 7f0562709640 1 --2- 172.21.15.28:0/2481493897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055c101830 0x7f055c198110 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.976+0000 7f0562709640 1 -- 172.21.15.28:0/2481493897 >> 172.21.15.28:0/2481493897 conn(0x7f055c0fd320 msgr2=0x7f055c0fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.976+0000 7f0562709640 1 -- 172.21.15.28:0/2481493897 shutdown_connections 2024-05-09T08:27:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:44.976+0000 7f0562709640 1 -- 172.21.15.28:0/2481493897 wait complete. 2024-05-09T08:27:44.998 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strtoq... yes 2024-05-09T08:27:45.061 INFO:tasks.workunit.client.0.smithi028.stdout:checking for snprintf... yes 2024-05-09T08:27:45.126 INFO:tasks.workunit.client.0.smithi028.stdout:checking for vsnprintf... yes 2024-05-09T08:27:45.190 INFO:tasks.workunit.client.0.smithi028.stdout:checking for vfprintf... yes 2024-05-09T08:27:45.254 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gethostname... yes 2024-05-09T08:27:45.323 INFO:tasks.workunit.client.0.smithi028.stdout:checking for setlocale... yes 2024-05-09T08:27:45.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:45 smithi165 ceph-mon[34540]: pgmap v46: 129 pgs: 129 active+clean; 3.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 152 KiB/s wr, 17 op/s 2024-05-09T08:27:45.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:45 smithi184 ceph-mon[34509]: pgmap v46: 129 pgs: 129 active+clean; 3.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 152 KiB/s wr, 17 op/s 2024-05-09T08:27:45.385 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sync... yes 2024-05-09T08:27:45.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:45 smithi028 ceph-mon[30119]: pgmap v46: 129 pgs: 129 active+clean; 3.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 152 KiB/s wr, 17 op/s 2024-05-09T08:27:45.454 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether snprintf is C99 conformant... done 2024-05-09T08:27:45.466 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-05-09T08:27:45.645 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating Makefile 2024-05-09T08:27:45.663 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating src/Makefile 2024-05-09T08:27:45.683 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating man/Makefile 2024-05-09T08:27:45.695 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-05-09T08:27:45.758 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing depfiles commands 2024-05-09T08:27:45.952 INFO:tasks.workunit.client.0.smithi028.stderr:+ make 2024-05-09T08:27:45.962 INFO:tasks.workunit.client.0.smithi028.stdout:make all-recursive 2024-05-09T08:27:45.964 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-05-09T08:27:45.967 INFO:tasks.workunit.client.0.smithi028.stdout:Making all in src 2024-05-09T08:27:45.972 INFO:tasks.workunit.client.0.smithi028.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-05-09T08:27:45.972 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-05-09T08:27:45.972 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-05-09T08:27:46.076 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-05-09T08:27:46.076 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-05-09T08:27:46.147 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-05-09T08:27:46.147 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-05-09T08:27:46.199 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-05-09T08:27:46.199 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-05-09T08:27:46.251 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-05-09T08:27:46.251 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-05-09T08:27:46.305 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-05-09T08:27:46.305 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-05-09T08:27:46.355 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-05-09T08:27:46.355 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-05-09T08:27:46.444 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-05-09T08:27:46.444 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-05-09T08:27:46.481 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-05-09T08:27:46.481 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-05-09T08:27:46.516 INFO:tasks.workunit.client.0.smithi028.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-05-09T08:27:46.516 INFO:tasks.workunit.client.0.smithi028.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-05-09T08:27:46.555 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-05-09T08:27:46.574 INFO:tasks.workunit.client.0.smithi028.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-05-09T08:27:46.575 INFO:tasks.workunit.client.0.smithi028.stdout:Making all in man 2024-05-09T08:27:46.600 INFO:tasks.workunit.client.0.smithi028.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-05-09T08:27:46.600 INFO:tasks.workunit.client.0.smithi028.stdout:make[2]: Nothing to be done for 'all'. 2024-05-09T08:27:46.600 INFO:tasks.workunit.client.0.smithi028.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-05-09T08:27:46.601 INFO:tasks.workunit.client.0.smithi028.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-05-09T08:27:46.602 INFO:tasks.workunit.client.0.smithi028.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-05-09T08:27:46.603 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-05-09T08:27:46.603 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd src 2024-05-09T08:27:46.603 INFO:tasks.workunit.client.0.smithi028.stderr:+ mkdir blogtest_in 2024-05-09T08:27:46.605 INFO:tasks.workunit.client.0.smithi028.stdout:running blogbench 2024-05-09T08:27:46.605 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'running blogbench' 2024-05-09T08:27:46.605 INFO:tasks.workunit.client.0.smithi028.stderr:+ ./blogbench -d blogtest_in 2024-05-09T08:27:46.606 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-05-09T08:27:46.606 INFO:tasks.workunit.client.0.smithi028.stdout:Frequency = 10 secs 2024-05-09T08:27:46.606 INFO:tasks.workunit.client.0.smithi028.stdout:Scratch dir = [blogtest_in] 2024-05-09T08:27:47.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:47 smithi165 ceph-mon[34540]: pgmap v47: 129 pgs: 129 active+clean; 3.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 152 KiB/s wr, 15 op/s 2024-05-09T08:27:47.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:47 smithi184 ceph-mon[34509]: pgmap v47: 129 pgs: 129 active+clean; 3.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 152 KiB/s wr, 15 op/s 2024-05-09T08:27:47.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:47 smithi028 ceph-mon[30119]: pgmap v47: 129 pgs: 129 active+clean; 3.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 152 KiB/s wr, 15 op/s 2024-05-09T08:27:48.701 INFO:tasks.workunit.client.0.smithi028.stdout:Spawning 3 writers... 2024-05-09T08:27:48.702 INFO:tasks.workunit.client.0.smithi028.stdout:Spawning 1 rewriters... 2024-05-09T08:27:48.702 INFO:tasks.workunit.client.0.smithi028.stdout:Spawning 5 commenters... 2024-05-09T08:27:48.702 INFO:tasks.workunit.client.0.smithi028.stdout:Spawning 100 readers... 2024-05-09T08:27:48.704 INFO:tasks.workunit.client.0.smithi028.stdout:Benchmarking for 30 iterations. 2024-05-09T08:27:48.704 INFO:tasks.workunit.client.0.smithi028.stdout:The test will run during 5 minutes. 2024-05-09T08:27:48.704 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-05-09T08:27:48.704 INFO:tasks.workunit.client.0.smithi028.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-05-09T08:27:49.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:49 smithi165 ceph-mon[34540]: pgmap v48: 129 pgs: 129 active+clean; 3.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 152 KiB/s wr, 13 op/s 2024-05-09T08:27:49.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:49 smithi184 ceph-mon[34509]: pgmap v48: 129 pgs: 129 active+clean; 3.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 152 KiB/s wr, 13 op/s 2024-05-09T08:27:49.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:49 smithi028 ceph-mon[30119]: pgmap v48: 129 pgs: 129 active+clean; 3.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 152 KiB/s wr, 13 op/s 2024-05-09T08:27:51.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:51 smithi165 ceph-mon[34540]: pgmap v49: 129 pgs: 129 active+clean; 6.3 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 510 KiB/s wr, 137 op/s 2024-05-09T08:27:51.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:51 smithi165 ceph-mon[34540]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-05-09T08:27:51.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:51 smithi184 ceph-mon[34509]: pgmap v49: 129 pgs: 129 active+clean; 6.3 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 510 KiB/s wr, 137 op/s 2024-05-09T08:27:51.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:51 smithi184 ceph-mon[34509]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-05-09T08:27:51.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:51 smithi028 ceph-mon[30119]: pgmap v49: 129 pgs: 129 active+clean; 6.3 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 510 KiB/s wr, 137 op/s 2024-05-09T08:27:51.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:51 smithi028 ceph-mon[30119]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-05-09T08:27:53.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:53 smithi165 ceph-mon[34540]: pgmap v50: 129 pgs: 129 active+clean; 6.3 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 510 KiB/s wr, 132 op/s 2024-05-09T08:27:53.359 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:53 smithi184 ceph-mon[34509]: pgmap v50: 129 pgs: 129 active+clean; 6.3 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 510 KiB/s wr, 132 op/s 2024-05-09T08:27:53.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:53 smithi028 ceph-mon[30119]: pgmap v50: 129 pgs: 129 active+clean; 6.3 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 510 KiB/s wr, 132 op/s 2024-05-09T08:27:55.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:55 smithi184 ceph-mon[34509]: pgmap v51: 129 pgs: 129 active+clean; 18 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-05-09T08:27:55.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:55 smithi028 ceph-mon[30119]: pgmap v51: 129 pgs: 129 active+clean; 18 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-05-09T08:27:55.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:55 smithi165 ceph-mon[34540]: pgmap v51: 129 pgs: 129 active+clean; 18 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-05-09T08:27:57.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:57 smithi028 ceph-mon[30119]: pgmap v52: 129 pgs: 129 active+clean; 37 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.1 MiB/s wr, 251 op/s 2024-05-09T08:27:57.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:57 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:27:57.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:57 smithi165 ceph-mon[34540]: pgmap v52: 129 pgs: 129 active+clean; 37 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.1 MiB/s wr, 251 op/s 2024-05-09T08:27:57.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:57 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:27:57.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:57 smithi184 ceph-mon[34509]: pgmap v52: 129 pgs: 129 active+clean; 37 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.1 MiB/s wr, 251 op/s 2024-05-09T08:27:57.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:57 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:27:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.067+0000 7fa926bce640 1 -- 172.21.15.28:0/4043025927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 msgr2=0x7fa920100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.067+0000 7fa926bce640 1 --2- 172.21.15.28:0/4043025927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 0x7fa920100be0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009930 tx=0x7fa90c02a420 comp rx=0 tx=0).stop 2024-05-09T08:27:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.067+0000 7fa926bce640 1 -- 172.21.15.28:0/4043025927 shutdown_connections 2024-05-09T08:27:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.067+0000 7fa926bce640 1 --2- 172.21.15.28:0/4043025927 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa920101120 0x7fa92010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.067+0000 7fa926bce640 1 --2- 172.21.15.28:0/4043025927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 0x7fa920100be0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.067+0000 7fa926bce640 1 --2- 172.21.15.28:0/4043025927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa920106720 0x7fa920106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.067+0000 7fa926bce640 1 -- 172.21.15.28:0/4043025927 >> 172.21.15.28:0/4043025927 conn(0x7fa9200778b0 msgr2=0x7fa920077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.067+0000 7fa926bce640 1 -- 172.21.15.28:0/4043025927 shutdown_connections 2024-05-09T08:27:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.067+0000 7fa926bce640 1 -- 172.21.15.28:0/4043025927 wait complete. 2024-05-09T08:27:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa926bce640 1 Processor -- start 2024-05-09T08:27:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa926bce640 1 -- start start 2024-05-09T08:27:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa926bce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 0x7fa920198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa926bce640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa920101120 0x7fa920199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa926bce640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa920106720 0x7fa9201a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa926bce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa920110b30 con 0x7fa920100760 2024-05-09T08:27:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa926bce640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa9201109b0 con 0x7fa920106720 2024-05-09T08:27:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa926bce640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa920110cb0 con 0x7fa920101120 2024-05-09T08:27:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa92496a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 0x7fa920198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa92496a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 0x7fa920198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33142/0 (socket says 172.21.15.28:33142) 2024-05-09T08:27:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa92496a640 1 -- 172.21.15.28:0/1985285577 learned_addr learned my addr 172.21.15.28:0/1985285577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.068+0000 7fa92516b640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa920106720 0x7fa9201a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa92496a640 1 -- 172.21.15.28:0/1985285577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa920101120 msgr2=0x7fa920199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa92496a640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa920101120 0x7fa920199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa92496a640 1 -- 172.21.15.28:0/1985285577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa920106720 msgr2=0x7fa9201a4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa92496a640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa920106720 0x7fa9201a4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa92496a640 1 -- 172.21.15.28:0/1985285577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9201a45b0 con 0x7fa920100760 2024-05-09T08:27:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa92496a640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 0x7fa920198ff0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa91400e9f0 tx=0x7fa91400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa91dffb640 1 -- 172.21.15.28:0/1985285577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91400cd80 con 0x7fa920100760 2024-05-09T08:27:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa91dffb640 1 -- 172.21.15.28:0/1985285577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa914010910 con 0x7fa920100760 2024-05-09T08:27:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa926bce640 1 -- 172.21.15.28:0/1985285577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9201a4780 con 0x7fa920100760 2024-05-09T08:27:58.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa91dffb640 1 -- 172.21.15.28:0/1985285577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914018990 con 0x7fa920100760 2024-05-09T08:27:58.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.069+0000 7fa926bce640 1 -- 172.21.15.28:0/1985285577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9201a4c70 con 0x7fa920100760 2024-05-09T08:27:58.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.071+0000 7fa91dffb640 1 -- 172.21.15.28:0/1985285577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa914010ac0 con 0x7fa920100760 2024-05-09T08:27:58.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa9299bb740 1 Processor -- start 2024-05-09T08:27:58.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa9299bb740 1 -- start start 2024-05-09T08:27:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 0x560a71a67830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a72630 0x560a71a74a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa9299bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560a719cc340 con 0x560a71a67d70 2024-05-09T08:27:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa9299bb740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560a71974290 con 0x560a71a67450 2024-05-09T08:27:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa9299bb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560a719c8300 con 0x560a71a72630 2024-05-09T08:27:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa92496a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa92496a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33154/0 (socket says 172.21.15.28:33154) 2024-05-09T08:27:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa92496a640 1 -- 172.21.15.28:0/2364606557 learned_addr learned my addr 172.21.15.28:0/2364606557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.072+0000 7fa92516b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 0x560a71a67830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa91dffb640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9100a2da0 0x7fa9100a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa91ffff640 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a72630 0x560a71a74a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa92516b640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9100a2da0 0x7fa9100a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa91dffb640 1 -- 172.21.15.28:0/1985285577 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa9140cfb50 con 0x7fa920100760 2024-05-09T08:27:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa92496a640 1 -- 172.21.15.28:0/2364606557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a72630 msgr2=0x560a71a74a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa92496a640 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a72630 0x560a71a74a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa92496a640 1 -- 172.21.15.28:0/2364606557 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 msgr2=0x560a71a67830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa92496a640 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 0x560a71a67830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa92496a640 1 -- 172.21.15.28:0/2364606557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a719988e0 con 0x560a71a67d70 2024-05-09T08:27:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa92516b640 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 0x560a71a67830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa91ffff640 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a72630 0x560a71a74a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa92496a640 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a720f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa914098100 tx=0x7fa914018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f735ab21efc68e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa92516b640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9100a2da0 0x7fa9100a5260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa918009fd0 tx=0x7fa918004a40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa90affd640 1 -- 172.21.15.28:0/2364606557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91409c3a0 con 0x560a71a67d70 2024-05-09T08:27:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa90affd640 1 -- 172.21.15.28:0/2364606557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa91409bca0 con 0x560a71a67d70 2024-05-09T08:27:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.073+0000 7fa90affd640 1 -- 172.21.15.28:0/2364606557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914018b30 con 0x560a71a67d70 2024-05-09T08:27:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 -- 172.21.15.28:0/2364606557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 msgr2=0x560a71a720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a720f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa914098100 tx=0x7fa914018420 comp rx=0 tx=0).stop 2024-05-09T08:27:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 -- 172.21.15.28:0/2364606557 shutdown_connections 2024-05-09T08:27:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a72630 0x560a71a74a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a720f0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2364606557 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 0x560a71a67830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 -- 172.21.15.28:0/2364606557 >> 172.21.15.28:0/2364606557 conn(0x560a71a63960 msgr2=0x560a71a63d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 -- 172.21.15.28:0/2364606557 shutdown_connections 2024-05-09T08:27:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 -- 172.21.15.28:0/2364606557 wait complete. 2024-05-09T08:27:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 -- start start 2024-05-09T08:27:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.074+0000 7fa9299bb740 1 Processor -- start 2024-05-09T08:27:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 0x560a71a6f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67d70 0x560a71a6fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a72630 0x560a71a70050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa9299bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560a71a751a0 con 0x560a71a72630 2024-05-09T08:27:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa9299bb740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560a71987790 con 0x560a71a67450 2024-05-09T08:27:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa9299bb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560a71a75320 con 0x560a71a67d70 2024-05-09T08:27:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa9299bb740 10 client.0 _ll_register_callbacks cb 0x560a719c2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92516b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a72630 0x560a71a70050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92516b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a72630 0x560a71a70050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33164/0 (socket says 172.21.15.28:33164) 2024-05-09T08:27:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92516b640 1 -- 172.21.15.28:0/2313346582 learned_addr learned my addr 172.21.15.28:0/2313346582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92496a640 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 0x560a71a6f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa91ffff640 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67d70 0x560a71a6fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92516b640 1 -- 172.21.15.28:0/2313346582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67d70 msgr2=0x560a71a6fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92516b640 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67d70 0x560a71a6fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92516b640 1 -- 172.21.15.28:0/2313346582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 msgr2=0x560a71a6f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92516b640 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 0x560a71a6f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92516b640 1 -- 172.21.15.28:0/2313346582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a716e0210 con 0x560a71a72630 2024-05-09T08:27:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa92516b640 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a72630 0x560a71a70050 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa91800af30 tx=0x7fa918004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353e7dc9de6f744e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.075+0000 7fa8fffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa8fffff640 1 -- 172.21.15.28:0/2313346582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918023610 con 0x560a71a72630 2024-05-09T08:27:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa8fffff640 1 -- 172.21.15.28:0/2313346582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa9180237b0 con 0x560a71a72630 2024-05-09T08:27:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa9299bb740 1 -- 172.21.15.28:0/2313346582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa91409bca0 con 0x560a71a72630 2024-05-09T08:27:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa8fffff640 1 -- 172.21.15.28:0/2313346582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa918031050 con 0x560a71a72630 2024-05-09T08:27:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa9299bb740 10 client.16206 fetch_fsmap learned FSMap version 29 2024-05-09T08:27:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa9299bb740 1 -- 172.21.15.28:0/2313346582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560a717ac620 con 0x560a71a72630 2024-05-09T08:27:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa8fffff640 1 -- 172.21.15.28:0/2313346582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa91409bca0 con 0x560a71a72630 2024-05-09T08:27:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa8fffff640 1 -- 172.21.15.28:0/2313346582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fa91802c840 con 0x560a71a72630 2024-05-09T08:27:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa9299bb740 10 client.16206 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:27:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa9299bb740 10 client.16206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.076+0000 7fa9299bb740 20 client.16206 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.077+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560a71a82370 0x560a71a847c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.077+0000 7fa9299bb740 4 client.16206 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.077+0000 7fa9299bb740 1 -- 172.21.15.28:0/2313346582 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a719d9a70 con 0x560a71a82370 2024-05-09T08:27:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.077+0000 7fa92496a640 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560a71a82370 0x560a71a847c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.077+0000 7fa8fffff640 10 client.16206 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.077+0000 7fa92496a640 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560a71a82370 0x560a71a847c0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d037da1944c37df1 server_cookie=bfe632f3c4a06890 in_seq=0 out_seq=0 2024-05-09T08:27:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.079+0000 7fa8fffff640 1 -- 172.21.15.28:0/2313346582 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a719d9a70 con 0x560a71a82370 2024-05-09T08:27:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.079+0000 7fa8fffff640 10 client.16206 handle_command_reply: tid=0 2024-05-09T08:27:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.079+0000 7fa9299bb740 1 client.16206 shutdown 2024-05-09T08:27:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.079+0000 7fa9299bb740 1 -- 172.21.15.28:0/2313346582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a72630 msgr2=0x560a71a70050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.079+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a72630 0x560a71a70050 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa91800af30 tx=0x7fa918004b50 comp rx=0 tx=0).stop 2024-05-09T08:27:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.079+0000 7fa9299bb740 1 -- 172.21.15.28:0/2313346582 shutdown_connections 2024-05-09T08:27:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.079+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560a71a82370 0x560a71a847c0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.079+0000 7fa8fffff640 0 client.16206 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.079+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a72630 0x560a71a70050 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.080+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67d70 0x560a71a6fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.080+0000 7fa9299bb740 1 --2- 172.21.15.28:0/2313346582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a67450 0x560a71a6f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.080+0000 7fa9299bb740 1 -- 172.21.15.28:0/2313346582 >> 172.21.15.28:0/2313346582 conn(0x560a71a63960 msgr2=0x560a71a696e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.080+0000 7fa9299bb740 1 -- 172.21.15.28:0/2313346582 shutdown_connections 2024-05-09T08:27:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.080+0000 7fa9299bb740 1 -- 172.21.15.28:0/2313346582 wait complete. 2024-05-09T08:27:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.080+0000 7fa9299bb740 20 client.16206 trim_cache size 0 max 16384 2024-05-09T08:27:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa9299bb740 1 Processor -- start 2024-05-09T08:27:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa9299bb740 1 -- start start 2024-05-09T08:27:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67450 0x560a71a7e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67d70 0x560a71a63d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 0x560a71a8a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa9299bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9201aeea0 con 0x560a71a67450 2024-05-09T08:27:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa9299bb740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa9201b0a10 con 0x560a71a72630 2024-05-09T08:27:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa9299bb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9201af570 con 0x560a71a67d70 2024-05-09T08:27:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa92496a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67450 0x560a71a7e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa92496a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67450 0x560a71a7e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33172/0 (socket says 172.21.15.28:33172) 2024-05-09T08:27:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa92496a640 1 -- 172.21.15.28:0/373818692 learned_addr learned my addr 172.21.15.28:0/373818692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.095+0000 7fa92516b640 1 --2- 172.21.15.28:0/373818692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 0x560a71a8a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.096+0000 7fa92496a640 1 -- 172.21.15.28:0/373818692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67d70 msgr2=0x560a71a63d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.096+0000 7fa92496a640 1 --2- 172.21.15.28:0/373818692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67d70 0x560a71a63d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.096+0000 7fa92496a640 1 -- 172.21.15.28:0/373818692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 msgr2=0x560a71a8a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.096+0000 7fa92496a640 1 --2- 172.21.15.28:0/373818692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 0x560a71a8a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.096+0000 7fa92496a640 1 -- 172.21.15.28:0/373818692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a71a6e580 con 0x560a71a67450 2024-05-09T08:27:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.096+0000 7fa92496a640 1 --2- 172.21.15.28:0/373818692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67450 0x560a71a7e060 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa914098670 tx=0x7fa914098070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbaebf06ec473eda server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.096+0000 7fa90a7fc640 1 -- 172.21.15.28:0/373818692 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91409bca0 con 0x560a71a67450 2024-05-09T08:27:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.096+0000 7fa90a7fc640 1 -- 172.21.15.28:0/373818692 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa914099d40 con 0x560a71a67450 2024-05-09T08:27:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.096+0000 7fa90a7fc640 1 -- 172.21.15.28:0/373818692 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91409b0f0 con 0x560a71a67450 2024-05-09T08:27:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 -- 172.21.15.28:0/373818692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67450 msgr2=0x560a71a7e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 --2- 172.21.15.28:0/373818692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67450 0x560a71a7e060 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa914098670 tx=0x7fa914098070 comp rx=0 tx=0).stop 2024-05-09T08:27:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 -- 172.21.15.28:0/373818692 shutdown_connections 2024-05-09T08:27:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 --2- 172.21.15.28:0/373818692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 0x560a71a8a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 --2- 172.21.15.28:0/373818692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67d70 0x560a71a63d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 --2- 172.21.15.28:0/373818692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67450 0x560a71a7e060 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 -- 172.21.15.28:0/373818692 >> 172.21.15.28:0/373818692 conn(0x560a71a63960 msgr2=0x560a71a62a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 -- 172.21.15.28:0/373818692 shutdown_connections 2024-05-09T08:27:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 -- 172.21.15.28:0/373818692 wait complete. 2024-05-09T08:27:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 -- start start 2024-05-09T08:27:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.097+0000 7fa9299bb740 1 Processor -- start 2024-05-09T08:27:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67450 0x560a71a76b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a770b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa9299bb740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 0x560a71ae0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa9299bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9201b0310 con 0x560a71a67d70 2024-05-09T08:27:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa9299bb740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa9201afc40 con 0x560a71a72630 2024-05-09T08:27:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa9299bb740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9201ace80 con 0x560a71a67450 2024-05-09T08:27:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa9299bb740 10 client.0 _ll_register_callbacks cb 0x7fa918015d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa91ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a770b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa91ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a770b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33188/0 (socket says 172.21.15.28:33188) 2024-05-09T08:27:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa91ffff640 1 -- 172.21.15.28:0/3506221002 learned_addr learned my addr 172.21.15.28:0/3506221002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa92516b640 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 0x560a71ae0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa92496a640 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67450 0x560a71a76b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa91ffff640 1 -- 172.21.15.28:0/3506221002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67450 msgr2=0x560a71a76b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa91ffff640 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67450 0x560a71a76b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa91ffff640 1 -- 172.21.15.28:0/3506221002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 msgr2=0x560a71ae0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa91ffff640 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 0x560a71ae0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa91ffff640 1 -- 172.21.15.28:0/3506221002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a71ae1130 con 0x560a71a67d70 2024-05-09T08:27:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa92516b640 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 0x560a71ae0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa91ffff640 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a770b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa920069a00 tx=0x7fa90c0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a965eae89ead9ba8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa8fffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:27:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa8fffff640 1 -- 172.21.15.28:0/3506221002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c040030 con 0x560a71a67d70 2024-05-09T08:27:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa9299bb740 1 -- 172.21.15.28:0/3506221002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa914099d40 con 0x560a71a67d70 2024-05-09T08:27:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.098+0000 7fa8fffff640 1 -- 172.21.15.28:0/3506221002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa90c004040 con 0x560a71a67d70 2024-05-09T08:27:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa8fffff640 1 -- 172.21.15.28:0/3506221002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa90c04d050 con 0x560a71a67d70 2024-05-09T08:27:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa9299bb740 10 client.16212 fetch_fsmap learned FSMap version 29 2024-05-09T08:27:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa9299bb740 1 -- 172.21.15.28:0/3506221002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560a71ae94e0 con 0x560a71a67d70 2024-05-09T08:27:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa8fffff640 1 -- 172.21.15.28:0/3506221002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa914099d40 con 0x560a71a67d70 2024-05-09T08:27:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa8fffff640 1 -- 172.21.15.28:0/3506221002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fa90c03a040 con 0x560a71a67d70 2024-05-09T08:27:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa9299bb740 10 client.16212 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:27:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa9299bb740 10 client.16212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa9299bb740 20 client.16212 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa9299bb740 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560a71a82370 0x560a71aebd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa9299bb740 4 client.16212 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:27:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa9299bb740 1 -- 172.21.15.28:0/3506221002 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560a719d3dc0 con 0x560a71a82370 2024-05-09T08:27:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa92496a640 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560a71a82370 0x560a71aebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa92496a640 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560a71a82370 0x560a71aebd50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ef77a0a16fe0e8c server_cookie=14a7d8b1dc68104b in_seq=0 out_seq=0 2024-05-09T08:27:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:27:58.099+0000 7fa8fffff640 10 client.16212 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:27:58.704 INFO:tasks.workunit.client.0.smithi028.stdout: 13 232160 799 193540 868 97889 978 2024-05-09T08:27:59.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:27:59.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:27:59.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:27:59 smithi028 ceph-mon[30119]: pgmap v53: 129 pgs: 129 active+clean; 37 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.1 MiB/s wr, 249 op/s 2024-05-09T08:27:59.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:27:59 smithi165 ceph-mon[34540]: pgmap v53: 129 pgs: 129 active+clean; 37 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.1 MiB/s wr, 249 op/s 2024-05-09T08:27:59.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:27:59 smithi184 ceph-mon[34509]: pgmap v53: 129 pgs: 129 active+clean; 37 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.1 MiB/s wr, 249 op/s 2024-05-09T08:27:59.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.671+0000 7f9afa93b640 1 -- 172.21.15.28:0/3879430028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af41010c0 msgr2=0x7f9af410d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:59.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.671+0000 7f9afa93b640 1 --2- 172.21.15.28:0/3879430028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af41010c0 0x7f9af410d5d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009cd0 tx=0x7f9aec02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.671+0000 7f9afa93b640 1 -- 172.21.15.28:0/3879430028 shutdown_connections 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.671+0000 7f9afa93b640 1 --2- 172.21.15.28:0/3879430028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af41010c0 0x7f9af410d5d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.671+0000 7f9afa93b640 1 --2- 172.21.15.28:0/3879430028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af4100700 0x7f9af4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.671+0000 7f9afa93b640 1 --2- 172.21.15.28:0/3879430028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9af4106710 0x7f9af4106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.671+0000 7f9afa93b640 1 -- 172.21.15.28:0/3879430028 >> 172.21.15.28:0/3879430028 conn(0x7f9af4077830 msgr2=0x7f9af4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.671+0000 7f9afa93b640 1 -- 172.21.15.28:0/3879430028 shutdown_connections 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.671+0000 7f9afa93b640 1 -- 172.21.15.28:0/3879430028 wait complete. 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9afa93b640 1 Processor -- start 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9afa93b640 1 -- start start 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9afa93b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af4100700 0x7f9af4197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9afa93b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af41010c0 0x7f9af41975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9afa93b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9af4106710 0x7f9af419b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9afa93b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9af4110b20 con 0x7f9af4100700 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9afa93b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9af41109a0 con 0x7f9af41010c0 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9afa93b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9af4110ca0 con 0x7f9af4106710 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af4100700 0x7f9af4197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af4100700 0x7f9af4197080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33200/0 (socket says 172.21.15.28:33200) 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af3fff640 1 -- 172.21.15.28:0/322856169 learned_addr learned my addr 172.21.15.28:0/322856169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9aebfff640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af41010c0 0x7f9af41975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af8ed8640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9af4106710 0x7f9af419b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af3fff640 1 -- 172.21.15.28:0/322856169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9af4106710 msgr2=0x7f9af419b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af3fff640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9af4106710 0x7f9af419b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af3fff640 1 -- 172.21.15.28:0/322856169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af41010c0 msgr2=0x7f9af41975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af3fff640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af41010c0 0x7f9af41975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af3fff640 1 -- 172.21.15.28:0/322856169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af419c070 con 0x7f9af4100700 2024-05-09T08:27:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.672+0000 7f9af3fff640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af4100700 0x7f9af4197080 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c890 tx=0x7f9ae400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.673+0000 7f9af1ffb640 1 -- 172.21.15.28:0/322856169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4004220 con 0x7f9af4100700 2024-05-09T08:27:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.673+0000 7f9af1ffb640 1 -- 172.21.15.28:0/322856169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ae40043c0 con 0x7f9af4100700 2024-05-09T08:27:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.673+0000 7f9afa93b640 1 -- 172.21.15.28:0/322856169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af419c360 con 0x7f9af4100700 2024-05-09T08:27:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.673+0000 7f9af1ffb640 1 -- 172.21.15.28:0/322856169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae40049e0 con 0x7f9af4100700 2024-05-09T08:27:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.673+0000 7f9afa93b640 1 -- 172.21.15.28:0/322856169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af41a3bf0 con 0x7f9af4100700 2024-05-09T08:27:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.675+0000 7f9af1ffb640 1 -- 172.21.15.28:0/322856169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9ae40079b0 con 0x7f9af4100700 2024-05-09T08:27:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.675+0000 7f9afd64a740 1 Processor -- start 2024-05-09T08:27:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9afd64a740 1 -- start start 2024-05-09T08:27:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9af1ffb640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9ae00a2d00 0x7f9ae00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 0x564d49e7a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e84f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e87870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9afd64a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d49dddd00 con 0x564d49e85480 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9afd64a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x564d49d854c0 con 0x564d49e7adb0 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9afd64a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564d49dd9cc0 con 0x564d49e7a490 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9af1ffb640 1 -- 172.21.15.28:0/322856169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f9ae40cf840 con 0x7f9af4100700 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9aebfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e84f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9af8ed8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e87870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9aebfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e84f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:59750/0 (socket says 172.21.15.28:59750) 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9aebfff640 1 -- 172.21.15.28:0/1262252696 learned_addr learned my addr 172.21.15.28:0/1262252696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9af8ed8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e87870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33202/0 (socket says 172.21.15.28:33202) 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9af3fff640 1 --2- 172.21.15.28:0/1262252696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 0x564d49e7a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9af8ed8640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9ae00a2d00 0x7f9ae00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9aebfff640 1 -- 172.21.15.28:0/1262252696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 msgr2=0x564d49e7a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9aebfff640 1 --2- 172.21.15.28:0/1262252696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 0x564d49e7a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9aebfff640 1 -- 172.21.15.28:0/1262252696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 msgr2=0x564d49e87870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9aebfff640 1 --2- 172.21.15.28:0/1262252696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e87870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.676+0000 7f9aebfff640 1 -- 172.21.15.28:0/1262252696 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d49bbb7d0 con 0x564d49e7adb0 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.677+0000 7f9aebfff640 1 --2- 172.21.15.28:0/1262252696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e84f40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009c90 tx=0x7f9adc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=279c0d334ddd95ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.677+0000 7f9af8ed8640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9ae00a2d00 0x7f9ae00a51c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009430 tx=0x7f9aec02a8e0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.677+0000 7f9aea7fc640 1 -- 172.21.15.28:0/1262252696 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc01e070 con 0x564d49e7adb0 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.677+0000 7f9aea7fc640 1 -- 172.21.15.28:0/1262252696 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9adc019b10 con 0x564d49e7adb0 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.677+0000 7f9aea7fc640 1 -- 172.21.15.28:0/1262252696 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00ea70 con 0x564d49e7adb0 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 -- 172.21.15.28:0/1262252696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 msgr2=0x564d49e84f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1262252696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e84f40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009c90 tx=0x7f9adc011810 comp rx=0 tx=0).stop 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 -- 172.21.15.28:0/1262252696 shutdown_connections 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1262252696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e87870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1262252696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e84f40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1262252696 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 0x564d49e7a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 -- 172.21.15.28:0/1262252696 >> 172.21.15.28:0/1262252696 conn(0x564d49e767b0 msgr2=0x564d49e76b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 -- 172.21.15.28:0/1262252696 shutdown_connections 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 -- 172.21.15.28:0/1262252696 wait complete. 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 -- start start 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.678+0000 7f9afd64a740 1 Processor -- start 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 0x564d49e82410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e82950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e82e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9afd64a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d49e88270 con 0x564d49e85480 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9afd64a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x564d49d98b90 con 0x564d49e7adb0 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9afd64a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564d49e883f0 con 0x564d49e7a490 2024-05-09T08:27:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9afd64a740 10 client.0 _ll_register_callbacks cb 0x564d49dd4da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9af8ed8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e82e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9af3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 0x564d49e82410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9aebfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e82950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9af8ed8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e82e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33210/0 (socket says 172.21.15.28:33210) 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9af8ed8640 1 -- 172.21.15.28:0/1066389071 learned_addr learned my addr 172.21.15.28:0/1066389071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9aebfff640 1 -- 172.21.15.28:0/1066389071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 msgr2=0x564d49e82410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9aebfff640 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 0x564d49e82410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9aebfff640 1 -- 172.21.15.28:0/1066389071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 msgr2=0x564d49e82e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9aebfff640 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e82e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.679+0000 7f9aebfff640 1 -- 172.21.15.28:0/1066389071 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d49af13f0 con 0x564d49e7adb0 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.680+0000 7f9aebfff640 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e82950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9adc000c00 tx=0x7f9adc009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a07a9e2b22646b3b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.680+0000 7f9ad37fe640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:27:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.680+0000 7f9ad37fe640 1 -- 172.21.15.28:0/1066389071 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc004230 con 0x564d49e7adb0 2024-05-09T08:27:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.680+0000 7f9afd64a740 1 -- 172.21.15.28:0/1066389071 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9adc019b10 con 0x564d49e7adb0 2024-05-09T08:27:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.680+0000 7f9ad37fe640 1 -- 172.21.15.28:0/1066389071 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f9adc0043d0 con 0x564d49e7adb0 2024-05-09T08:27:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.680+0000 7f9ad37fe640 1 -- 172.21.15.28:0/1066389071 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9adc019430 con 0x564d49e7adb0 2024-05-09T08:27:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.680+0000 7f9afd64a740 10 client.25697 fetch_fsmap learned FSMap version 29 2024-05-09T08:27:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.680+0000 7f9afd64a740 1 -- 172.21.15.28:0/1066389071 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564d49ab10f0 con 0x564d49e7adb0 2024-05-09T08:27:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.680+0000 7f9ad37fe640 1 -- 172.21.15.28:0/1066389071 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9adc019b10 con 0x564d49e7adb0 2024-05-09T08:27:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9ad37fe640 1 -- 172.21.15.28:0/1066389071 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f9adc00e770 con 0x564d49e7adb0 2024-05-09T08:27:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9afd64a740 10 client.25697 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:27:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9afd64a740 10 client.25697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:27:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9afd64a740 20 client.25697 populate_metadata read hostname 'smithi028' 2024-05-09T08:27:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564d49e750f0 0x564d49e75530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9afd64a740 4 client.25697 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:27:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9afd64a740 1 -- 172.21.15.28:0/1066389071 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d49deb430 con 0x564d49e750f0 2024-05-09T08:27:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9af3fff640 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564d49e750f0 0x564d49e75530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9af3fff640 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564d49e750f0 0x564d49e75530 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab25a89abe2c88e7 server_cookie=f54fc3ac3a33a321 in_seq=0 out_seq=0 2024-05-09T08:27:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:27:59.681+0000 7f9ad37fe640 10 client.25697 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.426+0000 7fa8fffff640 1 -- 172.21.15.28:0/3506221002 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560a719d3dc0 con 0x560a71a82370 2024-05-09T08:28:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.426+0000 7fa8fffff640 10 client.16212 handle_command_reply: tid=0 2024-05-09T08:28:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.426+0000 7fa9299bb740 1 client.16212 shutdown 2024-05-09T08:28:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.426+0000 7fa9299bb740 1 -- 172.21.15.28:0/3506221002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 msgr2=0x560a71a770b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a770b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa920069a00 tx=0x7fa90c0317e0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 1 -- 172.21.15.28:0/3506221002 shutdown_connections 2024-05-09T08:28:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560a71a82370 0x560a71aebd50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560a71a72630 0x560a71ae0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a71a67d70 0x560a71a770b0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa8fffff640 0 client.16212 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 1 --2- 172.21.15.28:0/3506221002 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560a71a67450 0x560a71a76b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 1 -- 172.21.15.28:0/3506221002 >> 172.21.15.28:0/3506221002 conn(0x560a71a63960 msgr2=0x560a71a62a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.427+0000 7f9ad37fe640 1 -- 172.21.15.28:0/1066389071 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d49deb430 con 0x564d49e750f0 2024-05-09T08:28:01.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.427+0000 7f9ad37fe640 10 client.25697 handle_command_reply: tid=0 2024-05-09T08:28:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 1 -- 172.21.15.28:0/3506221002 shutdown_connections 2024-05-09T08:28:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.427+0000 7f9afd64a740 1 client.25697 shutdown 2024-05-09T08:28:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 1 -- 172.21.15.28:0/3506221002 wait complete. 2024-05-09T08:28:01.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.427+0000 7f9afd64a740 1 -- 172.21.15.28:0/1066389071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 msgr2=0x564d49e82950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.427+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e82950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9adc000c00 tx=0x7f9adc009850 comp rx=0 tx=0).stop 2024-05-09T08:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.427+0000 7fa9299bb740 20 client.16212 trim_cache size 0 max 16384 2024-05-09T08:28:01.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.427+0000 7f9afd64a740 1 -- 172.21.15.28:0/1066389071 shutdown_connections 2024-05-09T08:28:01.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.427+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564d49e750f0 0x564d49e75530 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.427+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e85480 0x564d49e82e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.428+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e82950 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.428+0000 7f9afd64a740 1 --2- 172.21.15.28:0/1066389071 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7a490 0x564d49e82410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.428+0000 7f9afd64a740 1 -- 172.21.15.28:0/1066389071 >> 172.21.15.28:0/1066389071 conn(0x564d49e767b0 msgr2=0x564d49e737e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:01.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.428+0000 7f9ad37fe640 0 client.25697 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:28:01.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.428+0000 7f9afd64a740 1 -- 172.21.15.28:0/1066389071 shutdown_connections 2024-05-09T08:28:01.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.428+0000 7f9afd64a740 1 -- 172.21.15.28:0/1066389071 wait complete. 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 -- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9100a2da0 msgr2=0x7fa9100a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9100a2da0 0x7fa9100a5260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa918009fd0 tx=0x7fa918004a40 comp rx=0 tx=0).stop 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 -- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 msgr2=0x7fa920198ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 0x7fa920198ff0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa91400e9f0 tx=0x7fa91400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 -- 172.21.15.28:0/1985285577 shutdown_connections 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa920106720 0x7fa9201a4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9100a2da0 0x7fa9100a5260 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa920101120 0x7fa920199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 --2- 172.21.15.28:0/1985285577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920100760 0x7fa920198ff0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 -- 172.21.15.28:0/1985285577 >> 172.21.15.28:0/1985285577 conn(0x7fa9200778b0 msgr2=0x7fa920103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 -- 172.21.15.28:0/1985285577 shutdown_connections 2024-05-09T08:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:01.429+0000 7fa926bce640 1 -- 172.21.15.28:0/1985285577 wait complete. 2024-05-09T08:28:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.428+0000 7f9afd64a740 20 client.25697 trim_cache size 0 max 16384 2024-05-09T08:28:01.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:01 smithi028 ceph-mon[30119]: pgmap v54: 129 pgs: 129 active+clean; 51 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 315 op/s 2024-05-09T08:28:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.450+0000 7f9afd64a740 1 Processor -- start 2024-05-09T08:28:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.450+0000 7f9afd64a740 1 -- start start 2024-05-09T08:28:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.450+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e6f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.450+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7adb0 0x564d49e7cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.450+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e85480 0x564d49e99400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.450+0000 7f9afd64a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9af41acc70 con 0x564d49e7a490 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.450+0000 7f9afd64a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9af41ae7e0 con 0x564d49e85480 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.450+0000 7f9afd64a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9af41ad340 con 0x564d49e7adb0 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e6f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af8ed8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e85480 0x564d49e99400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e6f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33214/0 (socket says 172.21.15.28:33214) 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af3fff640 1 -- 172.21.15.28:0/2291379067 learned_addr learned my addr 172.21.15.28:0/2291379067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9aebfff640 1 --2- 172.21.15.28:0/2291379067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7adb0 0x564d49e7cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af3fff640 1 -- 172.21.15.28:0/2291379067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7adb0 msgr2=0x564d49e7cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af3fff640 1 --2- 172.21.15.28:0/2291379067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7adb0 0x564d49e7cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af3fff640 1 -- 172.21.15.28:0/2291379067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e85480 msgr2=0x564d49e99400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af3fff640 1 --2- 172.21.15.28:0/2291379067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e85480 0x564d49e99400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af3fff640 1 -- 172.21.15.28:0/2291379067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d49e813e0 con 0x564d49e7a490 2024-05-09T08:28:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.451+0000 7f9af3fff640 1 --2- 172.21.15.28:0/2291379067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e6f670 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4000c00 tx=0x7f9ae4005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76331bfed21b5633 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9ae9ffb640 1 -- 172.21.15.28:0/2291379067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae401d040 con 0x564d49e7a490 2024-05-09T08:28:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9ae9ffb640 1 -- 172.21.15.28:0/2291379067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ae409fd40 con 0x564d49e7a490 2024-05-09T08:28:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9ae9ffb640 1 -- 172.21.15.28:0/2291379067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4075040 con 0x564d49e7a490 2024-05-09T08:28:01.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9afd64a740 1 -- 172.21.15.28:0/2291379067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 msgr2=0x564d49e6f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9afd64a740 1 --2- 172.21.15.28:0/2291379067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e6f670 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4000c00 tx=0x7f9ae4005d20 comp rx=0 tx=0).stop 2024-05-09T08:28:01.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9afd64a740 1 -- 172.21.15.28:0/2291379067 shutdown_connections 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9afd64a740 1 --2- 172.21.15.28:0/2291379067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e85480 0x564d49e99400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9afd64a740 1 --2- 172.21.15.28:0/2291379067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e7adb0 0x564d49e7cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9afd64a740 1 --2- 172.21.15.28:0/2291379067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e6f670 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.452+0000 7f9afd64a740 1 -- 172.21.15.28:0/2291379067 >> 172.21.15.28:0/2291379067 conn(0x564d49e767b0 msgr2=0x564d49e737e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.453+0000 7f9afd64a740 1 -- 172.21.15.28:0/2291379067 shutdown_connections 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.453+0000 7f9afd64a740 1 -- 172.21.15.28:0/2291379067 wait complete. 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.453+0000 7f9afd64a740 1 -- start start 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9afd64a740 1 Processor -- start 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e990e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e99620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9afd64a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e85480 0x564d49e97580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9afd64a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9af41ae0e0 con 0x564d49e7a490 2024-05-09T08:28:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9afd64a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9af41ada10 con 0x564d49e7adb0 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9afd64a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9af41aac50 con 0x564d49e85480 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9afd64a740 10 client.0 _ll_register_callbacks cb 0x7f9adc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9af3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9aebfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e99620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9af8ed8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e85480 0x564d49e97580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9af3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33216/0 (socket says 172.21.15.28:33216) 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9af3fff640 1 -- 172.21.15.28:0/3988757172 learned_addr learned my addr 172.21.15.28:0/3988757172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.454+0000 7f9aebfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e99620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:59780/0 (socket says 172.21.15.28:59780) 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9af3fff640 1 -- 172.21.15.28:0/3988757172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e85480 msgr2=0x564d49e97580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9af3fff640 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e85480 0x564d49e97580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9af3fff640 1 -- 172.21.15.28:0/3988757172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 msgr2=0x564d49e99620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9af3fff640 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e99620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9af3fff640 1 -- 172.21.15.28:0/3988757172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d49e97c90 con 0x564d49e7a490 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9af8ed8640 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e85480 0x564d49e97580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9aebfff640 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e99620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9af3fff640 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e990e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40ccc70 tx=0x7f9ae409b370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b1aab737b85016 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9ad37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9ad37fe640 1 -- 172.21.15.28:0/3988757172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae401d030 con 0x564d49e7a490 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9afd64a740 1 -- 172.21.15.28:0/3988757172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ae409fd40 con 0x564d49e7a490 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9ad37fe640 1 -- 172.21.15.28:0/3988757172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f9ae4007ca0 con 0x564d49e7a490 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9ad37fe640 1 -- 172.21.15.28:0/3988757172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9ae40b9050 con 0x564d49e7a490 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9afd64a740 10 client.16230 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.455+0000 7f9afd64a740 1 -- 172.21.15.28:0/3988757172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564d49efa7f0 con 0x564d49e7a490 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9ad37fe640 1 -- 172.21.15.28:0/3988757172 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ae409fd40 con 0x564d49e7a490 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9ad37fe640 1 -- 172.21.15.28:0/3988757172 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f9ae407a300 con 0x564d49e7a490 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9afd64a740 10 client.16230 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9afd64a740 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9afd64a740 20 client.16230 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9afd64a740 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564d49efaf00 0x564d49efd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9afd64a740 4 client.16230 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9afd64a740 1 -- 172.21.15.28:0/3988757172 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564d49de5780 con 0x564d49efaf00 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9aebfff640 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564d49efaf00 0x564d49efd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.456+0000 7f9aebfff640 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564d49efaf00 0x564d49efd320 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=549b0b08cb7828fc server_cookie=2f13a3afa870d2d1 in_seq=0 out_seq=0 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.457+0000 7f9ad37fe640 10 client.16230 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9ad37fe640 1 -- 172.21.15.28:0/3988757172 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564d49de5780 con 0x564d49efaf00 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9ad37fe640 10 client.16230 handle_command_reply: tid=0 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 client.16230 shutdown 2024-05-09T08:28:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 -- 172.21.15.28:0/3988757172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 msgr2=0x564d49e990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e990e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40ccc70 tx=0x7f9ae409b370 comp rx=0 tx=0).stop 2024-05-09T08:28:01.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 -- 172.21.15.28:0/3988757172 shutdown_connections 2024-05-09T08:28:01.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564d49efaf00 0x564d49efd320 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564d49e85480 0x564d49e97580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d49e7adb0 0x564d49e99620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9ad37fe640 0 client.16230 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:01.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 --2- 172.21.15.28:0/3988757172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d49e7a490 0x564d49e990e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 -- 172.21.15.28:0/3988757172 >> 172.21.15.28:0/3988757172 conn(0x564d49e767b0 msgr2=0x564d49e72f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.458+0000 7f9afd64a740 1 -- 172.21.15.28:0/3988757172 shutdown_connections 2024-05-09T08:28:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.459+0000 7f9afd64a740 1 -- 172.21.15.28:0/3988757172 wait complete. 2024-05-09T08:28:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.459+0000 7f9afd64a740 20 client.16230 trim_cache size 0 max 16384 2024-05-09T08:28:01.465 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:28:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-09T08:28:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-09T08:28:01.465 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:28:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.459+0000 7f9afa93b640 1 -- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9ae00a2d00 msgr2=0x7f9ae00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.459+0000 7f9afa93b640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9ae00a2d00 0x7f9ae00a51c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009430 tx=0x7f9aec02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.459+0000 7f9afa93b640 1 -- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af4100700 msgr2=0x7f9af4197080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.459+0000 7f9afa93b640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af4100700 0x7f9af4197080 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c890 tx=0x7f9ae400cd50 comp rx=0 tx=0).stop 2024-05-09T08:28:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.460+0000 7f9afa93b640 1 -- 172.21.15.28:0/322856169 shutdown_connections 2024-05-09T08:28:01.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.460+0000 7f9afa93b640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9ae00a2d00 0x7f9ae00a51c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.460+0000 7f9afa93b640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9af4106710 0x7f9af419b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.460+0000 7f9afa93b640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af41010c0 0x7f9af41975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.460+0000 7f9afa93b640 1 --2- 172.21.15.28:0/322856169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9af4100700 0x7f9af4197080 secure :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c890 tx=0x7f9ae400cd50 comp rx=0 tx=0).stop 2024-05-09T08:28:01.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.460+0000 7f9afa93b640 1 -- 172.21.15.28:0/322856169 >> 172.21.15.28:0/322856169 conn(0x7f9af4077830 msgr2=0x7f9af4102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:01.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.460+0000 7f9afa93b640 1 -- 172.21.15.28:0/322856169 shutdown_connections 2024-05-09T08:28:01.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.460+0000 7f9afa93b640 1 -- 172.21.15.28:0/322856169 wait complete. 2024-05-09T08:28:01.474 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:28:01.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:28:01.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:01 smithi165 ceph-mon[34540]: pgmap v54: 129 pgs: 129 active+clean; 51 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 315 op/s 2024-05-09T08:28:01.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:01 smithi184 ceph-mon[34509]: pgmap v54: 129 pgs: 129 active+clean; 51 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 315 op/s 2024-05-09T08:28:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.663+0000 7f2465c95640 1 -- 172.21.15.28:0/2999588472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460106720 msgr2=0x7f2460106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.663+0000 7f2465c95640 1 --2- 172.21.15.28:0/2999588472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460106720 0x7f2460106b00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2450009930 tx=0x7f245002a420 comp rx=0 tx=0).stop 2024-05-09T08:28:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.663+0000 7f2465c95640 1 -- 172.21.15.28:0/2999588472 shutdown_connections 2024-05-09T08:28:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.663+0000 7f2465c95640 1 --2- 172.21.15.28:0/2999588472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24601010d0 0x7f246010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.663+0000 7f2465c95640 1 --2- 172.21.15.28:0/2999588472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2460100710 0x7f2460100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.663+0000 7f2465c95640 1 --2- 172.21.15.28:0/2999588472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460106720 0x7f2460106b00 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.663+0000 7f2465c95640 1 -- 172.21.15.28:0/2999588472 >> 172.21.15.28:0/2999588472 conn(0x7f2460077830 msgr2=0x7f2460077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.663+0000 7f2465c95640 1 -- 172.21.15.28:0/2999588472 shutdown_connections 2024-05-09T08:28:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.663+0000 7f2465c95640 1 -- 172.21.15.28:0/2999588472 wait complete. 2024-05-09T08:28:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f2465c95640 1 Processor -- start 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f2465c95640 1 -- start start 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f2465c95640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2460100710 0x7f2460198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f2465c95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24601010d0 0x7f2460198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f2465c95640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2460106720 0x7f246019c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f2465c95640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2460110b30 con 0x7f24601010d0 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f2465c95640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f24601109b0 con 0x7f2460106720 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f2465c95640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2460110cb0 con 0x7f2460100710 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f245effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24601010d0 0x7f2460198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2460100710 0x7f2460198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2460100710 0x7f2460198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42844/0 (socket says 172.21.15.28:42844) 2024-05-09T08:28:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.664+0000 7f245f7fe640 1 -- 172.21.15.28:0/557471504 learned_addr learned my addr 172.21.15.28:0/557471504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245ffff640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2460106720 0x7f246019c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245f7fe640 1 -- 172.21.15.28:0/557471504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2460106720 msgr2=0x7f246019c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245f7fe640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2460106720 0x7f246019c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245f7fe640 1 -- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24601010d0 msgr2=0x7f2460198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245f7fe640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24601010d0 0x7f2460198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245f7fe640 1 -- 172.21.15.28:0/557471504 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246019d110 con 0x7f2460100710 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245effd640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24601010d0 0x7f2460198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245ffff640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2460106720 0x7f246019c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245f7fe640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2460100710 0x7f2460198120 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2450009230 tx=0x7f2450004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245cff9640 1 -- 172.21.15.28:0/557471504 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245003f030 con 0x7f2460100710 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246019d3a0 con 0x7f2460100710 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245cff9640 1 -- 172.21.15.28:0/557471504 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2450004440 con 0x7f2460100710 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f245cff9640 1 -- 172.21.15.28:0/557471504 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2450035070 con 0x7f2460100710 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.665+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24601a4c90 con 0x7f2460100710 2024-05-09T08:28:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.667+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2460068650 con 0x7f2460100710 2024-05-09T08:28:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.668+0000 7f245cff9640 1 -- 172.21.15.28:0/557471504 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2450042030 con 0x7f2460100710 2024-05-09T08:28:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.668+0000 7f245cff9640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f24480a2d00 0x7f24480a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.669+0000 7f245effd640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f24480a2d00 0x7f24480a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.669+0000 7f245cff9640 1 -- 172.21.15.28:0/557471504 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f24500efb40 con 0x7f2460100710 2024-05-09T08:28:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.669+0000 7f245effd640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f24480a2d00 0x7f24480a51c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2460100310 tx=0x7f2454005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:01.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.673+0000 7f245cff9640 1 -- 172.21.15.28:0/557471504 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f24500bbce0 con 0x7f2460100710 2024-05-09T08:28:01.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.955+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2460109c90 con 0x7f2460100710 2024-05-09T08:28:01.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.956+0000 7f245cff9640 1 -- 172.21.15.28:0/557471504 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11939 (secure 0 0 0) 0x7f24500bffe0 con 0x7f2460100710 2024-05-09T08:28:01.958 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:28:01.959 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:28:00.415243+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24512,24463,24476,24499,24481]},"id":1}]} 2024-05-09T08:28:01.959 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.958+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f24480a2d00 msgr2=0x7f24480a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.958+0000 7f2465c95640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f24480a2d00 0x7f24480a51c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2460100310 tx=0x7f2454005f90 comp rx=0 tx=0).stop 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.958+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2460100710 msgr2=0x7f2460198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.958+0000 7f2465c95640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2460100710 0x7f2460198120 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2450009230 tx=0x7f2450004040 comp rx=0 tx=0).stop 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.959+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 shutdown_connections 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.959+0000 7f2465c95640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2460106720 0x7f246019c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.959+0000 7f2465c95640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24601010d0 0x7f2460198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.959+0000 7f2465c95640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f24480a2d00 0x7f24480a51c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.959+0000 7f2465c95640 1 --2- 172.21.15.28:0/557471504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2460100710 0x7f2460198120 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.959+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 >> 172.21.15.28:0/557471504 conn(0x7f2460077830 msgr2=0x7f2460068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.959+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 shutdown_connections 2024-05-09T08:28:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:01.959+0000 7f2465c95640 1 -- 172.21.15.28:0/557471504 wait complete. 2024-05-09T08:28:01.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:28:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 -- 172.21.15.28:0/728824939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14701010c0 msgr2=0x7f147010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 --2- 172.21.15.28:0/728824939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14701010c0 0x7f147010d5d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1468009cd0 tx=0x7f146802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 -- 172.21.15.28:0/728824939 shutdown_connections 2024-05-09T08:28:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 --2- 172.21.15.28:0/728824939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14701010c0 0x7f147010d5d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 --2- 172.21.15.28:0/728824939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1470100700 0x7f1470100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 --2- 172.21.15.28:0/728824939 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1470106710 0x7f1470106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 -- 172.21.15.28:0/728824939 >> 172.21.15.28:0/728824939 conn(0x7f1470077830 msgr2=0x7f1470077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 -- 172.21.15.28:0/728824939 shutdown_connections 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 -- 172.21.15.28:0/728824939 wait complete. 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 Processor -- start 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.086+0000 7f1476bb6640 1 -- start start 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1476bb6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1470100700 0x7f1470196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1476bb6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14701010c0 0x7f1470197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1476bb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1470106710 0x7f1470197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1476bb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1470110b20 con 0x7f1470106710 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1476bb6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f14701109a0 con 0x7f14701010c0 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1476bb6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1470110ca0 con 0x7f1470100700 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1475153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1470106710 0x7f1470197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1475153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1470106710 0x7f1470197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33256/0 (socket says 172.21.15.28:33256) 2024-05-09T08:28:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1475153640 1 -- 172.21.15.28:0/646000054 learned_addr learned my addr 172.21.15.28:0/646000054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1474952640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1470100700 0x7f1470196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1467fff640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14701010c0 0x7f1470197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1475153640 1 -- 172.21.15.28:0/646000054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1470100700 msgr2=0x7f1470196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1475153640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1470100700 0x7f1470196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1475153640 1 -- 172.21.15.28:0/646000054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14701010c0 msgr2=0x7f1470197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1475153640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14701010c0 0x7f1470197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1475153640 1 -- 172.21.15.28:0/646000054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14701a6660 con 0x7f1470106710 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1475153640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1470106710 0x7f1470197870 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1468005cc0 tx=0x7f14680093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1465ffb640 1 -- 172.21.15.28:0/646000054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468003e00 con 0x7f1470106710 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1465ffb640 1 -- 172.21.15.28:0/646000054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1468004520 con 0x7f1470106710 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1476bb6640 1 -- 172.21.15.28:0/646000054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14701a68f0 con 0x7f1470106710 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.087+0000 7f1465ffb640 1 -- 172.21.15.28:0/646000054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468030db0 con 0x7f1470106710 2024-05-09T08:28:02.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.088+0000 7f1476bb6640 1 -- 172.21.15.28:0/646000054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14701a6e90 con 0x7f1470106710 2024-05-09T08:28:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.090+0000 7f1465ffb640 1 -- 172.21.15.28:0/646000054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f14680046d0 con 0x7f1470106710 2024-05-09T08:28:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.090+0000 7f147999b740 1 Processor -- start 2024-05-09T08:28:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f147999b740 1 -- start start 2024-05-09T08:28:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f147999b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5ab710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f147999b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f1465ffb640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14580a2d80 0x7f14580a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f147999b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f147999b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5ca510c40 con 0x55b5ca5b6320 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f147999b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55b5ca4b8580 con 0x55b5ca5ab330 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f147999b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b5ca50cc00 con 0x55b5ca5abc50 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f1467fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f1475153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f1467fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5ab710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:59826/0 (socket says 172.21.15.28:59826) 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f1467fff640 1 -- 172.21.15.28:0/3810212178 learned_addr learned my addr 172.21.15.28:0/3810212178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f1467fff640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14580a2d80 0x7f14580a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f1465ffb640 1 -- 172.21.15.28:0/646000054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f146802e030 con 0x7f1470106710 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.091+0000 7f1474952640 1 --2- 172.21.15.28:0/3810212178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1467fff640 1 -- 172.21.15.28:0/3810212178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5abc50 msgr2=0x55b5ca5b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1467fff640 1 --2- 172.21.15.28:0/3810212178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1467fff640 1 -- 172.21.15.28:0/3810212178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 msgr2=0x55b5ca5b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1467fff640 1 --2- 172.21.15.28:0/3810212178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1467fff640 1 -- 172.21.15.28:0/3810212178 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ca2243f0 con 0x55b5ca5ab330 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1475153640 1 --2- 172.21.15.28:0/3810212178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1467fff640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14580a2d80 0x7f14580a5240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1460011fd0 tx=0x7f1460018000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1467fff640 1 --2- 172.21.15.28:0/3810212178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5ab710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f14600058c0 tx=0x7f1460011ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=150e5918edbde37a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1456ffd640 1 -- 172.21.15.28:0/3810212178 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460011050 con 0x55b5ca5ab330 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1456ffd640 1 -- 172.21.15.28:0/3810212178 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1460009a20 con 0x55b5ca5ab330 2024-05-09T08:28:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.092+0000 7f1456ffd640 1 -- 172.21.15.28:0/3810212178 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14600035f0 con 0x55b5ca5ab330 2024-05-09T08:28:02.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 -- 172.21.15.28:0/3810212178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 msgr2=0x55b5ca5ab710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 --2- 172.21.15.28:0/3810212178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5ab710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f14600058c0 tx=0x7f1460011ea0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 -- 172.21.15.28:0/3810212178 shutdown_connections 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 --2- 172.21.15.28:0/3810212178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 --2- 172.21.15.28:0/3810212178 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 --2- 172.21.15.28:0/3810212178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5ab710 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 -- 172.21.15.28:0/3810212178 >> 172.21.15.28:0/3810212178 conn(0x55b5ca5a7650 msgr2=0x55b5ca5a7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 -- 172.21.15.28:0/3810212178 shutdown_connections 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 -- 172.21.15.28:0/3810212178 wait complete. 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 -- start start 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.093+0000 7f147999b740 1 Processor -- start 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f147999b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f147999b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f147999b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f147999b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5ca5b90b0 con 0x55b5ca5b6320 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f147999b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55b5ca4cbc40 con 0x55b5ca5abc50 2024-05-09T08:28:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f147999b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b5ca5b9230 con 0x55b5ca5ab330 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f147999b740 10 client.0 _ll_register_callbacks cb 0x55b5ca223be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1475153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1475153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33272/0 (socket says 172.21.15.28:33272) 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1475153640 1 -- 172.21.15.28:0/166367180 learned_addr learned my addr 172.21.15.28:0/166367180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1474952640 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1467fff640 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1475153640 1 -- 172.21.15.28:0/166367180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 msgr2=0x55b5ca5b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1475153640 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1475153640 1 -- 172.21.15.28:0/166367180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5abc50 msgr2=0x55b5ca5b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1475153640 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1475153640 1 -- 172.21.15.28:0/166367180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ca2ee7d0 con 0x55b5ca5b6320 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.094+0000 7f1474952640 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f1475153640 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b3d50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f14680b9920 tx=0x7f14680bc8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f86ca7436a7eba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f144ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f144ffff640 1 -- 172.21.15.28:0/166367180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468031e60 con 0x55b5ca5b6320 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f144ffff640 1 -- 172.21.15.28:0/166367180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f14680de380 con 0x55b5ca5b6320 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f147999b740 1 -- 172.21.15.28:0/166367180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1460009a20 con 0x55b5ca5b6320 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f144ffff640 1 -- 172.21.15.28:0/166367180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f14680bd650 con 0x55b5ca5b6320 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f147999b740 10 client.16248 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f147999b740 1 -- 172.21.15.28:0/166367180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b5ca5c5300 con 0x55b5ca5b6320 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f144ffff640 1 -- 172.21.15.28:0/166367180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1460009a20 con 0x55b5ca5b6320 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f144ffff640 1 -- 172.21.15.28:0/166367180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f146804e070 con 0x55b5ca5b6320 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f147999b740 10 client.16248 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f147999b740 10 client.16248 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f147999b740 20 client.16248 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.095+0000 7f147999b740 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55b5ca5c67b0 0x55b5ca5c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.096+0000 7f147999b740 4 client.16248 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.096+0000 7f147999b740 1 -- 172.21.15.28:0/166367180 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5ca51e370 con 0x55b5ca5c67b0 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.096+0000 7f1474952640 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55b5ca5c67b0 0x55b5ca5c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.096+0000 7f144ffff640 10 client.16248 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.096+0000 7f1474952640 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55b5ca5c67b0 0x55b5ca5c8c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fe68a1f9152555b8 server_cookie=5bef1d5def26fc2 in_seq=0 out_seq=0 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f144ffff640 1 -- 172.21.15.28:0/166367180 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5ca51e370 con 0x55b5ca5c67b0 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f144ffff640 10 client.16248 handle_command_reply: tid=0 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 client.16248 shutdown 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 -- 172.21.15.28:0/166367180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 msgr2=0x55b5ca5b3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b3d50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f14680b9920 tx=0x7f14680bc8b0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 -- 172.21.15.28:0/166367180 shutdown_connections 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b3d50 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55b5ca5c67b0 0x55b5ca5c8c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 --2- 172.21.15.28:0/166367180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f144ffff640 0 client.16248 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:28:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 -- 172.21.15.28:0/166367180 >> 172.21.15.28:0/166367180 conn(0x55b5ca5a7650 msgr2=0x55b5ca5ad450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 -- 172.21.15.28:0/166367180 shutdown_connections 2024-05-09T08:28:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 1 -- 172.21.15.28:0/166367180 wait complete. 2024-05-09T08:28:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.097+0000 7f147999b740 20 client.16248 trim_cache size 0 max 16384 2024-05-09T08:28:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f147999b740 1 Processor -- start 2024-05-09T08:28:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f147999b740 1 -- start start 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f147999b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f147999b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5adc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f147999b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5b6320 0x55b5ca5ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f147999b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14701b0e00 con 0x55b5ca5abc50 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f147999b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f14701b2a00 con 0x55b5ca5ab330 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f147999b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14701b14d0 con 0x55b5ca5b6320 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f1467fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.113+0000 7f1467fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5adc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33276/0 (socket says 172.21.15.28:33276) 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f1467fff640 1 -- 172.21.15.28:0/3916387194 learned_addr learned my addr 172.21.15.28:0/3916387194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f1474952640 1 --2- 172.21.15.28:0/3916387194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f1475153640 1 --2- 172.21.15.28:0/3916387194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5b6320 0x55b5ca5ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f1467fff640 1 -- 172.21.15.28:0/3916387194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5b6320 msgr2=0x55b5ca5ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f1467fff640 1 --2- 172.21.15.28:0/3916387194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5b6320 0x55b5ca5ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f1467fff640 1 -- 172.21.15.28:0/3916387194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 msgr2=0x55b5ca5c1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f1467fff640 1 --2- 172.21.15.28:0/3916387194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5c1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f1467fff640 1 -- 172.21.15.28:0/3916387194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ca5aba80 con 0x55b5ca5abc50 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f1467fff640 1 --2- 172.21.15.28:0/3916387194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5adc60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1460006010 tx=0x7f1460005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573040ce651dc739 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f14567fc640 1 -- 172.21.15.28:0/3916387194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14600099a0 con 0x55b5ca5abc50 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f14567fc640 1 -- 172.21.15.28:0/3916387194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1460011050 con 0x55b5ca5abc50 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.114+0000 7f14567fc640 1 -- 172.21.15.28:0/3916387194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14600054f0 con 0x55b5ca5abc50 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 -- 172.21.15.28:0/3916387194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 msgr2=0x55b5ca5adc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 --2- 172.21.15.28:0/3916387194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5adc60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1460006010 tx=0x7f1460005d00 comp rx=0 tx=0).stop 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 -- 172.21.15.28:0/3916387194 shutdown_connections 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 --2- 172.21.15.28:0/3916387194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5b6320 0x55b5ca5ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 --2- 172.21.15.28:0/3916387194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5adc60 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 --2- 172.21.15.28:0/3916387194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5ab330 0x55b5ca5c1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 -- 172.21.15.28:0/3916387194 >> 172.21.15.28:0/3916387194 conn(0x55b5ca5a7650 msgr2=0x55b5ca5a7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 -- 172.21.15.28:0/3916387194 shutdown_connections 2024-05-09T08:28:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 -- 172.21.15.28:0/3916387194 wait complete. 2024-05-09T08:28:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 -- start start 2024-05-09T08:28:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.115+0000 7f147999b740 1 Processor -- start 2024-05-09T08:28:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f147999b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5affd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f147999b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f147999b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f147999b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14701b2270 con 0x55b5ca5abc50 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f147999b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f14701b1ba0 con 0x55b5ca5b6320 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f147999b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14701aede0 con 0x55b5ca5ab330 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f147999b740 10 client.0 _ll_register_callbacks cb 0x7f1468009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1467fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1474952640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1467fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33280/0 (socket says 172.21.15.28:33280) 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1467fff640 1 -- 172.21.15.28:0/3259943593 learned_addr learned my addr 172.21.15.28:0/3259943593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1475153640 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1467fff640 1 -- 172.21.15.28:0/3259943593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 msgr2=0x55b5ca5affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1467fff640 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5affd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1467fff640 1 -- 172.21.15.28:0/3259943593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5b6320 msgr2=0x55b5ca5b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1467fff640 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1467fff640 1 -- 172.21.15.28:0/3259943593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ca62bd80 con 0x55b5ca5abc50 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1474952640 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1475153640 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f1467fff640 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b0510 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f146000a010 tx=0x7f14600035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157de47d22505ad4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.116+0000 7f144ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f144ffff640 1 -- 172.21.15.28:0/3259943593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146003c060 con 0x55b5ca5abc50 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f144ffff640 1 -- 172.21.15.28:0/3259943593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f1460005270 con 0x55b5ca5abc50 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f147999b740 1 -- 172.21.15.28:0/3259943593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1460011050 con 0x55b5ca5abc50 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f144ffff640 1 -- 172.21.15.28:0/3259943593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1460043050 con 0x55b5ca5abc50 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f147999b740 10 client.16254 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f147999b740 1 -- 172.21.15.28:0/3259943593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b5ca62d080 con 0x55b5ca5abc50 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f144ffff640 1 -- 172.21.15.28:0/3259943593 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1460011050 con 0x55b5ca5abc50 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f144ffff640 1 -- 172.21.15.28:0/3259943593 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f1460031020 con 0x55b5ca5abc50 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f147999b740 10 client.16254 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f147999b740 10 client.16254 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f147999b740 20 client.16254 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f147999b740 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55b5ca62d580 0x55b5ca62f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f147999b740 4 client.16254 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f147999b740 1 -- 172.21.15.28:0/3259943593 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5ca5186c0 con 0x55b5ca62d580 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.117+0000 7f1474952640 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55b5ca62d580 0x55b5ca62f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f1474952640 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55b5ca62d580 0x55b5ca62f9a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=66fedc25b054853b server_cookie=8edc74769b54abd6 in_seq=0 out_seq=0 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f144ffff640 10 client.16254 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f144ffff640 1 -- 172.21.15.28:0/3259943593 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5ca5186c0 con 0x55b5ca62d580 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f144ffff640 10 client.16254 handle_command_reply: tid=0 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 client.16254 shutdown 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 -- 172.21.15.28:0/3259943593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 msgr2=0x55b5ca5b0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b0510 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f146000a010 tx=0x7f14600035f0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 -- 172.21.15.28:0/3259943593 shutdown_connections 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b5ca5b6320 0x55b5ca5b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ca5abc50 0x55b5ca5b0510 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55b5ca62d580 0x55b5ca62f9a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 --2- 172.21.15.28:0/3259943593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b5ca5ab330 0x55b5ca5affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 -- 172.21.15.28:0/3259943593 >> 172.21.15.28:0/3259943593 conn(0x55b5ca5a7650 msgr2=0x55b5ca5a5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f144ffff640 0 client.16254 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:28:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 -- 172.21.15.28:0/3259943593 shutdown_connections 2024-05-09T08:28:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 1 -- 172.21.15.28:0/3259943593 wait complete. 2024-05-09T08:28:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.118+0000 7f147999b740 20 client.16254 trim_cache size 0 max 16384 2024-05-09T08:28:02.125 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:28:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 -- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14580a2d80 msgr2=0x7f14580a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14580a2d80 0x7f14580a5240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1460011fd0 tx=0x7f1460018000 comp rx=0 tx=0).stop 2024-05-09T08:28:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 -- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1470106710 msgr2=0x7f1470197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1470106710 0x7f1470197870 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1468005cc0 tx=0x7f14680093d0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 -- 172.21.15.28:0/646000054 shutdown_connections 2024-05-09T08:28:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1470106710 0x7f1470197870 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14701010c0 0x7f1470197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f14580a2d80 0x7f14580a5240 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 --2- 172.21.15.28:0/646000054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1470100700 0x7f1470196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 -- 172.21.15.28:0/646000054 >> 172.21.15.28:0/646000054 conn(0x7f1470077830 msgr2=0x7f1470102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 -- 172.21.15.28:0/646000054 shutdown_connections 2024-05-09T08:28:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.119+0000 7f1476bb6640 1 -- 172.21.15.28:0/646000054 wait complete. 2024-05-09T08:28:02.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:28:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.253+0000 7fda7f709640 1 -- 172.21.15.28:0/3859707946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda78106710 msgr2=0x7fda78106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.253+0000 7fda7f709640 1 --2- 172.21.15.28:0/3859707946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda78106710 0x7fda78106af0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fda74009930 tx=0x7fda7402a420 comp rx=0 tx=0).stop 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.253+0000 7fda7f709640 1 -- 172.21.15.28:0/3859707946 shutdown_connections 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.253+0000 7fda7f709640 1 --2- 172.21.15.28:0/3859707946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda781010c0 0x7fda7810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.253+0000 7fda7f709640 1 --2- 172.21.15.28:0/3859707946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78100700 0x7fda78100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.253+0000 7fda7f709640 1 --2- 172.21.15.28:0/3859707946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda78106710 0x7fda78106af0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.253+0000 7fda7f709640 1 -- 172.21.15.28:0/3859707946 >> 172.21.15.28:0/3859707946 conn(0x7fda78077830 msgr2=0x7fda78077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.253+0000 7fda7f709640 1 -- 172.21.15.28:0/3859707946 shutdown_connections 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.253+0000 7fda7f709640 1 -- 172.21.15.28:0/3859707946 wait complete. 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7f709640 1 Processor -- start 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7f709640 1 -- start start 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7f709640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda78100700 0x7fda78102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7f709640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda781010c0 0x7fda78103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7f709640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78106710 0x7fda781037c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7f709640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda78110b20 con 0x7fda781010c0 2024-05-09T08:28:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7f709640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fda781109a0 con 0x7fda78100700 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7f709640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fda78110ca0 con 0x7fda78106710 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7d4a5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda78100700 0x7fda78102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7cca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda781010c0 0x7fda78103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7d4a5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda78100700 0x7fda78102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:59852/0 (socket says 172.21.15.28:59852) 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7d4a5640 1 -- 172.21.15.28:0/56639365 learned_addr learned my addr 172.21.15.28:0/56639365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7dca6640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78106710 0x7fda781037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7d4a5640 1 -- 172.21.15.28:0/56639365 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78106710 msgr2=0x7fda781037c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78106710 0x7fda781037c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7d4a5640 1 -- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda781010c0 msgr2=0x7fda78103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda781010c0 0x7fda78103280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7d4a5640 1 -- 172.21.15.28:0/56639365 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda781a6630 con 0x7fda78100700 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.254+0000 7fda7cca4640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda781010c0 0x7fda78103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.255+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda78100700 0x7fda78102d40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda74009230 tx=0x7fda74004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.255+0000 7fda6e7fc640 1 -- 172.21.15.28:0/56639365 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7403f030 con 0x7fda78100700 2024-05-09T08:28:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.255+0000 7fda7f709640 1 -- 172.21.15.28:0/56639365 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda781a68c0 con 0x7fda78100700 2024-05-09T08:28:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.255+0000 7fda6e7fc640 1 -- 172.21.15.28:0/56639365 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda74004440 con 0x7fda78100700 2024-05-09T08:28:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.255+0000 7fda6e7fc640 1 -- 172.21.15.28:0/56639365 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74035070 con 0x7fda78100700 2024-05-09T08:28:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.255+0000 7fda7f709640 1 -- 172.21.15.28:0/56639365 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda781a6e00 con 0x7fda78100700 2024-05-09T08:28:02.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.258+0000 7fda6e7fc640 1 -- 172.21.15.28:0/56639365 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fda74042030 con 0x7fda78100700 2024-05-09T08:28:02.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.258+0000 7fda82411740 1 Processor -- start 2024-05-09T08:28:02.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.258+0000 7fda82411740 1 -- start start 2024-05-09T08:28:02.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda6e7fc640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2d00 0x7fda640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda82411740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 0x561fb179f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda82411740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda82411740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb17aa320 0x561fb17ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda82411740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561fb1704c40 con 0x561fb179fc50 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda82411740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x561fb16ac580 con 0x561fb17aa320 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda82411740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561fb1700c00 con 0x561fb179f330 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7dca6640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2d00 0x7fda640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7cca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7cca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33300/0 (socket says 172.21.15.28:33300) 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7cca4640 1 -- 172.21.15.28:0/531384125 learned_addr learned my addr 172.21.15.28:0/531384125 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 0x561fb179f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7dca6640 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb17aa320 0x561fb17ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda6e7fc640 1 -- 172.21.15.28:0/56639365 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fda740efb40 con 0x7fda78100700 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7cca4640 1 -- 172.21.15.28:0/531384125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 msgr2=0x561fb179f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7cca4640 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 0x561fb179f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7cca4640 1 -- 172.21.15.28:0/531384125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb17aa320 msgr2=0x561fb17ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7cca4640 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb17aa320 0x561fb17ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7cca4640 1 -- 172.21.15.28:0/531384125 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fb14183f0 con 0x561fb179fc50 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 0x561fb179f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7cca4640 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a9de0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fda680114f0 tx=0x7fda680119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76fc5afd0783866d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.259+0000 7fda7dca6640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2d00 0x7fda640a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fda70005fd0 tx=0x7fda70009b00 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda637fe640 1 -- 172.21.15.28:0/531384125 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda68011030 con 0x561fb179fc50 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda637fe640 1 -- 172.21.15.28:0/531384125 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda680111d0 con 0x561fb179fc50 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda637fe640 1 -- 172.21.15.28:0/531384125 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6800f5c0 con 0x561fb179fc50 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda82411740 1 -- 172.21.15.28:0/531384125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 msgr2=0x561fb17a9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda82411740 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a9de0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fda680114f0 tx=0x7fda680119b0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda82411740 1 -- 172.21.15.28:0/531384125 shutdown_connections 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda82411740 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb17aa320 0x561fb17ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda82411740 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a9de0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda82411740 1 --2- 172.21.15.28:0/531384125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 0x561fb179f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda82411740 1 -- 172.21.15.28:0/531384125 >> 172.21.15.28:0/531384125 conn(0x561fb179b650 msgr2=0x561fb179ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda82411740 1 -- 172.21.15.28:0/531384125 shutdown_connections 2024-05-09T08:28:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.260+0000 7fda82411740 1 -- 172.21.15.28:0/531384125 wait complete. 2024-05-09T08:28:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda82411740 1 -- start start 2024-05-09T08:28:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda82411740 1 Processor -- start 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda82411740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 0x561fb17a7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda82411740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179fc50 0x561fb17a7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda82411740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb17a7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda82411740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561fb17ad110 con 0x561fb17aa320 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda82411740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x561fb16bfc40 con 0x561fb179f330 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda82411740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561fb17ad290 con 0x561fb179fc50 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda82411740 10 client.0 _ll_register_callbacks cb 0x561fb1417be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda7dca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb17a7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda7dca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb17a7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33314/0 (socket says 172.21.15.28:33314) 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda7dca6640 1 -- 172.21.15.28:0/2629261930 learned_addr learned my addr 172.21.15.28:0/2629261930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.261+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 0x561fb17a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda7cca4640 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179fc50 0x561fb17a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda7dca6640 1 -- 172.21.15.28:0/2629261930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179fc50 msgr2=0x561fb17a7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda7dca6640 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179fc50 0x561fb17a7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda7dca6640 1 -- 172.21.15.28:0/2629261930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 msgr2=0x561fb17a7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda7dca6640 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 0x561fb17a7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda7dca6640 1 -- 172.21.15.28:0/2629261930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fb14e27d0 con 0x561fb17aa320 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda7dca6640 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb17a7ca0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fda70013430 tx=0x7fda70004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffd3b70349ed705 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda60ff9640 1 -- 172.21.15.28:0/2629261930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7000fc90 con 0x561fb17aa320 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda60ff9640 1 -- 172.21.15.28:0/2629261930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fda700045b0 con 0x561fb17aa320 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda82411740 1 -- 172.21.15.28:0/2629261930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda680111d0 con 0x561fb17aa320 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda60ff9640 1 -- 172.21.15.28:0/2629261930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fda70032050 con 0x561fb17aa320 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda82411740 10 client.16266 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda82411740 1 -- 172.21.15.28:0/2629261930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561fb17b92b0 con 0x561fb17aa320 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.262+0000 7fda60ff9640 1 -- 172.21.15.28:0/2629261930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda680111d0 con 0x561fb17aa320 2024-05-09T08:28:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda60ff9640 1 -- 172.21.15.28:0/2629261930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fda7001f070 con 0x561fb17aa320 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda82411740 10 client.16266 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda82411740 10 client.16266 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda82411740 20 client.16266 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda82411740 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561fb17ba770 0x561fb17bcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda82411740 4 client.16266 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda82411740 1 -- 172.21.15.28:0/2629261930 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fb1712370 con 0x561fb17ba770 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561fb17ba770 0x561fb17bcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561fb17ba770 0x561fb17bcba0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4b3a05046f581490 server_cookie=d327af82584c2e0 in_seq=0 out_seq=0 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.263+0000 7fda60ff9640 10 client.16266 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.264+0000 7fda60ff9640 1 -- 172.21.15.28:0/2629261930 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561fb1712370 con 0x561fb17ba770 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.264+0000 7fda60ff9640 10 client.16266 handle_command_reply: tid=0 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 client.16266 shutdown 2024-05-09T08:28:02.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 -- 172.21.15.28:0/2629261930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 msgr2=0x561fb17a7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb17a7ca0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fda70013430 tx=0x7fda70004ab0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 -- 172.21.15.28:0/2629261930 shutdown_connections 2024-05-09T08:28:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561fb17ba770 0x561fb17bcba0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb17a7ca0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda60ff9640 0 client.16266 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:28:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179fc50 0x561fb17a7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 --2- 172.21.15.28:0/2629261930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 0x561fb17a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 -- 172.21.15.28:0/2629261930 >> 172.21.15.28:0/2629261930 conn(0x561fb179b650 msgr2=0x561fb17a1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 -- 172.21.15.28:0/2629261930 shutdown_connections 2024-05-09T08:28:02.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 1 -- 172.21.15.28:0/2629261930 wait complete. 2024-05-09T08:28:02.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.265+0000 7fda82411740 20 client.16266 trim_cache size 0 max 16384 2024-05-09T08:28:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.279+0000 7fda82411740 1 Processor -- start 2024-05-09T08:28:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.279+0000 7fda82411740 1 -- start start 2024-05-09T08:28:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda82411740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 0x561fb17b6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda82411740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179fc50 0x561fb17a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda82411740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb179a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda82411740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b1020 con 0x561fb17aa320 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda82411740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b2c20 con 0x561fb179fc50 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda82411740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b16f0 con 0x561fb179f330 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7dca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb179a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7cca4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179fc50 0x561fb17a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7d4a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 0x561fb17b6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7dca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb179a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33326/0 (socket says 172.21.15.28:33326) 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7dca6640 1 -- 172.21.15.28:0/899162286 learned_addr learned my addr 172.21.15.28:0/899162286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7dca6640 1 -- 172.21.15.28:0/899162286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 msgr2=0x561fb17b6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7dca6640 1 --2- 172.21.15.28:0/899162286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 0x561fb17b6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7dca6640 1 -- 172.21.15.28:0/899162286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179fc50 msgr2=0x561fb17a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7dca6640 1 --2- 172.21.15.28:0/899162286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179fc50 0x561fb17a63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7dca6640 1 -- 172.21.15.28:0/899162286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fb179fa80 con 0x561fb17aa320 2024-05-09T08:28:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda7dca6640 1 --2- 172.21.15.28:0/899162286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb179a630 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fda70003fc0 tx=0x7fda70013360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739f42546afc28c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda62ffd640 1 -- 172.21.15.28:0/899162286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70004b20 con 0x561fb17aa320 2024-05-09T08:28:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda62ffd640 1 -- 172.21.15.28:0/899162286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda70024070 con 0x561fb17aa320 2024-05-09T08:28:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.280+0000 7fda62ffd640 1 -- 172.21.15.28:0/899162286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70020ab0 con 0x561fb17aa320 2024-05-09T08:28:02.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.281+0000 7fda82411740 1 -- 172.21.15.28:0/899162286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 msgr2=0x561fb179a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.281+0000 7fda82411740 1 --2- 172.21.15.28:0/899162286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb179a630 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fda70003fc0 tx=0x7fda70013360 comp rx=0 tx=0).stop 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.281+0000 7fda82411740 1 -- 172.21.15.28:0/899162286 shutdown_connections 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.281+0000 7fda82411740 1 --2- 172.21.15.28:0/899162286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb17aa320 0x561fb179a630 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.281+0000 7fda82411740 1 --2- 172.21.15.28:0/899162286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179fc50 0x561fb17a63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.281+0000 7fda82411740 1 --2- 172.21.15.28:0/899162286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb179f330 0x561fb17b6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.281+0000 7fda82411740 1 -- 172.21.15.28:0/899162286 >> 172.21.15.28:0/899162286 conn(0x561fb179b650 msgr2=0x561fb1799f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.281+0000 7fda82411740 1 -- 172.21.15.28:0/899162286 shutdown_connections 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.281+0000 7fda82411740 1 -- 172.21.15.28:0/899162286 wait complete. 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda82411740 1 -- start start 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda82411740 1 Processor -- start 2024-05-09T08:28:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda82411740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 0x561fb17b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda82411740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda82411740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb17aa320 0x561fb17a4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda82411740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b2490 con 0x561fb179fc50 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda82411740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fda781b1dc0 con 0x561fb179f330 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda82411740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fda781af000 con 0x561fb17aa320 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda82411740 10 client.0 _ll_register_callbacks cb 0x7fda70009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda7cca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda7cca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33330/0 (socket says 172.21.15.28:33330) 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda7cca4640 1 -- 172.21.15.28:0/3332543835 learned_addr learned my addr 172.21.15.28:0/3332543835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 0x561fb17b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.282+0000 7fda7dca6640 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb17aa320 0x561fb17a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda7d4a5640 1 -- 172.21.15.28:0/3332543835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb17aa320 msgr2=0x561fb17a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb17aa320 0x561fb17a4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda7d4a5640 1 -- 172.21.15.28:0/3332543835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 msgr2=0x561fb17a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda7d4a5640 1 -- 172.21.15.28:0/3332543835 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fb17a4a90 con 0x561fb179f330 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda7dca6640 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb17aa320 0x561fb17a4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda7cca4640 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda7d4a5640 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 0x561fb17b29f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fda740ecec0 tx=0x7fda7402a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e60e97c9f803f03 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.283+0000 7fda60ff9640 1 -- 172.21.15.28:0/3332543835 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74049040 con 0x561fb179f330 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda82411740 1 -- 172.21.15.28:0/3332543835 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda70024070 con 0x561fb179f330 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda60ff9640 1 -- 172.21.15.28:0/3332543835 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fda740ed140 con 0x561fb179f330 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda60ff9640 1 -- 172.21.15.28:0/3332543835 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fda740b83b0 con 0x561fb179f330 2024-05-09T08:28:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda82411740 10 client.25745 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda82411740 1 -- 172.21.15.28:0/3332543835 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561fb1820e30 con 0x561fb179f330 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda60ff9640 1 -- 172.21.15.28:0/3332543835 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda70024070 con 0x561fb179f330 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda60ff9640 1 -- 172.21.15.28:0/3332543835 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fda74031bd0 con 0x561fb179f330 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda82411740 10 client.25745 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda82411740 10 client.25745 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda82411740 20 client.25745 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda82411740 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561fb17ba770 0x561fb18235e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda82411740 4 client.25745 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda82411740 1 -- 172.21.15.28:0/3332543835 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fb170c6c0 con 0x561fb17ba770 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.284+0000 7fda7cca4640 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561fb17ba770 0x561fb18235e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda7cca4640 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561fb17ba770 0x561fb18235e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b3aa7c85ad031492 server_cookie=fc057ed52981bedd in_seq=0 out_seq=0 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda60ff9640 10 client.25745 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:28:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda60ff9640 1 -- 172.21.15.28:0/3332543835 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fb170c6c0 con 0x561fb17ba770 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda60ff9640 10 client.25745 handle_command_reply: tid=0 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 client.25745 shutdown 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 -- 172.21.15.28:0/3332543835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 msgr2=0x561fb17b29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 0x561fb17b29f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fda740ecec0 tx=0x7fda7402a420 comp rx=0 tx=0).stop 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 -- 172.21.15.28:0/3332543835 shutdown_connections 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x561fb17ba770 0x561fb18235e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561fb17aa320 0x561fb17a4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda60ff9640 0 client.25745 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fb179fc50 0x561fb17a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 --2- 172.21.15.28:0/3332543835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561fb179f330 0x561fb17b29f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 -- 172.21.15.28:0/3332543835 >> 172.21.15.28:0/3332543835 conn(0x561fb179b650 msgr2=0x561fb17a7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 -- 172.21.15.28:0/3332543835 shutdown_connections 2024-05-09T08:28:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 1 -- 172.21.15.28:0/3332543835 wait complete. 2024-05-09T08:28:02.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.285+0000 7fda82411740 20 client.25745 trim_cache size 0 max 16384 2024-05-09T08:28:02.293 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 -- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2d00 msgr2=0x7fda640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2d00 0x7fda640a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fda70005fd0 tx=0x7fda70009b00 comp rx=0 tx=0).stop 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 -- 172.21.15.28:0/56639365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda78100700 msgr2=0x7fda78102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda78100700 0x7fda78102d40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda74009230 tx=0x7fda74004040 comp rx=0 tx=0).stop 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 -- 172.21.15.28:0/56639365 shutdown_connections 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fda640a2d00 0x7fda640a51c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda78106710 0x7fda781037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda781010c0 0x7fda78103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 --2- 172.21.15.28:0/56639365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda78100700 0x7fda78102d40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 -- 172.21.15.28:0/56639365 >> 172.21.15.28:0/56639365 conn(0x7fda78077830 msgr2=0x7fda78068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 -- 172.21.15.28:0/56639365 shutdown_connections 2024-05-09T08:28:02.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.286+0000 7fda7f709640 1 -- 172.21.15.28:0/56639365 wait complete. 2024-05-09T08:28:02.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:02 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:02.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:02 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/557471504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:28:02.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:28:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 -- 172.21.15.28:0/790968791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb81010c0 msgr2=0x7f0cb810d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 --2- 172.21.15.28:0/790968791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb81010c0 0x7f0cb810d5d0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0009cd0 tx=0x7f0cb002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 -- 172.21.15.28:0/790968791 shutdown_connections 2024-05-09T08:28:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 --2- 172.21.15.28:0/790968791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb81010c0 0x7f0cb810d5d0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 --2- 172.21.15.28:0/790968791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb8100700 0x7f0cb8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 --2- 172.21.15.28:0/790968791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb8106710 0x7f0cb8106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 -- 172.21.15.28:0/790968791 >> 172.21.15.28:0/790968791 conn(0x7f0cb8077830 msgr2=0x7f0cb8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 -- 172.21.15.28:0/790968791 shutdown_connections 2024-05-09T08:28:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 -- 172.21.15.28:0/790968791 wait complete. 2024-05-09T08:28:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 Processor -- start 2024-05-09T08:28:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.416+0000 7f0cbf210640 1 -- start start 2024-05-09T08:28:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbf210640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb8100700 0x7f0cb8102d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbf210640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb81010c0 0x7f0cb81032b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbf210640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb8106710 0x7f0cb81037f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbf210640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8110b20 con 0x7f0cb8100700 2024-05-09T08:28:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbf210640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81109a0 con 0x7f0cb8106710 2024-05-09T08:28:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbf210640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8110ca0 con 0x7f0cb81010c0 2024-05-09T08:28:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbcfac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb8100700 0x7f0cb8102d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbcfac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb8100700 0x7f0cb8102d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33350/0 (socket says 172.21.15.28:33350) 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbcfac640 1 -- 172.21.15.28:0/1366098034 learned_addr learned my addr 172.21.15.28:0/1366098034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbd7ad640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb8106710 0x7f0cb81037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0caffff640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb81010c0 0x7f0cb81032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbcfac640 1 -- 172.21.15.28:0/1366098034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb81010c0 msgr2=0x7f0cb81032b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb81010c0 0x7f0cb81032b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbcfac640 1 -- 172.21.15.28:0/1366098034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb8106710 msgr2=0x7f0cb81037f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb8106710 0x7f0cb81037f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbcfac640 1 -- 172.21.15.28:0/1366098034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb81a65a0 con 0x7f0cb8100700 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.417+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb8100700 0x7f0cb8102d70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400e9f0 tx=0x7f0cb400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.418+0000 7f0cadffb640 1 -- 172.21.15.28:0/1366098034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb400cd80 con 0x7f0cb8100700 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.418+0000 7f0cadffb640 1 -- 172.21.15.28:0/1366098034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb4005500 con 0x7f0cb8100700 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.418+0000 7f0cbf210640 1 -- 172.21.15.28:0/1366098034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb81a6890 con 0x7f0cb8100700 2024-05-09T08:28:02.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.418+0000 7f0cadffb640 1 -- 172.21.15.28:0/1366098034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4010690 con 0x7f0cb8100700 2024-05-09T08:28:02.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.418+0000 7f0cbf210640 1 -- 172.21.15.28:0/1366098034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb81a6d90 con 0x7f0cb8100700 2024-05-09T08:28:02.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.420+0000 7f0cadffb640 1 -- 172.21.15.28:0/1366098034 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0cb40046c0 con 0x7f0cb8100700 2024-05-09T08:28:02.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cc1ff3740 1 Processor -- start 2024-05-09T08:28:02.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cc1ff3740 1 -- start start 2024-05-09T08:28:02.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cadffb640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ca00a2d00 0x7f0ca00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e4990b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cbd7ad640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ca00a2d00 0x7f0ca00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e49915de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 0x562e49918710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cadffb640 1 -- 172.21.15.28:0/1366098034 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0cb40cfbc0 con 0x7f0cb8100700 2024-05-09T08:28:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e49870c40 con 0x562e4990b330 2024-05-09T08:28:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x562e49818580 con 0x562e49916320 2024-05-09T08:28:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562e4986cc00 con 0x562e4990bc50 2024-05-09T08:28:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cbcfac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e4990b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0caffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e49915de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0caffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e49915de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42982/0 (socket says 172.21.15.28:42982) 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0caffff640 1 -- 172.21.15.28:0/889589301 learned_addr learned my addr 172.21.15.28:0/889589301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.421+0000 7f0cbd7ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 0x562e49918710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0caffff640 1 -- 172.21.15.28:0/889589301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 msgr2=0x562e49918710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0caffff640 1 --2- 172.21.15.28:0/889589301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 0x562e49918710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0caffff640 1 -- 172.21.15.28:0/889589301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 msgr2=0x562e4990b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0caffff640 1 --2- 172.21.15.28:0/889589301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e4990b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0caffff640 1 -- 172.21.15.28:0/889589301 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e495843f0 con 0x562e4990bc50 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0cbd7ad640 1 --2- 172.21.15.28:0/889589301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 0x562e49918710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/889589301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e4990b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0cbd7ad640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ca00a2d00 0x7f0ca00a51c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0004ab0 tx=0x7f0cb0031750 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0caffff640 1 --2- 172.21.15.28:0/889589301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e49915de0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009c90 tx=0x7f0ca8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=807a6788465ed4a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0c9effd640 1 -- 172.21.15.28:0/889589301 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca801e070 con 0x562e4990bc50 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0c9effd640 1 -- 172.21.15.28:0/889589301 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ca8009d80 con 0x562e4990bc50 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.422+0000 7f0c9effd640 1 -- 172.21.15.28:0/889589301 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8018770 con 0x562e4990bc50 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/889589301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 msgr2=0x562e49915de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/889589301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e49915de0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009c90 tx=0x7f0ca8011810 comp rx=0 tx=0).stop 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/889589301 shutdown_connections 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/889589301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 0x562e49918710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/889589301 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e49915de0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/889589301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e4990b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/889589301 >> 172.21.15.28:0/889589301 conn(0x562e49907650 msgr2=0x562e49907a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/889589301 shutdown_connections 2024-05-09T08:28:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/889589301 wait complete. 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 -- start start 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.423+0000 7f0cc1ff3740 1 Processor -- start 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990b330 0x562e49913290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e499137d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49916320 0x562e49913d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e49919110 con 0x562e49916320 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x562e4982bc40 con 0x562e4990b330 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562e49919290 con 0x562e4990bc50 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cc1ff3740 10 client.0 _ll_register_callbacks cb 0x562e49583be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbcfac640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990b330 0x562e49913290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbcfac640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990b330 0x562e49913290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:59938/0 (socket says 172.21.15.28:59938) 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbcfac640 1 -- 172.21.15.28:0/2705095369 learned_addr learned my addr 172.21.15.28:0/2705095369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbd7ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49916320 0x562e49913d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0caffff640 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e499137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbcfac640 1 -- 172.21.15.28:0/2705095369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 msgr2=0x562e499137d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e499137d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbcfac640 1 -- 172.21.15.28:0/2705095369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49916320 msgr2=0x562e49913d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49916320 0x562e49913d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbcfac640 1 -- 172.21.15.28:0/2705095369 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e4964e7d0 con 0x562e4990b330 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990b330 0x562e49913290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400c6a0 tx=0x7f0cb409c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0c3f4aa5edbc1e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0c97fff640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.424+0000 7f0c97fff640 1 -- 172.21.15.28:0/2705095369 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40b9060 con 0x562e4990b330 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0c97fff640 1 -- 172.21.15.28:0/2705095369 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0cb40a46f0 con 0x562e4990b330 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2705095369 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ca8009d80 con 0x562e4990b330 2024-05-09T08:28:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0c97fff640 1 -- 172.21.15.28:0/2705095369 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0cb40a4980 con 0x562e4990b330 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0cc1ff3740 10 client.25757 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2705095369 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562e49925340 con 0x562e4990b330 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0c97fff640 1 -- 172.21.15.28:0/2705095369 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ca8009d80 con 0x562e4990b330 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0c97fff640 1 -- 172.21.15.28:0/2705095369 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f0cb409ad60 con 0x562e4990b330 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0cc1ff3740 10 client.25757 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0cc1ff3740 10 client.25757 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0cc1ff3740 20 client.25757 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x562e499268a0 0x562e49928cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0cc1ff3740 4 client.25757 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2705095369 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e4987e370 con 0x562e499268a0 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.425+0000 7f0caffff640 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x562e499268a0 0x562e49928cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.426+0000 7f0caffff640 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x562e499268a0 0x562e49928cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a73457f39e25bf96 server_cookie=43277fd89fdadbc in_seq=0 out_seq=0 2024-05-09T08:28:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.426+0000 7f0c97fff640 10 client.25757 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0c97fff640 1 -- 172.21.15.28:0/2705095369 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e4987e370 con 0x562e499268a0 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0c97fff640 10 client.25757 handle_command_reply: tid=0 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 client.25757 shutdown 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2705095369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990b330 msgr2=0x562e49913290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990b330 0x562e49913290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400c6a0 tx=0x7f0cb409c4d0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2705095369 shutdown_connections 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x562e499268a0 0x562e49928cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49916320 0x562e49913d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990bc50 0x562e499137d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0c97fff640 0 client.25757 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:28:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2705095369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990b330 0x562e49913290 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2705095369 >> 172.21.15.28:0/2705095369 conn(0x562e49907650 msgr2=0x562e4990d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2705095369 shutdown_connections 2024-05-09T08:28:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2705095369 wait complete. 2024-05-09T08:28:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.428+0000 7f0cc1ff3740 20 client.25757 trim_cache size 0 max 16384 2024-05-09T08:28:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.442+0000 7f0cc1ff3740 1 Processor -- start 2024-05-09T08:28:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.442+0000 7f0cc1ff3740 1 -- start start 2024-05-09T08:28:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.442+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e49921fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.442+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990bc50 0x562e4992e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.442+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e49916320 0x562e4992ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.442+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81b0f90 con 0x562e4990b330 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.442+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81b2b90 con 0x562e4990bc50 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.442+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81b1660 con 0x562e49916320 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbcfac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e49921fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbcfac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e49921fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33386/0 (socket says 172.21.15.28:33386) 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbcfac640 1 -- 172.21.15.28:0/1277154340 learned_addr learned my addr 172.21.15.28:0/1277154340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0caffff640 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990bc50 0x562e4992e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbd7ad640 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e49916320 0x562e4992ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbcfac640 1 -- 172.21.15.28:0/1277154340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e49916320 msgr2=0x562e4992ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e49916320 0x562e4992ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbcfac640 1 -- 172.21.15.28:0/1277154340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990bc50 msgr2=0x562e4992e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990bc50 0x562e4992e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbcfac640 1 -- 172.21.15.28:0/1277154340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e4990ba80 con 0x562e4990b330 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0caffff640 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990bc50 0x562e4992e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e49921fd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40cd540 tx=0x7f0cb40a0240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8e09948ff6736d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0c9e7fc640 1 -- 172.21.15.28:0/1277154340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40a4dd0 con 0x562e4990b330 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0c9e7fc640 1 -- 172.21.15.28:0/1277154340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb40a44f0 con 0x562e4990b330 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.443+0000 7f0c9e7fc640 1 -- 172.21.15.28:0/1277154340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb409ae30 con 0x562e4990b330 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/1277154340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 msgr2=0x562e49921fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e49921fd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40cd540 tx=0x7f0cb40a0240 comp rx=0 tx=0).stop 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/1277154340 shutdown_connections 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e49916320 0x562e4992ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e4990bc50 0x562e4992e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/1277154340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990b330 0x562e49921fd0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/1277154340 >> 172.21.15.28:0/1277154340 conn(0x562e49907650 msgr2=0x562e499067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/1277154340 shutdown_connections 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/1277154340 wait complete. 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 -- start start 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.444+0000 7f0cc1ff3740 1 Processor -- start 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990b330 0x562e4990fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990bc50 0x562e499103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cc1ff3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 0x562e499108e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81b2400 con 0x562e4990bc50 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81b1d30 con 0x562e49916320 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cc1ff3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81aef70 con 0x562e4990b330 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cc1ff3740 10 client.0 _ll_register_callbacks cb 0x7f0cb409bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0caffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990bc50 0x562e499103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cbcfac640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990b330 0x562e4990fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0caffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990bc50 0x562e499103a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33394/0 (socket says 172.21.15.28:33394) 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0caffff640 1 -- 172.21.15.28:0/2747163653 learned_addr learned my addr 172.21.15.28:0/2747163653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cbd7ad640 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 0x562e499108e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0caffff640 1 -- 172.21.15.28:0/2747163653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990b330 msgr2=0x562e4990fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0caffff640 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990b330 0x562e4990fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0caffff640 1 -- 172.21.15.28:0/2747163653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 msgr2=0x562e499108e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0caffff640 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 0x562e499108e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0caffff640 1 -- 172.21.15.28:0/2747163653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e4998bd30 con 0x562e4990bc50 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990b330 0x562e4990fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0caffff640 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990bc50 0x562e499103a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8014040 tx=0x7f0ca80096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355ccc7be97749e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.445+0000 7f0c97fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0c97fff640 1 -- 172.21.15.28:0/2747163653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca802ae60 con 0x562e4990bc50 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0c97fff640 1 -- 172.21.15.28:0/2747163653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0ca8016040 con 0x562e4990bc50 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2747163653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cb40a44f0 con 0x562e4990bc50 2024-05-09T08:28:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0c97fff640 1 -- 172.21.15.28:0/2747163653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0ca8036050 con 0x562e4990bc50 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cc1ff3740 10 client.16290 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2747163653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562e4998cf30 con 0x562e4990bc50 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0c97fff640 1 -- 172.21.15.28:0/2747163653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb40a44f0 con 0x562e4990bc50 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0c97fff640 1 -- 172.21.15.28:0/2747163653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f0ca802a520 con 0x562e4990bc50 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cc1ff3740 10 client.16290 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cc1ff3740 10 client.16290 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cc1ff3740 20 client.16290 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x562e4998d510 0x562e4998f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cc1ff3740 4 client.16290 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2747163653 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e498786c0 con 0x562e4998d510 2024-05-09T08:28:02.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.446+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x562e4998d510 0x562e4998f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.447+0000 7f0cbcfac640 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x562e4998d510 0x562e4998f930 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=206340b3a6cd32e8 server_cookie=41243e1ce043d327 in_seq=0 out_seq=0 2024-05-09T08:28:02.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.447+0000 7f0c97fff640 10 client.16290 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:28:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0c97fff640 1 -- 172.21.15.28:0/2747163653 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e498786c0 con 0x562e4998d510 2024-05-09T08:28:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0c97fff640 10 client.16290 handle_command_reply: tid=0 2024-05-09T08:28:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 client.16290 shutdown 2024-05-09T08:28:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2747163653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990bc50 msgr2=0x562e499103a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990bc50 0x562e499103a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8014040 tx=0x7f0ca80096d0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2747163653 shutdown_connections 2024-05-09T08:28:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x562e4998d510 0x562e4998f930 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562e49916320 0x562e499108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e4990bc50 0x562e499103a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0c97fff640 0 client.16290 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:28:02.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 --2- 172.21.15.28:0/2747163653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562e4990b330 0x562e4990fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2747163653 >> 172.21.15.28:0/2747163653 conn(0x562e49907650 msgr2=0x562e49911a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2747163653 shutdown_connections 2024-05-09T08:28:02.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 1 -- 172.21.15.28:0/2747163653 wait complete. 2024-05-09T08:28:02.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.448+0000 7f0cc1ff3740 20 client.16290 trim_cache size 0 max 16384 2024-05-09T08:28:02.455 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:28:02.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 -- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ca00a2d00 msgr2=0x7f0ca00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ca00a2d00 0x7f0ca00a51c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0004ab0 tx=0x7f0cb0031750 comp rx=0 tx=0).stop 2024-05-09T08:28:02.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 -- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb8100700 msgr2=0x7f0cb8102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb8100700 0x7f0cb8102d70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400e9f0 tx=0x7f0cb400edf0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 -- 172.21.15.28:0/1366098034 shutdown_connections 2024-05-09T08:28:02.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb8106710 0x7f0cb81037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ca00a2d00 0x7f0ca00a51c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cb81010c0 0x7f0cb81032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 --2- 172.21.15.28:0/1366098034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cb8100700 0x7f0cb8102d70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 -- 172.21.15.28:0/1366098034 >> 172.21.15.28:0/1366098034 conn(0x7f0cb8077830 msgr2=0x7f0cb8068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 -- 172.21.15.28:0/1366098034 shutdown_connections 2024-05-09T08:28:02.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.449+0000 7f0cbf210640 1 -- 172.21.15.28:0/1366098034 wait complete. 2024-05-09T08:28:02.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:28:02.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.575+0000 7f81189f9640 1 -- 172.21.15.28:0/484444950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114107800 msgr2=0x7f8114107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.575+0000 7f81189f9640 1 --2- 172.21.15.28:0/484444950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114107800 0x7f8114107be0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8104009930 tx=0x7f810402a420 comp rx=0 tx=0).stop 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.575+0000 7f81189f9640 1 -- 172.21.15.28:0/484444950 shutdown_connections 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.575+0000 7f81189f9640 1 --2- 172.21.15.28:0/484444950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81141021b0 0x7f811410e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.575+0000 7f81189f9640 1 --2- 172.21.15.28:0/484444950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81141017f0 0x7f8114101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.575+0000 7f81189f9640 1 --2- 172.21.15.28:0/484444950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114107800 0x7f8114107be0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.575+0000 7f81189f9640 1 -- 172.21.15.28:0/484444950 >> 172.21.15.28:0/484444950 conn(0x7f81140fd2a0 msgr2=0x7f81140ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.575+0000 7f81189f9640 1 -- 172.21.15.28:0/484444950 shutdown_connections 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.575+0000 7f81189f9640 1 -- 172.21.15.28:0/484444950 wait complete. 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f81189f9640 1 Processor -- start 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f81189f9640 1 -- start start 2024-05-09T08:28:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f81189f9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81141017f0 0x7f8114198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f81189f9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81141021b0 0x7f8114198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f81189f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114107800 0x7f811419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f81189f9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8114111c10 con 0x7f8114107800 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f81189f9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8114111a90 con 0x7f81141021b0 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f81189f9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8114111d90 con 0x7f81141017f0 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f8112d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114107800 0x7f811419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f811259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81141017f0 0x7f8114198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f811259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81141017f0 0x7f8114198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:43018/0 (socket says 172.21.15.28:43018) 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f8111d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81141021b0 0x7f8114198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f811259c640 1 -- 172.21.15.28:0/2925096229 learned_addr learned my addr 172.21.15.28:0/2925096229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f8112d9d640 1 -- 172.21.15.28:0/2925096229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81141017f0 msgr2=0x7f8114198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f8112d9d640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81141017f0 0x7f8114198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f8112d9d640 1 -- 172.21.15.28:0/2925096229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81141021b0 msgr2=0x7f8114198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f8112d9d640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81141021b0 0x7f8114198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f8112d9d640 1 -- 172.21.15.28:0/2925096229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811419d140 con 0x7f8114107800 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.576+0000 7f8111d9b640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81141021b0 0x7f8114198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.577+0000 7f8112d9d640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114107800 0x7f811419ca10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f810c009fd0 tx=0x7f810c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.577+0000 7f81037fe640 1 -- 172.21.15.28:0/2925096229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c00ce20 con 0x7f8114107800 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.577+0000 7f81189f9640 1 -- 172.21.15.28:0/2925096229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811419d430 con 0x7f8114107800 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.577+0000 7f81037fe640 1 -- 172.21.15.28:0/2925096229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f810c005020 con 0x7f8114107800 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.577+0000 7f81037fe640 1 -- 172.21.15.28:0/2925096229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c010720 con 0x7f8114107800 2024-05-09T08:28:02.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.577+0000 7f81189f9640 1 -- 172.21.15.28:0/2925096229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81141a4cc0 con 0x7f8114107800 2024-05-09T08:28:02.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.580+0000 7f81037fe640 1 -- 172.21.15.28:0/2925096229 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f810c005540 con 0x7f8114107800 2024-05-09T08:28:02.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.580+0000 7f811b7e8740 1 Processor -- start 2024-05-09T08:28:02.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.580+0000 7f811b7e8740 1 -- start start 2024-05-09T08:28:02.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.580+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 0x564215ab9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.580+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9a30 0x564215ac81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.580+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215acab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f811b7e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564215a1ec40 con 0x564215ac8730 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f811b7e8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5642159c6580 con 0x564215ab9080 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f811b7e8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564215a1ac00 con 0x564215ab9a30 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215acab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215acab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33428/0 (socket says 172.21.15.28:33428) 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 -- 172.21.15.28:0/3584472803 learned_addr learned my addr 172.21.15.28:0/3584472803 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f81037fe640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3584472803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 0x564215ab9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f811259c640 1 --2- 172.21.15.28:0/3584472803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9a30 0x564215ac81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8112d9d640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 -- 172.21.15.28:0/3584472803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9a30 msgr2=0x564215ac81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3584472803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9a30 0x564215ac81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 -- 172.21.15.28:0/3584472803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 msgr2=0x564215acab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3584472803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215acab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 -- 172.21.15.28:0/3584472803 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642157323f0 con 0x564215ab9080 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f81037fe640 1 -- 172.21.15.28:0/2925096229 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f810c0cfe90 con 0x7f8114107800 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3584472803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 0x564215ab9460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8108012b80 tx=0x7f810800b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e37ea1990f4fbcb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.581+0000 7f8112d9d640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f810c0cd2c0 tx=0x7f810c0bc040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f8100ff9640 1 -- 172.21.15.28:0/3584472803 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108009b30 con 0x564215ab9080 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f8100ff9640 1 -- 172.21.15.28:0/3584472803 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8108009cd0 con 0x564215ab9080 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f8100ff9640 1 -- 172.21.15.28:0/3584472803 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108003b40 con 0x564215ab9080 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f811b7e8740 1 -- 172.21.15.28:0/3584472803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 msgr2=0x564215ab9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3584472803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 0x564215ab9460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8108012b80 tx=0x7f810800b390 comp rx=0 tx=0).stop 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f811b7e8740 1 -- 172.21.15.28:0/3584472803 shutdown_connections 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3584472803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215acab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3584472803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9a30 0x564215ac81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3584472803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 0x564215ab9460 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f811b7e8740 1 -- 172.21.15.28:0/3584472803 >> 172.21.15.28:0/3584472803 conn(0x564215ab5650 msgr2=0x564215ab5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f811b7e8740 1 -- 172.21.15.28:0/3584472803 shutdown_connections 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.582+0000 7f811b7e8740 1 -- 172.21.15.28:0/3584472803 wait complete. 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811b7e8740 1 -- start start 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811b7e8740 1 Processor -- start 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 0x564215ac5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ac60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811b7e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564215acb540 con 0x564215ac8730 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811b7e8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5642159d9c40 con 0x564215ab9a30 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811b7e8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564215acb6c0 con 0x564215ab9080 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811b7e8740 10 client.0 _ll_register_callbacks cb 0x564215731be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f8111d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ac60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f8111d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ac60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33434/0 (socket says 172.21.15.28:33434) 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f8111d9b640 1 -- 172.21.15.28:0/3648442049 learned_addr learned my addr 172.21.15.28:0/3648442049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f811259c640 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.583+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 0x564215ac5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f811259c640 1 -- 172.21.15.28:0/3648442049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 msgr2=0x564215ac5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f811259c640 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 0x564215ac5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f811259c640 1 -- 172.21.15.28:0/3648442049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 msgr2=0x564215ac60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f811259c640 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ac60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f811259c640 1 -- 172.21.15.28:0/3648442049 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642157fc7d0 con 0x564215ab9a30 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f811259c640 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8104009900 tx=0x7f8104031920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df644b2f657a7fdd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f80f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3648442049 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104035040 con 0x564215ab9a30 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3648442049 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f8104004240 con 0x564215ab9a30 2024-05-09T08:28:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f811b7e8740 1 -- 172.21.15.28:0/3648442049 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8108009cd0 con 0x564215ab9a30 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3648442049 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f810404f050 con 0x564215ab9a30 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f811b7e8740 10 client.25775 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f811b7e8740 1 -- 172.21.15.28:0/3648442049 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564215ad7720 con 0x564215ab9a30 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.584+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3648442049 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8108009cd0 con 0x564215ab9a30 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3648442049 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f8104046530 con 0x564215ab9a30 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f811b7e8740 10 client.25775 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f811b7e8740 10 client.25775 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f811b7e8740 20 client.25775 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564215ad8be0 0x564215adb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f811b7e8740 4 client.25775 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f811b7e8740 1 -- 172.21.15.28:0/3648442049 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564215a2c370 con 0x564215ad8be0 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564215ad8be0 0x564215adb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564215ad8be0 0x564215adb020 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=653c507ba0a02649 server_cookie=50f64f2e71b90896 in_seq=0 out_seq=0 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.585+0000 7f80f9ffb640 10 client.25775 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3648442049 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564215a2c370 con 0x564215ad8be0 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f80f9ffb640 10 client.25775 handle_command_reply: tid=0 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 client.25775 shutdown 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 -- 172.21.15.28:0/3648442049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 msgr2=0x564215ac5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8104009900 tx=0x7f8104031920 comp rx=0 tx=0).stop 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 -- 172.21.15.28:0/3648442049 shutdown_connections 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564215ad8be0 0x564215adb020 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ac60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5b70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3648442049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 0x564215ac5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f80f9ffb640 0 client.25775 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 -- 172.21.15.28:0/3648442049 >> 172.21.15.28:0/3648442049 conn(0x564215ab5650 msgr2=0x564215abb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 -- 172.21.15.28:0/3648442049 shutdown_connections 2024-05-09T08:28:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 1 -- 172.21.15.28:0/3648442049 wait complete. 2024-05-09T08:28:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.586+0000 7f811b7e8740 20 client.25775 trim_cache size 0 max 16384 2024-05-09T08:28:02.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:02 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:02.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:02 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/557471504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:28:02.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811b7e8740 1 Processor -- start 2024-05-09T08:28:02.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811b7e8740 1 -- start start 2024-05-09T08:28:02.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 0x564215ad4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ab4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811b7e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81141acba0 con 0x564215ac8730 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811b7e8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f81141ae7a0 con 0x564215ab9a30 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811b7e8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f81141ad270 con 0x564215ab9080 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f8111d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ab4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f8111d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ab4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33444/0 (socket says 172.21.15.28:33444) 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f8111d9b640 1 -- 172.21.15.28:0/4127744722 learned_addr learned my addr 172.21.15.28:0/4127744722 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f8111d9b640 1 --2- 172.21.15.28:0/4127744722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 0x564215ad4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f8111d9b640 1 -- 172.21.15.28:0/4127744722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 msgr2=0x564215ad4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f8111d9b640 1 --2- 172.21.15.28:0/4127744722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 0x564215ad4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f8111d9b640 1 -- 172.21.15.28:0/4127744722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 msgr2=0x564215ac5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f8111d9b640 1 --2- 172.21.15.28:0/4127744722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f8111d9b640 1 -- 172.21.15.28:0/4127744722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564215aa6d20 con 0x564215ac8730 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.600+0000 7f811259c640 1 --2- 172.21.15.28:0/4127744722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f8111d9b640 1 --2- 172.21.15.28:0/4127744722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ab4630 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8108003890 tx=0x7f81080123f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c8cbe30262df3f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f80fbfff640 1 -- 172.21.15.28:0/4127744722 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108009c90 con 0x564215ac8730 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f80fbfff640 1 -- 172.21.15.28:0/4127744722 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f810800bb70 con 0x564215ac8730 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f80fbfff640 1 -- 172.21.15.28:0/4127744722 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108031870 con 0x564215ac8730 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f811b7e8740 1 -- 172.21.15.28:0/4127744722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 msgr2=0x564215ab4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f811b7e8740 1 --2- 172.21.15.28:0/4127744722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ab4630 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8108003890 tx=0x7f81080123f0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f811b7e8740 1 -- 172.21.15.28:0/4127744722 shutdown_connections 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f811b7e8740 1 --2- 172.21.15.28:0/4127744722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ac8730 0x564215ab4630 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f811b7e8740 1 --2- 172.21.15.28:0/4127744722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9a30 0x564215ac5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f811b7e8740 1 --2- 172.21.15.28:0/4127744722 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ab9080 0x564215ad4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f811b7e8740 1 -- 172.21.15.28:0/4127744722 >> 172.21.15.28:0/4127744722 conn(0x564215ab5650 msgr2=0x564215ab3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f811b7e8740 1 -- 172.21.15.28:0/4127744722 shutdown_connections 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.601+0000 7f811b7e8740 1 -- 172.21.15.28:0/4127744722 wait complete. 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811b7e8740 1 -- start start 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811b7e8740 1 Processor -- start 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 0x564215ad0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ab9a30 0x564215ac2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811b7e8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ac8730 0x564215ac2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811b7e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81141ae010 con 0x564215ab9a30 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811b7e8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f81141ad940 con 0x564215ab9080 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811b7e8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f81141aab80 con 0x564215ac8730 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811b7e8740 10 client.0 _ll_register_callbacks cb 0x7f8104009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ab9a30 0x564215ac2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ab9a30 0x564215ac2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33452/0 (socket says 172.21.15.28:33452) 2024-05-09T08:28:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811259c640 1 -- 172.21.15.28:0/3303675950 learned_addr learned my addr 172.21.15.28:0/3303675950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 0x564215ad0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ac8730 0x564215ac2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811259c640 1 -- 172.21.15.28:0/3303675950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ac8730 msgr2=0x564215ac2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811259c640 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ac8730 0x564215ac2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811259c640 1 -- 172.21.15.28:0/3303675950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 msgr2=0x564215ad0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811259c640 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 0x564215ad0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.602+0000 7f811259c640 1 -- 172.21.15.28:0/3303675950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564215ac2da0 con 0x564215ab9a30 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811259c640 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ab9a30 0x564215ac2030 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8104046930 tx=0x7f8104046d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea39fa0d5ff5edf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f80f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3303675950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104035070 con 0x564215ab9a30 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811b7e8740 1 -- 172.21.15.28:0/3303675950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f810800bb70 con 0x564215ab9a30 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3303675950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f8104046690 con 0x564215ab9a30 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3303675950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8104060050 con 0x564215ab9a30 2024-05-09T08:28:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811b7e8740 10 client.16308 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811b7e8740 1 -- 172.21.15.28:0/3303675950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564215b3f190 con 0x564215ab9a30 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3303675950 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f810800bb70 con 0x564215ab9a30 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3303675950 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f810405b030 con 0x564215ab9a30 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811b7e8740 10 client.16308 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811b7e8740 10 client.16308 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811b7e8740 20 client.16308 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564215b3f960 0x564215b41d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811b7e8740 4 client.16308 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.603+0000 7f811b7e8740 1 -- 172.21.15.28:0/3303675950 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564215a266c0 con 0x564215b3f960 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564215b3f960 0x564215b41d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f8111d9b640 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564215b3f960 0x564215b41d80 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ad819d777b0e6a35 server_cookie=26d61ef9cbbf4bc1 in_seq=0 out_seq=0 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f80f9ffb640 10 client.16308 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f80f9ffb640 1 -- 172.21.15.28:0/3303675950 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564215a266c0 con 0x564215b3f960 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f80f9ffb640 10 client.16308 handle_command_reply: tid=0 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 client.16308 shutdown 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 -- 172.21.15.28:0/3303675950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ab9a30 msgr2=0x564215ac2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ab9a30 0x564215ac2030 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8104046930 tx=0x7f8104046d50 comp rx=0 tx=0).stop 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 -- 172.21.15.28:0/3303675950 shutdown_connections 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x564215b3f960 0x564215b41d80 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564215ac8730 0x564215ac2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f80f9ffb640 0 client.16308 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564215ab9a30 0x564215ac2030 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 --2- 172.21.15.28:0/3303675950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564215ab9080 0x564215ad0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 -- 172.21.15.28:0/3303675950 >> 172.21.15.28:0/3303675950 conn(0x564215ab5650 msgr2=0x564215ab47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 -- 172.21.15.28:0/3303675950 shutdown_connections 2024-05-09T08:28:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.604+0000 7f811b7e8740 1 -- 172.21.15.28:0/3303675950 wait complete. 2024-05-09T08:28:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f811b7e8740 20 client.16308 trim_cache size 0 max 16384 2024-05-09T08:28:02.613 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:28:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 -- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f80fc0a2d00 msgr2=0x7f80fc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f810c0cd2c0 tx=0x7f810c0bc040 comp rx=0 tx=0).stop 2024-05-09T08:28:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 -- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114107800 msgr2=0x7f811419ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114107800 0x7f811419ca10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f810c009fd0 tx=0x7f810c00ef10 comp rx=0 tx=0).stop 2024-05-09T08:28:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 -- 172.21.15.28:0/2925096229 shutdown_connections 2024-05-09T08:28:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114107800 0x7f811419ca10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81141021b0 0x7f8114198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 --2- 172.21.15.28:0/2925096229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81141017f0 0x7f8114198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 -- 172.21.15.28:0/2925096229 >> 172.21.15.28:0/2925096229 conn(0x7f81140fd2a0 msgr2=0x7f81140fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 -- 172.21.15.28:0/2925096229 shutdown_connections 2024-05-09T08:28:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.605+0000 7f81189f9640 1 -- 172.21.15.28:0/2925096229 wait complete. 2024-05-09T08:28:02.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:02 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:02.615 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:02 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/557471504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:28:02.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:28:02.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.732+0000 7f0ee253b640 1 -- 172.21.15.28:0/3442093688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 msgr2=0x7f0edc10b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3442093688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 0x7f0edc10b8b0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009900 tx=0x7f0ed002a510 comp rx=0 tx=0).stop 2024-05-09T08:28:02.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 -- 172.21.15.28:0/3442093688 shutdown_connections 2024-05-09T08:28:02.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3442093688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0edc10bdf0 0x7f0edc10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3442093688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 0x7f0edc10b8b0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3442093688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0edc102ab0 0x7f0edc102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 -- 172.21.15.28:0/3442093688 >> 172.21.15.28:0/3442093688 conn(0x7f0edc077830 msgr2=0x7f0edc077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 -- 172.21.15.28:0/3442093688 shutdown_connections 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 -- 172.21.15.28:0/3442093688 wait complete. 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 Processor -- start 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.733+0000 7f0ee253b640 1 -- start start 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0ee253b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0edc102ab0 0x7f0edc100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0ee253b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 0x7f0edc100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0ee253b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0edc10bdf0 0x7f0edc101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0ee253b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc111bf0 con 0x7f0edc1033d0 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0ee253b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc111a70 con 0x7f0edc10bdf0 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0ee253b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc111d70 con 0x7f0edc102ab0 2024-05-09T08:28:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 0x7f0edc100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 0x7f0edc100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33478/0 (socket says 172.21.15.28:33478) 2024-05-09T08:28:02.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edb7fe640 1 -- 172.21.15.28:0/3988935683 learned_addr learned my addr 172.21.15.28:0/3988935683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edbfff640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0edc102ab0 0x7f0edc100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0ee0ad8640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0edc10bdf0 0x7f0edc101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edb7fe640 1 -- 172.21.15.28:0/3988935683 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0edc102ab0 msgr2=0x7f0edc100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0edc102ab0 0x7f0edc100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edb7fe640 1 -- 172.21.15.28:0/3988935683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0edc10bdf0 msgr2=0x7f0edc101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0edc10bdf0 0x7f0edc101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edb7fe640 1 -- 172.21.15.28:0/3988935683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0edc1a65d0 con 0x7f0edc1033d0 2024-05-09T08:28:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 0x7f0edc100c40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0006010 tx=0x7f0ed0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.734+0000 7f0ed97fa640 1 -- 172.21.15.28:0/3988935683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed003f030 con 0x7f0edc1033d0 2024-05-09T08:28:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.735+0000 7f0ed97fa640 1 -- 172.21.15.28:0/3988935683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed0004440 con 0x7f0edc1033d0 2024-05-09T08:28:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.735+0000 7f0ee253b640 1 -- 172.21.15.28:0/3988935683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0edc1a6860 con 0x7f0edc1033d0 2024-05-09T08:28:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.735+0000 7f0ed97fa640 1 -- 172.21.15.28:0/3988935683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0035070 con 0x7f0edc1033d0 2024-05-09T08:28:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.735+0000 7f0ee253b640 1 -- 172.21.15.28:0/3988935683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0edc1a6da0 con 0x7f0edc1033d0 2024-05-09T08:28:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.738+0000 7f0ed97fa640 1 -- 172.21.15.28:0/3988935683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0ed0042030 con 0x7f0edc1033d0 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee5248740 1 Processor -- start 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee5248740 1 -- start start 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7ae7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 0x55d4c7af4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee5248740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d4c7a4cc40 con 0x55d4c7af2320 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee5248740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55d4c79f4580 con 0x55d4c7ae7330 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee5248740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d4c7a48c00 con 0x55d4c7ae7c50 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee0ad8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7ae7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee0ad8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7ae7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60006/0 (socket says 172.21.15.28:60006) 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee0ad8640 1 -- 172.21.15.28:0/4168310813 learned_addr learned my addr 172.21.15.28:0/4168310813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0edbfff640 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 0x55d4c7af4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ed97fa640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ec40a2db0 0x7f0ec40a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee0ad8640 1 -- 172.21.15.28:0/4168310813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 msgr2=0x55d4c7af1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee0ad8640 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee0ad8640 1 -- 172.21.15.28:0/4168310813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 msgr2=0x55d4c7af4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee0ad8640 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 0x55d4c7af4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee0ad8640 1 -- 172.21.15.28:0/4168310813 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4c77603f0 con 0x55d4c7ae7330 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0edbfff640 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ed97fa640 1 -- 172.21.15.28:0/3988935683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0ed00efbe0 con 0x7f0edc1033d0 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.739+0000 7f0ee0ad8640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ec40a2db0 0x7f0ec40a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.740+0000 7f0ee0ad8640 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7ae7710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4011590 tx=0x7f0ed4011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94b75b5b50c29f0a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.740+0000 7f0ee0ad8640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ec40a2db0 0x7f0ec40a5270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4022000 tx=0x7f0ed4023000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.740+0000 7f0ec27fc640 1 -- 172.21.15.28:0/4168310813 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4017050 con 0x55d4c7ae7330 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.740+0000 7f0ec27fc640 1 -- 172.21.15.28:0/4168310813 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed401bb90 con 0x55d4c7ae7330 2024-05-09T08:28:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.740+0000 7f0ec27fc640 1 -- 172.21.15.28:0/4168310813 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed400e3b0 con 0x55d4c7ae7330 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 -- 172.21.15.28:0/4168310813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 msgr2=0x55d4c7ae7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7ae7710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4011590 tx=0x7f0ed4011a50 comp rx=0 tx=0).stop 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 -- 172.21.15.28:0/4168310813 shutdown_connections 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 0x55d4c7af4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 --2- 172.21.15.28:0/4168310813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7ae7710 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 -- 172.21.15.28:0/4168310813 >> 172.21.15.28:0/4168310813 conn(0x55d4c7ae3650 msgr2=0x55d4c7ae3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 -- 172.21.15.28:0/4168310813 shutdown_connections 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 -- 172.21.15.28:0/4168310813 wait complete. 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 -- start start 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.741+0000 7f0ee5248740 1 Processor -- start 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7aef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7c50 0x55d4c7aef820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7af2320 0x55d4c7aefd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0ee5248740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d4c7af5110 con 0x55d4c7ae7330 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0ee5248740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55d4c7a07c40 con 0x55d4c7ae7c50 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0ee5248740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d4c7af5290 con 0x55d4c7af2320 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7aef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7aef2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33502/0 (socket says 172.21.15.28:33502) 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0edbfff640 1 -- 172.21.15.28:0/1666921357 learned_addr learned my addr 172.21.15.28:0/1666921357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7c50 0x55d4c7aef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0ee5248740 10 client.0 _ll_register_callbacks cb 0x55d4c775fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.742+0000 7f0ee0ad8640 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7af2320 0x55d4c7aefd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0edbfff640 1 -- 172.21.15.28:0/1666921357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7af2320 msgr2=0x55d4c7aefd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0edbfff640 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7af2320 0x55d4c7aefd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0edbfff640 1 -- 172.21.15.28:0/1666921357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7c50 msgr2=0x55d4c7aef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0edbfff640 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7c50 0x55d4c7aef820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0edbfff640 1 -- 172.21.15.28:0/1666921357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4c782a7d0 con 0x55d4c7ae7330 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0ee0ad8640 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7af2320 0x55d4c7aefd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7c50 0x55d4c7aef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0edbfff640 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7aef2e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc012710 tx=0x7f0ecc012bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa33201c7c63bee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0ebb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/1666921357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc004050 con 0x55d4c7ae7330 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/1666921357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0ecc0041f0 con 0x55d4c7ae7330 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0ee5248740 1 -- 172.21.15.28:0/1666921357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ed401bb90 con 0x55d4c7ae7330 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/1666921357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0ecc0043f0 con 0x55d4c7ae7330 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0ee5248740 10 client.16320 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0ee5248740 1 -- 172.21.15.28:0/1666921357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d4c7b01390 con 0x55d4c7ae7330 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.743+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/1666921357 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed401bb90 con 0x55d4c7ae7330 2024-05-09T08:28:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/1666921357 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f0ecc003bf0 con 0x55d4c7ae7330 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0ee5248740 10 client.16320 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0ee5248740 10 client.16320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0ee5248740 20 client.16320 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d4c7b02940 0x55d4c7b04da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0ee5248740 4 client.16320 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0ee5248740 1 -- 172.21.15.28:0/1666921357 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4c7a5a370 con 0x55d4c7b02940 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d4c7b02940 0x55d4c7b04da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0ebb7fe640 10 client.16320 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.744+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d4c7b02940 0x55d4c7b04da0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=388fbbe466b2fade server_cookie=f05e1f002515beb1 in_seq=0 out_seq=0 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/1666921357 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d4c7a5a370 con 0x55d4c7b02940 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ebb7fe640 10 client.16320 handle_command_reply: tid=0 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 client.16320 shutdown 2024-05-09T08:28:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 -- 172.21.15.28:0/1666921357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 msgr2=0x55d4c7aef2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7aef2e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc012710 tx=0x7f0ecc012bd0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 -- 172.21.15.28:0/1666921357 shutdown_connections 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d4c7b02940 0x55d4c7b04da0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7af2320 0x55d4c7aefd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7c50 0x55d4c7aef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1666921357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7aef2e0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ebb7fe640 0 client.16320 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 -- 172.21.15.28:0/1666921357 >> 172.21.15.28:0/1666921357 conn(0x55d4c7ae3650 msgr2=0x55d4c7ae9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 -- 172.21.15.28:0/1666921357 shutdown_connections 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 1 -- 172.21.15.28:0/1666921357 wait complete. 2024-05-09T08:28:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.746+0000 7f0ee5248740 20 client.16320 trim_cache size 0 max 16384 2024-05-09T08:28:02.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0ee5248740 1 Processor -- start 2024-05-09T08:28:02.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0ee5248740 1 -- start start 2024-05-09T08:28:02.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7afdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7aee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7af2320 0x55d4c7b0a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0ee5248740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1b1090 con 0x55d4c7ae7330 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0ee5248740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1b2c90 con 0x55d4c7af2320 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0ee5248740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1b1760 con 0x55d4c7ae7c50 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7afdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7afdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33514/0 (socket says 172.21.15.28:33514) 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0edbfff640 1 -- 172.21.15.28:0/1017292613 learned_addr learned my addr 172.21.15.28:0/1017292613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0ee0ad8640 1 --2- 172.21.15.28:0/1017292613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7af2320 0x55d4c7b0a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/1017292613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7aee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0edbfff640 1 -- 172.21.15.28:0/1017292613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 msgr2=0x55d4c7aee2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0edbfff640 1 --2- 172.21.15.28:0/1017292613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7aee2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0edbfff640 1 -- 172.21.15.28:0/1017292613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7af2320 msgr2=0x55d4c7b0a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0edbfff640 1 --2- 172.21.15.28:0/1017292613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7af2320 0x55d4c7b0a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.762+0000 7f0edbfff640 1 -- 172.21.15.28:0/1017292613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4c7ad4d20 con 0x55d4c7ae7330 2024-05-09T08:28:02.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.763+0000 7f0edbfff640 1 --2- 172.21.15.28:0/1017292613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7afdfd0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc010500 tx=0x7f0ecc009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab14b7710e071ad server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.763+0000 7f0ec1ffb640 1 -- 172.21.15.28:0/1017292613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc02d070 con 0x55d4c7ae7330 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.763+0000 7f0ec1ffb640 1 -- 172.21.15.28:0/1017292613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc00f790 con 0x55d4c7ae7330 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.763+0000 7f0ec1ffb640 1 -- 172.21.15.28:0/1017292613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0052e0 con 0x55d4c7ae7330 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 -- 172.21.15.28:0/1017292613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 msgr2=0x55d4c7afdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1017292613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7afdfd0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc010500 tx=0x7f0ecc009ea0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 -- 172.21.15.28:0/1017292613 shutdown_connections 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1017292613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7af2320 0x55d4c7b0a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1017292613 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7aee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 --2- 172.21.15.28:0/1017292613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7ae7330 0x55d4c7afdfd0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 -- 172.21.15.28:0/1017292613 >> 172.21.15.28:0/1017292613 conn(0x55d4c7ae3650 msgr2=0x55d4c7aedbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 -- 172.21.15.28:0/1017292613 shutdown_connections 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 -- 172.21.15.28:0/1017292613 wait complete. 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 -- start start 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 Processor -- start 2024-05-09T08:28:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.764+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7afa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ee5248740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 0x55d4c7af6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ee5248740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1b2500 con 0x55d4c7af2320 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ee5248740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1b1e30 con 0x55d4c7ae7330 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ee5248740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1af070 con 0x55d4c7ae7c50 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ee5248740 10 client.0 _ll_register_callbacks cb 0x7f0ecc009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7afa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7afa880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60044/0 (socket says 172.21.15.28:60044) 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ee0ad8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 0x55d4c7af6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edbfff640 1 -- 172.21.15.28:0/3891335607 learned_addr learned my addr 172.21.15.28:0/3891335607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edb7fe640 1 -- 172.21.15.28:0/3891335607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 msgr2=0x55d4c7afa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7afa880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edb7fe640 1 -- 172.21.15.28:0/3891335607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 msgr2=0x55d4c7af6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 0x55d4c7af6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edb7fe640 1 -- 172.21.15.28:0/3891335607 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4c7af6960 con 0x55d4c7ae7c50 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edbfff640 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7afa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af5bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00ecf00 tx=0x7f0ed00b7fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15d1d6532f5506ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ebb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/3891335607 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0049040 con 0x55d4c7ae7c50 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/3891335607 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f0ed00b7b90 con 0x55d4c7ae7c50 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.765+0000 7f0ee5248740 1 -- 172.21.15.28:0/3891335607 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ecc00f790 con 0x55d4c7ae7c50 2024-05-09T08:28:02.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/3891335607 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0ed00bb7a0 con 0x55d4c7ae7c50 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ee5248740 10 client.25717 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ee5248740 1 -- 172.21.15.28:0/3891335607 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d4c7b68f70 con 0x55d4c7ae7c50 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/3891335607 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc00f790 con 0x55d4c7ae7c50 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/3891335607 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f0ed00ed180 con 0x55d4c7ae7c50 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ee5248740 10 client.25717 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ee5248740 10 client.25717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ee5248740 20 client.25717 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ee5248740 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d4c7b696c0 0x55d4c7b6bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ee5248740 4 client.25717 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0ee5248740 1 -- 172.21.15.28:0/3891335607 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d4c7a546c0 con 0x55d4c7b696c0 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.766+0000 7f0edbfff640 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d4c7b696c0 0x55d4c7b6bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.767+0000 7f0edbfff640 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d4c7b696c0 0x55d4c7b6bae0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2277c444c00e6ff4 server_cookie=2f2e00142d56f579 in_seq=0 out_seq=0 2024-05-09T08:28:02.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.767+0000 7f0ebb7fe640 10 client.25717 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.770+0000 7f0ebb7fe640 1 -- 172.21.15.28:0/3891335607 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d4c7a546c0 con 0x55d4c7b696c0 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.770+0000 7f0ebb7fe640 10 client.25717 handle_command_reply: tid=0 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.770+0000 7f0ee5248740 1 client.25717 shutdown 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 -- 172.21.15.28:0/3891335607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 msgr2=0x55d4c7af5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af5bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00ecf00 tx=0x7f0ed00b7fc0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 -- 172.21.15.28:0/3891335607 shutdown_connections 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d4c7b696c0 0x55d4c7b6bae0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4c7af2320 0x55d4c7af6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d4c7ae7c50 0x55d4c7af5bf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 --2- 172.21.15.28:0/3891335607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c7ae7330 0x55d4c7afa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ebb7fe640 0 client.25717 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 -- 172.21.15.28:0/3891335607 >> 172.21.15.28:0/3891335607 conn(0x55d4c7ae3650 msgr2=0x55d4c7ae9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 -- 172.21.15.28:0/3891335607 shutdown_connections 2024-05-09T08:28:02.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 1 -- 172.21.15.28:0/3891335607 wait complete. 2024-05-09T08:28:02.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee5248740 20 client.25717 trim_cache size 0 max 16384 2024-05-09T08:28:02.775 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:28:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee253b640 1 -- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ec40a2db0 msgr2=0x7f0ec40a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ec40a2db0 0x7f0ec40a5270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4022000 tx=0x7f0ed4023000 comp rx=0 tx=0).stop 2024-05-09T08:28:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee253b640 1 -- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 msgr2=0x7f0edc100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.771+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 0x7f0edc100c40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0006010 tx=0x7f0ed0004040 comp rx=0 tx=0).stop 2024-05-09T08:28:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.772+0000 7f0ee253b640 1 -- 172.21.15.28:0/3988935683 shutdown_connections 2024-05-09T08:28:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.772+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0edc10bdf0 0x7f0edc101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.772+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc1033d0 0x7f0edc100c40 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.772+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ec40a2db0 0x7f0ec40a5270 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.772+0000 7f0ee253b640 1 --2- 172.21.15.28:0/3988935683 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0edc102ab0 0x7f0edc100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.772+0000 7f0ee253b640 1 -- 172.21.15.28:0/3988935683 >> 172.21.15.28:0/3988935683 conn(0x7f0edc077830 msgr2=0x7f0edc0fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.772+0000 7f0ee253b640 1 -- 172.21.15.28:0/3988935683 shutdown_connections 2024-05-09T08:28:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:02.772+0000 7f0ee253b640 1 -- 172.21.15.28:0/3988935683 wait complete. 2024-05-09T08:28:03.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:03 smithi028 ceph-mon[30119]: pgmap v55: 129 pgs: 129 active+clean; 51 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-09T08:28:03.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:03 smithi165 ceph-mon[34540]: pgmap v55: 129 pgs: 129 active+clean; 51 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-09T08:28:03.507 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:03 smithi184 ceph-mon[34509]: pgmap v55: 129 pgs: 129 active+clean; 51 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 192 op/s 2024-05-09T08:28:03.784 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:28:03.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:28:03.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:28:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.903+0000 7f8955c7d640 1 -- 172.21.15.28:0/3709920655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950100700 msgr2=0x7f8950100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.903+0000 7f8955c7d640 1 --2- 172.21.15.28:0/3709920655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950100700 0x7f8950100b80 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8944009930 tx=0x7f894402a420 comp rx=0 tx=0).stop 2024-05-09T08:28:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.903+0000 7f8955c7d640 1 -- 172.21.15.28:0/3709920655 shutdown_connections 2024-05-09T08:28:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.903+0000 7f8955c7d640 1 --2- 172.21.15.28:0/3709920655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89501010c0 0x7f895010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.903+0000 7f8955c7d640 1 --2- 172.21.15.28:0/3709920655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950100700 0x7f8950100b80 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.903+0000 7f8955c7d640 1 --2- 172.21.15.28:0/3709920655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8950106710 0x7f8950106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.903+0000 7f8955c7d640 1 -- 172.21.15.28:0/3709920655 >> 172.21.15.28:0/3709920655 conn(0x7f8950077830 msgr2=0x7f8950077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.903+0000 7f8955c7d640 1 -- 172.21.15.28:0/3709920655 shutdown_connections 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.903+0000 7f8955c7d640 1 -- 172.21.15.28:0/3709920655 wait complete. 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f8955c7d640 1 Processor -- start 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f8955c7d640 1 -- start start 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f8955c7d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950100700 0x7f895019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f8955c7d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89501010c0 0x7f895019a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f8955c7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950106710 0x7f895019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f8955c7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8950110b70 con 0x7f8950106710 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f8955c7d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f89501109f0 con 0x7f8950100700 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f8955c7d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8950110cf0 con 0x7f89501010c0 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f894ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950106710 0x7f895019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f894f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950100700 0x7f895019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.904+0000 7f894ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950106710 0x7f895019ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33548/0 (socket says 172.21.15.28:33548) 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894ffff640 1 -- 172.21.15.28:0/1984176147 learned_addr learned my addr 172.21.15.28:0/1984176147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894effd640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89501010c0 0x7f895019a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894ffff640 1 -- 172.21.15.28:0/1984176147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89501010c0 msgr2=0x7f895019a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894ffff640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89501010c0 0x7f895019a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894ffff640 1 -- 172.21.15.28:0/1984176147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950100700 msgr2=0x7f895019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894ffff640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950100700 0x7f895019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894ffff640 1 -- 172.21.15.28:0/1984176147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895019f340 con 0x7f8950106710 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894f7fe640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950100700 0x7f895019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894ffff640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950106710 0x7f895019ec10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f894800efd0 tx=0x7f894800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894cff9640 1 -- 172.21.15.28:0/1984176147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948003b70 con 0x7f8950106710 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894cff9640 1 -- 172.21.15.28:0/1984176147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8948003d10 con 0x7f8950106710 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f8955c7d640 1 -- 172.21.15.28:0/1984176147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895019f630 con 0x7f8950106710 2024-05-09T08:28:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f894cff9640 1 -- 172.21.15.28:0/1984176147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89480109f0 con 0x7f8950106710 2024-05-09T08:28:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.905+0000 7f8955c7d640 1 -- 172.21.15.28:0/1984176147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89501a6ec0 con 0x7f8950106710 2024-05-09T08:28:03.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.908+0000 7f894cff9640 1 -- 172.21.15.28:0/1984176147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8948004020 con 0x7f8950106710 2024-05-09T08:28:03.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.908+0000 7f895898d740 1 Processor -- start 2024-05-09T08:28:03.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.908+0000 7f895898d740 1 -- start start 2024-05-09T08:28:03.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f895898d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0efdd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f895898d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 0x55fa0efdde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f895898d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0efeab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f895898d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa0ef42990 con 0x55fa0efdf0b0 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f895898d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fa0eeea5b0 con 0x55fa0efdd9c0 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f895898d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fa0ef3e950 con 0x55fa0efe8740 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0efdd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 0x55fa0efdde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0efdd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33564/0 (socket says 172.21.15.28:33564) 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 -- 172.21.15.28:0/3505800679 learned_addr learned my addr 172.21.15.28:0/3505800679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894cff9640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f89380a2d00 0x7f89380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 --2- 172.21.15.28:0/3505800679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0efeab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 -- 172.21.15.28:0/3505800679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 msgr2=0x55fa0efeab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 --2- 172.21.15.28:0/3505800679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0efeab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894ffff640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f89380a2d00 0x7f89380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 -- 172.21.15.28:0/3505800679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 msgr2=0x55fa0efdde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 --2- 172.21.15.28:0/3505800679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 0x55fa0efdde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 -- 172.21.15.28:0/3505800679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa0ed1e7e0 con 0x55fa0efdf0b0 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894f7fe640 1 --2- 172.21.15.28:0/3505800679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 0x55fa0efdde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894cff9640 1 -- 172.21.15.28:0/1984176147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f89480cfc40 con 0x7f8950106710 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.909+0000 7f894effd640 1 --2- 172.21.15.28:0/3505800679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0efdd3f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8944009930 tx=0x7f8944005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a2cd2fd00f9a31 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f8935ffb640 1 -- 172.21.15.28:0/3505800679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944044020 con 0x55fa0efdf0b0 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f894ffff640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f89380a2d00 0x7f89380a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f894800cba0 tx=0x7f8948013b60 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f8935ffb640 1 -- 172.21.15.28:0/3505800679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8944005520 con 0x55fa0efdf0b0 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f8935ffb640 1 -- 172.21.15.28:0/3505800679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944035040 con 0x55fa0efdf0b0 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f895898d740 1 -- 172.21.15.28:0/3505800679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 msgr2=0x55fa0efdd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f895898d740 1 --2- 172.21.15.28:0/3505800679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0efdd3f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8944009930 tx=0x7f8944005b50 comp rx=0 tx=0).stop 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f895898d740 1 -- 172.21.15.28:0/3505800679 shutdown_connections 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f895898d740 1 --2- 172.21.15.28:0/3505800679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0efeab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f895898d740 1 --2- 172.21.15.28:0/3505800679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 0x55fa0efdde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f895898d740 1 --2- 172.21.15.28:0/3505800679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0efdd3f0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f895898d740 1 -- 172.21.15.28:0/3505800679 >> 172.21.15.28:0/3505800679 conn(0x55fa0efa60a0 msgr2=0x55fa0efd9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f895898d740 1 -- 172.21.15.28:0/3505800679 shutdown_connections 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.910+0000 7f895898d740 1 -- 172.21.15.28:0/3505800679 wait complete. 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f895898d740 1 -- start start 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f895898d740 1 Processor -- start 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f895898d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efdd9c0 0x55fa0efe5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f895898d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 0x55fa0efe5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f895898d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efe8740 0x55fa0efe60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f895898d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa0efeb570 con 0x55fa0efe8740 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f895898d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fa0eefdb40 con 0x55fa0efdf0b0 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f895898d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fa0efeb6f0 con 0x55fa0efdd9c0 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f895898d740 10 client.0 _ll_register_callbacks cb 0x55fa0eef20f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f894effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efe8740 0x55fa0efe60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f894f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 0x55fa0efe5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.911+0000 7f894effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efe8740 0x55fa0efe60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33574/0 (socket says 172.21.15.28:33574) 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f894effd640 1 -- 172.21.15.28:0/3259184628 learned_addr learned my addr 172.21.15.28:0/3259184628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f894effd640 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efdd9c0 0x55fa0efe5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f894effd640 1 -- 172.21.15.28:0/3259184628 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efdd9c0 msgr2=0x55fa0efe5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f894effd640 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efdd9c0 0x55fa0efe5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f894effd640 1 -- 172.21.15.28:0/3259184628 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 msgr2=0x55fa0efe5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f894effd640 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 0x55fa0efe5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f894effd640 1 -- 172.21.15.28:0/3259184628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa0ed7ef70 con 0x55fa0efe8740 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f894f7fe640 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 0x55fa0efe5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f894effd640 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efe8740 0x55fa0efe60c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8944004da0 tx=0x7f8944005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b290e24dc32b94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f892effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f892effd640 1 -- 172.21.15.28:0/3259184628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944035030 con 0x55fa0efe8740 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f895898d740 1 -- 172.21.15.28:0/3259184628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8944005520 con 0x55fa0efe8740 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f892effd640 1 -- 172.21.15.28:0/3259184628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f8944004b10 con 0x55fa0efe8740 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f892effd640 1 -- 172.21.15.28:0/3259184628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8944044070 con 0x55fa0efe8740 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f895898d740 10 client.16338 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.912+0000 7f895898d740 1 -- 172.21.15.28:0/3259184628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fa0ec16170 con 0x55fa0efe8740 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f892effd640 1 -- 172.21.15.28:0/3259184628 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8944005520 con 0x55fa0efe8740 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f892effd640 1 -- 172.21.15.28:0/3259184628 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f8944003780 con 0x55fa0efe8740 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f895898d740 10 client.16338 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f895898d740 10 client.16338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f895898d740 20 client.16338 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:03.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f895898d740 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fa0eff8670 0x55fa0effaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f895898d740 4 client.16338 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:03.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f895898d740 1 -- 172.21.15.28:0/3259184628 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa0ef500c0 con 0x55fa0eff8670 2024-05-09T08:28:03.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f894f7fe640 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fa0eff8670 0x55fa0effaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f892effd640 10 client.16338 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:03.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.913+0000 7f894f7fe640 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fa0eff8670 0x55fa0effaa60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3b2c4c34c5f5e78 server_cookie=892d0f49b5dce9f in_seq=0 out_seq=0 2024-05-09T08:28:03.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f892effd640 1 -- 172.21.15.28:0/3259184628 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa0ef500c0 con 0x55fa0eff8670 2024-05-09T08:28:03.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f892effd640 10 client.16338 handle_command_reply: tid=0 2024-05-09T08:28:03.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 client.16338 shutdown 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 -- 172.21.15.28:0/3259184628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efe8740 msgr2=0x55fa0efe60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efe8740 0x55fa0efe60c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8944004da0 tx=0x7f8944005850 comp rx=0 tx=0).stop 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 -- 172.21.15.28:0/3259184628 shutdown_connections 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fa0eff8670 0x55fa0effaa60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efe8740 0x55fa0efe60c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 0x55fa0efe5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f892effd640 0 client.16338 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 --2- 172.21.15.28:0/3259184628 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efdd9c0 0x55fa0efe5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 -- 172.21.15.28:0/3259184628 >> 172.21.15.28:0/3259184628 conn(0x55fa0efa60a0 msgr2=0x55fa0efdfd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 -- 172.21.15.28:0/3259184628 shutdown_connections 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 1 -- 172.21.15.28:0/3259184628 wait complete. 2024-05-09T08:28:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.915+0000 7f895898d740 20 client.16338 trim_cache size 0 max 16384 2024-05-09T08:28:03.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f895898d740 1 Processor -- start 2024-05-09T08:28:03.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f895898d740 1 -- start start 2024-05-09T08:28:03.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f895898d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdd9c0 0x55fa0efd8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f895898d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 0x55fa0efe9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f895898d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0efea070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f895898d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89501b0ef0 con 0x55fa0efdd9c0 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f895898d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f89501b2af0 con 0x55fa0efdf0b0 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f895898d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89501b15c0 con 0x55fa0efe8740 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f894effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdd9c0 0x55fa0efd8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f894effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdd9c0 0x55fa0efd8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33576/0 (socket says 172.21.15.28:33576) 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f894effd640 1 -- 172.21.15.28:0/1286256118 learned_addr learned my addr 172.21.15.28:0/1286256118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f894f7fe640 1 --2- 172.21.15.28:0/1286256118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 0x55fa0efe9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f894effd640 1 -- 172.21.15.28:0/1286256118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 msgr2=0x55fa0efea070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.934+0000 7f894effd640 1 --2- 172.21.15.28:0/1286256118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0efea070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f894effd640 1 -- 172.21.15.28:0/1286256118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 msgr2=0x55fa0efe9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f894effd640 1 --2- 172.21.15.28:0/1286256118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 0x55fa0efe9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f894effd640 1 -- 172.21.15.28:0/1286256118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa0efe45f0 con 0x55fa0efdd9c0 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f894effd640 1 --2- 172.21.15.28:0/1286256118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdd9c0 0x55fa0efd8f00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8944005900 tx=0x7f8944009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e2a4e275e4f3c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f89357fa640 1 -- 172.21.15.28:0/1286256118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944035040 con 0x55fa0efdd9c0 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f89357fa640 1 -- 172.21.15.28:0/1286256118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8944005430 con 0x55fa0efdd9c0 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f89357fa640 1 -- 172.21.15.28:0/1286256118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944003850 con 0x55fa0efdd9c0 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f895898d740 1 -- 172.21.15.28:0/1286256118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdd9c0 msgr2=0x55fa0efd8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f895898d740 1 --2- 172.21.15.28:0/1286256118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdd9c0 0x55fa0efd8f00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8944005900 tx=0x7f8944009290 comp rx=0 tx=0).stop 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f895898d740 1 -- 172.21.15.28:0/1286256118 shutdown_connections 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f895898d740 1 --2- 172.21.15.28:0/1286256118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0efea070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f895898d740 1 --2- 172.21.15.28:0/1286256118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdf0b0 0x55fa0efe9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f895898d740 1 --2- 172.21.15.28:0/1286256118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdd9c0 0x55fa0efd8f00 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.935+0000 7f895898d740 1 -- 172.21.15.28:0/1286256118 >> 172.21.15.28:0/1286256118 conn(0x55fa0efa60a0 msgr2=0x55fa0efd87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 -- 172.21.15.28:0/1286256118 shutdown_connections 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 -- 172.21.15.28:0/1286256118 wait complete. 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 -- start start 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 Processor -- start 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 0x55fa0effa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0effa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0f056800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89501b2360 con 0x55fa0efdf0b0 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f89501b1c90 con 0x55fa0efdd9c0 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89501aeed0 con 0x55fa0efe8740 2024-05-09T08:28:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f895898d740 10 client.0 _ll_register_callbacks cb 0x7f8940009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f894f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0effa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f894effd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 0x55fa0effa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f894f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0effa5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33588/0 (socket says 172.21.15.28:33588) 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.936+0000 7f894f7fe640 1 -- 172.21.15.28:0/4199430121 learned_addr learned my addr 172.21.15.28:0/4199430121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f894effd640 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0f056800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f894f7fe640 1 -- 172.21.15.28:0/4199430121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 msgr2=0x55fa0f056800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f894f7fe640 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0f056800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f894f7fe640 1 -- 172.21.15.28:0/4199430121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 msgr2=0x55fa0effa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f894f7fe640 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 0x55fa0effa0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f894f7fe640 1 -- 172.21.15.28:0/4199430121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa0f056fe0 con 0x55fa0efdf0b0 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f894f7fe640 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0effa5f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x55fa0efe6c10 tx=0x7f8940022410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c02271cf2a3e63c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f892effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f892effd640 1 -- 172.21.15.28:0/4199430121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940005020 con 0x55fa0efdf0b0 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f895898d740 1 -- 172.21.15.28:0/4199430121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8944005430 con 0x55fa0efdf0b0 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f892effd640 1 -- 172.21.15.28:0/4199430121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f89400051c0 con 0x55fa0efdf0b0 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f892effd640 1 -- 172.21.15.28:0/4199430121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f89400053c0 con 0x55fa0efdf0b0 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f895898d740 10 client.16344 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f895898d740 1 -- 172.21.15.28:0/4199430121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fa0f05f410 con 0x55fa0efdf0b0 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.937+0000 7f892effd640 1 -- 172.21.15.28:0/4199430121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8944005430 con 0x55fa0efdf0b0 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f892effd640 1 -- 172.21.15.28:0/4199430121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f8940029070 con 0x55fa0efdf0b0 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f895898d740 10 client.16344 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f895898d740 10 client.16344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f895898d740 20 client.16344 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f895898d740 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fa0f05fc80 0x55fa0f0620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f895898d740 4 client.16344 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:28:03.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f895898d740 1 -- 172.21.15.28:0/4199430121 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fa0ef4a410 con 0x55fa0f05fc80 2024-05-09T08:28:03.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f894effd640 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fa0f05fc80 0x55fa0f0620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:03.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f894effd640 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fa0f05fc80 0x55fa0f0620a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e78ba785651b7cbc server_cookie=6e7525dbe03e3f24 in_seq=0 out_seq=0 2024-05-09T08:28:03.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.938+0000 7f892effd640 10 client.16344 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:03.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f892effd640 1 -- 172.21.15.28:0/4199430121 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fa0ef4a410 con 0x55fa0f05fc80 2024-05-09T08:28:03.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f892effd640 10 client.16344 handle_command_reply: tid=0 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 client.16344 shutdown 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 -- 172.21.15.28:0/4199430121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 msgr2=0x55fa0effa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0effa5f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x55fa0efe6c10 tx=0x7f8940022410 comp rx=0 tx=0).stop 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 -- 172.21.15.28:0/4199430121 shutdown_connections 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fa0f05fc80 0x55fa0f0620a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fa0efe8740 0x55fa0f056800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa0efdf0b0 0x55fa0effa5f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f892effd640 0 client.16344 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 --2- 172.21.15.28:0/4199430121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fa0efdd9c0 0x55fa0effa0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 -- 172.21.15.28:0/4199430121 >> 172.21.15.28:0/4199430121 conn(0x55fa0efa60a0 msgr2=0x55fa0efd9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 -- 172.21.15.28:0/4199430121 shutdown_connections 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 1 -- 172.21.15.28:0/4199430121 wait complete. 2024-05-09T08:28:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.941+0000 7f895898d740 20 client.16344 trim_cache size 0 max 16384 2024-05-09T08:28:03.948 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:28:03.948 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-09T08:28:03.948 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "6e261940-0110-44b1-9fb5-ddd00a0e95a4", 2024-05-09T08:28:03.948 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-09T08:28:03.948 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:28:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 -- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f89380a2d00 msgr2=0x7f89380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f89380a2d00 0x7f89380a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f894800cba0 tx=0x7f8948013b60 comp rx=0 tx=0).stop 2024-05-09T08:28:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 -- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950106710 msgr2=0x7f895019ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950106710 0x7f895019ec10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f894800efd0 tx=0x7f894800c6a0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 -- 172.21.15.28:0/1984176147 shutdown_connections 2024-05-09T08:28:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8950106710 0x7f895019ec10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f89380a2d00 0x7f89380a51c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89501010c0 0x7f895019a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 --2- 172.21.15.28:0/1984176147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8950100700 0x7f895019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:03.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 -- 172.21.15.28:0/1984176147 >> 172.21.15.28:0/1984176147 conn(0x7f8950077830 msgr2=0x7f895010c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:03.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 -- 172.21.15.28:0/1984176147 shutdown_connections 2024-05-09T08:28:03.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:03.942+0000 7f8955c7d640 1 -- 172.21.15.28:0/1984176147 wait complete. 2024-05-09T08:28:03.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:28:03.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:28:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.064+0000 7fdbb7b70640 1 -- 172.21.15.28:0/910911205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb01021c0 msgr2=0x7fdbb010e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.064+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/910911205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb01021c0 0x7fdbb010e6d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009cd0 tx=0x7fdba802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.064+0000 7fdbb7b70640 1 -- 172.21.15.28:0/910911205 shutdown_connections 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.064+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/910911205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb01021c0 0x7fdbb010e6d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.064+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/910911205 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbb0101800 0x7fdbb0101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.064+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/910911205 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbb0107810 0x7fdbb0107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.064+0000 7fdbb7b70640 1 -- 172.21.15.28:0/910911205 >> 172.21.15.28:0/910911205 conn(0x7fdbb0078c80 msgr2=0x7fdbb0079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.064+0000 7fdbb7b70640 1 -- 172.21.15.28:0/910911205 shutdown_connections 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.064+0000 7fdbb7b70640 1 -- 172.21.15.28:0/910911205 wait complete. 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb7b70640 1 Processor -- start 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb7b70640 1 -- start start 2024-05-09T08:28:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb7b70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0101800 0x7fdbb0198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb7b70640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbb01021c0 0x7fdbb0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb7b70640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbb0107810 0x7fdbb019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb7b70640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb0111c20 con 0x7fdbb0101800 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb7b70640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb0111aa0 con 0x7fdbb01021c0 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb7b70640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb0111da0 con 0x7fdbb0107810 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb590c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0101800 0x7fdbb0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb590c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0101800 0x7fdbb0198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33602/0 (socket says 172.21.15.28:33602) 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb590c640 1 -- 172.21.15.28:0/501292313 learned_addr learned my addr 172.21.15.28:0/501292313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb510b640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbb01021c0 0x7fdbb0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.065+0000 7fdbb610d640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbb0107810 0x7fdbb019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdbb590c640 1 -- 172.21.15.28:0/501292313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbb0107810 msgr2=0x7fdbb019c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdbb590c640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbb0107810 0x7fdbb019c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdbb590c640 1 -- 172.21.15.28:0/501292313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbb01021c0 msgr2=0x7fdbb0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdbb590c640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbb01021c0 0x7fdbb0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdbb590c640 1 -- 172.21.15.28:0/501292313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb019d0e0 con 0x7fdbb0101800 2024-05-09T08:28:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdbb590c640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0101800 0x7fdbb0198140 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00d950 tx=0x7fdbac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdba6ffd640 1 -- 172.21.15.28:0/501292313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac017030 con 0x7fdbb0101800 2024-05-09T08:28:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdbb7b70640 1 -- 172.21.15.28:0/501292313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb019d3d0 con 0x7fdbb0101800 2024-05-09T08:28:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdba6ffd640 1 -- 172.21.15.28:0/501292313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbac004020 con 0x7fdbb0101800 2024-05-09T08:28:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdba6ffd640 1 -- 172.21.15.28:0/501292313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac0047a0 con 0x7fdbb0101800 2024-05-09T08:28:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.066+0000 7fdbb7b70640 1 -- 172.21.15.28:0/501292313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb01a4c60 con 0x7fdbb0101800 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.068+0000 7fdba6ffd640 1 -- 172.21.15.28:0/501292313 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdbac0041d0 con 0x7fdbb0101800 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.069+0000 7fdba6ffd640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb9c0a2d00 0x7fdb9c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.069+0000 7fdbba883740 1 Processor -- start 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.069+0000 7fdbb510b640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb9c0a2d00 0x7fdb9c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.069+0000 7fdba6ffd640 1 -- 172.21.15.28:0/501292313 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fdbac0cf210 con 0x7fdbb0101800 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbba883740 1 -- start start 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbba883740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb510b640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb9c0a2d00 0x7fdb9c0a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdba00095f0 tx=0x7fdba0005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbba883740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8ba0 0x561f00ec2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbba883740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00ec3270 0x561f00ec5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbba883740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561f00e1dd00 con 0x561f00eb8280 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbba883740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x561f00dc54c0 con 0x561f00ec3270 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbba883740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561f00e19cc0 con 0x561f00eb8ba0 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb590c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb590c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33612/0 (socket says 172.21.15.28:33612) 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb590c640 1 -- 172.21.15.28:0/419894369 learned_addr learned my addr 172.21.15.28:0/419894369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb610d640 1 --2- 172.21.15.28:0/419894369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00ec3270 0x561f00ec5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb510b640 1 --2- 172.21.15.28:0/419894369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8ba0 0x561f00ec2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb590c640 1 -- 172.21.15.28:0/419894369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8ba0 msgr2=0x561f00ec2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb590c640 1 --2- 172.21.15.28:0/419894369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8ba0 0x561f00ec2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb590c640 1 -- 172.21.15.28:0/419894369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00ec3270 msgr2=0x561f00ec5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb590c640 1 --2- 172.21.15.28:0/419894369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00ec3270 0x561f00ec5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb590c640 1 -- 172.21.15.28:0/419894369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f00bfb7d0 con 0x561f00eb8280 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.070+0000 7fdbb590c640 1 --2- 172.21.15.28:0/419894369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb8660 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0a3a40 tx=0x7fdbac09af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be420c394b5bc578 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdb9bfff640 1 -- 172.21.15.28:0/419894369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac09b300 con 0x561f00eb8280 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdb9bfff640 1 -- 172.21.15.28:0/419894369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbac09f6b0 con 0x561f00eb8280 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdb9bfff640 1 -- 172.21.15.28:0/419894369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac0bb330 con 0x561f00eb8280 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdbba883740 1 -- 172.21.15.28:0/419894369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 msgr2=0x561f00eb8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdbba883740 1 --2- 172.21.15.28:0/419894369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb8660 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0a3a40 tx=0x7fdbac09af70 comp rx=0 tx=0).stop 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdbba883740 1 -- 172.21.15.28:0/419894369 shutdown_connections 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdbba883740 1 --2- 172.21.15.28:0/419894369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00ec3270 0x561f00ec5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdbba883740 1 --2- 172.21.15.28:0/419894369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8ba0 0x561f00ec2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdbba883740 1 --2- 172.21.15.28:0/419894369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb8660 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.071+0000 7fdbba883740 1 -- 172.21.15.28:0/419894369 >> 172.21.15.28:0/419894369 conn(0x561f00eb45a0 msgr2=0x561f00eb4980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 -- 172.21.15.28:0/419894369 shutdown_connections 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 -- 172.21.15.28:0/419894369 wait complete. 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 -- start start 2024-05-09T08:28:04.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 Processor -- start 2024-05-09T08:28:04.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8280 0x561f00ec0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 0x561f00ec07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00ec3270 0x561f00ec0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561f00ec6060 con 0x561f00ec3270 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x561f00dd8b90 con 0x561f00eb8ba0 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.072+0000 7fdbba883740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x561f00ec61e0 con 0x561f00eb8280 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbba883740 10 client.0 _ll_register_callbacks cb 0x561f00e14da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb610d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00ec3270 0x561f00ec0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb610d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00ec3270 0x561f00ec0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33628/0 (socket says 172.21.15.28:33628) 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb610d640 1 -- 172.21.15.28:0/902097652 learned_addr learned my addr 172.21.15.28:0/902097652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb510b640 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 0x561f00ec07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb590c640 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8280 0x561f00ec0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb610d640 1 -- 172.21.15.28:0/902097652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8280 msgr2=0x561f00ec0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb610d640 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8280 0x561f00ec0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb610d640 1 -- 172.21.15.28:0/902097652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 msgr2=0x561f00ec07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb610d640 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 0x561f00ec07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb610d640 1 -- 172.21.15.28:0/902097652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f00b313f0 con 0x561f00ec3270 2024-05-09T08:28:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbb610d640 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00ec3270 0x561f00ec0cf0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009cd0 tx=0x7fdba8031af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3904043d8956351 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdb997fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdb997fa640 1 -- 172.21.15.28:0/902097652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8049060 con 0x561f00ec3270 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdb997fa640 1 -- 172.21.15.28:0/902097652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fdba8003970 con 0x561f00ec3270 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbba883740 1 -- 172.21.15.28:0/902097652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbac09f6b0 con 0x561f00ec3270 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdb997fa640 1 -- 172.21.15.28:0/902097652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdba8050050 con 0x561f00ec3270 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbba883740 10 client.16356 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.073+0000 7fdbba883740 1 -- 172.21.15.28:0/902097652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561f00af10f0 con 0x561f00ec3270 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdb997fa640 1 -- 172.21.15.28:0/902097652 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbac09f6b0 con 0x561f00ec3270 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdb997fa640 1 -- 172.21.15.28:0/902097652 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fdba8043070 con 0x561f00ec3270 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdbba883740 10 client.16356 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdbba883740 10 client.16356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdbba883740 20 client.16356 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdbba883740 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x561f00ed3710 0x561f00ed5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdbba883740 4 client.16356 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdbba883740 1 -- 172.21.15.28:0/902097652 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f00e2b430 con 0x561f00ed3710 2024-05-09T08:28:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdbb590c640 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x561f00ed3710 0x561f00ed5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdbb590c640 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x561f00ed3710 0x561f00ed5b70 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61ab4c5ae3980e16 server_cookie=ac4274a1cce71911 in_seq=0 out_seq=0 2024-05-09T08:28:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.074+0000 7fdb997fa640 10 client.16356 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdb997fa640 1 -- 172.21.15.28:0/902097652 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f00e2b430 con 0x561f00ed3710 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdb997fa640 10 client.16356 handle_command_reply: tid=0 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 client.16356 shutdown 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 -- 172.21.15.28:0/902097652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00ec3270 msgr2=0x561f00ec0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00ec3270 0x561f00ec0cf0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009cd0 tx=0x7fdba8031af0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 -- 172.21.15.28:0/902097652 shutdown_connections 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x561f00ed3710 0x561f00ed5b70 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00ec3270 0x561f00ec0cf0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 0x561f00ec07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdb997fa640 0 client.16356 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 --2- 172.21.15.28:0/902097652 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00eb8280 0x561f00ec0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 -- 172.21.15.28:0/902097652 >> 172.21.15.28:0/902097652 conn(0x561f00eb45a0 msgr2=0x561f00eba3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 -- 172.21.15.28:0/902097652 shutdown_connections 2024-05-09T08:28:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 1 -- 172.21.15.28:0/902097652 wait complete. 2024-05-09T08:28:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.078+0000 7fdbba883740 20 client.16356 trim_cache size 0 max 16384 2024-05-09T08:28:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbba883740 1 Processor -- start 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbba883740 1 -- start start 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbba883740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbba883740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 0x561f00ec0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbba883740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 0x561f00eba9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbba883740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb01aef20 con 0x561f00eb8280 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbba883740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb01b0b20 con 0x561f00eb8ba0 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbba883740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb01af5f0 con 0x561f00ec3270 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb590c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb590c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33632/0 (socket says 172.21.15.28:33632) 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb590c640 1 -- 172.21.15.28:0/408870289 learned_addr learned my addr 172.21.15.28:0/408870289 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb510b640 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 0x561f00ec0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb590c640 1 -- 172.21.15.28:0/408870289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 msgr2=0x561f00eba9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb610d640 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 0x561f00eba9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb590c640 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 0x561f00eba9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb590c640 1 -- 172.21.15.28:0/408870289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 msgr2=0x561f00ec0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb590c640 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 0x561f00ec0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb590c640 1 -- 172.21.15.28:0/408870289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f00ec59d0 con 0x561f00eb8280 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.097+0000 7fdbb510b640 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 0x561f00ec0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbb590c640 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb3f60 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0995b0 tx=0x7fdbac005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd23e4a60a63d92f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdb9b7fe640 1 -- 172.21.15.28:0/408870289 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac09f6b0 con 0x561f00eb8280 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdb9b7fe640 1 -- 172.21.15.28:0/408870289 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbac097970 con 0x561f00eb8280 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdb9b7fe640 1 -- 172.21.15.28:0/408870289 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac0bb330 con 0x561f00eb8280 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbba883740 1 -- 172.21.15.28:0/408870289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 msgr2=0x561f00eb3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbba883740 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb3f60 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0995b0 tx=0x7fdbac005bf0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbba883740 1 -- 172.21.15.28:0/408870289 shutdown_connections 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbba883740 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 0x561f00eba9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbba883740 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8ba0 0x561f00ec0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbba883740 1 --2- 172.21.15.28:0/408870289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8280 0x561f00eb3f60 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbba883740 1 -- 172.21.15.28:0/408870289 >> 172.21.15.28:0/408870289 conn(0x561f00eb45a0 msgr2=0x561f00ec1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbba883740 1 -- 172.21.15.28:0/408870289 shutdown_connections 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.098+0000 7fdbba883740 1 -- 172.21.15.28:0/408870289 wait complete. 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbba883740 1 -- start start 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbba883740 1 Processor -- start 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbba883740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8280 0x561f00ecb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbba883740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8ba0 0x561f00ecb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbba883740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 0x561f00ec8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbba883740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb01b0390 con 0x561f00eb8ba0 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbba883740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb01afcc0 con 0x561f00eb8280 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbba883740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb01acf00 con 0x561f00ec3270 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbba883740 10 client.0 _ll_register_callbacks cb 0x7fdba8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbb510b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8ba0 0x561f00ecb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbb590c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8280 0x561f00ecb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbb510b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8ba0 0x561f00ecb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33642/0 (socket says 172.21.15.28:33642) 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.099+0000 7fdbb510b640 1 -- 172.21.15.28:0/4079720991 learned_addr learned my addr 172.21.15.28:0/4079720991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbb610d640 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 0x561f00ec8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbb510b640 1 -- 172.21.15.28:0/4079720991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 msgr2=0x561f00ec8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbb510b640 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 0x561f00ec8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbb510b640 1 -- 172.21.15.28:0/4079720991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8280 msgr2=0x561f00ecb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbb510b640 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8280 0x561f00ecb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbb510b640 1 -- 172.21.15.28:0/4079720991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f00ec8bc0 con 0x561f00eb8ba0 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbb590c640 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8280 0x561f00ecb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbb610d640 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 0x561f00ec8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbb510b640 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8ba0 0x561f00ecb9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdba00099c0 tx=0x7fdba0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25a18bae8b63f50 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdb997fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdb997fa640 1 -- 172.21.15.28:0/4079720991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba002d060 con 0x561f00eb8ba0 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdb997fa640 1 -- 172.21.15.28:0/4079720991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fdba000e8f0 con 0x561f00eb8ba0 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbba883740 1 -- 172.21.15.28:0/4079720991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbac097970 con 0x561f00eb8ba0 2024-05-09T08:28:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdb997fa640 1 -- 172.21.15.28:0/4079720991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdba0034050 con 0x561f00eb8ba0 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbba883740 10 client.16362 fetch_fsmap learned FSMap version 30 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdbba883740 1 -- 172.21.15.28:0/4079720991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561f00f3a850 con 0x561f00eb8ba0 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.100+0000 7fdb997fa640 1 -- 172.21.15.28:0/4079720991 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbac097970 con 0x561f00eb8ba0 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdb997fa640 1 -- 172.21.15.28:0/4079720991 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fdba0028c10 con 0x561f00eb8ba0 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdbba883740 10 client.16362 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdbba883740 10 client.16362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdbba883740 20 client.16362 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdbba883740 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x561f00f3b0f0 0x561f00f3d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdbba883740 4 client.16362 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdbba883740 1 -- 172.21.15.28:0/4079720991 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561f00e25780 con 0x561f00f3b0f0 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdbb590c640 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x561f00f3b0f0 0x561f00f3d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdb997fa640 10 client.16362 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.101+0000 7fdbb590c640 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x561f00f3b0f0 0x561f00f3d510 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=322963cbae08b213 server_cookie=df2b995fd219eb28 in_seq=0 out_seq=0 2024-05-09T08:28:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdb997fa640 1 -- 172.21.15.28:0/4079720991 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x561f00e25780 con 0x561f00f3b0f0 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdb997fa640 10 client.16362 handle_command_reply: tid=0 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 client.16362 shutdown 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 -- 172.21.15.28:0/4079720991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8ba0 msgr2=0x561f00ecb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8ba0 0x561f00ecb9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdba00099c0 tx=0x7fdba0004730 comp rx=0 tx=0).stop 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 -- 172.21.15.28:0/4079720991 shutdown_connections 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x561f00f3b0f0 0x561f00f3d510 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x561f00ec3270 0x561f00ec8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f00eb8ba0 0x561f00ecb9c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 --2- 172.21.15.28:0/4079720991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561f00eb8280 0x561f00ecb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdb997fa640 0 client.16362 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 -- 172.21.15.28:0/4079720991 >> 172.21.15.28:0/4079720991 conn(0x561f00eb45a0 msgr2=0x561f00ec0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 -- 172.21.15.28:0/4079720991 shutdown_connections 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 1 -- 172.21.15.28:0/4079720991 wait complete. 2024-05-09T08:28:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbba883740 20 client.16362 trim_cache size 0 max 16384 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (27 inodes in the stack)", 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout: "6e261940-0110-44b1-9fb5-ddd00a0e95a4": { 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "6e261940-0110-44b1-9fb5-ddd00a0e95a4", 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:28:04.109 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:28:04.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbb7b70640 1 -- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb9c0a2d00 msgr2=0x7fdb9c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb9c0a2d00 0x7fdb9c0a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdba00095f0 tx=0x7fdba0005f90 comp rx=0 tx=0).stop 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbb7b70640 1 -- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0101800 msgr2=0x7fdbb0198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0101800 0x7fdbb0198140 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00d950 tx=0x7fdbac00de10 comp rx=0 tx=0).stop 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.105+0000 7fdbb7b70640 1 -- 172.21.15.28:0/501292313 shutdown_connections 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.106+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb9c0a2d00 0x7fdb9c0a51c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.106+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbb0107810 0x7fdbb019c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.106+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbb01021c0 0x7fdbb0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.106+0000 7fdbb7b70640 1 --2- 172.21.15.28:0/501292313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0101800 0x7fdbb0198140 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.106+0000 7fdbb7b70640 1 -- 172.21.15.28:0/501292313 >> 172.21.15.28:0/501292313 conn(0x7fdbb0078c80 msgr2=0x7fdbb00ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.106+0000 7fdbb7b70640 1 -- 172.21.15.28:0/501292313 shutdown_connections 2024-05-09T08:28:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:04.106+0000 7fdbb7b70640 1 -- 172.21.15.28:0/501292313 wait complete. 2024-05-09T08:28:04.116 INFO:tasks.cephfs.filesystem:scrub status for tag:6e261940-0110-44b1-9fb5-ddd00a0e95a4 - {'path': '/', 'tag': '6e261940-0110-44b1-9fb5-ddd00a0e95a4', 'options': 'recursive,force'} 2024-05-09T08:28:04.311 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:03.954+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000207 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.in auth v12 ap=1 s=9817 n(v0 rc2024-05-09T08:27:39.663178+0000 b9817 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c79c5a5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000207:[<0x10000000204/Makefile.in v12>,<0x10000000201/man v1450>,<0x10000000006/blogbench-1.0 v1456>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v14>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v41>,<0x1/volumes v39>]//[3]","memoryvalue":"(4)0x10000000207:[<0x10000000204/Makefile.in v12>,<0x10000000201/man v1943>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.312 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:03.954+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000205 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/blogbench.8 auth v10 ap=1 s=964 n(v0 rc2024-05-09T08:27:39.661178+0000 b964 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d748580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000205:[<0x10000000204/blogbench.8 v10>,<0x10000000201/man v1450>,<0x10000000006/blogbench-1.0 v1456>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v14>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v41>,<0x1/volumes v39>]//[3]","memoryvalue":"(4)0x10000000205:[<0x10000000204/blogbench.8 v10>,<0x10000000201/man v1943>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.312 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:03.954+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000206 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.am auth v8 ap=1 s=55 n(v0 rc2024-05-09T08:27:39.662178+0000 b55 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c79c5a5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000206:[<0x10000000204/Makefile.am v8>,<0x10000000201/man v1450>,<0x10000000006/blogbench-1.0 v1456>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v14>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v41>,<0x1/volumes v39>]//[3]","memoryvalue":"(4)0x10000000206:[<0x10000000204/Makefile.am v8>,<0x10000000201/man v1943>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.312 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.311+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000203 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/m4/acx_pthread.m4 auth v4 ap=1 s=9243 n(v0 rc2024-05-09T08:27:39.659178+0000 b9243 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d2ac680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000203:[<0x10000000202/acx_pthread.m4 v4>,<0x10000000201/m4 v1414>,<0x10000000006/blogbench-1.0 v1456>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v14>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v30>,<0x10000000001/sv_1 v53>,<0x10000000000/qa v41>,<0x1/volumes v39>]//[3]","memoryvalue":"(4)0x10000000203:[<0x10000000202/acx_pthread.m4 v4>,<0x10000000201/m4 v1907>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.607 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.367+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c1b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-59.xml auth v220 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.120680+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7beb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c1b:[<0x10000000ba3/article-59.xml v220>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c1b:[<0x10000000ba3/article-59.xml v220>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.607 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.368+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bae [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-92.xml auth v22 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.793689+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7c1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bae:[<0x10000000ba3/article-92.xml v22>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bae:[<0x10000000ba3/article-92.xml v22>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.608 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.386+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dde [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-76.xml auth v16 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.547639+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@40},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea5a4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dde:[<0x10000000dd1/article-76.xml v16>,<0x10000000399/blog-15 v20287>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dde:[<0x10000000dd1/article-76.xml v16>,<0x10000000399/blog-15 v22907>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.608 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.393+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dd4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-78.xml auth v4 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.502640+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@40},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea5bb080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dd4:[<0x10000000dd1/article-78.xml v4>,<0x10000000399/blog-15 v20287>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dd4:[<0x10000000dd1/article-78.xml v4>,<0x10000000399/blog-15 v22907>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.367+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007d3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-72.xml auth v939 ap=1 s=18181 n(v0 rc2024-05-09T08:27:52.159821+0000 b18181 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2248580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007d3:[<0x100000007b9/article-72.xml v939>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007d3:[<0x100000007b9/article-72.xml v939>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.368+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000800 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-35.xml auth v951 ap=1 s=11196 n(v0 rc2024-05-09T08:27:52.292817+0000 b11196 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2249600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000800:[<0x100000007b9/article-35.xml v951>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000800:[<0x100000007b9/article-35.xml v951>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.608 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.368+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c9a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-18.xml auth v466 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.554667+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d2feb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c9a:[<0x10000000ba3/article-18.xml v466>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c9a:[<0x10000000ba3/article-18.xml v466>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.609 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.369+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bed [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-74.xml auth v130 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.972684+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d2fe580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bed:[<0x10000000ba3/article-74.xml v130>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bed:[<0x10000000ba3/article-74.xml v130>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.609 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.368+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000830 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-75.jpg auth v959 ap=1 s=30518 n(v0 rc2024-05-09T08:27:52.467812+0000 b30518 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2249b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000830:[<0x100000007b9/picture-75.jpg v959>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000830:[<0x100000007b9/picture-75.jpg v959>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.609 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.368+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007ba [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-97.xml auth v937 ap=1 s=4630 n(v0 rc2024-05-09T08:27:52.083823+0000 b4630 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007ba:[<0x100000007b9/article-97.xml v937>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007ba:[<0x100000007b9/article-97.xml v937>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.609 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.411+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dd8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-77.xml auth v10 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.524639+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@41},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea5e7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dd8:[<0x10000000dd1/article-77.xml v10>,<0x10000000399/blog-15 v20287>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dd8:[<0x10000000dd1/article-77.xml v10>,<0x10000000399/blog-15 v24697>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.609 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.412+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000404 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-64.xml auth v999 ap=1 s=23496 n(v0 rc2024-05-09T08:27:48.875915+0000 b23496 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea5f5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000404:[<0x1000000039a/article-64.xml v999>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000404:[<0x1000000039a/article-64.xml v999>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.610 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.370+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d12 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-35.jpg auth v598 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.906657+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d300c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d12:[<0x10000000ba3/picture-35.jpg v598>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d12:[<0x10000000ba3/picture-35.jpg v598>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.610 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.370+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c36 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-51.xml auth v268 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.200677+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d2ffb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c36:[<0x10000000ba3/article-51.xml v268>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c36:[<0x10000000ba3/article-51.xml v268>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.610 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.369+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007fb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-40.xml auth v953 ap=1 s=24238 n(v0 rc2024-05-09T08:27:52.277818+0000 b24238 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007fb:[<0x100000007b9/article-40.xml v953>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007fb:[<0x100000007b9/article-40.xml v953>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.610 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.369+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000832 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-73.jpg auth v957 ap=1 s=37816 n(v0 rc2024-05-09T08:27:52.472812+0000 b37816 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2248b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000832:[<0x100000007b9/picture-73.jpg v957>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000832:[<0x100000007b9/picture-73.jpg v957>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.610 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.369+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009a1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-21.jpg auth v969 ap=1 s=35006 n(v0 rc2024-05-09T08:27:53.723777+0000 b35006 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2249080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009a1:[<0x100000007b9/picture-21.jpg v969>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009a1:[<0x100000007b9/picture-21.jpg v969>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.611 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.412+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dce [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-13.xml auth v1291 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.473641+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea361600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dce:[<0x1000000039a/comment-13.xml v1291>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dce:[<0x1000000039a/comment-13.xml v1291>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.611 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.412+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003be [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-87.xml auth v1025 ap=1 s=22494 n(v0 rc2024-05-09T08:27:48.763918+0000 b22494 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea5f5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003be:[<0x1000000039a/article-87.xml v1025>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003be:[<0x1000000039a/article-87.xml v1025>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.611 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.412+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d92 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-21.xml auth v1263 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.213648+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea361b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d92:[<0x1000000039a/comment-21.xml v1263>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d92:[<0x1000000039a/comment-21.xml v1263>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.611 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.370+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c7e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-27.xml auth v412 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.445670+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d300100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c7e:[<0x10000000ba3/article-27.xml v412>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c7e:[<0x10000000ba3/article-27.xml v412>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.611 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.370+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bf9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-70.xml auth v154 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.008683+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d301700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bf9:[<0x10000000ba3/article-70.xml v154>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bf9:[<0x10000000ba3/article-70.xml v154>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.611 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.371+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c93 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-20.xml auth v454 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.534668+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7cc000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c93:[<0x10000000ba3/article-20.xml v454>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c93:[<0x10000000ba3/article-20.xml v454>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.612 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.413+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d75 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-26.xml auth v1255 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.098652+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea362680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d75:[<0x1000000039a/comment-26.xml v1255>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d75:[<0x1000000039a/comment-26.xml v1255>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.612 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.413+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-75.xml auth v1015 ap=1 s=17929 n(v0 rc2024-05-09T08:27:48.822917+0000 b17929 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea5f4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e3:[<0x1000000039a/article-75.xml v1015>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003e3:[<0x1000000039a/article-75.xml v1015>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.369+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007c5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-86.xml auth v943 ap=1 s=23278 n(v0 rc2024-05-09T08:27:52.114823+0000 b23278 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007c5:[<0x100000007b9/article-86.xml v943>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007c5:[<0x100000007b9/article-86.xml v943>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.370+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007c9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-82.xml auth v941 ap=1 s=25886 n(v0 rc2024-05-09T08:27:52.125822+0000 b25886 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007c9:[<0x100000007b9/article-82.xml v941>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007c9:[<0x100000007b9/article-82.xml v941>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.613 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.371+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c15 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-61.xml auth v208 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.102680+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7cd080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c15:[<0x10000000ba3/article-61.xml v208>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c15:[<0x10000000ba3/article-61.xml v208>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.613 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.371+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bcf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-83.xml auth v76 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.891686+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7ccb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bcf:[<0x10000000ba3/article-83.xml v76>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bcf:[<0x10000000ba3/article-83.xml v76>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.613 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.413+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000044c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-37.xml auth v1011 ap=1 s=28278 n(v0 rc2024-05-09T08:27:49.032911+0000 b28278 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea5f3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000044c:[<0x1000000039a/article-37.xml v1011>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000044c:[<0x1000000039a/article-37.xml v1011>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.613 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.413+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008d7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-27.jpg auth v1027 ap=1 s=29004 n(v0 rc2024-05-09T08:27:53.320788+0000 b29004 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea5f3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008d7:[<0x1000000039a/picture-27.jpg v1027>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008d7:[<0x1000000039a/picture-27.jpg v1027>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.614 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.370+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000971 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-30.jpg auth v961 ap=1 s=56898 n(v0 rc2024-05-09T08:27:53.548782+0000 b56898 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000971:[<0x100000007b9/picture-30.jpg v961>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000971:[<0x100000007b9/picture-30.jpg v961>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.614 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.370+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000857 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-36.jpg auth v963 ap=1 s=21022 n(v0 rc2024-05-09T08:27:52.584809+0000 b21022 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22a8580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000857:[<0x100000007b9/picture-36.jpg v963>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000857:[<0x100000007b9/picture-36.jpg v963>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.614 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.371+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c54 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-41.xml auth v328 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.304674+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7cd600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c54:[<0x10000000ba3/article-41.xml v328>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c54:[<0x10000000ba3/article-41.xml v328>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.614 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.372+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c48 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-45.xml auth v304 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.265675+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7cdb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c48:[<0x10000000ba3/article-45.xml v304>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c48:[<0x10000000ba3/article-45.xml v304>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.615 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.413+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-73.xml auth v1007 ap=1 s=3869 n(v0 rc2024-05-09T08:27:48.832916+0000 b3869 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3d0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e9:[<0x1000000039a/article-73.xml v1007>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003e9:[<0x1000000039a/article-73.xml v1007>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.615 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.413+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000447 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-39.xml auth v991 ap=1 s=24289 n(v0 rc2024-05-09T08:27:49.012911+0000 b24289 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3d0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000447:[<0x1000000039a/article-39.xml v991>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000447:[<0x1000000039a/article-39.xml v991>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.615 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.414+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000044e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-36.xml auth v1017 ap=1 s=4001 n(v0 rc2024-05-09T08:27:49.040910+0000 b4001 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3d0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000044e:[<0x1000000039a/article-36.xml v1017>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000044e:[<0x1000000039a/article-36.xml v1017>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.615 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.371+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000810 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-19.xml auth v949 ap=1 s=18405 n(v0 rc2024-05-09T08:27:52.352816+0000 b18405 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22a9080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000810:[<0x100000007b9/article-19.xml v949>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000810:[<0x100000007b9/article-19.xml v949>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.616 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.371+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000855 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-38.jpg auth v965 ap=1 s=30772 n(v0 rc2024-05-09T08:27:52.578809+0000 b30772 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22aa680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000855:[<0x100000007b9/picture-38.jpg v965>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000855:[<0x100000007b9/picture-38.jpg v965>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.616 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.372+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009a4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-20.jpg auth v971 ap=1 s=18478 n(v0 rc2024-05-09T08:27:53.731776+0000 b18478 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22a9600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009a4:[<0x100000007b9/picture-20.jpg v971>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009a4:[<0x100000007b9/picture-20.jpg v971>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.616 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.372+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bfd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-69.xml auth v160 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.020682+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7ce100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bfd:[<0x10000000ba3/article-69.xml v160>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bfd:[<0x10000000ba3/article-69.xml v160>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.616 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.372+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c72 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-31.xml auth v388 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.404671+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7cec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c72:[<0x10000000ba3/article-31.xml v388>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c72:[<0x10000000ba3/article-31.xml v388>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.616 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.373+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d66 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-11.xml auth v642 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.082652+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7cf700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d66:[<0x10000000ba3/comment-11.xml v642>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d66:[<0x10000000ba3/comment-11.xml v642>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.617 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.414+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ed [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-52.jpg auth v1029 ap=1 s=13282 n(v0 rc2024-05-09T08:27:49.581895+0000 b13282 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3d1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ed:[<0x1000000039a/picture-52.jpg v1029>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004ed:[<0x1000000039a/picture-52.jpg v1029>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.617 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.414+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000491 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-12.xml auth v985 ap=1 s=27084 n(v0 rc2024-05-09T08:27:49.266904+0000 b27084 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3d1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000491:[<0x1000000039a/article-12.xml v985>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000491:[<0x1000000039a/article-12.xml v985>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.618 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.372+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007f9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-42.xml auth v955 ap=1 s=19037 n(v0 rc2024-05-09T08:27:52.271818+0000 b19037 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22aa100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007f9:[<0x100000007b9/article-42.xml v955>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007f9:[<0x100000007b9/article-42.xml v955>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.618 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.372+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007c2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-89.xml auth v945 ap=1 s=26333 n(v0 rc2024-05-09T08:27:52.106823+0000 b26333 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22a9b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007c2:[<0x100000007b9/article-89.xml v945>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007c2:[<0x100000007b9/article-89.xml v945>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.618 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.373+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bd9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-80.xml auth v94 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.921685+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7d4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bd9:[<0x10000000ba3/article-80.xml v94>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bd9:[<0x10000000ba3/article-80.xml v94>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.619 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.373+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cc9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-9.xml auth v520 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.699663+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7d5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cc9:[<0x10000000ba3/article-9.xml v520>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cc9:[<0x10000000ba3/article-9.xml v520>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.619 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.372+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000082c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-79.jpg auth v974 ap=1 s=53636 n(v0 rc2024-05-09T08:27:52.455813+0000 b53636 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000082c:[<0x100000007b9/picture-79.jpg v974>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000082c:[<0x100000007b9/picture-79.jpg v974>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.619 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.373+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000099a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-23.jpg auth v994 ap=1 s=51977 n(v0 rc2024-05-09T08:27:53.703777+0000 b51977 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da208bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000099a:[<0x100000007b9/picture-23.jpg v994>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000099a:[<0x100000007b9/picture-23.jpg v994>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.619 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.414+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004b2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-1.xml auth v1013 ap=1 s=10969 n(v0 rc2024-05-09T08:27:49.380901+0000 b10969 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3d1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004b2:[<0x1000000039a/article-1.xml v1013>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004b2:[<0x1000000039a/article-1.xml v1013>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.620 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.415+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004c7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-67.jpg auth v995 ap=1 s=30770 n(v0 rc2024-05-09T08:27:49.451899+0000 b30770 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3d3180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004c7:[<0x1000000039a/picture-67.jpg v995>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004c7:[<0x1000000039a/picture-67.jpg v995>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.620 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.415+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000da9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-18.xml auth v1278 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.306646+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3da000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000da9:[<0x1000000039a/comment-18.xml v1278>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000da9:[<0x1000000039a/comment-18.xml v1278>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.620 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.374+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c09 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-65.xml auth v184 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.068681+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7d6100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c09:[<0x10000000ba3/article-65.xml v184>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c09:[<0x10000000ba3/article-65.xml v184>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.620 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.374+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c12 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-62.xml auth v202 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.093680+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7d6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c12:[<0x10000000ba3/article-62.xml v202>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c12:[<0x10000000ba3/article-62.xml v202>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.621 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.416+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ea [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-53.jpg auth v1009 ap=1 s=61099 n(v0 rc2024-05-09T08:27:49.570895+0000 b61099 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3d2c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ea:[<0x1000000039a/picture-53.jpg v1009>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004ea:[<0x1000000039a/picture-53.jpg v1009>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.621 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.416+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000075f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-34.jpg auth v1037 ap=1 s=13017 n(v0 rc2024-05-09T08:27:52.196820+0000 b13017 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3d2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000075f:[<0x1000000039a/picture-34.jpg v1037>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000075f:[<0x1000000039a/picture-34.jpg v1037>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.621 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.373+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000084e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-45.jpg auth v990 ap=1 s=58025 n(v0 rc2024-05-09T08:27:52.557810+0000 b58025 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000084e:[<0x100000007b9/picture-45.jpg v990>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000084e:[<0x100000007b9/picture-45.jpg v990>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.622 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.374+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000852 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-41.jpg auth v986 ap=1 s=34469 n(v0 rc2024-05-09T08:27:52.568810+0000 b34469 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000852:[<0x100000007b9/picture-41.jpg v986>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000852:[<0x100000007b9/picture-41.jpg v986>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.622 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.375+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000850 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-43.jpg auth v967 ap=1 s=13637 n(v0 rc2024-05-09T08:27:52.562810+0000 b13637 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22a8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000850:[<0x100000007b9/picture-43.jpg v967>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000850:[<0x100000007b9/picture-43.jpg v967>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.622 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.374+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c88 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-24.xml auth v430 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.491669+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7d6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c88:[<0x10000000ba3/article-24.xml v430>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c88:[<0x10000000ba3/article-24.xml v430>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.622 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.375+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c4a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-44.xml auth v310 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.275675+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7d5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c4a:[<0x10000000ba3/article-44.xml v310>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c4a:[<0x10000000ba3/article-44.xml v310>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.623 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.375+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000da4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-28.jpg auth v682 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.311645+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7d7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000da4:[<0x10000000ba3/picture-28.jpg v682>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000da4:[<0x10000000ba3/picture-28.jpg v682>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.623 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.416+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bef [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-16.jpg auth v1081 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.988683+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3dab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bef:[<0x1000000039a/picture-16.jpg v1081>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bef:[<0x1000000039a/picture-16.jpg v1081>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.623 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.416+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003fd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-66.xml auth v987 ap=1 s=9631 n(v0 rc2024-05-09T08:27:48.866915+0000 b9631 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3da580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003fd:[<0x1000000039a/article-66.xml v987>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003fd:[<0x1000000039a/article-66.xml v987>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.623 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.375+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007c7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-84.xml auth v942 ap=1 s=27145 n(v0 rc2024-05-09T08:27:52.120822+0000 b27145 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f4580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007c7:[<0x100000007b9/article-84.xml v942>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007c7:[<0x100000007b9/article-84.xml v942>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.624 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.376+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000815 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-14.xml auth v946 ap=1 s=30305 n(v0 rc2024-05-09T08:27:52.377815+0000 b30305 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000815:[<0x100000007b9/article-14.xml v946>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000815:[<0x100000007b9/article-14.xml v946>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.624 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.375+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bca [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-85.xml auth v64 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.876686+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7d7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bca:[<0x10000000ba3/article-85.xml v64>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bca:[<0x10000000ba3/article-85.xml v64>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.626 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.375+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bbf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-88.xml auth v46 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.852687+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d80a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bbf:[<0x10000000ba3/article-88.xml v46>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bbf:[<0x10000000ba3/article-88.xml v46>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.626 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.376+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000083f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-60.jpg auth v968 ap=1 s=31177 n(v0 rc2024-05-09T08:27:52.512811+0000 b31177 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000083f:[<0x100000007b9/picture-60.jpg v968>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000083f:[<0x100000007b9/picture-60.jpg v968>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.626 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.376+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009df [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-46.xml auth v1013 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.875744+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009df:[<0x100000007b9/article-46.xml v1013>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009df:[<0x100000007b9/article-46.xml v1013>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.627 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.375+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c03 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-67.xml auth v172 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.050682+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d80a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c03:[<0x10000000ba3/article-67.xml v172>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c03:[<0x10000000ba3/article-67.xml v172>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.627 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.376+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ce6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-3.xml auth v556 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.760661+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d80bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ce6:[<0x10000000ba3/article-3.xml v556>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ce6:[<0x10000000ba3/article-3.xml v556>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.627 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.417+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003dc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-77.xml auth v1005 ap=1 s=13736 n(v0 rc2024-05-09T08:27:48.813917+0000 b13736 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3db080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003dc:[<0x1000000039a/article-77.xml v1005>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003dc:[<0x1000000039a/article-77.xml v1005>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.627 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.417+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dd3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-12.xml auth v1298 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.496640+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3db600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dd3:[<0x1000000039a/comment-12.xml v1298>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dd3:[<0x1000000039a/comment-12.xml v1298>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.628 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.418+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cad [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-49.xml auth v1168 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.636665+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3dbb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cad:[<0x1000000039a/comment-49.xml v1168>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cad:[<0x1000000039a/comment-49.xml v1168>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.628 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.376+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007c0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-91.xml auth v944 ap=1 s=22001 n(v0 rc2024-05-09T08:27:52.100823+0000 b22001 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007c0:[<0x100000007b9/article-91.xml v944>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007c0:[<0x100000007b9/article-91.xml v944>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.628 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.377+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000839 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-66.jpg auth v970 ap=1 s=21106 n(v0 rc2024-05-09T08:27:52.494812+0000 b21106 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000839:[<0x100000007b9/picture-66.jpg v970>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000839:[<0x100000007b9/picture-66.jpg v970>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.628 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.377+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007d4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-71.xml auth v940 ap=1 s=23446 n(v0 rc2024-05-09T08:27:52.161821+0000 b23446 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007d4:[<0x100000007b9/article-71.xml v940>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007d4:[<0x100000007b9/article-71.xml v940>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.629 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.376+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cfb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-37.jpg auth v580 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.812660+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d80b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cfb:[<0x10000000ba3/picture-37.jpg v580>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cfb:[<0x10000000ba3/picture-37.jpg v580>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.629 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.376+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cd3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-7.xml auth v532 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.719662+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d80cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cd3:[<0x10000000ba3/article-7.xml v532>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cd3:[<0x10000000ba3/article-7.xml v532>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.629 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.377+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c2a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-55.xml auth v244 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.160678+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d80d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c2a:[<0x10000000ba3/article-55.xml v244>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c2a:[<0x10000000ba3/article-55.xml v244>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.630 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.377+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000082b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-80.jpg auth v976 ap=1 s=50704 n(v0 rc2024-05-09T08:27:52.452813+0000 b50704 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f6100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000082b:[<0x100000007b9/picture-80.jpg v976>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000082b:[<0x100000007b9/picture-80.jpg v976>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.630 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.377+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007f4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-47.xml auth v958 ap=1 s=23140 n(v0 rc2024-05-09T08:27:52.256818+0000 b23140 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@8},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007f4:[<0x100000007b9/article-47.xml v958>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007f4:[<0x100000007b9/article-47.xml v958>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.630 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.418+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000755 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-87.xml auth v1035 ap=1 s=4504 n(v0 rc2024-05-09T08:27:51.818831+0000 b4504 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3dc100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000755:[<0x1000000039a/comment-87.xml v1035>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000755:[<0x1000000039a/comment-87.xml v1035>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.631 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.419+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cf3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-42.xml auth v1202 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.792660+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3dc680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cf3:[<0x1000000039a/comment-42.xml v1202>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cf3:[<0x1000000039a/comment-42.xml v1202>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.631 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.377+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ba5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-95.xml auth v4 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.745690+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d814000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ba5:[<0x10000000ba3/article-95.xml v4>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ba5:[<0x10000000ba3/article-95.xml v4>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.631 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.377+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ccf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-8.xml auth v526 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.710663+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d814580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ccf:[<0x10000000ba3/article-8.xml v526>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ccf:[<0x10000000ba3/article-8.xml v526>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.631 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.377+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000099e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-22.jpg auth v996 ap=1 s=36551 n(v0 rc2024-05-09T08:27:53.714777+0000 b36551 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000099e:[<0x100000007b9/picture-22.jpg v996>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000099e:[<0x100000007b9/picture-22.jpg v996>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.632 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.378+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000813 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-16.xml auth v948 ap=1 s=31710 n(v0 rc2024-05-09T08:27:52.368815+0000 b31710 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21fe000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000813:[<0x100000007b9/article-16.xml v948>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000813:[<0x100000007b9/article-16.xml v948>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.632 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.419+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000039c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-98.xml auth v983 ap=1 s=26029 n(v0 rc2024-05-09T08:27:48.706920+0000 b26029 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3dcc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000039c:[<0x1000000039a/article-98.xml v983>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000039c:[<0x1000000039a/article-98.xml v983>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.632 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.419+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c0b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-65.xml auth v1102 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.087680+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3dd180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c0b:[<0x1000000039a/comment-65.xml v1102>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c0b:[<0x1000000039a/comment-65.xml v1102>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.633 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.378+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000be7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-76.xml auth v118 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.956684+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d815600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000be7:[<0x10000000ba3/article-76.xml v118>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000be7:[<0x10000000ba3/article-76.xml v118>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.633 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.378+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d6c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-31.jpg auth v646 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.095652+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d814b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d6c:[<0x10000000ba3/picture-31.jpg v646>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d6c:[<0x10000000ba3/picture-31.jpg v646>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.633 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.378+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007e8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-54.xml auth v962 ap=1 s=16589 n(v0 rc2024-05-09T08:27:52.221819+0000 b16589 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21fe580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007e8:[<0x100000007b9/article-54.xml v962>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007e8:[<0x100000007b9/article-54.xml v962>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.634 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.378+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000809 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-26.xml auth v952 ap=1 s=20795 n(v0 rc2024-05-09T08:27:52.317817+0000 b20795 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21feb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000809:[<0x100000007b9/article-26.xml v952>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000809:[<0x100000007b9/article-26.xml v952>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.634 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.378+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009b6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-15.jpg auth v1000 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.774747+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21ff080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009b6:[<0x100000007b9/picture-15.jpg v1000>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009b6:[<0x100000007b9/picture-15.jpg v1000>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.634 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.419+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000473 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-23.xml auth v1023 ap=1 s=20055 n(v0 rc2024-05-09T08:27:49.162907+0000 b20055 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3dd700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000473:[<0x1000000039a/article-23.xml v1023>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000473:[<0x1000000039a/article-23.xml v1023>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.634 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.420+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004f8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-100.xml auth v1001 ap=1 s=7919 n(v0 rc2024-05-09T08:27:49.625894+0000 b7919 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004f8:[<0x1000000039a/comment-100.xml v1001>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004f8:[<0x1000000039a/comment-100.xml v1001>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.635 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.420+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000482 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-17.xml auth v989 ap=1 s=6983 n(v0 rc2024-05-09T08:27:49.212905+0000 b6983 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e4580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000482:[<0x1000000039a/article-17.xml v989>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000482:[<0x1000000039a/article-17.xml v989>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.635 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.378+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bff [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-68.xml auth v166 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.037682+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d815080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bff:[<0x10000000ba3/article-68.xml v166>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bff:[<0x10000000ba3/article-68.xml v166>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.635 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.381+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d41 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-33.jpg auth v622 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.002654+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d80c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d41:[<0x10000000ba3/picture-33.jpg v622>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d41:[<0x10000000ba3/picture-33.jpg v622>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.635 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.381+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dcb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-26.jpg auth v706 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.501640+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d816100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dcb:[<0x10000000ba3/picture-26.jpg v706>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dcb:[<0x10000000ba3/picture-26.jpg v706>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.636 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.378+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007be [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-93.xml auth v938 ap=1 s=3968 n(v0 rc2024-05-09T08:27:52.094823+0000 b3968 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21ff600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007be:[<0x100000007b9/article-93.xml v938>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007be:[<0x100000007b9/article-93.xml v938>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.636 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.379+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000843 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-56.jpg auth v964 ap=1 s=62531 n(v0 rc2024-05-09T08:27:52.524811+0000 b62531 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21ffb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000843:[<0x100000007b9/picture-56.jpg v964>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000843:[<0x100000007b9/picture-56.jpg v964>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.636 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.420+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000085a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-32.jpg auth v1039 ap=1 s=10734 n(v0 rc2024-05-09T08:27:52.615808+0000 b10734 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000085a:[<0x1000000039a/picture-32.jpg v1039>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000085a:[<0x1000000039a/picture-32.jpg v1039>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.637 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.420+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000406 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-63.xml auth v1021 ap=1 s=23044 n(v0 rc2024-05-09T08:27:48.880915+0000 b23044 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000406:[<0x1000000039a/article-63.xml v1021>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000406:[<0x1000000039a/article-63.xml v1021>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.637 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.381+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c6f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-32.xml auth v382 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.394672+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d816680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c6f:[<0x10000000ba3/article-32.xml v382>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c6f:[<0x10000000ba3/article-32.xml v382>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.637 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.382+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c92 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-21.xml auth v448 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.525668+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d816c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c92:[<0x10000000ba3/article-21.xml v448>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c92:[<0x10000000ba3/article-21.xml v448>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.638 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.379+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009d7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-47.jpg auth v1008 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.844745+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2200100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009d7:[<0x100000007b9/picture-47.jpg v1008>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009d7:[<0x100000007b9/picture-47.jpg v1008>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.638 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.379+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000818 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-11.xml auth v984 ap=1 s=26000 n(v0 rc2024-05-09T08:27:52.391815+0000 b26000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21f7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000818:[<0x100000007b9/article-11.xml v984>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000818:[<0x100000007b9/article-11.xml v984>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.638 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.420+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004f6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-48.jpg auth v993 ap=1 s=60161 n(v0 rc2024-05-09T08:27:49.610894+0000 b60161 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004f6:[<0x1000000039a/picture-48.jpg v993>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004f6:[<0x1000000039a/picture-48.jpg v993>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.638 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.420+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c99 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-51.xml auth v1148 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.567667+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c99:[<0x1000000039a/comment-51.xml v1148>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c99:[<0x1000000039a/comment-51.xml v1148>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.639 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.421+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000865 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-81.xml auth v1023 ap=1 s=6907 n(v0 rc2024-05-09T08:27:52.659807+0000 b6907 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea715180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000865:[<0x1000000039a/comment-81.xml v1023>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000865:[<0x1000000039a/comment-81.xml v1023>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.639 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.382+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c74 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-30.xml auth v394 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.418671+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d817180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c74:[<0x10000000ba3/article-30.xml v394>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c74:[<0x10000000ba3/article-30.xml v394>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.639 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.382+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ba8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-94.xml auth v10 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.758690+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d817700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ba8:[<0x10000000ba3/article-94.xml v10>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ba8:[<0x10000000ba3/article-94.xml v10>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.640 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.380+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007f2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-48.xml auth v960 ap=1 s=14172 n(v0 rc2024-05-09T08:27:52.250819+0000 b14172 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2201700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007f2:[<0x100000007b9/article-48.xml v960>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007f2:[<0x100000007b9/article-48.xml v960>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.640 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.380+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000081f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-4.xml auth v980 ap=1 s=16276 n(v0 rc2024-05-09T08:27:52.419814+0000 b16276 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000081f:[<0x100000007b9/article-4.xml v980>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000081f:[<0x100000007b9/article-4.xml v980>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.640 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.380+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000819 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-10.xml auth v982 ap=1 s=23755 n(v0 rc2024-05-09T08:27:52.396814+0000 b23755 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2200680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000819:[<0x100000007b9/article-10.xml v982>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000819:[<0x100000007b9/article-10.xml v982>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.640 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.421+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c7d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-3.jpg auth v1146 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.459670+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea522000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c7d:[<0x1000000039a/picture-3.jpg v1146>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c7d:[<0x1000000039a/picture-3.jpg v1146>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.641 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.421+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c84 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-53.xml auth v1154 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.496669+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea522580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c84:[<0x1000000039a/comment-53.xml v1154>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c84:[<0x1000000039a/comment-53.xml v1154>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.641 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.382+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cb1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-13.xml auth v496 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.636665+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d820000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cb1:[<0x10000000ba3/article-13.xml v496>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cb1:[<0x10000000ba3/article-13.xml v496>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.641 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.383+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cd8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-6.xml auth v538 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.732662+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d821080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cd8:[<0x10000000ba3/article-6.xml v538>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cd8:[<0x10000000ba3/article-6.xml v538>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.641 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.381+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009c8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-12.jpg auth v1004 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.811745+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x555da2200c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009c8:[<0x100000007b9/picture-12.jpg v1004>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009c8:[<0x100000007b9/picture-12.jpg v1004>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.641 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.385+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000084f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-44.jpg auth v988 ap=1 s=39675 n(v0 rc2024-05-09T08:27:52.560810+0000 b39675 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2201180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000084f:[<0x100000007b9/picture-44.jpg v988>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000084f:[<0x100000007b9/picture-44.jpg v988>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.642 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.421+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004e4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-55.jpg auth v1017 ap=1 s=45075 n(v0 rc2024-05-09T08:27:49.550896+0000 b45075 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea523080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004e4:[<0x1000000039a/picture-55.jpg v1017>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004e4:[<0x1000000039a/picture-55.jpg v1017>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.642 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.421+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000763 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-85.xml auth v1031 ap=1 s=6798 n(v0 rc2024-05-09T08:27:52.202820+0000 b6798 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea523600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000763:[<0x1000000039a/comment-85.xml v1031>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000763:[<0x1000000039a/comment-85.xml v1031>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.642 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.383+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c39 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-50.xml auth v274 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.210677+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d820b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c39:[<0x10000000ba3/article-50.xml v274>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c39:[<0x10000000ba3/article-50.xml v274>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.642 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.383+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cea [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-2.xml auth v562 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.769661+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d821600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cea:[<0x10000000ba3/article-2.xml v562>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cea:[<0x10000000ba3/article-2.xml v562>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.642 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.384+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bb7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-90.xml auth v34 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.826688+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d822100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bb7:[<0x10000000ba3/article-90.xml v34>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bb7:[<0x10000000ba3/article-90.xml v34>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.643 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.385+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000840 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-59.jpg auth v966 ap=1 s=54316 n(v0 rc2024-05-09T08:27:52.515811+0000 b54316 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000840:[<0x100000007b9/picture-59.jpg v966>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000840:[<0x100000007b9/picture-59.jpg v966>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.643 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.386+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000833 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-72.jpg auth v972 ap=1 s=11178 n(v0 rc2024-05-09T08:27:52.477812+0000 b11178 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000833:[<0x100000007b9/picture-72.jpg v972>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000833:[<0x100000007b9/picture-72.jpg v972>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.643 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.422+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c29 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-11.jpg auth v1116 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.170678+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea523b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c29:[<0x1000000039a/picture-11.jpg v1116>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c29:[<0x1000000039a/picture-11.jpg v1116>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.644 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.422+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004bd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-70.jpg auth v1015 ap=1 s=27186 n(v0 rc2024-05-09T08:27:49.417900+0000 b27186 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea524100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004bd:[<0x1000000039a/picture-70.jpg v1015>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004bd:[<0x1000000039a/picture-70.jpg v1015>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.644 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.422+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-5.xml auth v1001 ap=1 s=30311 n(v0 rc2024-05-09T08:27:49.346902+0000 b30311 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea524680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a6:[<0x1000000039a/article-5.xml v1001>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004a6:[<0x1000000039a/article-5.xml v1001>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.644 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.384+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d29 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-14.xml auth v606 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.941656+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d821b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d29:[<0x10000000ba3/comment-14.xml v606>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d29:[<0x10000000ba3/comment-14.xml v606>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.644 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.384+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d91 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-29.jpg auth v670 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.235648+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d822680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d91:[<0x10000000ba3/picture-29.jpg v670>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d91:[<0x10000000ba3/picture-29.jpg v670>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.644 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.387+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000080a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-25.xml auth v950 ap=1 s=11160 n(v0 rc2024-05-09T08:27:52.320817+0000 b11160 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000080a:[<0x100000007b9/article-25.xml v950>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000080a:[<0x100000007b9/article-25.xml v950>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.644 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.388+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007fd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-38.xml auth v954 ap=1 s=18856 n(v0 rc2024-05-09T08:27:52.283818+0000 b18856 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007fd:[<0x100000007b9/article-38.xml v954>,<0x10000000399/blog-7 v18009>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007fd:[<0x100000007b9/article-38.xml v954>,<0x10000000399/blog-7 v20507>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.645 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.391+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000454 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-12.xml auth v1473 ap=1 s=24963 n(v0 rc2024-05-09T08:27:49.095909+0000 b24963 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2233080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000454:[<0x1000000039b/article-12.xml v1473>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000454:[<0x1000000039b/article-12.xml v1473>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.645 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.422+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000455 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-34.xml auth v995 ap=1 s=13455 n(v0 rc2024-05-09T08:27:49.063910+0000 b13455 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea524c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000455:[<0x1000000039a/article-34.xml v995>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000455:[<0x1000000039a/article-34.xml v995>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.645 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.422+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d33 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-35.xml auth v1222 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.959656+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea525180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d33:[<0x1000000039a/comment-35.xml v1222>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d33:[<0x1000000039a/comment-35.xml v1222>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.645 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.384+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c26 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-56.xml auth v238 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.150679+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d822c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c26:[<0x10000000ba3/article-56.xml v238>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c26:[<0x10000000ba3/article-56.xml v238>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.645 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.384+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c4e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-43.xml auth v316 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.283675+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d823700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c4e:[<0x10000000ba3/article-43.xml v316>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c4e:[<0x10000000ba3/article-43.xml v316>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.646 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.422+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d6a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-28.xml auth v1237 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.072652+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea525700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d6a:[<0x1000000039a/comment-28.xml v1237>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d6a:[<0x1000000039a/comment-28.xml v1237>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.646 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.422+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c5c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-57.xml auth v1129 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.345673+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c5c:[<0x1000000039a/comment-57.xml v1129>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c5c:[<0x1000000039a/comment-57.xml v1129>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.646 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.391+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006d1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-27.jpg auth v1501 ap=1 s=46341 n(v0 rc2024-05-09T08:27:51.391843+0000 b46341 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2232b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006d1:[<0x1000000039b/picture-27.jpg v1501>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006d1:[<0x1000000039b/picture-27.jpg v1501>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.646 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.391+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000005ba [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-52.jpg auth v1505 ap=1 s=34780 n(v0 rc2024-05-09T08:27:50.404871+0000 b34780 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2233600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000005ba:[<0x1000000039b/picture-52.jpg v1505>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000005ba:[<0x1000000039b/picture-52.jpg v1505>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.646 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.385+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c6b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-33.xml auth v376 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.384672+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c6b:[<0x10000000ba3/article-33.xml v376>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c6b:[<0x10000000ba3/article-33.xml v376>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.647 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.386+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c22 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-57.xml auth v232 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.138679+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c22:[<0x10000000ba3/article-57.xml v232>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c22:[<0x10000000ba3/article-57.xml v232>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.647 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.386+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000db1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-7.xml auth v690 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.373644+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000db1:[<0x10000000ba3/comment-7.xml v690>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000db1:[<0x10000000ba3/comment-7.xml v690>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.647 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.392+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000517 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-1.xml auth v1499 ap=1 s=28475 n(v0 rc2024-05-09T08:27:49.785889+0000 b28475 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2233b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000517:[<0x1000000039b/article-1.xml v1499>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000517:[<0x1000000039b/article-1.xml v1499>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.647 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.393+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006bc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-34.jpg auth v1511 ap=1 s=8778 n(v0 rc2024-05-09T08:27:51.326845+0000 b8778 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2234680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006bc:[<0x1000000039b/picture-34.jpg v1511>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006bc:[<0x1000000039b/picture-34.jpg v1511>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.647 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.423+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-81.xml auth v1007 ap=1 s=16091 n(v0 rc2024-05-09T08:27:48.793917+0000 b16091 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d0:[<0x1000000039a/article-81.xml v1007>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003d0:[<0x1000000039a/article-81.xml v1007>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.647 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.423+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-78.xml auth v991 ap=1 s=26058 n(v0 rc2024-05-09T08:27:48.808917+0000 b26058 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d9:[<0x1000000039a/article-78.xml v991>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003d9:[<0x1000000039a/article-78.xml v991>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.647 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.423+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ba [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-71.jpg auth v999 ap=1 s=10517 n(v0 rc2024-05-09T08:27:49.407900+0000 b10517 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ba:[<0x1000000039a/picture-71.jpg v999>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004ba:[<0x1000000039a/picture-71.jpg v999>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.648 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.387+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bdd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-79.xml auth v100 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.932685+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bdd:[<0x10000000ba3/article-79.xml v100>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bdd:[<0x10000000ba3/article-79.xml v100>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.648 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.387+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d0f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-15.xml auth v594 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.884658+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d0f:[<0x10000000ba3/comment-15.xml v594>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d0f:[<0x10000000ba3/comment-15.xml v594>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.648 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.423+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000461 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-29.xml auth v1009 ap=1 s=12170 n(v0 rc2024-05-09T08:27:49.105908+0000 b12170 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000461:[<0x1000000039a/article-29.xml v1009>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000461:[<0x1000000039a/article-29.xml v1009>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.648 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.423+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000040f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-60.xml auth v1005 ap=1 s=16429 n(v0 rc2024-05-09T08:27:48.895914+0000 b16429 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000040f:[<0x1000000039a/article-60.xml v1005>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000040f:[<0x1000000039a/article-60.xml v1005>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.649 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.393+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000005ae [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-53.jpg auth v1509 ap=1 s=22090 n(v0 rc2024-05-09T08:27:50.329873+0000 b22090 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2234c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000005ae:[<0x1000000039b/picture-53.jpg v1509>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000005ae:[<0x1000000039b/picture-53.jpg v1509>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.649 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.394+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006f2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-16.jpg auth v1503 ap=1 s=43696 n(v0 rc2024-05-09T08:27:51.491840+0000 b43696 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da223a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006f2:[<0x1000000039b/picture-16.jpg v1503>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006f2:[<0x1000000039b/picture-16.jpg v1503>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.649 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.396+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-23.xml auth v1477 ap=1 s=12981 n(v0 rc2024-05-09T08:27:48.806917+0000 b12981 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da223b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d2:[<0x1000000039b/article-23.xml v1477>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003d2:[<0x1000000039b/article-23.xml v1477>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.649 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.387+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c1e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-58.xml auth v226 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.129679+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c1e:[<0x10000000ba3/article-58.xml v226>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c1e:[<0x10000000ba3/article-58.xml v226>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.649 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.387+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c30 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-53.xml auth v256 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.178678+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c30:[<0x10000000ba3/article-53.xml v256>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c30:[<0x10000000ba3/article-53.xml v256>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.649 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.396+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000411 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-17.xml auth v1475 ap=1 s=6910 n(v0 rc2024-05-09T08:27:48.909914+0000 b6910 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da223bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000411:[<0x1000000039b/article-17.xml v1475>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000411:[<0x1000000039b/article-17.xml v1475>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.650 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.396+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006c2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-32.jpg auth v1507 ap=1 s=8951 n(v0 rc2024-05-09T08:27:51.346844+0000 b8951 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da223c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006c2:[<0x1000000039b/picture-32.jpg v1507>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006c2:[<0x1000000039b/picture-32.jpg v1507>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.650 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.423+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004d1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-63.jpg auth v1003 ap=1 s=43967 n(v0 rc2024-05-09T08:27:49.483898+0000 b43967 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004d1:[<0x1000000039a/picture-63.jpg v1003>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004d1:[<0x1000000039a/picture-63.jpg v1003>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.650 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.423+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000536 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-40.jpg auth v993 ap=1 s=50204 n(v0 rc2024-05-09T08:27:49.886886+0000 b50204 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000536:[<0x1000000039a/picture-40.jpg v993>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000536:[<0x1000000039a/picture-40.jpg v993>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.650 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.424+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c35 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-61.xml auth v1122 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.209677+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c35:[<0x1000000039a/comment-61.xml v1122>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c35:[<0x1000000039a/comment-61.xml v1122>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.651 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.387+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c7b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-28.xml auth v406 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.437670+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c7b:[<0x10000000ba3/article-28.xml v406>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c7b:[<0x10000000ba3/article-28.xml v406>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.651 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.388+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d8d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-9.xml auth v666 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.216648+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d8d:[<0x10000000ba3/comment-9.xml v666>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d8d:[<0x10000000ba3/comment-9.xml v666>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.651 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.388+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c33 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-52.xml auth v262 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.189678+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d84d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c33:[<0x10000000ba3/article-52.xml v262>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c33:[<0x10000000ba3/article-52.xml v262>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.651 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.396+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000605 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-48.jpg auth v1513 ap=1 s=59015 n(v0 rc2024-05-09T08:27:50.650864+0000 b59015 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da223c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000605:[<0x1000000039b/picture-48.jpg v1513>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000605:[<0x1000000039b/picture-48.jpg v1513>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.651 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.398+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006e3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-21.jpg auth v1498 ap=1 s=48662 n(v0 rc2024-05-09T08:27:51.446842+0000 b48662 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22d3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006e3:[<0x1000000039b/picture-21.jpg v1498>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006e3:[<0x1000000039b/picture-21.jpg v1498>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.652 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.424+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d1b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-38.xml auth v1200 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.904657+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d1b:[<0x1000000039a/comment-38.xml v1200>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d1b:[<0x1000000039a/comment-38.xml v1200>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.652 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.424+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000476 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-22.xml auth v1011 ap=1 s=11637 n(v0 rc2024-05-09T08:27:49.176906+0000 b11637 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea52d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000476:[<0x1000000039a/article-22.xml v1011>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000476:[<0x1000000039a/article-22.xml v1011>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.652 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.389+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dc5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-6.xml auth v702 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.477641+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d85cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dc5:[<0x10000000ba3/comment-6.xml v702>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dc5:[<0x10000000ba3/comment-6.xml v702>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.652 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.389+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cdc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-5.xml auth v544 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.741662+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d85d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cdc:[<0x10000000ba3/article-5.xml v544>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cdc:[<0x10000000ba3/article-5.xml v544>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.652 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.424+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000be9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-68.xml auth v1085 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.975684+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea622000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000be9:[<0x1000000039a/comment-68.xml v1085>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000be9:[<0x1000000039a/comment-68.xml v1085>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.652 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.424+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000430 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-49.xml auth v989 ap=1 s=8491 n(v0 rc2024-05-09T08:27:48.946913+0000 b8491 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea622580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000430:[<0x1000000039a/article-49.xml v989>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000430:[<0x1000000039a/article-49.xml v989>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.653 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.398+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006c8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-30.jpg auth v1510 ap=1 s=48128 n(v0 rc2024-05-09T08:27:51.364844+0000 b48128 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22d4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006c8:[<0x1000000039b/picture-30.jpg v1510>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006c8:[<0x1000000039b/picture-30.jpg v1510>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.653 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.398+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003bf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-60.xml auth v1494 ap=1 s=4925 n(v0 rc2024-05-09T08:27:48.774918+0000 b4925 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22d3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003bf:[<0x1000000039b/comment-60.xml v1494>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003bf:[<0x1000000039b/comment-60.xml v1494>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.653 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.400+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000068d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-36.jpg auth v1508 ap=1 s=19591 n(v0 rc2024-05-09T08:27:51.202849+0000 b19591 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da1eb2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000068d:[<0x1000000039b/picture-36.jpg v1508>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000068d:[<0x1000000039b/picture-36.jpg v1508>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.653 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.390+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bd5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-81.xml auth v88 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.908686+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d85e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bd5:[<0x10000000ba3/article-81.xml v88>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bd5:[<0x10000000ba3/article-81.xml v88>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.653 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.390+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000be1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-78.xml auth v106 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.938685+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d85e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000be1:[<0x10000000ba3/article-78.xml v106>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000be1:[<0x10000000ba3/article-78.xml v106>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.654 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.424+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d8b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-22.xml auth v1257 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.183649+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea623b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d8b:[<0x1000000039a/comment-22.xml v1257>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d8b:[<0x1000000039a/comment-22.xml v1257>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.654 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.425+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008c7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-80.xml auth v1029 ap=1 s=9713 n(v0 rc2024-05-09T08:27:52.984798+0000 b9713 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea623080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008c7:[<0x1000000039a/comment-80.xml v1029>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008c7:[<0x1000000039a/comment-80.xml v1029>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.654 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.425+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cda [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-44.xml auth v1195 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.740662+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea624680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cda:[<0x1000000039a/comment-44.xml v1195>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cda:[<0x1000000039a/comment-44.xml v1195>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.654 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.400+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003fa [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-19.xml auth v1482 ap=1 s=22457 n(v0 rc2024-05-09T08:27:48.869915+0000 b22457 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da1eb2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003fa:[<0x1000000039b/article-19.xml v1482>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003fa:[<0x1000000039b/article-19.xml v1482>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.654 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.400+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006e6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-20.jpg auth v1502 ap=1 s=14912 n(v0 rc2024-05-09T08:27:51.455841+0000 b14912 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da1eb3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006e6:[<0x1000000039b/picture-20.jpg v1502>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006e6:[<0x1000000039b/picture-20.jpg v1502>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.655 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.425+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000043e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-43.xml auth v1017 ap=1 s=31888 n(v0 rc2024-05-09T08:27:48.981912+0000 b31888 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea624c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000043e:[<0x1000000039a/article-43.xml v1017>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000043e:[<0x1000000039a/article-43.xml v1017>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.655 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.425+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000458 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-33.xml auth v1027 ap=1 s=20964 n(v0 rc2024-05-09T08:27:49.077909+0000 b20964 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea625180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000458:[<0x1000000039a/article-33.xml v1027>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000458:[<0x1000000039a/article-33.xml v1027>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.655 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.401+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-63.xml auth v1496 ap=1 s=6755 n(v0 rc2024-05-09T08:27:48.727919+0000 b6755 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da1eb3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003a4:[<0x1000000039b/comment-63.xml v1496>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003a4:[<0x1000000039b/comment-63.xml v1496>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.655 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.401+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000422 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-15.xml auth v1480 ap=1 s=24970 n(v0 rc2024-05-09T08:27:48.933913+0000 b24970 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22d4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000422:[<0x1000000039b/article-15.xml v1480>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000422:[<0x1000000039b/article-15.xml v1480>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.655 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.391+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c78 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-29.xml auth v400 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.428671+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d85ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c78:[<0x10000000ba3/article-29.xml v400>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c78:[<0x10000000ba3/article-29.xml v400>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.655 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.391+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c18 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-60.xml auth v214 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.110680+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d85f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c18:[<0x10000000ba3/article-60.xml v214>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c18:[<0x10000000ba3/article-60.xml v214>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.656 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.391+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d7c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-10.xml auth v654 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.141650+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d85c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d7c:[<0x10000000ba3/comment-10.xml v654>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d7c:[<0x10000000ba3/comment-10.xml v654>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.656 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.425+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000052d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-93.xml auth v1029 ap=1 s=6656 n(v0 rc2024-05-09T08:27:49.840887+0000 b6656 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea625700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000052d:[<0x1000000039a/comment-93.xml v1029>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000052d:[<0x1000000039a/comment-93.xml v1029>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.656 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.425+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c6d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-5.jpg auth v1141 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.397672+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c6d:[<0x1000000039a/picture-5.jpg v1141>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c6d:[<0x1000000039a/picture-5.jpg v1141>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.656 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.426+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d70 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-27.xml auth v1242 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.085652+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d70:[<0x1000000039a/comment-27.xml v1242>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d70:[<0x1000000039a/comment-27.xml v1242>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.656 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.402+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000063c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-43.jpg auth v1504 ap=1 s=48488 n(v0 rc2024-05-09T08:27:50.878858+0000 b48488 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da1eb2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000063c:[<0x1000000039b/picture-43.jpg v1504>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000063c:[<0x1000000039b/picture-43.jpg v1504>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.656 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.402+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000963 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-38.jpg auth v1506 ap=1 s=51013 n(v0 rc2024-05-09T08:27:53.495783+0000 b51013 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da1eb4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000963:[<0x1000000039b/picture-38.jpg v1506>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000963:[<0x1000000039b/picture-38.jpg v1506>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.656 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.403+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000719 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-3.jpg auth v1503 ap=1 s=11595 n(v0 rc2024-05-09T08:27:51.614837+0000 b11595 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20d2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000719:[<0x1000000039b/picture-3.jpg v1503>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000719:[<0x1000000039b/picture-3.jpg v1503>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.657 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.392+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c69 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-34.xml auth v370 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.372672+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d85db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c69:[<0x10000000ba3/article-34.xml v370>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c69:[<0x10000000ba3/article-34.xml v370>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.657 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.392+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c8f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-22.xml auth v442 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.516668+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d85f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c8f:[<0x10000000ba3/article-22.xml v442>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c8f:[<0x10000000ba3/article-22.xml v442>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.657 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.426+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004f1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-50.jpg auth v1021 ap=1 s=45889 n(v0 rc2024-05-09T08:27:49.594894+0000 b45889 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004f1:[<0x1000000039a/picture-50.jpg v1021>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004f1:[<0x1000000039a/picture-50.jpg v1021>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.657 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.426+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000098e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-76.xml auth v1039 ap=1 s=7118 n(v0 rc2024-05-09T08:27:53.666778+0000 b7118 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000098e:[<0x1000000039a/comment-76.xml v1039>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000098e:[<0x1000000039a/comment-76.xml v1039>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.658 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.403+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000591 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-55.jpg auth v1507 ap=1 s=44690 n(v0 rc2024-05-09T08:27:50.251876+0000 b44690 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20d3700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000591:[<0x1000000039b/picture-55.jpg v1507>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000591:[<0x1000000039b/picture-55.jpg v1507>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.658 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.403+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004bb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-5.xml auth v1501 ap=1 s=17049 n(v0 rc2024-05-09T08:27:49.424899+0000 b17049 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20f0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004bb:[<0x1000000039b/article-5.xml v1501>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004bb:[<0x1000000039b/article-5.xml v1501>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.658 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.392+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d9d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-8.xml auth v678 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.292646+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d868580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d9d:[<0x10000000ba3/comment-8.xml v678>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d9d:[<0x10000000ba3/comment-8.xml v678>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.658 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.392+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c3c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-49.xml auth v280 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.222677+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d868000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c3c:[<0x10000000ba3/article-49.xml v280>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c3c:[<0x10000000ba3/article-49.xml v280>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.658 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.426+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004db [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-58.jpg auth v1005 ap=1 s=52028 n(v0 rc2024-05-09T08:27:49.517897+0000 b52028 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004db:[<0x1000000039a/picture-58.jpg v1005>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004db:[<0x1000000039a/picture-58.jpg v1005>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.658 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.426+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000418 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-57.xml auth v993 ap=1 s=20589 n(v0 rc2024-05-09T08:27:48.909914+0000 b20589 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000418:[<0x1000000039a/article-57.xml v993>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000418:[<0x1000000039a/article-57.xml v993>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.659 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.404+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000701 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-11.jpg auth v1505 ap=1 s=44416 n(v0 rc2024-05-09T08:27:51.537839+0000 b44416 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20f0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000701:[<0x1000000039b/picture-11.jpg v1505>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000701:[<0x1000000039b/picture-11.jpg v1505>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.659 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.404+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003de [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-57.xml auth v1497 ap=1 s=4296 n(v0 rc2024-05-09T08:27:48.831916+0000 b4296 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20f1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003de:[<0x1000000039b/comment-57.xml v1497>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003de:[<0x1000000039b/comment-57.xml v1497>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.659 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.393+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cab [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-14.xml auth v490 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.626665+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d869080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cab:[<0x10000000ba3/article-14.xml v490>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cab:[<0x10000000ba3/article-14.xml v490>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.659 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.393+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bcc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-84.xml auth v70 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.885686+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d869600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bcc:[<0x10000000ba3/article-84.xml v70>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bcc:[<0x10000000ba3/article-84.xml v70>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.659 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.393+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c45 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-46.xml auth v298 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.254676+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d869b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c45:[<0x10000000ba3/article-46.xml v298>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c45:[<0x10000000ba3/article-46.xml v298>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.660 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.426+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c91 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-52.xml auth v1160 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.523668+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c91:[<0x1000000039a/comment-52.xml v1160>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c91:[<0x1000000039a/comment-52.xml v1160>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.660 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.426+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c01 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-66.xml auth v1094 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.053681+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c01:[<0x1000000039a/comment-66.xml v1094>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c01:[<0x1000000039a/comment-66.xml v1094>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.660 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.427+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004cd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-65.jpg auth v997 ap=1 s=62841 n(v0 rc2024-05-09T08:27:49.468898+0000 b62841 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004cd:[<0x1000000039a/picture-65.jpg v997>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004cd:[<0x1000000039a/picture-65.jpg v997>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.660 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.405+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-61.xml auth v1499 ap=1 s=3085 n(v0 rc2024-05-09T08:27:48.762918+0000 b3085 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20f2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b9:[<0x1000000039b/comment-61.xml v1499>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003b9:[<0x1000000039b/comment-61.xml v1499>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.660 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.406+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000066d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-40.jpg auth v1509 ap=1 s=24079 n(v0 rc2024-05-09T08:27:51.056853+0000 b24079 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20f1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000066d:[<0x1000000039b/picture-40.jpg v1509>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000066d:[<0x1000000039b/picture-40.jpg v1509>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.660 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.406+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003da [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-22.xml auth v1481 ap=1 s=30285 n(v0 rc2024-05-09T08:27:48.821917+0000 b30285 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20f2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003da:[<0x1000000039b/article-22.xml v1481>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003da:[<0x1000000039b/article-22.xml v1481>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.661 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.394+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c42 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-47.xml auth v292 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.244676+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d86a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c42:[<0x10000000ba3/article-47.xml v292>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c42:[<0x10000000ba3/article-47.xml v292>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.661 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.394+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bf6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-71.xml auth v148 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.998683+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d86ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bf6:[<0x10000000ba3/article-71.xml v148>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bf6:[<0x10000000ba3/article-71.xml v148>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.661 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.427+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-79.xml auth v991 ap=1 s=22083 n(v0 rc2024-05-09T08:27:48.803917+0000 b22083 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea73f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d6:[<0x1000000039a/article-79.xml v991>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003d6:[<0x1000000039a/article-79.xml v991>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.661 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.427+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dc2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-15.xml auth v1272 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.420642+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea748000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dc2:[<0x1000000039a/comment-15.xml v1272>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dc2:[<0x1000000039a/comment-15.xml v1272>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.661 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.407+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000713 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-5.jpg auth v1495 ap=1 s=63617 n(v0 rc2024-05-09T08:27:51.596837+0000 b63617 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da210f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000713:[<0x1000000039b/picture-5.jpg v1495>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000713:[<0x1000000039b/picture-5.jpg v1495>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.661 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.407+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000005ec [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-50.jpg auth v1503 ap=1 s=37571 n(v0 rc2024-05-09T08:27:50.570867+0000 b37571 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da210fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000005ec:[<0x1000000039b/picture-50.jpg v1503>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000005ec:[<0x1000000039b/picture-50.jpg v1503>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.662 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.395+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bb2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-91.xml auth v28 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.808688+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d86a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bb2:[<0x10000000ba3/article-91.xml v28>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bb2:[<0x10000000ba3/article-91.xml v28>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.662 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.395+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cc1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-11.xml auth v508 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.680664+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d86b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cc1:[<0x10000000ba3/article-11.xml v508>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cc1:[<0x10000000ba3/article-11.xml v508>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.662 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.427+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000415 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-58.xml auth v999 ap=1 s=11868 n(v0 rc2024-05-09T08:27:48.904914+0000 b11868 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea748580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000415:[<0x1000000039a/article-58.xml v999>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000415:[<0x1000000039a/article-58.xml v999>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.662 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.427+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ce8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-43.xml auth v1204 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.773661+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea749080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ce8:[<0x1000000039a/comment-43.xml v1204>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ce8:[<0x1000000039a/comment-43.xml v1204>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.663 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.407+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000056b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-58.jpg auth v1497 ap=1 s=23907 n(v0 rc2024-05-09T08:27:50.096880+0000 b23907 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2110100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000056b:[<0x1000000039b/picture-58.jpg v1497>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000056b:[<0x1000000039b/picture-58.jpg v1497>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.663 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.409+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000005df [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-51.jpg auth v1499 ap=1 s=25609 n(v0 rc2024-05-09T08:27:50.497869+0000 b25609 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2117080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000005df:[<0x1000000039b/picture-51.jpg v1499>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000005df:[<0x1000000039b/picture-51.jpg v1499>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.663 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.395+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ca3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-16.xml auth v478 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.587666+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d87e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ca3:[<0x10000000ba3/article-16.xml v478>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ca3:[<0x10000000ba3/article-16.xml v478>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.663 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.396+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c2d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-54.xml auth v250 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.166678+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d87e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c2d:[<0x10000000ba3/article-54.xml v250>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c2d:[<0x10000000ba3/article-54.xml v250>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.663 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.396+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c81 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-26.xml auth v418 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.461670+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d87eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c81:[<0x10000000ba3/article-26.xml v418>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c81:[<0x10000000ba3/article-26.xml v418>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.663 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.427+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000424 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-53.xml auth v1013 ap=1 s=27765 n(v0 rc2024-05-09T08:27:48.927914+0000 b27765 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea748b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000424:[<0x1000000039a/article-53.xml v1013>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000424:[<0x1000000039a/article-53.xml v1013>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.664 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.428+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dae [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-17.xml auth v1267 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.337645+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea749600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dae:[<0x1000000039a/comment-17.xml v1267>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dae:[<0x1000000039a/comment-17.xml v1267>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.664 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.428+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000464 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-28.xml auth v995 ap=1 s=22967 n(v0 rc2024-05-09T08:27:49.115908+0000 b22967 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea749b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000464:[<0x1000000039a/article-28.xml v995>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000464:[<0x1000000039a/article-28.xml v995>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.664 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.409+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000061a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-46.jpg auth v1505 ap=1 s=46887 n(v0 rc2024-05-09T08:27:50.735862+0000 b46887 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2118100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000061a:[<0x1000000039b/picture-46.jpg v1505>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000061a:[<0x1000000039b/picture-46.jpg v1505>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.664 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.410+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000530 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-62.jpg auth v1501 ap=1 s=47730 n(v0 rc2024-05-09T08:27:49.861887+0000 b47730 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2117b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000530:[<0x1000000039b/picture-62.jpg v1501>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000530:[<0x1000000039b/picture-62.jpg v1501>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.664 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.410+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006d7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-25.jpg auth v1501 ap=1 s=12534 n(v0 rc2024-05-09T08:27:51.410843+0000 b12534 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22aac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006d7:[<0x1000000039b/picture-25.jpg v1501>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006d7:[<0x1000000039b/picture-25.jpg v1501>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.664 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.396+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bab [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-93.xml auth v16 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.775689+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d87f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bab:[<0x10000000ba3/article-93.xml v16>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bab:[<0x10000000ba3/article-93.xml v16>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.665 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.396+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cc5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-10.xml auth v514 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.690663+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d87fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cc5:[<0x10000000ba3/article-10.xml v514>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cc5:[<0x10000000ba3/article-10.xml v514>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.665 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.410+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000497 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-8.xml auth v1499 ap=1 s=25978 n(v0 rc2024-05-09T08:27:49.318902+0000 b25978 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22ab180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000497:[<0x1000000039b/article-8.xml v1499>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000497:[<0x1000000039b/article-8.xml v1499>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.665 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.411+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006c5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-31.jpg auth v1511 ap=1 s=28731 n(v0 rc2024-05-09T08:27:51.355844+0000 b28731 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22be580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006c5:[<0x1000000039b/picture-31.jpg v1511>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006c5:[<0x1000000039b/picture-31.jpg v1511>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.665 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.428+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000427 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-52.xml auth v1015 ap=1 s=25450 n(v0 rc2024-05-09T08:27:48.932913+0000 b25450 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea74a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000427:[<0x1000000039a/article-52.xml v1015>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000427:[<0x1000000039a/article-52.xml v1015>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.665 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.428+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004c1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-69.jpg auth v1019 ap=1 s=24221 n(v0 rc2024-05-09T08:27:49.428899+0000 b24221 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea74ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004c1:[<0x1000000039a/picture-69.jpg v1019>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004c1:[<0x1000000039a/picture-69.jpg v1019>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.666 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.397+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c3f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-48.xml auth v286 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.233676+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d880680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c3f:[<0x10000000ba3/article-48.xml v286>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c3f:[<0x10000000ba3/article-48.xml v286>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.666 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.397+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ce1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-4.xml auth v550 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.751661+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d880c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ce1:[<0x10000000ba3/article-4.xml v550>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ce1:[<0x10000000ba3/article-4.xml v550>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.666 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.411+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000005fc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-49.jpg auth v1507 ap=1 s=60535 n(v0 rc2024-05-09T08:27:50.601866+0000 b60535 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22beb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000005fc:[<0x1000000039b/picture-49.jpg v1507>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000005fc:[<0x1000000039b/picture-49.jpg v1507>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.666 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.411+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000070a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-8.jpg auth v1505 ap=1 s=13148 n(v0 rc2024-05-09T08:27:51.564838+0000 b13148 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22bf080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000070a:[<0x1000000039b/picture-8.jpg v1505>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000070a:[<0x1000000039b/picture-8.jpg v1505>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.666 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.398+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c5d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-38.xml auth v346 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.334673+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d881700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c5d:[<0x10000000ba3/article-38.xml v346>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c5d:[<0x10000000ba3/article-38.xml v346>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.666 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.398+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bf3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-72.xml auth v142 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.990683+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d888000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bf3:[<0x10000000ba3/article-72.xml v142>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bf3:[<0x10000000ba3/article-72.xml v142>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.667 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.428+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ff [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-99.xml auth v1003 ap=1 s=4091 n(v0 rc2024-05-09T08:27:49.649893+0000 b4091 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea74b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ff:[<0x1000000039a/comment-99.xml v1003>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004ff:[<0x1000000039a/comment-99.xml v1003>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.667 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.429+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ef [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-51.jpg auth v1011 ap=1 s=56265 n(v0 rc2024-05-09T08:27:49.588895+0000 b56265 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea74b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ef:[<0x1000000039a/picture-51.jpg v1011>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004ef:[<0x1000000039a/picture-51.jpg v1011>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.667 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.429+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cd2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-45.xml auth v1187 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.723662+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea752000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cd2:[<0x1000000039a/comment-45.xml v1187>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cd2:[<0x1000000039a/comment-45.xml v1187>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.667 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.411+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006d4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-26.jpg auth v1503 ap=1 s=37348 n(v0 rc2024-05-09T08:27:51.401843+0000 b37348 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22bf600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006d4:[<0x1000000039b/picture-26.jpg v1503>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006d4:[<0x1000000039b/picture-26.jpg v1503>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.667 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.411+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-21.xml auth v1481 ap=1 s=6963 n(v0 rc2024-05-09T08:27:48.836916+0000 b6963 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22bfb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e4:[<0x1000000039b/article-21.xml v1481>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003e4:[<0x1000000039b/article-21.xml v1481>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.667 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.412+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000967 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-13.xml auth v1515 ap=1 s=18907 n(v0 rc2024-05-09T08:27:53.508783+0000 b18907 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22c0680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000967:[<0x1000000039b/article-13.xml v1515>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000967:[<0x1000000039b/article-13.xml v1515>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.668 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.429+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007e4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-84.xml auth v1023 ap=1 s=4404 n(v0 rc2024-05-09T08:27:52.235819+0000 b4404 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea752b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007e4:[<0x1000000039a/comment-84.xml v1023>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007e4:[<0x1000000039a/comment-84.xml v1023>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.668 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.429+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004d3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-62.jpg auth v1009 ap=1 s=37212 n(v0 rc2024-05-09T08:27:49.489897+0000 b37212 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea752580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004d3:[<0x1000000039a/picture-62.jpg v1009>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004d3:[<0x1000000039a/picture-62.jpg v1009>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.668 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.398+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c85 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-25.xml auth v424 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.474669+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d881180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c85:[<0x10000000ba3/article-25.xml v424>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c85:[<0x10000000ba3/article-25.xml v424>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.668 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.399+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c66 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-35.xml auth v364 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.362673+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d888b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c66:[<0x10000000ba3/article-35.xml v364>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c66:[<0x10000000ba3/article-35.xml v364>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.668 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.399+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bc7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-86.xml auth v58 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.866687+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d889080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bc7:[<0x10000000ba3/article-86.xml v58>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bc7:[<0x10000000ba3/article-86.xml v58>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.669 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.412+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004b3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-6.xml auth v1497 ap=1 s=17411 n(v0 rc2024-05-09T08:27:49.400900+0000 b17411 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22c1180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004b3:[<0x1000000039b/article-6.xml v1497>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004b3:[<0x1000000039b/article-6.xml v1497>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.669 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.412+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000509 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-2.xml auth v1495 ap=1 s=18744 n(v0 rc2024-05-09T08:27:49.693892+0000 b18744 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22c1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000509:[<0x1000000039b/article-2.xml v1495>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000509:[<0x1000000039b/article-2.xml v1495>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.669 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.429+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004fd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-46.jpg auth v1035 ap=1 s=42411 n(v0 rc2024-05-09T08:27:49.645893+0000 b42411 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea753080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004fd:[<0x1000000039a/picture-46.jpg v1035>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004fd:[<0x1000000039a/picture-46.jpg v1035>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.669 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.429+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004cf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-64.jpg auth v1016 ap=1 s=53704 n(v0 rc2024-05-09T08:27:49.478898+0000 b53704 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea5ad180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004cf:[<0x1000000039a/picture-64.jpg v1016>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004cf:[<0x1000000039a/picture-64.jpg v1016>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.669 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.399+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c56 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-40.xml auth v334 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.314674+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d889b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c56:[<0x10000000ba3/article-40.xml v334>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c56:[<0x10000000ba3/article-40.xml v334>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.669 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.399+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d7f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-30.jpg auth v658 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.163650+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d88a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d7f:[<0x10000000ba3/picture-30.jpg v658>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d7f:[<0x10000000ba3/picture-30.jpg v658>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.670 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.413+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000059e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-54.jpg auth v1513 ap=1 s=22051 n(v0 rc2024-05-09T08:27:50.294874+0000 b22051 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22d2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000059e:[<0x1000000039b/picture-54.jpg v1513>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000059e:[<0x1000000039b/picture-54.jpg v1513>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.670 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.413+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006cb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-29.jpg auth v1509 ap=1 s=38853 n(v0 rc2024-05-09T08:27:51.373844+0000 b38853 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22d2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006cb:[<0x1000000039b/picture-29.jpg v1509>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006cb:[<0x1000000039b/picture-29.jpg v1509>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.670 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.413+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006fb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-13.jpg auth v1499 ap=1 s=44887 n(v0 rc2024-05-09T08:27:51.518840+0000 b44887 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2087600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006fb:[<0x1000000039b/picture-13.jpg v1499>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006fb:[<0x1000000039b/picture-13.jpg v1499>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.670 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.400+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bd2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-82.xml auth v82 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.900686+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d88a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bd2:[<0x10000000ba3/article-82.xml v82>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bd2:[<0x10000000ba3/article-82.xml v82>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.670 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.400+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ca7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-15.xml auth v484 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.607666+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d88ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ca7:[<0x10000000ba3/article-15.xml v484>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ca7:[<0x10000000ba3/article-15.xml v484>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.671 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.430+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-80.xml auth v1020 ap=1 s=6998 n(v0 rc2024-05-09T08:27:48.798917+0000 b6998 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d4:[<0x1000000039a/article-80.xml v1020>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003d4:[<0x1000000039a/article-80.xml v1020>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.671 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.430+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d25 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-37.xml auth v1209 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.924657+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d25:[<0x1000000039a/comment-37.xml v1209>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d25:[<0x1000000039a/comment-37.xml v1209>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.671 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.430+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bc2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-71.xml auth v1064 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.890686+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bc2:[<0x1000000039a/comment-71.xml v1064>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bc2:[<0x1000000039a/comment-71.xml v1064>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.671 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.413+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000402 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-18.xml auth v1475 ap=1 s=24488 n(v0 rc2024-05-09T08:27:48.884915+0000 b24488 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2088680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000402:[<0x1000000039b/article-18.xml v1475>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000402:[<0x1000000039b/article-18.xml v1475>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.671 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.414+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006ec [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-18.jpg auth v1503 ap=1 s=61232 n(v0 rc2024-05-09T08:27:51.472841+0000 b61232 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2088100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006ec:[<0x1000000039b/picture-18.jpg v1503>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006ec:[<0x1000000039b/picture-18.jpg v1503>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.671 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.430+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c32 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-10.jpg auth v1119 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.203677+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea3e7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c32:[<0x1000000039a/picture-10.jpg v1119>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c32:[<0x1000000039a/picture-10.jpg v1119>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.672 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.430+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000051c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-95.xml auth v1032 ap=1 s=4151 n(v0 rc2024-05-09T08:27:49.770889+0000 b4151 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea361080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000051c:[<0x1000000039a/comment-95.xml v1032>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000051c:[<0x1000000039a/comment-95.xml v1032>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.672 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.414+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006e9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-19.jpg auth v1505 ap=1 s=32195 n(v0 rc2024-05-09T08:27:51.464841+0000 b32195 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2089180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006e9:[<0x1000000039b/picture-19.jpg v1505>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006e9:[<0x1000000039b/picture-19.jpg v1505>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.672 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.414+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000039d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-27.xml auth v1479 ap=1 s=10226 n(v0 rc2024-05-09T08:27:48.707920+0000 b10226 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da208c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000039d:[<0x1000000039b/article-27.xml v1479>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000039d:[<0x1000000039b/article-27.xml v1479>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.672 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.400+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d04 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-36.jpg auth v586 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.835659+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d88b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d04:[<0x10000000ba3/picture-36.jpg v586>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d04:[<0x10000000ba3/picture-36.jpg v586>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.672 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.400+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c98 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-19.xml auth v460 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.544667+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d88b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c98:[<0x10000000ba3/article-19.xml v460>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c98:[<0x10000000ba3/article-19.xml v460>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.672 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.401+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bbb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-89.xml auth v40 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.840688+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bbb:[<0x10000000ba3/article-89.xml v40>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bbb:[<0x10000000ba3/article-89.xml v40>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.673 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.430+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c97 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-1.jpg auth v1166 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.560667+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea592000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c97:[<0x1000000039a/picture-1.jpg v1166>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c97:[<0x1000000039a/picture-1.jpg v1166>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.673 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.431+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000db8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-16.xml auth v1284 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.366644+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea592580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000db8:[<0x1000000039a/comment-16.xml v1284>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000db8:[<0x1000000039a/comment-16.xml v1284>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.673 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.402+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c51 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-42.xml auth v322 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.294675+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c51:[<0x10000000ba3/article-42.xml v322>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c51:[<0x10000000ba3/article-42.xml v322>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.673 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.402+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cf4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-38.jpg auth v574 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.793660+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d80ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cf4:[<0x10000000ba3/picture-38.jpg v574>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cf4:[<0x10000000ba3/picture-38.jpg v574>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.673 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.414+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000716 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-4.jpg auth v1493 ap=1 s=56870 n(v0 rc2024-05-09T08:27:51.605837+0000 b56870 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da208c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000716:[<0x1000000039b/picture-4.jpg v1493>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000716:[<0x1000000039b/picture-4.jpg v1493>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.674 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.415+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006a9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-35.jpg auth v1509 ap=1 s=17562 n(v0 rc2024-05-09T08:27:51.313845+0000 b17562 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da208cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006a9:[<0x1000000039b/picture-35.jpg v1509>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006a9:[<0x1000000039b/picture-35.jpg v1509>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.674 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.415+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000071c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-2.jpg auth v1491 ap=1 s=26874 n(v0 rc2024-05-09T08:27:51.623837+0000 b26874 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da208d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000071c:[<0x1000000039b/picture-2.jpg v1491>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000071c:[<0x1000000039b/picture-2.jpg v1491>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.674 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.403+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bc5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-87.xml auth v52 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.861687+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bc5:[<0x10000000ba3/article-87.xml v52>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bc5:[<0x10000000ba3/article-87.xml v52>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.674 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.403+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c0c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-64.xml auth v190 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.076681+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c0c:[<0x10000000ba3/article-64.xml v190>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c0c:[<0x10000000ba3/article-64.xml v190>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.674 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.431+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000499 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-9.xml auth v1008 ap=1 s=15328 n(v0 rc2024-05-09T08:27:49.305903+0000 b15328 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea592b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000499:[<0x1000000039a/article-9.xml v1008>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000499:[<0x1000000039a/article-9.xml v1008>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.674 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.431+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c62 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-6.jpg auth v1130 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.360673+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea593600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c62:[<0x1000000039a/picture-6.jpg v1130>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c62:[<0x1000000039a/picture-6.jpg v1130>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.674 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.431+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000043c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-44.xml auth v1012 ap=1 s=19088 n(v0 rc2024-05-09T08:27:48.974912+0000 b19088 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea593080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000043c:[<0x1000000039a/article-44.xml v1012>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000043c:[<0x1000000039a/article-44.xml v1012>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.675 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.415+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-20.xml auth v1477 ap=1 s=4640 n(v0 rc2024-05-09T08:27:48.857916+0000 b4640 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da208d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f3:[<0x1000000039b/article-20.xml v1477>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003f3:[<0x1000000039b/article-20.xml v1477>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.675 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.415+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006f8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-14.jpg auth v1501 ap=1 s=10932 n(v0 rc2024-05-09T08:27:51.509840+0000 b10932 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20d0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006f8:[<0x1000000039b/picture-14.jpg v1501>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006f8:[<0x1000000039b/picture-14.jpg v1501>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.675 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.416+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000540 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-61.jpg auth v1507 ap=1 s=57011 n(v0 rc2024-05-09T08:27:49.949884+0000 b57011 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20d0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000540:[<0x1000000039b/picture-61.jpg v1507>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000540:[<0x1000000039b/picture-61.jpg v1507>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.675 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.416+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000070d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-7.jpg auth v1495 ap=1 s=24640 n(v0 rc2024-05-09T08:27:51.573838+0000 b24640 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20d0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000070d:[<0x1000000039b/picture-7.jpg v1495>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000070d:[<0x1000000039b/picture-7.jpg v1495>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.675 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.431+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000401 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-65.xml auth v992 ap=1 s=32034 n(v0 rc2024-05-09T08:27:48.870915+0000 b32034 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea593b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000401:[<0x1000000039a/article-65.xml v992>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000401:[<0x1000000039a/article-65.xml v992>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.676 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.431+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c50 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-58.xml auth v1127 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.302674+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea594100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c50:[<0x1000000039a/comment-58.xml v1127>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c50:[<0x1000000039a/comment-58.xml v1127>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.676 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.403+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d3d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-13.xml auth v618 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.986655+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d3d:[<0x10000000ba3/comment-13.xml v618>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d3d:[<0x10000000ba3/comment-13.xml v618>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.676 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.404+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bea [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-75.xml auth v124 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.964684+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bea:[<0x10000000ba3/article-75.xml v124>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bea:[<0x10000000ba3/article-75.xml v124>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.676 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.404+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c60 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-37.xml auth v352 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.342673+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c60:[<0x10000000ba3/article-37.xml v352>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c60:[<0x10000000ba3/article-37.xml v352>,<0x10000000399/blog-13 v21441>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.676 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.416+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000707 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-9.jpg auth v1497 ap=1 s=60373 n(v0 rc2024-05-09T08:27:51.555838+0000 b60373 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da20d1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000707:[<0x1000000039b/picture-9.jpg v1497>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000707:[<0x1000000039b/picture-9.jpg v1497>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.676 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.438+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000974 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-21.jpg auth v916 ap=1 s=38449 n(v0 rc2024-05-09T08:27:53.555781+0000 b38449 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da223d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000974:[<0x1000000088d/picture-21.jpg v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000974:[<0x1000000088d/picture-21.jpg v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.676 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.438+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008fc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-73.jpg auth v918 ap=1 s=25975 n(v0 rc2024-05-09T08:27:53.146793+0000 b25975 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da223d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008fc:[<0x1000000088d/picture-73.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008fc:[<0x1000000088d/picture-73.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.677 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.404+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c5a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-39.xml auth v340 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.324674+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c5a:[<0x10000000ba3/article-39.xml v340>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c5a:[<0x10000000ba3/article-39.xml v340>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.677 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.405+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000db6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-27.jpg auth v694 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.403643+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000db6:[<0x10000000ba3/picture-27.jpg v694>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000db6:[<0x10000000ba3/picture-27.jpg v694>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.677 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.432+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000409 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-62.xml auth v998 ap=1 s=30452 n(v0 rc2024-05-09T08:27:48.885915+0000 b30452 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea594680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000409:[<0x1000000039a/article-62.xml v998>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000409:[<0x1000000039a/article-62.xml v998>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.677 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.432+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c6e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-55.xml auth v1138 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.403671+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea594c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c6e:[<0x1000000039a/comment-55.xml v1138>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c6e:[<0x1000000039a/comment-55.xml v1138>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.677 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.432+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000470 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-24.xml auth v1006 ap=1 s=32559 n(v0 rc2024-05-09T08:27:49.151907+0000 b32559 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea595180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000470:[<0x1000000039a/article-24.xml v1006>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000470:[<0x1000000039a/article-24.xml v1006>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.678 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.439+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a9e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-73.xml auth v973 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.395729+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a9e:[<0x1000000088d/comment-73.xml v973>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a9e:[<0x1000000088d/comment-73.xml v973>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.678 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.439+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-35.xml auth v908 ap=1 s=24451 n(v0 rc2024-05-09T08:27:52.864801+0000 b24451 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a3:[<0x1000000088d/article-35.xml v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a3:[<0x1000000088d/article-35.xml v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.678 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.405+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bf0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-73.xml auth v136 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.981683+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d89f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bf0:[<0x10000000ba3/article-73.xml v136>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bf0:[<0x10000000ba3/article-73.xml v136>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.678 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.405+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c63 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-36.xml auth v358 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.352673+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8b2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c63:[<0x10000000ba3/article-36.xml v358>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c63:[<0x10000000ba3/article-36.xml v358>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.678 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.432+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d3b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-34.xml auth v1224 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.974655+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea595700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d3b:[<0x1000000039a/comment-34.xml v1224>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d3b:[<0x1000000039a/comment-34.xml v1224>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.678 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.432+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d7b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-25.xml auth v1249 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.116651+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea72e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d7b:[<0x1000000039a/comment-25.xml v1249>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d7b:[<0x1000000039a/comment-25.xml v1249>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.679 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.439+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000acc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-60.xml auth v987 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.523725+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x555da224cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000acc:[<0x1000000088d/comment-60.xml v987>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000acc:[<0x1000000088d/comment-60.xml v987>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.679 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.439+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a8a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-78.xml auth v971 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.349730+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a8a:[<0x1000000088d/comment-78.xml v971>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a8a:[<0x1000000088d/comment-78.xml v971>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.679 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.432+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008cc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-28.jpg auth v1038 ap=1 s=14864 n(v0 rc2024-05-09T08:27:52.997797+0000 b14864 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea72e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008cc:[<0x1000000039a/picture-28.jpg v1038>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008cc:[<0x1000000039a/picture-28.jpg v1038>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.679 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.432+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-85.xml auth v996 ap=1 s=11522 n(v0 rc2024-05-09T08:27:48.773918+0000 b11522 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea72eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c5:[<0x1000000039a/article-85.xml v996>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003c5:[<0x1000000039a/article-85.xml v996>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.680 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.406+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cb8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-12.xml auth v502 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.664664+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8b2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cb8:[<0x10000000ba3/article-12.xml v502>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cb8:[<0x10000000ba3/article-12.xml v502>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.680 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.406+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cee [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-1.xml auth v568 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.780661+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8b2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cee:[<0x10000000ba3/article-1.xml v568>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cee:[<0x10000000ba3/article-1.xml v568>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.680 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.407+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d2e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-34.jpg auth v610 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.951656+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8b3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d2e:[<0x10000000ba3/picture-34.jpg v610>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d2e:[<0x10000000ba3/picture-34.jpg v610>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.680 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.439+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008f8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-75.jpg auth v924 ap=1 s=63981 n(v0 rc2024-05-09T08:27:53.134793+0000 b63981 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008f8:[<0x1000000088d/picture-75.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008f8:[<0x1000000088d/picture-75.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.680 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.439+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b7a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-1.xml auth v1078 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.995712+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b7a:[<0x1000000088d/comment-1.xml v1078>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b7a:[<0x1000000088d/comment-1.xml v1078>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.680 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.440+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000089e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-40.xml auth v920 ap=1 s=2502 n(v0 rc2024-05-09T08:27:52.847802+0000 b2502 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000089e:[<0x1000000088d/article-40.xml v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000089e:[<0x1000000088d/article-40.xml v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.681 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.407+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c06 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-66.xml auth v178 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.059681+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8b3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c06:[<0x10000000ba3/article-66.xml v178>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c06:[<0x10000000ba3/article-66.xml v178>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.681 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.407+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000be4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-77.xml auth v112 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.946684+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8b4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000be4:[<0x10000000ba3/article-77.xml v112>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000be4:[<0x10000000ba3/article-77.xml v112>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.681 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.433+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cf9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-41.xml auth v1196 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.812660+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea72f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cf9:[<0x1000000039a/comment-41.xml v1196>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cf9:[<0x1000000039a/comment-41.xml v1196>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.681 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.433+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003bb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-88.xml auth v1018 ap=1 s=7356 n(v0 rc2024-05-09T08:27:48.758918+0000 b7356 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea72f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003bb:[<0x1000000039a/article-88.xml v1018>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003bb:[<0x1000000039a/article-88.xml v1018>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.681 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.433+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-67.xml auth v1002 ap=1 s=3224 n(v0 rc2024-05-09T08:27:48.861915+0000 b3224 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea72fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f9:[<0x1000000039a/article-67.xml v1002>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003f9:[<0x1000000039a/article-67.xml v1002>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.682 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.440+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000095b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-30.jpg auth v910 ap=1 s=65449 n(v0 rc2024-05-09T08:27:53.470784+0000 b65449 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000095b:[<0x1000000088d/picture-30.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000095b:[<0x1000000088d/picture-30.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.682 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.440+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-15.xml auth v912 ap=1 s=2358 n(v0 rc2024-05-09T08:27:52.921799+0000 b2358 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b7:[<0x1000000088d/article-15.xml v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b7:[<0x1000000088d/article-15.xml v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.682 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.407+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c9d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-17.xml auth v472 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.569667+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8b5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c9d:[<0x10000000ba3/article-17.xml v472>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c9d:[<0x10000000ba3/article-17.xml v472>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.682 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.408+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d56 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-32.jpg auth v634 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.044653+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d56:[<0x10000000ba3/picture-32.jpg v634>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d56:[<0x10000000ba3/picture-32.jpg v634>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.682 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.433+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000099c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-75.xml auth v1048 ap=1 s=5803 n(v0 rc2024-05-09T08:27:54.787746+0000 b5803 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea730100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000099c:[<0x1000000039a/comment-75.xml v1048>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000099c:[<0x1000000039a/comment-75.xml v1048>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.682 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.433+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000074b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-37.jpg auth v1036 ap=1 s=58483 n(v0 rc2024-05-09T08:27:51.790832+0000 b58483 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea730680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000074b:[<0x1000000039a/picture-37.jpg v1036>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000074b:[<0x1000000039a/picture-37.jpg v1036>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.683 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.440+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a86 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-79.xml auth v969 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.340730+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a86:[<0x1000000088d/comment-79.xml v969>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a86:[<0x1000000088d/comment-79.xml v969>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.683 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.440+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b11 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-40.xml auth v1011 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.707720+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da224f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b11:[<0x1000000088d/comment-40.xml v1011>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b11:[<0x1000000088d/comment-40.xml v1011>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.683 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.408+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d51 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-12.xml auth v630 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.031653+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8b4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d51:[<0x10000000ba3/comment-12.xml v630>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d51:[<0x10000000ba3/comment-12.xml v630>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.683 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.409+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c8c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-23.xml auth v436 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.508668+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8b5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c8c:[<0x10000000ba3/article-23.xml v436>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c8c:[<0x10000000ba3/article-23.xml v436>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.683 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.409+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c0f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-63.xml auth v196 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.084681+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c0f:[<0x10000000ba3/article-63.xml v196>,<0x10000000399/blog-13 v20275>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c0f:[<0x10000000ba3/article-63.xml v196>,<0x10000000399/blog-13 v24699>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.684 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.433+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000be3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-17.jpg auth v1081 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.954684+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea730c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000be3:[<0x1000000039a/picture-17.jpg v1081>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000be3:[<0x1000000039a/picture-17.jpg v1081>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.684 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.434+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ab [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-3.xml auth v1022 ap=1 s=26026 n(v0 rc2024-05-09T08:27:49.363901+0000 b26026 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea731700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ab:[<0x1000000039a/article-3.xml v1022>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004ab:[<0x1000000039a/article-3.xml v1022>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.684 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.434+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-96.xml auth v1024 ap=1 s=16578 n(v0 rc2024-05-09T08:27:48.716920+0000 b16578 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea731180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003a3:[<0x1000000039a/article-96.xml v1024>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003a3:[<0x1000000039a/article-96.xml v1024>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.684 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.440+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b29 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-33.xml auth v1023 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.765718+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2260000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b29:[<0x1000000088d/comment-33.xml v1023>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b29:[<0x1000000088d/comment-33.xml v1023>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.684 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.441+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b14 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-39.xml auth v1015 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.715720+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2260580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b14:[<0x1000000088d/comment-39.xml v1015>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b14:[<0x1000000088d/comment-39.xml v1015>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.684 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.441+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000094b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-36.jpg auth v904 ap=1 s=53803 n(v0 rc2024-05-09T08:27:53.407786+0000 b53803 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2260b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000094b:[<0x1000000088d/picture-36.jpg v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000094b:[<0x1000000088d/picture-36.jpg v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.684 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.452+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d68 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg auth v637 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.071652+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d68:[<0x10000000ba2/picture-13.jpg v637>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d68:[<0x10000000ba2/picture-13.jpg v637>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.685 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.453+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d4f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-18.jpg auth v607 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.011654+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d4f:[<0x10000000ba2/picture-18.jpg v607>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d4f:[<0x10000000ba2/picture-18.jpg v607>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.685 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.434+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000998 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-24.jpg auth v1040 ap=1 s=34644 n(v0 rc2024-05-09T08:27:53.696777+0000 b34644 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea712000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000998:[<0x1000000039a/picture-24.jpg v1040>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000998:[<0x1000000039a/picture-24.jpg v1040>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.685 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.434+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007ed [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-33.jpg auth v1034 ap=1 s=14360 n(v0 rc2024-05-09T08:27:52.245819+0000 b14360 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea712580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007ed:[<0x1000000039a/picture-33.jpg v1034>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007ed:[<0x1000000039a/picture-33.jpg v1034>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.685 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.441+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000093b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-43.jpg auth v922 ap=1 s=53348 n(v0 rc2024-05-09T08:27:53.349787+0000 b53348 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2261080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000093b:[<0x1000000088d/picture-43.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000093b:[<0x1000000088d/picture-43.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.685 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.441+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-19.xml auth v914 ap=1 s=5929 n(v0 rc2024-05-09T08:27:52.910800+0000 b5929 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2261600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b3:[<0x1000000088d/article-19.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b3:[<0x1000000088d/article-19.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.686 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.453+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bd4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-18.xml auth v76 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.902686+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bd4:[<0x10000000ba2/article-18.xml v76>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bd4:[<0x10000000ba2/article-18.xml v76>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.686 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.453+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d4a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-19.jpg auth v601 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.002654+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d4a:[<0x10000000ba2/picture-19.jpg v601>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d4a:[<0x10000000ba2/picture-19.jpg v601>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.686 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.434+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004df [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-57.jpg auth v1004 ap=1 s=24922 n(v0 rc2024-05-09T08:27:49.526896+0000 b24922 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea712b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004df:[<0x1000000039a/picture-57.jpg v1004>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004df:[<0x1000000039a/picture-57.jpg v1004>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.686 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.435+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-7.xml auth v1028 ap=1 s=11112 n(v0 rc2024-05-09T08:27:49.328902+0000 b11112 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea713080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a0:[<0x1000000039a/article-7.xml v1028>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004a0:[<0x1000000039a/article-7.xml v1028>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.687 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.441+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a7a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-82.xml auth v967 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.312731+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2262100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a7a:[<0x1000000088d/comment-82.xml v967>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a7a:[<0x1000000088d/comment-82.xml v967>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.687 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.441+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b2c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-32.xml auth v1025 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.774718+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2262680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b2c:[<0x1000000088d/comment-32.xml v1025>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b2c:[<0x1000000088d/comment-32.xml v1025>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.687 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.453+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c0d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-82.jpg auth v184 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.077681+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c0d:[<0x10000000ba2/picture-82.jpg v184>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c0d:[<0x10000000ba2/picture-82.jpg v184>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.687 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.454+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bb3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-27.xml auth v22 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.809688+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bb3:[<0x10000000ba2/article-27.xml v22>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bb3:[<0x10000000ba2/article-27.xml v22>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.687 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.454+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dbc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-4.jpg auth v697 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.391643+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dbc:[<0x10000000ba2/picture-4.jpg v697>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dbc:[<0x10000000ba2/picture-4.jpg v697>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.687 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.435+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000514 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-42.jpg auth v1030 ap=1 s=17719 n(v0 rc2024-05-09T08:27:49.747890+0000 b17719 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea713600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000514:[<0x1000000039a/picture-42.jpg v1030>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000514:[<0x1000000039a/picture-42.jpg v1030>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.688 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.435+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004c4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-68.jpg auth v1010 ap=1 s=53102 n(v0 rc2024-05-09T08:27:49.440899+0000 b53102 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea713b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004c4:[<0x1000000039a/picture-68.jpg v1010>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004c4:[<0x1000000039a/picture-68.jpg v1010>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.688 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.435+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000546 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-39.jpg auth v994 ap=1 s=34478 n(v0 rc2024-05-09T08:27:49.957884+0000 b34478 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea714100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000546:[<0x1000000039a/picture-39.jpg v994>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000546:[<0x1000000039a/picture-39.jpg v994>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.688 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.442+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000976 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-20.jpg auth v926 ap=1 s=14740 n(v0 rc2024-05-09T08:27:53.562781+0000 b14740 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2261b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000976:[<0x1000000088d/picture-20.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000976:[<0x1000000088d/picture-20.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.688 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.442+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ac2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-63.xml auth v983 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.497726+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2262c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ac2:[<0x1000000088d/comment-63.xml v983>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ac2:[<0x1000000088d/comment-63.xml v983>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.688 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.442+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ae1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-54.xml auth v991 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.579724+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2263180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ae1:[<0x1000000088d/comment-54.xml v991>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ae1:[<0x1000000088d/comment-54.xml v991>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.688 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.435+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d87 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-23.xml auth v1259 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.161650+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea714680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d87:[<0x1000000039a/comment-23.xml v1259>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d87:[<0x1000000039a/comment-23.xml v1259>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.688 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:04.435+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d82 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-24.xml auth v1254 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.139650+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea714c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d82:[<0x1000000039a/comment-24.xml v1254>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d82:[<0x1000000039a/comment-24.xml v1254>,<0x10000000399/blog-0 v24705>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.689 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.454+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cdd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-35.jpg auth v495 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.756661+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cdd:[<0x10000000ba2/picture-35.jpg v495>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cdd:[<0x10000000ba2/picture-35.jpg v495>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.689 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.454+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dc6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-2.jpg auth v709 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.446642+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8c5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dc6:[<0x10000000ba2/picture-2.jpg v709>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dc6:[<0x10000000ba2/picture-2.jpg v709>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.689 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.442+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000089c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-42.xml auth v928 ap=1 s=7642 n(v0 rc2024-05-09T08:27:52.840802+0000 b7642 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2263700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000089c:[<0x1000000088d/article-42.xml v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000089c:[<0x1000000088d/article-42.xml v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.689 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.442+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000947 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-38.jpg auth v906 ap=1 s=52261 n(v0 rc2024-05-09T08:27:53.391786+0000 b52261 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da226e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000947:[<0x1000000088d/picture-38.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000947:[<0x1000000088d/picture-38.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.689 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.455+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c19 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-78.jpg auth v208 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.111680+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fa000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c19:[<0x10000000ba2/picture-78.jpg v208>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c19:[<0x10000000ba2/picture-78.jpg v208>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.690 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.455+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bce [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-20.xml auth v64 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.887686+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fa580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bce:[<0x10000000ba2/article-20.xml v64>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bce:[<0x10000000ba2/article-20.xml v64>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.690 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.442+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000096f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-23.jpg auth v954 ap=1 s=52506 n(v0 rc2024-05-09T08:27:53.539782+0000 b52506 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2186c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000096f:[<0x1000000088d/picture-23.jpg v954>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000096f:[<0x1000000088d/picture-23.jpg v954>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.690 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.442+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008f0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-79.jpg auth v906 ap=1 s=29993 n(v0 rc2024-05-09T08:27:53.112794+0000 b29993 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2187180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008f0:[<0x1000000088d/picture-79.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008f0:[<0x1000000088d/picture-79.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.690 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.455+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d62 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-14.jpg auth v631 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.058653+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d62:[<0x10000000ba2/picture-14.jpg v631>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d62:[<0x10000000ba2/picture-14.jpg v631>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.690 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.455+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c4f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-61.jpg auth v310 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.285675+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fb080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c4f:[<0x10000000ba2/picture-61.jpg v310>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c4f:[<0x10000000ba2/picture-61.jpg v310>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.690 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.456+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dac [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-7.jpg auth v675 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.322645+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fb600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dac:[<0x10000000ba2/picture-7.jpg v675>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dac:[<0x10000000ba2/picture-7.jpg v675>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.691 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.443+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-14.xml auth v926 ap=1 s=2689 n(v0 rc2024-05-09T08:27:52.923799+0000 b2689 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2187700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b8:[<0x1000000088d/article-14.xml v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b8:[<0x1000000088d/article-14.xml v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.691 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.443+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000935 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-45.jpg auth v938 ap=1 s=38993 n(v0 rc2024-05-09T08:27:53.331788+0000 b38993 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000935:[<0x1000000088d/picture-45.jpg v938>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000935:[<0x1000000088d/picture-45.jpg v938>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.691 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.443+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000940 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-41.jpg auth v930 ap=1 s=37039 n(v0 rc2024-05-09T08:27:53.364787+0000 b37039 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000940:[<0x1000000088d/picture-41.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000940:[<0x1000000088d/picture-41.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.691 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.456+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d7e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-9.jpg auth v661 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.131651+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fbb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d7e:[<0x10000000ba2/picture-9.jpg v661>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d7e:[<0x10000000ba2/picture-9.jpg v661>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.691 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.456+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c46 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-64.jpg auth v292 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.255676+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fc100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c46:[<0x10000000ba2/picture-64.jpg v292>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c46:[<0x10000000ba2/picture-64.jpg v292>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.692 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.443+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ac5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-62.xml auth v970 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.506726+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ac5:[<0x1000000088d/comment-62.xml v970>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ac5:[<0x1000000088d/comment-62.xml v970>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.692 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.443+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000916 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-60.jpg auth v958 ap=1 s=30006 n(v0 rc2024-05-09T08:27:53.221791+0000 b30006 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000916:[<0x1000000088d/picture-60.jpg v958>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000916:[<0x1000000088d/picture-60.jpg v958>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.692 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.457+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d78 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-10.jpg auth v655 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.110651+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fc680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d78:[<0x10000000ba2/picture-10.jpg v655>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d78:[<0x10000000ba2/picture-10.jpg v655>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.692 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.457+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dcc [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-1.jpg auth{0=1} v719 pv723 ap=2 s=15961 n(v0 rc2024-05-09T08:27:58.468641+0000 b15961 1=1+0)->n(v0 rc2024-05-09T08:27:58.468641+0000 b15961 1=1+0) (ifile lock w=1) cr={15549=0-4194304@2} caps={15549=pAsLsXsFc/pFscr@33} mcw={0=Fscr} | ptrwaiter=1 request=0 lock=1 importingcaps=0 caps=1 needsnapflush=0 dirtyrstat=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x55c79d8fcc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dcc:[<0x10000000ba2/picture-1.jpg v715>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dcc:[<0x10000000ba2/picture-1.jpg v719>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.692 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.443+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000898 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-46.xml auth v904 ap=1 s=6439 n(v0 rc2024-05-09T08:27:52.829802+0000 b6439 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000898:[<0x1000000088d/article-46.xml v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000898:[<0x1000000088d/article-46.xml v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.692 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.443+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000090a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-66.jpg auth v922 ap=1 s=41993 n(v0 rc2024-05-09T08:27:53.186792+0000 b41993 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000090a:[<0x1000000088d/picture-66.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000090a:[<0x1000000088d/picture-66.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.693 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.457+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000db2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-6.jpg auth v684 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.343645+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fd700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000db2:[<0x10000000ba2/picture-6.jpg v684>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000db2:[<0x10000000ba2/picture-6.jpg v684>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.693 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.458+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bf1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-9.xml auth v130 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.983683+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8fd180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bf1:[<0x10000000ba2/article-9.xml v130>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bf1:[<0x10000000ba2/article-9.xml v130>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.693 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.444+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008ee [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-80.jpg auth v950 ap=1 s=55792 n(v0 rc2024-05-09T08:27:53.105794+0000 b55792 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008ee:[<0x1000000088d/picture-80.jpg v950>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008ee:[<0x1000000088d/picture-80.jpg v950>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.693 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.444+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000897 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-47.xml auth v936 ap=1 s=1044 n(v0 rc2024-05-09T08:27:52.826802+0000 b1044 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000897:[<0x1000000088d/article-47.xml v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000897:[<0x1000000088d/article-47.xml v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.693 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.444+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008d3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-91.jpg auth v920 ap=1 s=61967 n(v0 rc2024-05-09T08:27:52.996797+0000 b61967 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a2c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008d3:[<0x1000000088d/picture-91.jpg v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008d3:[<0x1000000088d/picture-91.jpg v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.694 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.458+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bc1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-24.xml auth v40 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.853687+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d906000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bc1:[<0x10000000ba2/article-24.xml v40>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bc1:[<0x10000000ba2/article-24.xml v40>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.694 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.458+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d22 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-28.jpg auth v547 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.913657+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d906580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d22:[<0x10000000ba2/picture-28.jpg v547>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d22:[<0x10000000ba2/picture-28.jpg v547>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.694 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.458+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cbe [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-37.jpg auth v475 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.685663+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d906b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cbe:[<0x10000000ba2/picture-37.jpg v475>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cbe:[<0x10000000ba2/picture-37.jpg v475>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.694 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.444+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000972 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-22.jpg auth v928 ap=1 s=16687 n(v0 rc2024-05-09T08:27:53.547782+0000 b16687 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a3180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000972:[<0x1000000088d/picture-22.jpg v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000972:[<0x1000000088d/picture-22.jpg v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.694 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.445+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008bb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-11.xml auth v916 ap=1 s=9307 n(v0 rc2024-05-09T08:27:52.932799+0000 b9307 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21a3700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008bb:[<0x1000000088d/article-11.xml v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008bb:[<0x1000000088d/article-11.xml v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.694 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.459+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d54 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-17.jpg auth v613 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.021654+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d907080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d54:[<0x10000000ba2/picture-17.jpg v613>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d54:[<0x10000000ba2/picture-17.jpg v613>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.695 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.459+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c04 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-3.xml auth v166 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.051681+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d907600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c04:[<0x10000000ba2/article-3.xml v166>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c04:[<0x10000000ba2/article-3.xml v166>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.695 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.445+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000acf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-59.xml auth v972 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.532725+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000acf:[<0x1000000088d/comment-59.xml v972>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000acf:[<0x1000000088d/comment-59.xml v972>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.695 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.445+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-16.xml auth v944 ap=1 s=5949 n(v0 rc2024-05-09T08:27:52.918800+0000 b5949 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b6:[<0x1000000088d/article-16.xml v944>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b6:[<0x1000000088d/article-16.xml v944>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.696 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.459+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d35 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-24.jpg auth v571 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.952656+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d907b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d35:[<0x10000000ba2/picture-24.jpg v571>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d35:[<0x10000000ba2/picture-24.jpg v571>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.696 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.459+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d02 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-33.jpg auth v517 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.837659+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d908100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d02:[<0x10000000ba2/picture-33.jpg v517>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d02:[<0x10000000ba2/picture-33.jpg v517>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.696 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.445+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a8e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-77.xml auth v964 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.358730+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a8e:[<0x1000000088d/comment-77.xml v964>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a8e:[<0x1000000088d/comment-77.xml v964>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.696 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.445+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000890 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-54.xml auth v914 ap=1 s=6747 n(v0 rc2024-05-09T08:27:52.808803+0000 b6747 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000890:[<0x1000000088d/article-54.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000890:[<0x1000000088d/article-54.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.696 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.445+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008ac [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-26.xml auth v908 ap=1 s=17605 n(v0 rc2024-05-09T08:27:52.891800+0000 b17605 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008ac:[<0x1000000088d/article-26.xml v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008ac:[<0x1000000088d/article-26.xml v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.696 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.460+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c5b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-57.jpg auth v334 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.326674+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d908680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c5b:[<0x10000000ba2/picture-57.jpg v334>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c5b:[<0x10000000ba2/picture-57.jpg v334>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.696 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.460+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bf7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-7.xml auth v142 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.000683+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d908c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bf7:[<0x10000000ba2/article-7.xml v142>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bf7:[<0x10000000ba2/article-7.xml v142>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.697 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.460+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c8d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-42.jpg auth v426 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.522668+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d909180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c8d:[<0x10000000ba2/picture-42.jpg v426>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c8d:[<0x10000000ba2/picture-42.jpg v426>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.697 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.446+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000981 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-15.jpg auth v924 ap=1 s=27459 n(v0 rc2024-05-09T08:27:53.599780+0000 b27459 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000981:[<0x1000000088d/picture-15.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000981:[<0x1000000088d/picture-15.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.697 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.446+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000930 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-47.jpg auth v918 ap=1 s=9577 n(v0 rc2024-05-09T08:27:53.313788+0000 b9577 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000930:[<0x1000000088d/picture-47.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000930:[<0x1000000088d/picture-47.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.697 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.460+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ca8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-39.jpg auth v457 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.626665+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d912000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ca8:[<0x10000000ba2/picture-39.jpg v457>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ca8:[<0x10000000ba2/picture-39.jpg v457>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.697 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.460+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c3a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-68.jpg auth v268 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.212677+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d909700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c3a:[<0x10000000ba2/picture-68.jpg v268>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c3a:[<0x10000000ba2/picture-68.jpg v268>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.698 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.446+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000091e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-56.jpg auth v942 ap=1 s=51160 n(v0 rc2024-05-09T08:27:53.253790+0000 b51160 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000091e:[<0x1000000088d/picture-56.jpg v942>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000091e:[<0x1000000088d/picture-56.jpg v942>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.698 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.446+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008bc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-10.xml auth v912 ap=1 s=23961 n(v0 rc2024-05-09T08:27:52.934799+0000 b23961 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2110680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008bc:[<0x1000000088d/article-10.xml v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008bc:[<0x1000000088d/article-10.xml v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.698 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.461+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d31 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-25.jpg auth v565 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.942656+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d912580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d31:[<0x10000000ba2/picture-25.jpg v565>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d31:[<0x10000000ba2/picture-25.jpg v565>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.698 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.461+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d84 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-8.jpg auth v667 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.152650+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d913b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d84:[<0x10000000ba2/picture-8.jpg v667>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d84:[<0x10000000ba2/picture-8.jpg v667>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.699 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.446+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000985 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-12.jpg auth v946 ap=1 s=13680 n(v0 rc2024-05-09T08:27:53.623779+0000 b13680 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000985:[<0x1000000088d/picture-12.jpg v946>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000985:[<0x1000000088d/picture-12.jpg v946>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.699 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.446+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aef [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-50.xml auth v994 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.615723+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aef:[<0x1000000088d/comment-50.xml v994>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aef:[<0x1000000088d/comment-50.xml v994>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.699 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000939 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-44.jpg auth v952 ap=1 s=56936 n(v0 rc2024-05-09T08:27:53.339788+0000 b56936 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21b5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000939:[<0x1000000088d/picture-44.jpg v952>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000939:[<0x1000000088d/picture-44.jpg v952>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.699 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.462+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c73 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-49.jpg auth v382 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.406671+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d913600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c73:[<0x10000000ba2/picture-49.jpg v382>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c73:[<0x10000000ba2/picture-49.jpg v382>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.699 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.462+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bf4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-8.xml auth v136 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.991683+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d912b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bf4:[<0x10000000ba2/article-8.xml v136>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bf4:[<0x10000000ba2/article-8.xml v136>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.699 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.462+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d2b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-26.jpg auth v559 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.933656+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d914100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d2b:[<0x10000000ba2/picture-26.jpg v559>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d2b:[<0x10000000ba2/picture-26.jpg v559>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.700 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000896 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-48.xml auth v948 ap=1 s=2074 n(v0 rc2024-05-09T08:27:52.824802+0000 b2074 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000896:[<0x1000000088d/article-48.xml v948>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000896:[<0x1000000088d/article-48.xml v948>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.700 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000af6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-48.xml auth v996 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.634722+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c4580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000af6:[<0x1000000088d/comment-48.xml v996>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000af6:[<0x1000000088d/comment-48.xml v996>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.700 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.462+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d17 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-31.jpg auth v529 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.882658+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d913080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d17:[<0x10000000ba2/picture-31.jpg v529>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d17:[<0x10000000ba2/picture-31.jpg v529>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.700 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.462+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bcb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-21.xml auth v58 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.877686+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d914680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bcb:[<0x10000000ba2/article-21.xml v58>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bcb:[<0x10000000ba2/article-21.xml v58>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.700 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008c2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-4.xml auth v940 ap=1 s=23282 n(v0 rc2024-05-09T08:27:52.950799+0000 b23282 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008c2:[<0x1000000088d/article-4.xml v940>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008c2:[<0x1000000088d/article-4.xml v940>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.700 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008ad [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-25.xml auth v910 ap=1 s=7883 n(v0 rc2024-05-09T08:27:52.894800+0000 b7883 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008ad:[<0x1000000088d/article-25.xml v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008ad:[<0x1000000088d/article-25.xml v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.701 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.462+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ba4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-30.xml auth v4 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.741690+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d914c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ba4:[<0x10000000ba2/article-30.xml v4>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ba4:[<0x10000000ba2/article-30.xml v4>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.701 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.463+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bfa [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-6.xml auth v148 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.010683+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d915700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bfa:[<0x10000000ba2/article-6.xml v148>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bfa:[<0x10000000ba2/article-6.xml v148>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.701 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008fe [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-72.jpg auth v932 ap=1 s=15020 n(v0 rc2024-05-09T08:27:53.152793+0000 b15020 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008fe:[<0x1000000088d/picture-72.jpg v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008fe:[<0x1000000088d/picture-72.jpg v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.701 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000918 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-59.jpg auth v934 ap=1 s=33054 n(v0 rc2024-05-09T08:27:53.227791+0000 b33054 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000918:[<0x1000000088d/picture-59.jpg v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000918:[<0x1000000088d/picture-59.jpg v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.701 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-38.xml auth v956 ap=1 s=12757 n(v0 rc2024-05-09T08:27:52.853801+0000 b12757 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c6100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a0:[<0x1000000088d/article-38.xml v956>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a0:[<0x1000000088d/article-38.xml v956>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.702 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.463+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c96 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-13.xml auth v444 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.551667+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d915180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c96:[<0x10000000ba2/article-13.xml v444>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c96:[<0x10000000ba2/article-13.xml v444>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.702 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.463+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c07 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-2.xml auth v172 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.061681+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d922000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c07:[<0x10000000ba2/article-2.xml v172>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c07:[<0x10000000ba2/article-2.xml v172>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.702 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.463+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c64 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-54.jpg auth v352 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.353673+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d922580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c64:[<0x10000000ba2/picture-54.jpg v352>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c64:[<0x10000000ba2/picture-54.jpg v352>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.702 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.448+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b2f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-31.xml auth v1028 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.782718+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da21c6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b2f:[<0x1000000088d/comment-31.xml v1028>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b2f:[<0x1000000088d/comment-31.xml v1028>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.702 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.448+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a7e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-81.xml auth v947 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.321731+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2153600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a7e:[<0x1000000088d/comment-81.xml v947>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a7e:[<0x1000000088d/comment-81.xml v947>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.703 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.464+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c10 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-81.jpg auth v190 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.086680+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d923080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c10:[<0x10000000ba2/picture-81.jpg v190>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c10:[<0x10000000ba2/picture-81.jpg v190>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.703 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.464+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d1f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-29.jpg auth v541 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.903657+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d922b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d1f:[<0x10000000ba2/picture-29.jpg v541>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d1f:[<0x10000000ba2/picture-29.jpg v541>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.703 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.448+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b56 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-10.xml auth v1053 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.888715+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2153b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b56:[<0x1000000088d/comment-10.xml v1053>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b56:[<0x1000000088d/comment-10.xml v1053>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.703 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.448+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ae4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-53.xml auth v999 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.588723+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2154680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ae4:[<0x1000000088d/comment-53.xml v999>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ae4:[<0x1000000088d/comment-53.xml v999>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.703 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.448+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b66 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-6.xml auth v1065 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.931713+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2154c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b66:[<0x1000000088d/comment-6.xml v1065>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b66:[<0x1000000088d/comment-6.xml v1065>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.703 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.464+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000db7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-5.jpg auth v691 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.365644+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d923600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000db7:[<0x10000000ba2/picture-5.jpg v691>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000db7:[<0x10000000ba2/picture-5.jpg v691>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.704 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.464+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c70 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-50.jpg auth v376 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.395672+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d923b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c70:[<0x10000000ba2/picture-50.jpg v376>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c70:[<0x10000000ba2/picture-50.jpg v376>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.704 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.448+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000920 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-55.jpg auth v916 ap=1 s=39115 n(v0 rc2024-05-09T08:27:53.263790+0000 b39115 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2155180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000920:[<0x1000000088d/picture-55.jpg v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000920:[<0x1000000088d/picture-55.jpg v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.704 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.449+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009a8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-3.jpg auth v908 ap=1 s=45604 n(v0 rc2024-05-09T08:27:53.745776+0000 b45604 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2154100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009a8:[<0x1000000088d/picture-3.jpg v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009a8:[<0x1000000088d/picture-3.jpg v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.704 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.465+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c57 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-58.jpg auth v328 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.316674+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d924100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c57:[<0x10000000ba2/picture-58.jpg v328>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c57:[<0x10000000ba2/picture-58.jpg v328>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.704 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.465+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c43 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-65.jpg auth v286 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.245676+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d924680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c43:[<0x10000000ba2/picture-65.jpg v286>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c43:[<0x10000000ba2/picture-65.jpg v286>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.704 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.465+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000baf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-28.xml auth v16 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.794689+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d924c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000baf:[<0x10000000ba2/article-28.xml v16>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000baf:[<0x10000000ba2/article-28.xml v16>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.705 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.449+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000987 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-11.jpg auth v926 ap=1 s=36402 n(v0 rc2024-05-09T08:27:53.628779+0000 b36402 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2155700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000987:[<0x1000000088d/picture-11.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000987:[<0x1000000088d/picture-11.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.705 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.449+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008c1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-5.xml auth v918 ap=1 s=26956 n(v0 rc2024-05-09T08:27:52.948799+0000 b26956 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2172580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008c1:[<0x1000000088d/article-5.xml v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008c1:[<0x1000000088d/article-5.xml v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.705 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.465+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c1c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-77.jpg auth v214 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.121679+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d925180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c1c:[<0x10000000ba2/picture-77.jpg v214>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c1c:[<0x10000000ba2/picture-77.jpg v214>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.705 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.466+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c37 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-69.jpg auth v262 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.201677+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d925700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c37:[<0x10000000ba2/picture-69.jpg v262>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c37:[<0x10000000ba2/picture-69.jpg v262>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.706 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.449+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-34.xml auth v904 ap=1 s=20716 n(v0 rc2024-05-09T08:27:52.867801+0000 b20716 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2172b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a4:[<0x1000000088d/article-34.xml v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a4:[<0x1000000088d/article-34.xml v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.706 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.449+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000902 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-70.jpg auth v912 ap=1 s=25398 n(v0 rc2024-05-09T08:27:53.163793+0000 b25398 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2172000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000902:[<0x1000000088d/picture-70.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000902:[<0x1000000088d/picture-70.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.706 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.450+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b22 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-35.xml auth v1019 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.748719+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2173080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b22:[<0x1000000088d/comment-35.xml v1019>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b22:[<0x1000000088d/comment-35.xml v1019>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.706 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.466+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c6c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-51.jpg auth v370 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.385672+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d7bfb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c6c:[<0x10000000ba2/picture-51.jpg v370>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c6c:[<0x10000000ba2/picture-51.jpg v370>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.706 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.466+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c4b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-62.jpg auth v304 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.276675+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c4b:[<0x10000000ba2/picture-62.jpg v304>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c4b:[<0x10000000ba2/picture-62.jpg v304>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.707 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.450+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b3a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-28.xml auth v1027 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.816717+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2173600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b3a:[<0x1000000088d/comment-28.xml v1027>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b3a:[<0x1000000088d/comment-28.xml v1027>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.707 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.450+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ad6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-57.xml auth v988 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.551724+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2173b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ad6:[<0x1000000088d/comment-57.xml v988>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ad6:[<0x1000000088d/comment-57.xml v988>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.707 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.450+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000900 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-71.jpg auth v906 ap=1 s=61555 n(v0 rc2024-05-09T08:27:53.157793+0000 b61555 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2174100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000900:[<0x1000000088d/picture-71.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000900:[<0x1000000088d/picture-71.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.707 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.450+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-29.xml auth v930 ap=1 s=10277 n(v0 rc2024-05-09T08:27:52.884801+0000 b10277 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2174680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a9:[<0x1000000088d/article-29.xml v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a9:[<0x1000000088d/article-29.xml v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.707 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.466+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c7c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-46.jpg auth v400 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.438670+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c7c:[<0x10000000ba2/picture-46.jpg v400>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c7c:[<0x10000000ba2/picture-46.jpg v400>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.707 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.466+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dc0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-3.jpg auth v703 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.417642+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dc0:[<0x10000000ba2/picture-3.jpg v703>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dc0:[<0x10000000ba2/picture-3.jpg v703>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.707 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.466+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c61 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-55.jpg auth v346 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.344673+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c61:[<0x10000000ba2/picture-55.jpg v346>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c61:[<0x10000000ba2/picture-55.jpg v346>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.708 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.450+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000942 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-40.jpg auth v920 ap=1 s=10153 n(v0 rc2024-05-09T08:27:53.373787+0000 b10153 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2175180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000942:[<0x1000000088d/picture-40.jpg v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000942:[<0x1000000088d/picture-40.jpg v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.708 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.451+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000910 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-63.jpg auth v924 ap=1 s=41074 n(v0 rc2024-05-09T08:27:53.204791+0000 b41074 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2174c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000910:[<0x1000000088d/picture-63.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000910:[<0x1000000088d/picture-63.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.708 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.451+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ac8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-61.xml auth v986 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.514725+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2175700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ac8:[<0x1000000088d/comment-61.xml v986>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ac8:[<0x1000000088d/comment-61.xml v986>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.708 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.467+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d73 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-11.jpg auth v649 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.096652+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d73:[<0x10000000ba2/picture-11.jpg v649>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d73:[<0x10000000ba2/picture-11.jpg v649>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.708 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.467+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c34 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-70.jpg auth v256 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.191677+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c34:[<0x10000000ba2/picture-70.jpg v256>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c34:[<0x10000000ba2/picture-70.jpg v256>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.708 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.452+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-22.xml auth v914 ap=1 s=17457 n(v0 rc2024-05-09T08:27:52.902800+0000 b17457 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2184000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b0:[<0x1000000088d/article-22.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b0:[<0x1000000088d/article-22.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.709 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.452+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b18 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-38.xml auth v1013 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.723719+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2184580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b18:[<0x1000000088d/comment-38.xml v1013>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b18:[<0x1000000088d/comment-38.xml v1013>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.709 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.467+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c31 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-71.jpg auth v250 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.179678+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c31:[<0x10000000ba2/picture-71.jpg v250>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c31:[<0x10000000ba2/picture-71.jpg v250>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.709 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.468+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bfe [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-5.xml auth v154 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.021682+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bfe:[<0x10000000ba2/article-5.xml v154>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bfe:[<0x10000000ba2/article-5.xml v154>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.709 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.453+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ab0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-68.xml auth v980 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.452727+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2184b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ab0:[<0x1000000088d/comment-68.xml v980>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ab0:[<0x1000000088d/comment-68.xml v980>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.710 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.453+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008de [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-87.jpg auth v928 ap=1 s=59713 n(v0 rc2024-05-09T08:27:53.040796+0000 b59713 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2185080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008de:[<0x1000000088d/picture-87.jpg v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008de:[<0x1000000088d/picture-87.jpg v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.710 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.453+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008e4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-84.jpg auth v910 ap=1 s=47192 n(v0 rc2024-05-09T08:27:53.062795+0000 b47192 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2185600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008e4:[<0x1000000088d/picture-84.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008e4:[<0x1000000088d/picture-84.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.710 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000895 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-49.xml auth v922 ap=1 s=20213 n(v0 rc2024-05-09T08:27:52.821802+0000 b20213 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2185b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000895:[<0x1000000088d/article-49.xml v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000895:[<0x1000000088d/article-49.xml v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.710 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b5e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-8.xml auth v1061 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.908714+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x555da2186100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b5e:[<0x1000000088d/comment-8.xml v1061>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b5e:[<0x1000000088d/comment-8.xml v1061>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.710 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a82 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-80.xml auth v949 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.331731+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2186680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a82:[<0x1000000088d/comment-80.xml v949>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a82:[<0x1000000088d/comment-80.xml v949>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.710 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b41 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-22.xml auth v1046 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.835716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2118680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b41:[<0x1000000088d/comment-22.xml v1046>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b41:[<0x1000000088d/comment-22.xml v1046>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.710 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.468+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ba9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-29.xml auth v10 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.770690+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e2c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ba9:[<0x10000000ba2/article-29.xml v10>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ba9:[<0x10000000ba2/article-29.xml v10>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.710 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.468+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c49 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-63.jpg auth v298 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.266675+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e3180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c49:[<0x10000000ba2/picture-63.jpg v298>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c49:[<0x10000000ba2/picture-63.jpg v298>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.710 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.468+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c9e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-40.jpg auth v451 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.581666+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8e3700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c9e:[<0x10000000ba2/picture-40.jpg v451>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c9e:[<0x10000000ba2/picture-40.jpg v451>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.711 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.455+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b76 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-2.xml auth v1074 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.980712+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2118c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b76:[<0x1000000088d/comment-2.xml v1074>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b76:[<0x1000000088d/comment-2.xml v1074>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.711 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.455+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008e0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-86.jpg auth v936 ap=1 s=35605 n(v0 rc2024-05-09T08:27:53.047796+0000 b35605 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2119180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008e0:[<0x1000000088d/picture-86.jpg v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008e0:[<0x1000000088d/picture-86.jpg v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.711 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.469+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bc8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-22.xml auth v52 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.867687+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8cc000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bc8:[<0x10000000ba2/article-22.xml v52>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bc8:[<0x10000000ba2/article-22.xml v52>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.711 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.469+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d39 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-23.jpg auth v577 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.962655+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8cc580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d39:[<0x10000000ba2/picture-23.jpg v577>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d39:[<0x10000000ba2/picture-23.jpg v577>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.712 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.456+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b03 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-44.xml auth v1006 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.672721+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2119700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b03:[<0x1000000088d/comment-44.xml v1006>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b03:[<0x1000000088d/comment-44.xml v1006>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.712 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.456+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009f0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-94.jpg auth v942 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.934742+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009f0:[<0x1000000088d/picture-94.jpg v942>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009f0:[<0x1000000088d/picture-94.jpg v942>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.712 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.456+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000089b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-43.xml auth v924 ap=1 s=15278 n(v0 rc2024-05-09T08:27:52.837802+0000 b15278 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000089b:[<0x1000000088d/article-43.xml v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000089b:[<0x1000000088d/article-43.xml v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.712 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.457+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-33.xml auth v928 ap=1 s=28954 n(v0 rc2024-05-09T08:27:52.870801+0000 b28954 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a5:[<0x1000000088d/article-33.xml v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a5:[<0x1000000088d/article-33.xml v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.712 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.457+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008ec [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-81.jpg auth v926 ap=1 s=57695 n(v0 rc2024-05-09T08:27:53.096794+0000 b57695 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008ec:[<0x1000000088d/picture-81.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008ec:[<0x1000000088d/picture-81.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.712 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.469+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c16 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-79.jpg auth v202 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.103680+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8ccb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c16:[<0x10000000ba2/picture-79.jpg v202>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c16:[<0x10000000ba2/picture-79.jpg v202>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.712 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.469+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000be2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-14.xml auth v100 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.939685+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8cd080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000be2:[<0x10000000ba2/article-14.xml v100>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000be2:[<0x10000000ba2/article-14.xml v100>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.713 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.458+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009a2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-5.jpg auth v934 ap=1 s=28683 n(v0 rc2024-05-09T08:27:53.726776+0000 b28683 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009a2:[<0x1000000088d/picture-5.jpg v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009a2:[<0x1000000088d/picture-5.jpg v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.713 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.459+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b3c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-27.xml auth v1030 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.822717+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b3c:[<0x1000000088d/comment-27.xml v1030>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b3c:[<0x1000000088d/comment-27.xml v1030>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.713 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.470+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c7f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-45.jpg auth v406 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.447670+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8cd600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c7f:[<0x10000000ba2/picture-45.jpg v406>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c7f:[<0x10000000ba2/picture-45.jpg v406>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.713 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.470+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c94 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-41.jpg auth v438 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.545667+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8cdb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c94:[<0x10000000ba2/picture-41.jpg v438>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c94:[<0x10000000ba2/picture-41.jpg v438>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.713 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.470+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c52 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-60.jpg auth v316 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.295675+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8ce100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c52:[<0x10000000ba2/picture-60.jpg v316>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c52:[<0x10000000ba2/picture-60.jpg v316>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.714 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.459+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000092a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-50.jpg auth v912 ap=1 s=17156 n(v0 rc2024-05-09T08:27:53.295789+0000 b17156 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000092a:[<0x1000000088d/picture-50.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000092a:[<0x1000000088d/picture-50.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.714 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.459+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a92 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-76.xml auth v955 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.368730+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a92:[<0x1000000088d/comment-76.xml v955>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a92:[<0x1000000088d/comment-76.xml v955>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.714 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.460+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000091a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-58.jpg auth v916 ap=1 s=57580 n(v0 rc2024-05-09T08:27:53.234791+0000 b57580 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000091a:[<0x1000000088d/picture-58.jpg v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000091a:[<0x1000000088d/picture-58.jpg v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.714 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.470+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c40 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-66.jpg auth v280 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.234676+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8ce680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c40:[<0x10000000ba2/picture-66.jpg v280>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c40:[<0x10000000ba2/picture-66.jpg v280>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.714 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.470+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c13 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-80.jpg auth v196 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.094680+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8cec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c13:[<0x10000000ba2/picture-80.jpg v196>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c13:[<0x10000000ba2/picture-80.jpg v196>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.714 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.460+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ae8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-52.xml auth v1002 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.598723+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da213d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ae8:[<0x1000000088d/comment-52.xml v1002>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ae8:[<0x1000000088d/comment-52.xml v1002>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.715 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.460+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000090c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-65.jpg auth v914 ap=1 s=47458 n(v0 rc2024-05-09T08:27:53.192792+0000 b47458 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2148000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000090c:[<0x1000000088d/picture-65.jpg v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000090c:[<0x1000000088d/picture-65.jpg v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.715 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.470+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d3e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-22.jpg auth v583 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.972655+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8cf180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d3e:[<0x10000000ba2/picture-22.jpg v583>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d3e:[<0x10000000ba2/picture-22.jpg v583>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.715 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.471+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000beb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-11.xml auth v118 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.965684+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d8cf700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000beb:[<0x10000000ba2/article-11.xml v118>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000beb:[<0x10000000ba2/article-11.xml v118>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.715 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.461+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ab7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-66.xml auth v959 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.470727+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x555da213d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ab7:[<0x1000000088d/comment-66.xml v959>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ab7:[<0x1000000088d/comment-66.xml v959>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.715 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.461+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b62 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-7.xml auth v1067 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.920714+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2148580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b62:[<0x1000000088d/comment-7.xml v1067>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b62:[<0x1000000088d/comment-7.xml v1067>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.716 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.471+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bb8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-26.xml auth v28 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.828688+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d932580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bb8:[<0x10000000ba2/article-26.xml v28>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bb8:[<0x10000000ba2/article-26.xml v28>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.716 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.471+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bda [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-16.xml auth v88 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.924685+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d932000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bda:[<0x10000000ba2/article-16.xml v88>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bda:[<0x10000000ba2/article-16.xml v88>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.716 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.471+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c5e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-56.jpg auth v340 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.335673+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d933080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c5e:[<0x10000000ba2/picture-56.jpg v340>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c5e:[<0x10000000ba2/picture-56.jpg v340>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.716 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.461+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b48 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-15.xml auth v1053 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.853716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2148b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b48:[<0x1000000088d/comment-15.xml v1053>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b48:[<0x1000000088d/comment-15.xml v1053>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.716 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.461+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000891 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-53.xml auth v920 ap=1 s=12646 n(v0 rc2024-05-09T08:27:52.810803+0000 b12646 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2149080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000891:[<0x1000000088d/article-53.xml v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000891:[<0x1000000088d/article-53.xml v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.716 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.461+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b07 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-43.xml auth v1008 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.682721+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2149600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b07:[<0x1000000088d/comment-43.xml v1008>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b07:[<0x1000000088d/comment-43.xml v1008>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.717 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.462+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b46 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-17.xml auth v1051 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.847716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2149b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b46:[<0x1000000088d/comment-17.xml v1051>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b46:[<0x1000000088d/comment-17.xml v1051>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.717 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.462+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008aa [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-28.xml auth v910 ap=1 s=14990 n(v0 rc2024-05-09T08:27:52.886800+0000 b14990 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da214a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008aa:[<0x1000000088d/article-28.xml v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008aa:[<0x1000000088d/article-28.xml v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.717 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.472+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d5d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-15.jpg auth v625 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.045653+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d932b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d5d:[<0x10000000ba2/picture-15.jpg v625>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d5d:[<0x10000000ba2/picture-15.jpg v625>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.717 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.472+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c79 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-47.jpg auth v394 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.429671+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d933600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c79:[<0x10000000ba2/picture-47.jpg v394>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c79:[<0x10000000ba2/picture-47.jpg v394>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.717 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.462+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000892 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-52.xml auth v908 ap=1 s=7818 n(v0 rc2024-05-09T08:27:52.813803+0000 b7818 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da214ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000892:[<0x1000000088d/article-52.xml v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000892:[<0x1000000088d/article-52.xml v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.717 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.462+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b5a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-9.xml auth v1059 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.898714+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da214a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b5a:[<0x1000000088d/comment-9.xml v1059>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b5a:[<0x1000000088d/comment-9.xml v1059>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.718 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.472+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bee [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-10.xml auth v124 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.974684+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d933b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bee:[<0x10000000ba2/article-10.xml v124>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bee:[<0x10000000ba2/article-10.xml v124>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.718 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.472+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d6e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-12.jpg auth v643 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.083652+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d934100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d6e:[<0x10000000ba2/picture-12.jpg v643>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d6e:[<0x10000000ba2/picture-12.jpg v643>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.718 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.463+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008f4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-77.jpg auth v918 ap=1 s=33164 n(v0 rc2024-05-09T08:27:53.123794+0000 b33164 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da214b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008f4:[<0x1000000088d/picture-77.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008f4:[<0x1000000088d/picture-77.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.718 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.463+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000904 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-69.jpg auth v930 ap=1 s=48785 n(v0 rc2024-05-09T08:27:53.169792+0000 b48785 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da214b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000904:[<0x1000000088d/picture-69.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000904:[<0x1000000088d/picture-69.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.718 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.463+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000928 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-51.jpg auth v906 ap=1 s=56769 n(v0 rc2024-05-09T08:27:53.290789+0000 b56769 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2152000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000928:[<0x1000000088d/picture-51.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000928:[<0x1000000088d/picture-51.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.719 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.472+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c83 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-44.jpg auth v412 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.471669+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d934680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c83:[<0x10000000ba2/picture-44.jpg v412>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c83:[<0x10000000ba2/picture-44.jpg v412>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.719 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.473+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bbc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-25.xml auth v34 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.843687+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d935180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bbc:[<0x10000000ba2/article-25.xml v34>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bbc:[<0x10000000ba2/article-25.xml v34>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.719 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.473+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c00 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-4.xml auth v160 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.038682+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@4},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d934c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c00:[<0x10000000ba2/article-4.xml v160>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c00:[<0x10000000ba2/article-4.xml v160>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.719 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.463+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b00 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-45.xml auth v1004 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.662721+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2152580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b00:[<0x1000000088d/comment-45.xml v1004>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b00:[<0x1000000088d/comment-45.xml v1004>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.719 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.464+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000912 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-62.jpg auth v932 ap=1 s=53326 n(v0 rc2024-05-09T08:27:53.210791+0000 b53326 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2152b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000912:[<0x1000000088d/picture-62.jpg v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000912:[<0x1000000088d/picture-62.jpg v932>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.720 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.473+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c2e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-72.jpg auth v244 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.168678+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d935700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c2e:[<0x10000000ba2/picture-72.jpg v244>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c2e:[<0x10000000ba2/picture-72.jpg v244>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.720 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.473+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c55 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-59.jpg auth v322 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.306674+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d93e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c55:[<0x10000000ba2/picture-59.jpg v322>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c55:[<0x10000000ba2/picture-59.jpg v322>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.720 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.473+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c2b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-73.jpg auth v238 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.160678+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d93e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c2b:[<0x10000000ba2/picture-73.jpg v238>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c2b:[<0x10000000ba2/picture-73.jpg v238>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.720 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.473+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d42 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-21.jpg auth v589 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.983655+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d93eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d42:[<0x10000000ba2/picture-21.jpg v589>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d42:[<0x10000000ba2/picture-21.jpg v589>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.720 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.464+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000933 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-46.jpg auth v922 ap=1 s=35098 n(v0 rc2024-05-09T08:27:53.322788+0000 b35098 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2153080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000933:[<0x1000000088d/picture-46.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000933:[<0x1000000088d/picture-46.jpg v922>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.720 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.465+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000088f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-55.xml auth v916 ap=1 s=23203 n(v0 rc2024-05-09T08:27:52.805803+0000 b23203 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da226e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000088f:[<0x1000000088d/article-55.xml v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000088f:[<0x1000000088d/article-55.xml v916>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.720 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.466+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000096a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-25.jpg auth v920 ap=1 s=28362 n(v0 rc2024-05-09T08:27:53.523782+0000 b28362 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da226eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000096a:[<0x1000000088d/picture-25.jpg v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000096a:[<0x1000000088d/picture-25.jpg v920>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.721 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.473+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c23 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-75.jpg auth v226 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.140679+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d93f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c23:[<0x10000000ba2/picture-75.jpg v226>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c23:[<0x10000000ba2/picture-75.jpg v226>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.721 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.474+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ce2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-100.jpg auth v499 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.765661+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d93f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ce2:[<0x10000000ba2/picture-100.jpg v499>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ce2:[<0x10000000ba2/picture-100.jpg v499>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.721 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.474+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cf1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-97.xml auth v511 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.808660+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d93fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cf1:[<0x10000000ba2/article-97.xml v511>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cf1:[<0x10000000ba2/article-97.xml v511>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.721 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.466+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008be [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-8.xml auth v940 ap=1 s=25693 n(v0 rc2024-05-09T08:27:52.940799+0000 b25693 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da226f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008be:[<0x1000000088d/article-8.xml v940>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008be:[<0x1000000088d/article-8.xml v940>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.721 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.466+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000af9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-47.xml auth v998 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.644722+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da226f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000af9:[<0x1000000088d/comment-47.xml v998>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000af9:[<0x1000000088d/comment-47.xml v998>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.721 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.474+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d1a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-30.jpg auth v535 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.892657+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d940100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d1a:[<0x10000000ba2/picture-30.jpg v535>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d1a:[<0x10000000ba2/picture-30.jpg v535>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.722 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.474+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bde [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-15.xml auth v94 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.932685+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d940680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bde:[<0x10000000ba2/article-15.xml v94>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bde:[<0x10000000ba2/article-15.xml v94>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.722 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.466+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000afc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-46.xml auth v1000 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.653721+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da226fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000afc:[<0x1000000088d/comment-46.xml v1000>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000afc:[<0x1000000088d/comment-46.xml v1000>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.722 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.466+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000958 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-31.jpg auth v930 ap=1 s=51091 n(v0 rc2024-05-09T08:27:53.460784+0000 b51091 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2270100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000958:[<0x1000000088d/picture-31.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000958:[<0x1000000088d/picture-31.jpg v930>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.722 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.474+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ccd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-36.jpg auth v487 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.719662+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d940c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ccd:[<0x10000000ba2/picture-36.jpg v487>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ccd:[<0x10000000ba2/picture-36.jpg v487>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.722 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.474+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c8a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-43.jpg auth v418 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.502669+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d941180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c8a:[<0x10000000ba2/picture-43.jpg v418>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c8a:[<0x10000000ba2/picture-43.jpg v418>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.723 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.467+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aa9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-70.xml auth v978 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.435728+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2270680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aa9:[<0x1000000088d/comment-70.xml v978>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aa9:[<0x1000000088d/comment-70.xml v978>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.723 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.467+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000092c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-49.jpg auth v918 ap=1 s=60564 n(v0 rc2024-05-09T08:27:53.301789+0000 b60564 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2270c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000092c:[<0x1000000088d/picture-49.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000092c:[<0x1000000088d/picture-49.jpg v918>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.723 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.468+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000968 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-26.jpg auth v938 ap=1 s=23282 n(v0 rc2024-05-09T08:27:53.511783+0000 b23282 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2282000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000968:[<0x1000000088d/picture-26.jpg v938>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000968:[<0x1000000088d/picture-26.jpg v938>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.723 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.474+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bd0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-19.xml auth v70 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.894686+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d941700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bd0:[<0x10000000ba2/article-19.xml v70>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bd0:[<0x10000000ba2/article-19.xml v70>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.723 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.475+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d47 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-20.jpg auth v595 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.992655+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d950000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d47:[<0x10000000ba2/picture-20.jpg v595>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d47:[<0x10000000ba2/picture-20.jpg v595>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.723 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.475+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cb3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-38.jpg auth v463 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.649664+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d950580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cb3:[<0x10000000ba2/picture-38.jpg v463>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cb3:[<0x10000000ba2/picture-38.jpg v463>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.724 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.468+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000997 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-8.jpg auth v924 ap=1 s=52342 n(v0 rc2024-05-09T08:27:53.695777+0000 b52342 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2271180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000997:[<0x1000000088d/picture-8.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000997:[<0x1000000088d/picture-8.jpg v924>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.724 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.468+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008dc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-88.jpg auth v922 ap=1 s=33972 n(v0 rc2024-05-09T08:27:53.025796+0000 b33972 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2271700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008dc:[<0x1000000088d/picture-88.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008dc:[<0x1000000088d/picture-88.jpg v922>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.724 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.475+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d27 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-27.jpg auth v553 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.922657+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d950b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d27:[<0x10000000ba2/picture-27.jpg v553>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d27:[<0x10000000ba2/picture-27.jpg v553>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.724 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.475+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000be8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-12.xml auth v112 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.957684+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d951080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000be8:[<0x10000000ba2/article-12.xml v112>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000be8:[<0x10000000ba2/article-12.xml v112>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.724 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.469+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-32.xml auth v936 ap=1 s=26398 n(v0 rc2024-05-09T08:27:52.873801+0000 b26398 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2282580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a6:[<0x1000000088d/article-32.xml v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a6:[<0x1000000088d/article-32.xml v936>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.724 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.469+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aad [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-69.xml auth v980 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.443727+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2283080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aad:[<0x1000000088d/comment-69.xml v980>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aad:[<0x1000000088d/comment-69.xml v980>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.725 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.475+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c6a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-52.jpg auth v364 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.373672+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d951600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c6a:[<0x10000000ba2/picture-52.jpg v364>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c6a:[<0x10000000ba2/picture-52.jpg v364>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.725 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.476+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c0a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-1.xml auth v178 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.069681+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@3},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d951b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c0a:[<0x10000000ba2/article-1.xml v178>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c0a:[<0x10000000ba2/article-1.xml v178>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.725 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.469+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a76 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-83.xml auth v964 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.303731+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2282b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a76:[<0x1000000088d/comment-83.xml v964>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a76:[<0x1000000088d/comment-83.xml v964>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.725 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.469+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008cb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-95.jpg auth v914 ap=1 s=20193 n(v0 rc2024-05-09T08:27:52.974798+0000 b20193 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2283600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008cb:[<0x1000000088d/picture-95.jpg v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008cb:[<0x1000000088d/picture-95.jpg v914>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.725 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.470+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-21.xml auth v934 ap=1 s=13593 n(v0 rc2024-05-09T08:27:52.905800+0000 b13593 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2283b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b1:[<0x1000000088d/article-21.xml v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b1:[<0x1000000088d/article-21.xml v934>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.726 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.476+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cef [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-34.jpg auth v507 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.793660+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d952100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cef:[<0x10000000ba2/picture-34.jpg v507>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cef:[<0x10000000ba2/picture-34.jpg v507>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.726 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.476+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c67 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-53.jpg auth v358 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.363673+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d952680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c67:[<0x10000000ba2/picture-53.jpg v358>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c67:[<0x10000000ba2/picture-53.jpg v358>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.726 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.476+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c3d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-67.jpg auth v274 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.223677+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d952c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c3d:[<0x10000000ba2/picture-67.jpg v274>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c3d:[<0x10000000ba2/picture-67.jpg v274>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.726 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.470+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-30.xml auth v928 ap=1 s=18575 n(v0 rc2024-05-09T08:27:52.880801+0000 b18575 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2284100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a8:[<0x1000000088d/article-30.xml v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a8:[<0x1000000088d/article-30.xml v928>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.726 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.470+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a9a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-74.xml auth v970 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.386729+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2284680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a9a:[<0x1000000088d/comment-74.xml v970>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a9a:[<0x1000000088d/comment-74.xml v970>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.727 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.476+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d58 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-16.jpg auth v619 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.032653+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d953180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d58:[<0x10000000ba2/picture-16.jpg v619>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d58:[<0x10000000ba2/picture-16.jpg v619>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.727 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.476+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c27 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-74.jpg auth v232 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.152679+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d953700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c27:[<0x10000000ba2/picture-74.jpg v232>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c27:[<0x10000000ba2/picture-74.jpg v232>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.727 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.470+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b1e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-36.xml auth v1019 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.740719+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2284c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b1e:[<0x1000000088d/comment-36.xml v1019>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b1e:[<0x1000000088d/comment-36.xml v1019>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.727 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.470+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000abe [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-64.xml auth v984 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.488726+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2285700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000abe:[<0x1000000088d/comment-64.xml v984>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000abe:[<0x1000000088d/comment-64.xml v984>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.728 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.477+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bc6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-23.xml auth v46 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.862687+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@7},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d95e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bc6:[<0x10000000ba2/article-23.xml v46>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bc6:[<0x10000000ba2/article-23.xml v46>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.728 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.477+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bd6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-17.xml auth v82 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.911685+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d95e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bd6:[<0x10000000ba2/article-17.xml v82>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bd6:[<0x10000000ba2/article-17.xml v82>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.728 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.477+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c1f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-76.jpg auth v220 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.131679+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d95eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c1f:[<0x10000000ba2/picture-76.jpg v220>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c1f:[<0x10000000ba2/picture-76.jpg v220>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.728 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.477+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d15 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-32.jpg auth v523 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.878658+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d95f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d15:[<0x10000000ba2/picture-32.jpg v523>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d15:[<0x10000000ba2/picture-32.jpg v523>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.728 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.477+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cca [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-63.xml auth v483 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.710663+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d95f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cca:[<0x10000000ba2/article-63.xml v483>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cca:[<0x10000000ba2/article-63.xml v483>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.728 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:28:04.477+0000 7feeb2b0e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c75 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-48.jpg auth v388 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.419671+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c79d95fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c75:[<0x10000000ba2/picture-48.jpg v388>,<0x10000000399/blog-12 v19517>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c75:[<0x10000000ba2/picture-48.jpg v388>,<0x10000000399/blog-12 v24725>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.729 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.470+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-13.xml auth v904 ap=1 s=22483 n(v0 rc2024-05-09T08:27:52.926799+0000 b22483 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2285180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b9:[<0x1000000088d/article-13.xml v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b9:[<0x1000000088d/article-13.xml v904>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.729 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.471+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aa2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-72.xml auth v976 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.407728+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aa2:[<0x1000000088d/comment-72.xml v976>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aa2:[<0x1000000088d/comment-72.xml v976>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.729 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.471+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b6e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-4.xml auth v1070 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.951713+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b6e:[<0x1000000088d/comment-4.xml v1070>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b6e:[<0x1000000088d/comment-4.xml v1070>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.729 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.471+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008c6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-2.xml auth{0=1} v1072 ap=1 s=24924 n(v0 rc2024-05-09T08:27:52.961798+0000 b24924 1=1+0) caps={15549=pAsLsXsFscr/pFscr@4} mcw={0=Fscr} | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x555da228b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008c6:[<0x1000000088d/article-2.xml v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008c6:[<0x1000000088d/article-2.xml v1072>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.729 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.471+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000894 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-50.xml auth v912 ap=1 s=21240 n(v0 rc2024-05-09T08:27:52.818802+0000 b21240 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000894:[<0x1000000088d/article-50.xml v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000894:[<0x1000000088d/article-50.xml v912>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.729 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.471+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b49 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-14.xml auth v1056 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.856716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b49:[<0x1000000088d/comment-14.xml v1056>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b49:[<0x1000000088d/comment-14.xml v1056>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.729 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.472+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008c0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-6.xml auth v908 ap=1 s=14007 n(v0 rc2024-05-09T08:27:52.945799+0000 b14007 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008c0:[<0x1000000088d/article-6.xml v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008c0:[<0x1000000088d/article-6.xml v908>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.729 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.472+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ab4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-67.xml auth v982 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.461727+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ab4:[<0x1000000088d/comment-67.xml v982>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ab4:[<0x1000000088d/comment-67.xml v982>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.730 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.472+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000922 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-54.jpg auth v926 ap=1 s=59737 n(v0 rc2024-05-09T08:27:53.271789+0000 b59737 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000922:[<0x1000000088d/picture-54.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000922:[<0x1000000088d/picture-54.jpg v926>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.730 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.472+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009e7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-90.xml auth v945 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.892743+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009e7:[<0x1000000088d/article-90.xml v945>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009e7:[<0x1000000088d/article-90.xml v945>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.730 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.473+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000095f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-29.jpg auth v932 ap=1 s=8835 n(v0 rc2024-05-09T08:27:53.478784+0000 b8835 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000095f:[<0x1000000088d/picture-29.jpg v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000095f:[<0x1000000088d/picture-29.jpg v932>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.730 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.473+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b44 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-19.xml auth v1054 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.842716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da228d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b44:[<0x1000000088d/comment-19.xml v1054>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b44:[<0x1000000088d/comment-19.xml v1054>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.730 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:04.473+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009d1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-56.xml auth v942 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.826745+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2248000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009d1:[<0x1000000088d/article-56.xml v942>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009d1:[<0x1000000088d/article-56.xml v942>,<0x10000000399/blog-9 v24723>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.852 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.364+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b9f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-73.xml auth v1060 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.769690+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984c0680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b9f:[<0x1000000039a/comment-73.xml v1060>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b9f:[<0x1000000039a/comment-73.xml v1060>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.852 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.364+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000baa [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-21.jpg auth v1064 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.804689+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984bfb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000baa:[<0x1000000039a/picture-21.jpg v1064>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000baa:[<0x1000000039a/picture-21.jpg v1064>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.852 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.364+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003eb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-72.xml auth v993 ap=1 s=13103 n(v0 rc2024-05-09T08:27:48.837916+0000 b13103 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984bf080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003eb:[<0x1000000039a/article-72.xml v993>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003eb:[<0x1000000039a/article-72.xml v993>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.852 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.364+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000452 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-35.xml auth v1005 ap=1 s=28714 n(v0 rc2024-05-09T08:27:49.051910+0000 b28714 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984c0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000452:[<0x1000000039a/article-35.xml v1005>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000452:[<0x1000000039a/article-35.xml v1005>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.852 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.364+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004b5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-73.jpg auth v987 ap=1 s=53005 n(v0 rc2024-05-09T08:27:49.388900+0000 b53005 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984bf600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004b5:[<0x1000000039a/picture-73.jpg v987>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004b5:[<0x1000000039a/picture-73.jpg v987>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.364+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c3b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-60.xml auth v1121 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.236676+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984c0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c3b:[<0x1000000039a/comment-60.xml v1121>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c3b:[<0x1000000039a/comment-60.xml v1121>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.365+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000938 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-78.xml auth v1019 ap=1 s=2352 n(v0 rc2024-05-09T08:27:53.363787+0000 b2352 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984c1180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000938:[<0x1000000039a/comment-78.xml v1019>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000938:[<0x1000000039a/comment-78.xml v1019>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.365+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-86.xml auth v1009 ap=1 s=2742 n(v0 rc2024-05-09T08:27:48.768918+0000 b2742 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984c1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c1:[<0x1000000039a/article-86.xml v1009>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003c1:[<0x1000000039a/article-86.xml v1009>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.365+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-97.xml auth v991 ap=1 s=13514 n(v0 rc2024-05-09T08:27:48.711920+0000 b13514 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003a0:[<0x1000000039a/article-97.xml v991>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003a0:[<0x1000000039a/article-97.xml v991>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.365+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000050d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-96.xml auth v1013 ap=1 s=5178 n(v0 rc2024-05-09T08:27:49.715891+0000 b5178 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000050d:[<0x1000000039a/comment-96.xml v1013>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000050d:[<0x1000000039a/comment-96.xml v1013>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.366+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000861 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-30.jpg auth v1017 ap=1 s=18007 n(v0 rc2024-05-09T08:27:52.650807+0000 b18007 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000861:[<0x1000000039a/picture-30.jpg v1017>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000861:[<0x1000000039a/picture-30.jpg v1017>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.366+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000445 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-40.xml auth v995 ap=1 s=31607 n(v0 rc2024-05-09T08:27:49.002911+0000 b31607 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000445:[<0x1000000039a/article-40.xml v995>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000445:[<0x1000000039a/article-40.xml v995>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.366+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003cd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-82.xml auth v1011 ap=1 s=28288 n(v0 rc2024-05-09T08:27:48.789917+0000 b28288 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003cd:[<0x1000000039a/article-82.xml v1011>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003cd:[<0x1000000039a/article-82.xml v1011>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.366+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000487 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-15.xml auth v989 ap=1 s=12389 n(v0 rc2024-05-09T08:27:49.232905+0000 b12389 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000487:[<0x1000000039a/article-15.xml v989>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000487:[<0x1000000039a/article-15.xml v989>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.366+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000074d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-88.xml auth v1029 ap=1 s=7933 n(v0 rc2024-05-09T08:27:51.798832+0000 b7933 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000074d:[<0x1000000039a/comment-88.xml v1029>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000074d:[<0x1000000039a/comment-88.xml v1029>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.853 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.366+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d01 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-40.xml auth v1208 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.835659+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d01:[<0x1000000039a/comment-40.xml v1208>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d01:[<0x1000000039a/comment-40.xml v1208>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.857 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.367+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008d0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-79.xml auth v1031 ap=1 s=9958 n(v0 rc2024-05-09T08:27:53.019797+0000 b9958 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008d0:[<0x1000000039a/comment-79.xml v1031>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008d0:[<0x1000000039a/comment-79.xml v1031>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.857 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.367+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d45 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-33.xml auth v1230 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.993655+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e8000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d45:[<0x1000000039a/comment-33.xml v1230>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d45:[<0x1000000039a/comment-33.xml v1230>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.857 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.367+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000075b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-86.xml auth v1033 ap=1 s=8851 n(v0 rc2024-05-09T08:27:51.837830+0000 b8851 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000075b:[<0x1000000039a/comment-86.xml v1033>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000075b:[<0x1000000039a/comment-86.xml v1033>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.857 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.367+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d0b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-39.xml auth v1213 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.884658+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e8580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d0b:[<0x1000000039a/comment-39.xml v1213>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d0b:[<0x1000000039a/comment-39.xml v1213>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.857 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.367+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000050b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-43.jpg auth v1023 ap=1 s=11695 n(v0 rc2024-05-09T08:27:49.702891+0000 b11695 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e9600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000050b:[<0x1000000039a/picture-43.jpg v1023>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000050b:[<0x1000000039a/picture-43.jpg v1023>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.858 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.368+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-89.xml auth v1007 ap=1 s=18157 n(v0 rc2024-05-09T08:27:48.753918+0000 b18157 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984ea680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b8:[<0x1000000039a/article-89.xml v1007>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003b8:[<0x1000000039a/article-89.xml v1007>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.858 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.368+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000751 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-36.jpg auth v1015 ap=1 s=58623 n(v0 rc2024-05-09T08:27:51.809831+0000 b58623 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e9080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000751:[<0x1000000039a/picture-36.jpg v1015>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000751:[<0x1000000039a/picture-36.jpg v1015>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.858 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.368+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000085e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-82.xml auth v1027 ap=1 s=7363 n(v0 rc2024-05-09T08:27:52.639808+0000 b7363 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984eac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000085e:[<0x1000000039a/comment-82.xml v1027>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000085e:[<0x1000000039a/comment-82.xml v1027>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.858 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.368+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000047c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-19.xml auth v985 ap=1 s=22369 n(v0 rc2024-05-09T08:27:49.196906+0000 b22369 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984e9b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000047c:[<0x1000000039a/article-19.xml v985>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000047c:[<0x1000000039a/article-19.xml v985>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.858 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.368+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bbd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-20.jpg auth v1074 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.876686+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984ea100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bbd:[<0x1000000039a/picture-20.jpg v1074>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bbd:[<0x1000000039a/picture-20.jpg v1074>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.858 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.369+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c20 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-63.xml auth v1112 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.141679+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984eb700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c20:[<0x1000000039a/comment-63.xml v1112>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c20:[<0x1000000039a/comment-63.xml v1112>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.858 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.369+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d4d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-32.xml auth v1235 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.018654+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984eb180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d4d:[<0x1000000039a/comment-32.xml v1235>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d4d:[<0x1000000039a/comment-32.xml v1235>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.858 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.369+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c7a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-54.xml auth v1138 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.444670+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984f8000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c7a:[<0x1000000039a/comment-54.xml v1138>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c7a:[<0x1000000039a/comment-54.xml v1138>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.858 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.369+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000544 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-91.xml auth v997 ap=1 s=6202 n(v0 rc2024-05-09T08:27:49.946884+0000 b6202 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984f8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000544:[<0x1000000039a/comment-91.xml v997>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000544:[<0x1000000039a/comment-91.xml v997>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.369+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000073f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-38.jpg auth v1001 ap=1 s=9316 n(v0 rc2024-05-09T08:27:51.761833+0000 b9316 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984f9080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000073f:[<0x1000000039a/picture-38.jpg v1001>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000073f:[<0x1000000039a/picture-38.jpg v1001>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.370+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009be [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-23.jpg auth v1047 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.812745+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598588680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009be:[<0x1000000039a/picture-23.jpg v1047>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009be:[<0x1000000039a/picture-23.jpg v1047>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.370+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000440 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-42.xml auth v999 ap=1 s=10782 n(v0 rc2024-05-09T08:27:48.988912+0000 b10782 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984f8580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000440:[<0x1000000039a/article-42.xml v999>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000440:[<0x1000000039a/article-42.xml v999>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.370+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000048b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-14.xml auth v994 ap=1 s=9203 n(v0 rc2024-05-09T08:27:49.240905+0000 b9203 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598588c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000048b:[<0x1000000039a/article-14.xml v994>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000048b:[<0x1000000039a/article-14.xml v994>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.370+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-84.xml auth v1018 ap=1 s=26700 n(v0 rc2024-05-09T08:27:48.779918+0000 b26700 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c7:[<0x1000000039a/article-84.xml v1018>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003c7:[<0x1000000039a/article-84.xml v1018>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.370+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000503 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-45.jpg auth v1000 ap=1 s=64163 n(v0 rc2024-05-09T08:27:49.663893+0000 b64163 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598589180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000503:[<0x1000000039a/picture-45.jpg v1000>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000503:[<0x1000000039a/picture-45.jpg v1000>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.371+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004d7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-60.jpg auth v988 ap=1 s=19661 n(v0 rc2024-05-09T08:27:49.503897+0000 b19661 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004d7:[<0x1000000039a/picture-60.jpg v988>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004d7:[<0x1000000039a/picture-60.jpg v988>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.371+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000438 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-46.xml auth v1012 ap=1 s=28769 n(v0 rc2024-05-09T08:27:48.962913+0000 b28769 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000438:[<0x1000000039a/article-46.xml v1012>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000438:[<0x1000000039a/article-46.xml v1012>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.371+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c2c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-62.xml auth v1106 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.181678+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c2c:[<0x1000000039a/comment-62.xml v1106>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c2c:[<0x1000000039a/comment-62.xml v1106>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.371+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004ca [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-66.jpg auth v1028 ap=1 s=39960 n(v0 rc2024-05-09T08:27:49.459898+0000 b39960 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004ca:[<0x1000000039a/picture-66.jpg v1028>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004ca:[<0x1000000039a/picture-66.jpg v1028>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.371+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000526 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-41.jpg auth v984 ap=1 s=23863 n(v0 rc2024-05-09T08:27:49.817888+0000 b23863 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598589700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000526:[<0x1000000039a/picture-41.jpg v984>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000526:[<0x1000000039a/picture-41.jpg v984>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.859 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.372+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-91.xml auth v976 ap=1 s=28209 n(v0 rc2024-05-09T08:27:48.743919+0000 b28209 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b2:[<0x1000000039a/article-91.xml v976>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003b2:[<0x1000000039a/article-91.xml v976>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.860 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.372+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ed [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-71.xml auth v978 ap=1 s=1865 n(v0 rc2024-05-09T08:27:48.842916+0000 b1865 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ed:[<0x1000000039a/article-71.xml v978>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003ed:[<0x1000000039a/article-71.xml v978>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.860 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.372+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b9a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-22.jpg auth v1049 ap=1 s=17739 n(v0 rc2024-05-09T08:27:56.695692+0000 b17739 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b9a:[<0x1000000039a/picture-22.jpg v1049>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b9a:[<0x1000000039a/picture-22.jpg v1049>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.860 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.372+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000494 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-11.xml auth v1010 ap=1 s=4847 n(v0 rc2024-05-09T08:27:49.278903+0000 b4847 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000494:[<0x1000000039a/article-11.xml v1010>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000494:[<0x1000000039a/article-11.xml v1010>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.860 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.372+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c47 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-59.xml auth v1121 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.263675+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c47:[<0x1000000039a/comment-59.xml v1121>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c47:[<0x1000000039a/comment-59.xml v1121>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.860 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.372+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000484 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-16.xml auth v1026 ap=1 s=11085 n(v0 rc2024-05-09T08:27:49.223905+0000 b11085 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985aa000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000484:[<0x1000000039a/article-16.xml v1026>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000484:[<0x1000000039a/article-16.xml v1026>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.860 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.373+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000436 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-47.xml auth v1016 ap=1 s=19556 n(v0 rc2024-05-09T08:27:48.956913+0000 b19556 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985a4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000436:[<0x1000000039a/article-47.xml v1016>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000436:[<0x1000000039a/article-47.xml v1016>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.860 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.373+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000941 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-77.xml auth v1030 ap=1 s=9199 n(v0 rc2024-05-09T08:27:53.394786+0000 b9199 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985aa580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000941:[<0x1000000039a/comment-77.xml v1030>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000941:[<0x1000000039a/comment-77.xml v1030>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.860 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.373+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000421 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-54.xml auth v1014 ap=1 s=18314 n(v0 rc2024-05-09T08:27:48.923914+0000 b18314 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985aab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000421:[<0x1000000039a/article-54.xml v1014>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000421:[<0x1000000039a/article-54.xml v1014>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.373+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000046a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-26.xml auth v1008 ap=1 s=6285 n(v0 rc2024-05-09T08:27:49.132908+0000 b6285 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ab080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000046a:[<0x1000000039a/article-26.xml v1008>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000046a:[<0x1000000039a/article-26.xml v1008>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.373+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bfb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-15.jpg auth v1088 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.024682+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ab600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bfb:[<0x1000000039a/picture-15.jpg v1088>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bfb:[<0x1000000039a/picture-15.jpg v1088>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.374+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ac [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-93.xml auth v980 ap=1 s=7436 n(v0 rc2024-05-09T08:27:48.732919+0000 b7436 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985abb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ac:[<0x1000000039a/article-93.xml v980>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003ac:[<0x1000000039a/article-93.xml v980>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.374+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004e1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-56.jpg auth v1002 ap=1 s=51638 n(v0 rc2024-05-09T08:27:49.537896+0000 b51638 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ac100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004e1:[<0x1000000039a/picture-56.jpg v1002>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004e1:[<0x1000000039a/picture-56.jpg v1002>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.374+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004fa [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-47.jpg auth v996 ap=1 s=45675 n(v0 rc2024-05-09T08:27:49.630893+0000 b45675 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ac680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004fa:[<0x1000000039a/picture-47.jpg v996>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004fa:[<0x1000000039a/picture-47.jpg v996>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.374+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000496 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-10.xml auth v1022 ap=1 s=14237 n(v0 rc2024-05-09T08:27:49.293903+0000 b14237 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985acc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000496:[<0x1000000039a/article-10.xml v1022>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000496:[<0x1000000039a/article-10.xml v1022>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.374+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c1d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-12.jpg auth v1092 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.136679+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ad180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c1d:[<0x1000000039a/picture-12.jpg v1092>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c1d:[<0x1000000039a/picture-12.jpg v1092>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.375+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ca2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-50.xml auth v1144 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.606666+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ca2:[<0x1000000039a/comment-50.xml v1144>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ca2:[<0x1000000039a/comment-50.xml v1144>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.375+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000505 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-44.jpg auth v998 ap=1 s=49463 n(v0 rc2024-05-09T08:27:49.674892+0000 b49463 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ad700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000505:[<0x1000000039a/picture-44.jpg v998>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000505:[<0x1000000039a/picture-44.jpg v998>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.375+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000433 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-48.xml auth v982 ap=1 s=24246 n(v0 rc2024-05-09T08:27:48.951913+0000 b24246 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b4580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000433:[<0x1000000039a/article-48.xml v982>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000433:[<0x1000000039a/article-48.xml v982>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.375+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cb6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-48.xml auth v1149 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.664664+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cb6:[<0x1000000039a/comment-48.xml v1149>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cb6:[<0x1000000039a/comment-48.xml v1149>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.375+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-4.xml auth v1004 ap=1 s=19686 n(v0 rc2024-05-09T08:27:49.354901+0000 b19686 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a9:[<0x1000000039a/article-4.xml v1004>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004a9:[<0x1000000039a/article-4.xml v1004>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.861 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.376+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000046d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-25.xml auth v986 ap=1 s=31630 n(v0 rc2024-05-09T08:27:49.142907+0000 b31630 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000046d:[<0x1000000039a/article-25.xml v986>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000046d:[<0x1000000039a/article-25.xml v986>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.376+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004b8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-72.jpg auth v1006 ap=1 s=41933 n(v0 rc2024-05-09T08:27:49.398900+0000 b41933 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004b8:[<0x1000000039a/picture-72.jpg v1006>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004b8:[<0x1000000039a/picture-72.jpg v1006>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.376+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004d9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-59.jpg auth v1024 ap=1 s=10239 n(v0 rc2024-05-09T08:27:49.510897+0000 b10239 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b6100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004d9:[<0x1000000039a/picture-59.jpg v1024>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004d9:[<0x1000000039a/picture-59.jpg v1024>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.376+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000044a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-38.xml auth v1020 ap=1 s=14176 n(v0 rc2024-05-09T08:27:49.021911+0000 b14176 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000044a:[<0x1000000039a/article-38.xml v1020>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000044a:[<0x1000000039a/article-38.xml v1020>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.377+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000041f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-55.xml auth v1020 ap=1 s=10303 n(v0 rc2024-05-09T08:27:48.918914+0000 b10303 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984f9b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000041f:[<0x1000000039a/article-55.xml v1020>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000041f:[<0x1000000039a/article-55.xml v1020>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.377+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000098b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-25.jpg auth v1036 ap=1 s=54680 n(v0 rc2024-05-09T08:27:53.654779+0000 b54680 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984fa100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000098b:[<0x1000000039a/picture-25.jpg v1036>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000098b:[<0x1000000039a/picture-25.jpg v1036>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.377+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000508 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-97.xml auth v1022 ap=1 s=9970 n(v0 rc2024-05-09T08:27:49.680892+0000 b9970 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984fac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000508:[<0x1000000039a/comment-97.xml v1022>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000508:[<0x1000000039a/comment-97.xml v1022>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.377+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-95.xml auth v1002 ap=1 s=3882 n(v0 rc2024-05-09T08:27:48.722919+0000 b3882 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984fa680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003a7:[<0x1000000039a/article-95.xml v1002>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003a7:[<0x1000000039a/article-95.xml v1002>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.377+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000049d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-8.xml auth v1014 ap=1 s=12189 n(v0 rc2024-05-09T08:27:49.316902+0000 b12189 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984fb180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000049d:[<0x1000000039a/article-8.xml v1014>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000049d:[<0x1000000039a/article-8.xml v1014>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.377+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cc0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-47.xml auth v1172 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.684663+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984fb700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cc0:[<0x1000000039a/comment-47.xml v1172>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cc0:[<0x1000000039a/comment-47.xml v1172>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.862 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.378+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cc8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-46.xml auth v1177 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.703663+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981e8000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cc8:[<0x1000000039a/comment-46.xml v1177>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cc8:[<0x1000000039a/comment-46.xml v1177>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.872 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.378+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000085d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-31.jpg auth v1038 ap=1 s=31575 n(v0 rc2024-05-09T08:27:52.633808+0000 b31575 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981e8580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000085d:[<0x1000000039a/picture-31.jpg v1038>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000085d:[<0x1000000039a/picture-31.jpg v1038>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.872 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.378+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-68.xml auth v998 ap=1 s=28009 n(v0 rc2024-05-09T08:27:48.856916+0000 b28009 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981e8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f7:[<0x1000000039a/article-68.xml v998>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003f7:[<0x1000000039a/article-68.xml v998>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.378+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bd1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-70.xml auth v1079 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.915685+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981e9080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bd1:[<0x1000000039a/comment-70.xml v1079>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bd1:[<0x1000000039a/comment-70.xml v1079>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.379+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-76.xml auth v994 ap=1 s=7431 n(v0 rc2024-05-09T08:27:48.818917+0000 b7431 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981e9b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e0:[<0x1000000039a/article-76.xml v994>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003e0:[<0x1000000039a/article-76.xml v994>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.379+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c4d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-8.jpg auth v1127 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.297674+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981ea100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c4d:[<0x1000000039a/picture-8.jpg v1127>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c4d:[<0x1000000039a/picture-8.jpg v1127>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.379+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000093d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-26.jpg auth v1050 ap=1 s=8677 n(v0 rc2024-05-09T08:27:53.385786+0000 b8677 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981ea680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000093d:[<0x1000000039a/picture-26.jpg v1050>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000093d:[<0x1000000039a/picture-26.jpg v1050>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.381+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004f3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-49.jpg auth v990 ap=1 s=56789 n(v0 rc2024-05-09T08:27:49.602894+0000 b56789 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981e9600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004f3:[<0x1000000039a/picture-49.jpg v990>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004f3:[<0x1000000039a/picture-49.jpg v990>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.381+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000501 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-98.xml auth v1024 ap=1 s=7410 n(v0 rc2024-05-09T08:27:49.659893+0000 b7410 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981eac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000501:[<0x1000000039a/comment-98.xml v1024>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000501:[<0x1000000039a/comment-98.xml v1024>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.381+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000045a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-32.xml auth v1016 ap=1 s=1996 n(v0 rc2024-05-09T08:27:49.084909+0000 b1996 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981eb180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000045a:[<0x1000000039a/article-32.xml v1016>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000045a:[<0x1000000039a/article-32.xml v1016>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.381+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000007f3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-83.xml auth v1000 ap=1 s=10182 n(v0 rc2024-05-09T08:27:52.603809+0000 b10182 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5981eb700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000007f3:[<0x1000000039a/comment-83.xml v1000>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000007f3:[<0x1000000039a/comment-83.xml v1000>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.381+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000be0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-69.xml auth v1084 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.945685+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598528000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000be0:[<0x1000000039a/comment-69.xml v1084>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000be0:[<0x1000000039a/comment-69.xml v1084>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.381+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000478 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-21.xml auth v996 ap=1 s=12462 n(v0 rc2024-05-09T08:27:49.185906+0000 b12462 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598528580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000478:[<0x1000000039a/article-21.xml v996>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000478:[<0x1000000039a/article-21.xml v996>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.382+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000045f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-30.xml auth v1006 ap=1 s=21480 n(v0 rc2024-05-09T08:27:49.097909+0000 b21480 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598528b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000045f:[<0x1000000039a/article-30.xml v1006>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000045f:[<0x1000000039a/article-30.xml v1006>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.873 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.382+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003aa [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-94.xml auth v986 ap=1 s=22536 n(v0 rc2024-05-09T08:27:48.727919+0000 b22536 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598529080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003aa:[<0x1000000039a/article-94.xml v986>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003aa:[<0x1000000039a/article-94.xml v986>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.874 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.382+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000747 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-89.xml auth v1040 ap=1 s=7068 n(v0 rc2024-05-09T08:27:51.779832+0000 b7068 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598529600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000747:[<0x1000000039a/comment-89.xml v1040>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000747:[<0x1000000039a/comment-89.xml v1040>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.874 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.382+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009d0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-74.xml auth v1056 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.088709+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598529b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009d0:[<0x1000000039a/comment-74.xml v1056>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009d0:[<0x1000000039a/comment-74.xml v1056>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.874 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.383+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d2c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-36.xml auth v1213 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.939656+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59852a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d2c:[<0x1000000039a/comment-36.xml v1213>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d2c:[<0x1000000039a/comment-36.xml v1213>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.874 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.383+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000054b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-90.xml auth v1018 ap=1 s=2353 n(v0 rc2024-05-09T08:27:51.750833+0000 b2353 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59852a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000054b:[<0x1000000039a/comment-90.xml v1018>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000054b:[<0x1000000039a/comment-90.xml v1018>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.874 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.383+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000048e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-13.xml auth v1008 ap=1 s=8994 n(v0 rc2024-05-09T08:27:49.255904+0000 b8994 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59852ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000048e:[<0x1000000039a/article-13.xml v1008>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000048e:[<0x1000000039a/article-13.xml v1008>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.874 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.383+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c17 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-64.xml auth v1092 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.122679+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59852b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c17:[<0x1000000039a/comment-64.xml v1092>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c17:[<0x1000000039a/comment-64.xml v1092>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.874 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.384+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000042d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-50.xml auth v1012 ap=1 s=9218 n(v0 rc2024-05-09T08:27:48.942913+0000 b9218 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598534580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000042d:[<0x1000000039a/article-50.xml v1012>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000042d:[<0x1000000039a/article-50.xml v1012>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.874 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.385+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bb4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-72.xml auth v1069 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.844687+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59852b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bb4:[<0x1000000039a/comment-72.xml v1069>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bb4:[<0x1000000039a/comment-72.xml v1069>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.874 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.385+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004af [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-2.xml auth v1030 ap=1 s=18294 n(v0 rc2024-05-09T08:27:49.371901+0000 b18294 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598535080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004af:[<0x1000000039a/article-2.xml v1030>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004af:[<0x1000000039a/article-2.xml v1030>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.385+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-6.xml auth v992 ap=1 s=31579 n(v0 rc2024-05-09T08:27:49.336902+0000 b31579 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598534b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a3:[<0x1000000039a/article-6.xml v992>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004a3:[<0x1000000039a/article-6.xml v992>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.386+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dc8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-14.xml auth v1276 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.452641+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598535600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dc8:[<0x1000000039a/comment-14.xml v1276>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dc8:[<0x1000000039a/comment-14.xml v1276>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.386+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bf5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-67.xml auth v1089 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.011683+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598535b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bf5:[<0x1000000039a/comment-67.xml v1089>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bf5:[<0x1000000039a/comment-67.xml v1089>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.386+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004e7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-54.jpg auth v988 ap=1 s=38135 n(v0 rc2024-05-09T08:27:49.560895+0000 b38135 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598536100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004e7:[<0x1000000039a/picture-54.jpg v988>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004e7:[<0x1000000039a/picture-54.jpg v988>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.387+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-90.xml auth v984 ap=1 s=25244 n(v0 rc2024-05-09T08:27:48.748919+0000 b25244 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598536680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b5:[<0x1000000039a/article-90.xml v984>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003b5:[<0x1000000039a/article-90.xml v984>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.387+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000864 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-29.jpg auth v1028 ap=1 s=29094 n(v0 rc2024-05-09T08:27:52.954799+0000 b29094 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598536c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000864:[<0x1000000039a/picture-29.jpg v1028>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000864:[<0x1000000039a/picture-29.jpg v1028>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.387+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000da3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-19.xml auth v1271 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.280646+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598537180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000da3:[<0x1000000039a/comment-19.xml v1271>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000da3:[<0x1000000039a/comment-19.xml v1271>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.387+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000041b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-56.xml auth v1010 ap=1 s=10279 n(v0 rc2024-05-09T08:27:48.913914+0000 b10279 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598537700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000041b:[<0x1000000039a/article-56.xml v1010>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000041b:[<0x1000000039a/article-56.xml v1010>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.388+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000413 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-59.xml auth v1012 ap=1 s=26133 n(v0 rc2024-05-09T08:27:48.899914+0000 b26133 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598459b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000413:[<0x1000000039a/article-59.xml v1012>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000413:[<0x1000000039a/article-59.xml v1012>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.388+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c11 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-13.jpg auth v1099 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.100680+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c59845a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c11:[<0x1000000039a/picture-13.jpg v1099>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c11:[<0x1000000039a/picture-13.jpg v1099>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.388+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c68 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-56.xml auth v1139 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.370672+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59845a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c68:[<0x1000000039a/comment-56.xml v1139>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c68:[<0x1000000039a/comment-56.xml v1139>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.875 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.388+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003af [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-92.xml auth v986 ap=1 s=22298 n(v0 rc2024-05-09T08:27:48.738919+0000 b22298 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59845ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003af:[<0x1000000039a/article-92.xml v986>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003af:[<0x1000000039a/article-92.xml v986>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.388+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bd8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-18.jpg auth v1077 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.931685+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59845b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bd8:[<0x1000000039a/picture-18.jpg v1077>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bd8:[<0x1000000039a/picture-18.jpg v1077>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.389+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-74.xml auth v998 ap=1 s=9813 n(v0 rc2024-05-09T08:27:48.827916+0000 b9813 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59845b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e6:[<0x1000000039a/article-74.xml v998>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003e6:[<0x1000000039a/article-74.xml v998>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.389+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000047f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-18.xml auth v984 ap=1 s=29628 n(v0 rc2024-05-09T08:27:49.204906+0000 b29628 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000047f:[<0x1000000039a/article-18.xml v984>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000047f:[<0x1000000039a/article-18.xml v984>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.389+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bcd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-19.jpg auth v1072 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.905686+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bcd:[<0x1000000039a/picture-19.jpg v1072>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bcd:[<0x1000000039a/picture-19.jpg v1072>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.389+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000042a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-51.xml auth v1002 ap=1 s=2490 n(v0 rc2024-05-09T08:27:48.937913+0000 b2490 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000042a:[<0x1000000039a/article-51.xml v1002>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000042a:[<0x1000000039a/article-51.xml v1002>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.389+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d64 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-29.xml auth v1250 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.060653+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d64:[<0x1000000039a/comment-29.xml v1250>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d64:[<0x1000000039a/comment-29.xml v1250>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.390+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000468 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-27.xml auth v1000 ap=1 s=3807 n(v0 rc2024-05-09T08:27:49.123908+0000 b3807 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000468:[<0x1000000039a/article-27.xml v1000>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000468:[<0x1000000039a/article-27.xml v1000>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.390+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c77 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-4.jpg auth v1144 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.435671+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c77:[<0x1000000039a/picture-4.jpg v1144>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c77:[<0x1000000039a/picture-4.jpg v1144>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.390+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000759 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-35.jpg auth v992 ap=1 s=14788 n(v0 rc2024-05-09T08:27:51.829831+0000 b14788 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000759:[<0x1000000039a/picture-35.jpg v992>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000759:[<0x1000000039a/picture-35.jpg v992>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.391+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c8b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-2.jpg auth v1148 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.515668+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c8b:[<0x1000000039a/picture-2.jpg v1148>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c8b:[<0x1000000039a/picture-2.jpg v1148>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.876 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.391+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-70.xml auth v996 ap=1 s=7840 n(v0 rc2024-05-09T08:27:48.846916+0000 b7840 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f1:[<0x1000000039a/article-70.xml v996>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003f1:[<0x1000000039a/article-70.xml v996>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.391+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000047a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-20.xml auth v994 ap=1 s=2715 n(v0 rc2024-05-09T08:27:49.190906+0000 b2715 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59849f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000047a:[<0x1000000039a/article-20.xml v994>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000047a:[<0x1000000039a/article-20.xml v994>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.392+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c05 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-14.jpg auth v1093 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.067681+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c05:[<0x1000000039a/picture-14.jpg v1093>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c05:[<0x1000000039a/picture-14.jpg v1093>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.392+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000053c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-92.xml auth v978 ap=1 s=8892 n(v0 rc2024-05-09T08:27:49.878886+0000 b8892 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b4580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000053c:[<0x1000000039a/comment-92.xml v978>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000053c:[<0x1000000039a/comment-92.xml v978>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.392+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ca [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-83.xml auth v976 ap=1 s=28872 n(v0 rc2024-05-09T08:27:48.783918+0000 b28872 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ca:[<0x1000000039a/article-83.xml v976>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003ca:[<0x1000000039a/article-83.xml v976>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.392+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000040c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-61.xml auth v980 ap=1 s=24907 n(v0 rc2024-05-09T08:27:48.890915+0000 b24907 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000040c:[<0x1000000039a/article-61.xml v980>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000040c:[<0x1000000039a/article-61.xml v980>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.393+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000442 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-41.xml auth v1006 ap=1 s=2984 n(v0 rc2024-05-09T08:27:48.994912+0000 b2984 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b5600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000442:[<0x1000000039a/article-41.xml v1006>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000442:[<0x1000000039a/article-41.xml v1006>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.393+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000043a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-45.xml auth v982 ap=1 s=12316 n(v0 rc2024-05-09T08:27:48.968912+0000 b12316 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000043a:[<0x1000000039a/article-45.xml v982>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000043a:[<0x1000000039a/article-45.xml v982>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.393+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-69.xml auth v990 ap=1 s=11000 n(v0 rc2024-05-09T08:27:48.851916+0000 b11000 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b6100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f5:[<0x1000000039a/article-69.xml v990>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003f5:[<0x1000000039a/article-69.xml v990>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.393+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000523 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-94.xml auth v1014 ap=1 s=2291 n(v0 rc2024-05-09T08:27:49.801889+0000 b2291 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000523:[<0x1000000039a/comment-94.xml v1014>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000523:[<0x1000000039a/comment-94.xml v1014>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.394+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004d5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-61.jpg auth v988 ap=1 s=31107 n(v0 rc2024-05-09T08:27:49.496897+0000 b31107 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004d5:[<0x1000000039a/picture-61.jpg v988>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004d5:[<0x1000000039a/picture-61.jpg v988>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.394+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000045c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-31.xml auth v1008 ap=1 s=24468 n(v0 rc2024-05-09T08:27:49.090909+0000 b24468 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000045c:[<0x1000000039a/article-31.xml v1008>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000045c:[<0x1000000039a/article-31.xml v1008>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.877 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.394+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c59 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-7.jpg auth v1132 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.336673+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984b7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c59:[<0x1000000039a/picture-7.jpg v1132>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c59:[<0x1000000039a/picture-7.jpg v1132>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.394+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ddb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-11.xml auth v1272 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.531639+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984be580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ddb:[<0x1000000039a/comment-11.xml v1272>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ddb:[<0x1000000039a/comment-11.xml v1272>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.395+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c41 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-9.jpg auth v1118 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.252676+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984be000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c41:[<0x1000000039a/picture-9.jpg v1118>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c41:[<0x1000000039a/picture-9.jpg v1118>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.395+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d99 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-20.xml auth v1267 ap=1 s=0 n(v0 rc2024-05-09T08:27:58.239648+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5984beb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d99:[<0x1000000039a/comment-20.xml v1267>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d99:[<0x1000000039a/comment-20.xml v1267>,<0x10000000399/blog-0 v20177>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.395+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000704 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-10.jpg auth v1514 ap=1 s=31267 n(v0 rc2024-05-09T08:27:51.546839+0000 b31267 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ee680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000704:[<0x1000000039b/picture-10.jpg v1514>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000704:[<0x1000000039b/picture-10.jpg v1514>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.395+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000866 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-1.jpg auth v1510 ap=1 s=46704 n(v0 rc2024-05-09T08:27:52.672807+0000 b46704 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985eec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000866:[<0x1000000039b/picture-1.jpg v1510>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000866:[<0x1000000039b/picture-1.jpg v1510>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.397+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000710 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-6.jpg auth v1512 ap=1 s=37851 n(v0 rc2024-05-09T08:27:51.585838+0000 b37851 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598606580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000710:[<0x1000000039b/picture-6.jpg v1512>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000710:[<0x1000000039b/picture-6.jpg v1512>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.397+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ce [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-58.xml auth v1498 ap=1 s=7472 n(v0 rc2024-05-09T08:27:48.799917+0000 b7472 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@5},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598606b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ce:[<0x1000000039b/comment-58.xml v1498>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003ce:[<0x1000000039b/comment-58.xml v1498>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.397+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000488 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-9.xml auth v1504 ap=1 s=22934 n(v0 rc2024-05-09T08:27:49.257904+0000 b22934 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ef700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000488:[<0x1000000039b/article-9.xml v1504>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000488:[<0x1000000039b/article-9.xml v1504>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.398+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-24.xml auth v1480 ap=1 s=13275 n(v0 rc2024-05-09T08:27:48.782918+0000 b13275 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598607600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c3:[<0x1000000039b/article-24.xml v1480>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003c3:[<0x1000000039b/article-24.xml v1480>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.878 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.398+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006ce [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-28.jpg auth v1508 ap=1 s=64286 n(v0 rc2024-05-09T08:27:51.382843+0000 b64286 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598608680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006ce:[<0x1000000039b/picture-28.jpg v1508>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006ce:[<0x1000000039b/picture-28.jpg v1508>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.399+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000689 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-37.jpg auth v1526 ap=1 s=9590 n(v0 rc2024-05-09T08:27:51.166850+0000 b9590 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598609180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000689:[<0x1000000039b/picture-37.jpg v1526>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000689:[<0x1000000039b/picture-37.jpg v1526>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.399+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006ef [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-17.jpg auth v1516 ap=1 s=53325 n(v0 rc2024-05-09T08:27:51.482841+0000 b53325 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598609700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006ef:[<0x1000000039b/picture-17.jpg v1516>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006ef:[<0x1000000039b/picture-17.jpg v1516>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.399+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004e9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-3.xml auth v1500 ap=1 s=18639 n(v0 rc2024-05-09T08:27:49.612894+0000 b18639 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598610000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004e9:[<0x1000000039b/article-3.xml v1500>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004e9:[<0x1000000039b/article-3.xml v1500>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.399+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006da [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-24.jpg auth v1506 ap=1 s=56047 n(v0 rc2024-05-09T08:27:51.419842+0000 b56047 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598610580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006da:[<0x1000000039b/picture-24.jpg v1506>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006da:[<0x1000000039b/picture-24.jpg v1506>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.400+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006bf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-33.jpg auth v1522 ap=1 s=22322 n(v0 rc2024-05-09T08:27:51.336845+0000 b22322 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598610b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006bf:[<0x1000000039b/picture-33.jpg v1522>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006bf:[<0x1000000039b/picture-33.jpg v1522>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.400+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000578 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-57.jpg auth v1518 ap=1 s=50592 n(v0 rc2024-05-09T08:27:50.142879+0000 b50592 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598611080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000578:[<0x1000000039b/picture-57.jpg v1518>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000578:[<0x1000000039b/picture-57.jpg v1518>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.400+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000004a1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-7.xml auth v1502 ap=1 s=18148 n(v0 rc2024-05-09T08:27:49.351901+0000 b18148 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598611600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000004a1:[<0x1000000039b/article-7.xml v1502>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000004a1:[<0x1000000039b/article-7.xml v1502>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.400+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000649 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-42.jpg auth v1524 ap=1 s=14173 n(v0 rc2024-05-09T08:27:50.911857+0000 b14173 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598611b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000649:[<0x1000000039b/picture-42.jpg v1524>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000649:[<0x1000000039b/picture-42.jpg v1524>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.401+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000670 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-39.jpg auth v1520 ap=1 s=8893 n(v0 rc2024-05-09T08:27:51.072852+0000 b8893 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598612100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000670:[<0x1000000039b/picture-39.jpg v1520>,<0x10000000399/blog-1 v20283>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000670:[<0x1000000039b/picture-39.jpg v1520>,<0x10000000399/blog-1 v21939>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.438+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b4b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-13.xml auth v1069 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.863715+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b4b:[<0x1000000088d/comment-13.xml v1069>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b4b:[<0x1000000088d/comment-13.xml v1069>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.879 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.438+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b42 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-21.xml auth v1047 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.837716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985b7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b42:[<0x1000000088d/comment-21.xml v1047>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b42:[<0x1000000088d/comment-21.xml v1047>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.438+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b3d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-26.xml auth v1037 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.825717+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985dc000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b3d:[<0x1000000088d/comment-26.xml v1037>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b3d:[<0x1000000088d/comment-26.xml v1037>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.438+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-37.xml auth v928 ap=1 s=30732 n(v0 rc2024-05-09T08:27:52.857801+0000 b30732 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985dc580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a1:[<0x1000000088d/article-37.xml v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a1:[<0x1000000088d/article-37.xml v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.439+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000964 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-27.jpg auth v920 ap=1 s=11455 n(v0 rc2024-05-09T08:27:53.498783+0000 b11455 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985dcb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000964:[<0x1000000088d/picture-27.jpg v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000964:[<0x1000000088d/picture-27.jpg v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.439+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000089f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-39.xml auth v908 ap=1 s=19458 n(v0 rc2024-05-09T08:27:52.850801+0000 b19458 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985dd080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000089f:[<0x1000000088d/article-39.xml v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000089f:[<0x1000000088d/article-39.xml v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.439+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-36.xml auth v914 ap=1 s=2287 n(v0 rc2024-05-09T08:27:52.860801+0000 b2287 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985dd600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a2:[<0x1000000088d/article-36.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a2:[<0x1000000088d/article-36.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.439+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008ba [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-12.xml auth v924 ap=1 s=16613 n(v0 rc2024-05-09T08:27:52.929799+0000 b16613 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ddb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008ba:[<0x1000000088d/article-12.xml v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008ba:[<0x1000000088d/article-12.xml v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.439+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bb1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-52.jpg auth v1081 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.806688+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985de100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bb1:[<0x1000000088d/picture-52.jpg v1081>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bb1:[<0x1000000088d/picture-52.jpg v1081>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.440+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008c8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-1.xml auth v938 ap=1 s=4854 n(v0 rc2024-05-09T08:27:52.968798+0000 b4854 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985de680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008c8:[<0x1000000088d/article-1.xml v938>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008c8:[<0x1000000088d/article-1.xml v938>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.440+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b33 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-30.xml auth v1031 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.791717+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985dec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b33:[<0x1000000088d/comment-30.xml v1031>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b33:[<0x1000000088d/comment-30.xml v1031>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.440+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000950 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-34.jpg auth v932 ap=1 s=30934 n(v0 rc2024-05-09T08:27:53.430785+0000 b30934 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985df180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000950:[<0x1000000088d/picture-34.jpg v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000950:[<0x1000000088d/picture-34.jpg v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.440+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000924 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-53.jpg auth v926 ap=1 s=17560 n(v0 rc2024-05-09T08:27:53.277789+0000 b17560 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985df700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000924:[<0x1000000088d/picture-53.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000924:[<0x1000000088d/picture-53.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.440+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000908 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-67.jpg auth v910 ap=1 s=63606 n(v0 rc2024-05-09T08:27:53.180792+0000 b63606 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000908:[<0x1000000088d/picture-67.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000908:[<0x1000000088d/picture-67.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.880 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.440+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008d5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-90.jpg auth v906 ap=1 s=62105 n(v0 rc2024-05-09T08:27:53.008797+0000 b62105 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008d5:[<0x1000000088d/picture-90.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008d5:[<0x1000000088d/picture-90.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.881 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.441+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b72 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-3.xml auth v1075 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.966712+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b72:[<0x1000000088d/comment-3.xml v1075>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b72:[<0x1000000088d/comment-3.xml v1075>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.881 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.441+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b45 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-18.xml auth v1059 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.844716+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b45:[<0x1000000088d/comment-18.xml v1059>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b45:[<0x1000000088d/comment-18.xml v1059>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.881 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.441+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000980 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-16.jpg auth v936 ap=1 s=62586 n(v0 rc2024-05-09T08:27:53.594780+0000 b62586 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000980:[<0x1000000088d/picture-16.jpg v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000980:[<0x1000000088d/picture-16.jpg v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.881 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.441+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b4e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-12.xml auth v1073 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.871715+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b4e:[<0x1000000088d/comment-12.xml v1073>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b4e:[<0x1000000088d/comment-12.xml v1073>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.881 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.441+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008fa [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-74.jpg auth v922 ap=1 s=11443 n(v0 rc2024-05-09T08:27:53.140793+0000 b11443 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008fa:[<0x1000000088d/picture-74.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008fa:[<0x1000000088d/picture-74.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.881 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.441+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b0a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-42.xml auth v1007 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.691720+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e2c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b0a:[<0x1000000088d/comment-42.xml v1007>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b0a:[<0x1000000088d/comment-42.xml v1007>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.881 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.442+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000abb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-65.xml auth v979 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.479726+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e3180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000abb:[<0x1000000088d/comment-65.xml v979>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000abb:[<0x1000000088d/comment-65.xml v979>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.881 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.442+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000af2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-49.xml auth v997 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.625722+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000af2:[<0x1000000088d/comment-49.xml v997>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000af2:[<0x1000000088d/comment-49.xml v997>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.442+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008af [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-23.xml auth v916 ap=1 s=21280 n(v0 rc2024-05-09T08:27:52.899800+0000 b21280 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985e3700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008af:[<0x1000000088d/article-23.xml v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008af:[<0x1000000088d/article-23.xml v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.442+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-17.xml auth v934 ap=1 s=24618 n(v0 rc2024-05-09T08:27:52.915800+0000 b24618 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ec000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b5:[<0x1000000088d/article-17.xml v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b5:[<0x1000000088d/article-17.xml v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.443+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008f6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-76.jpg auth v912 ap=1 s=16493 n(v0 rc2024-05-09T08:27:53.129794+0000 b16493 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ec580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008f6:[<0x1000000088d/picture-76.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008f6:[<0x1000000088d/picture-76.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.443+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000955 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-32.jpg auth v930 ap=1 s=11178 n(v0 rc2024-05-09T08:27:53.448784+0000 b11178 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ecb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000955:[<0x1000000088d/picture-32.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000955:[<0x1000000088d/picture-32.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.443+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009f5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-63.xml auth v961 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.956741+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ed080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009f5:[<0x1000000088d/article-63.xml v961>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009f5:[<0x1000000088d/article-63.xml v961>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.444+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000092e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-48.jpg auth v904 ap=1 s=39971 n(v0 rc2024-05-09T08:27:53.308788+0000 b39971 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985ed600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000092e:[<0x1000000088d/picture-48.jpg v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000092e:[<0x1000000088d/picture-48.jpg v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.444+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aeb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-51.xml auth v993 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.606723+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5985edb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aeb:[<0x1000000088d/comment-51.xml v993>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aeb:[<0x1000000088d/comment-51.xml v993>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.444+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000090e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-64.jpg auth v924 ap=1 s=53234 n(v0 rc2024-05-09T08:27:53.198792+0000 b53234 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598613180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000090e:[<0x1000000088d/picture-64.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000090e:[<0x1000000088d/picture-64.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.444+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b1b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-37.xml auth v1016 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.732719+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598613700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b1b:[<0x1000000088d/comment-37.xml v1016>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b1b:[<0x1000000088d/comment-37.xml v1016>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.444+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aa6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-71.xml auth v972 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.426728+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598628000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aa6:[<0x1000000088d/comment-71.xml v972>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aa6:[<0x1000000088d/comment-71.xml v972>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.882 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.445+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009b0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-1.jpg auth{0=1} v1087 ap=1 s=26409 n(v0 rc2024-05-09T08:27:53.784775+0000 b26409 1=1+0) caps={15549=pAsLsXsFscr/pFscr@4} mcw={0=Fscr} | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x55c598628b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009b0:[<0x1000000088d/picture-1.jpg v944>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009b0:[<0x1000000088d/picture-1.jpg v1087>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.445+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000988 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-10.jpg auth v910 ap=1 s=16623 n(v0 rc2024-05-09T08:27:53.674778+0000 b16623 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598628580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000988:[<0x1000000088d/picture-10.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000988:[<0x1000000088d/picture-10.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.445+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008bd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-9.xml auth v932 ap=1 s=16043 n(v0 rc2024-05-09T08:27:52.937799+0000 b16043 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598629600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008bd:[<0x1000000088d/article-9.xml v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008bd:[<0x1000000088d/article-9.xml v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.445+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000089a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-44.xml auth v904 ap=1 s=17558 n(v0 rc2024-05-09T08:27:52.834802+0000 b17558 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59862a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000089a:[<0x1000000088d/article-44.xml v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000089a:[<0x1000000088d/article-44.xml v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.445+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b47 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-16.xml auth v1064 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.851716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598629080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b47:[<0x1000000088d/comment-16.xml v1064>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b47:[<0x1000000088d/comment-16.xml v1064>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.446+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008e6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-83.jpg auth v918 ap=1 s=12716 n(v0 rc2024-05-09T08:27:53.073795+0000 b12716 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598629b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008e6:[<0x1000000088d/picture-83.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008e6:[<0x1000000088d/picture-83.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.446+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000099f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-6.jpg auth v922 ap=1 s=59652 n(v0 rc2024-05-09T08:27:53.717777+0000 b59652 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59862a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000099f:[<0x1000000088d/picture-6.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000099f:[<0x1000000088d/picture-6.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.446+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ad3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-58.xml auth v990 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.542725+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59862ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ad3:[<0x1000000088d/comment-58.xml v990>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ad3:[<0x1000000088d/comment-58.xml v990>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.446+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000add [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-55.xml auth v996 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.570724+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59862b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000add:[<0x1000000088d/comment-55.xml v996>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000add:[<0x1000000088d/comment-55.xml v996>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.446+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bb5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-24.xml auth v1081 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.823688+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59862b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bb5:[<0x1000000088d/article-24.xml v1081>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bb5:[<0x1000000088d/article-24.xml v1081>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.447+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b25 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-34.xml auth v1024 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.757718+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x55c598632000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b25:[<0x1000000088d/comment-34.xml v1024>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b25:[<0x1000000088d/comment-34.xml v1024>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.447+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b3e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-25.xml auth v1038 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.827716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598632580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b3e:[<0x1000000088d/comment-25.xml v1038>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b3e:[<0x1000000088d/comment-25.xml v1038>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.447+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008d1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-92.jpg auth v926 ap=1 s=63114 n(v0 rc2024-05-09T08:27:52.991797+0000 b63114 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598633080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008d1:[<0x1000000088d/picture-92.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008d1:[<0x1000000088d/picture-92.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.883 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.447+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b0e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-41.xml auth v1010 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.699720+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598633600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b0e:[<0x1000000088d/comment-41.xml v1010>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b0e:[<0x1000000088d/comment-41.xml v1010>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.447+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a96 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-75.xml auth v967 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.377729+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598633b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a96:[<0x1000000088d/comment-75.xml v967>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a96:[<0x1000000088d/comment-75.xml v967>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.447+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000961 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-28.jpg auth v906 ap=1 s=37909 n(v0 rc2024-05-09T08:27:53.488783+0000 b37909 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598632b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000961:[<0x1000000088d/picture-28.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000961:[<0x1000000088d/picture-28.jpg v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.448+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000949 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-37.jpg auth v930 ap=1 s=60052 n(v0 rc2024-05-09T08:27:53.401786+0000 b60052 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598634100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000949:[<0x1000000088d/picture-37.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000949:[<0x1000000088d/picture-37.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.448+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000097e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-17.jpg auth v934 ap=1 s=65276 n(v0 rc2024-05-09T08:27:53.585780+0000 b65276 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598634680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000097e:[<0x1000000088d/picture-17.jpg v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000097e:[<0x1000000088d/picture-17.jpg v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.448+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008e1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-85.jpg auth v928 ap=1 s=52991 n(v0 rc2024-05-09T08:27:53.056796+0000 b52991 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598634c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008e1:[<0x1000000088d/picture-85.jpg v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008e1:[<0x1000000088d/picture-85.jpg v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.448+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bbe [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-3.xml auth v1084 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.851687+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598635180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bbe:[<0x1000000088d/article-3.xml v1084>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bbe:[<0x1000000088d/article-3.xml v1084>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.449+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000096d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-24.jpg auth v920 ap=1 s=62501 n(v0 rc2024-05-09T08:27:53.531782+0000 b62501 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598635700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000096d:[<0x1000000088d/picture-24.jpg v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000096d:[<0x1000000088d/picture-24.jpg v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.449+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000953 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-33.jpg auth v942 ap=1 s=49404 n(v0 rc2024-05-09T08:27:53.439785+0000 b49404 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598646000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000953:[<0x1000000088d/picture-33.jpg v942>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000953:[<0x1000000088d/picture-33.jpg v942>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.449+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000091c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-57.jpg auth v908 ap=1 s=27853 n(v0 rc2024-05-09T08:27:53.243790+0000 b27853 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598646580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000091c:[<0x1000000088d/picture-57.jpg v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000091c:[<0x1000000088d/picture-57.jpg v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.449+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008bf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-7.xml auth v936 ap=1 s=1902 n(v0 rc2024-05-09T08:27:52.942799+0000 b1902 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598646b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008bf:[<0x1000000088d/article-7.xml v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008bf:[<0x1000000088d/article-7.xml v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.450+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000093e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-42.jpg auth v914 ap=1 s=55168 n(v0 rc2024-05-09T08:27:53.357787+0000 b55168 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598647080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000093e:[<0x1000000088d/picture-42.jpg v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000093e:[<0x1000000088d/picture-42.jpg v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.884 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.450+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000906 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-68.jpg auth v940 ap=1 s=55672 n(v0 rc2024-05-09T08:27:53.175792+0000 b55672 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598647600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000906:[<0x1000000088d/picture-68.jpg v940>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000906:[<0x1000000088d/picture-68.jpg v940>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.885 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.450+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000945 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-39.jpg auth v912 ap=1 s=16040 n(v0 rc2024-05-09T08:27:53.383786+0000 b16040 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598647b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000945:[<0x1000000088d/picture-39.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000945:[<0x1000000088d/picture-39.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.885 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.450+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b40 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-23.xml auth v1044 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.832716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598648100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b40:[<0x1000000088d/comment-23.xml v1044>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b40:[<0x1000000088d/comment-23.xml v1044>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.885 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.450+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b3f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-24.xml auth v1042 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.830716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598648680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b3f:[<0x1000000088d/comment-24.xml v1042>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b3f:[<0x1000000088d/comment-24.xml v1042>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.885 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.451+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000984 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-13.jpg auth v908 ap=1 s=47629 n(v0 rc2024-05-09T08:27:53.608780+0000 b47629 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598648c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000984:[<0x1000000088d/picture-13.jpg v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000984:[<0x1000000088d/picture-13.jpg v908>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.885 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.451+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ada [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-56.xml auth v993 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.560724+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598649180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ada:[<0x1000000088d/comment-56.xml v993>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ada:[<0x1000000088d/comment-56.xml v993>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.885 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.451+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000097b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-18.jpg auth v934 ap=1 s=58185 n(v0 rc2024-05-09T08:27:53.578781+0000 b58185 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598649700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000097b:[<0x1000000088d/picture-18.jpg v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000097b:[<0x1000000088d/picture-18.jpg v934>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.885 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.451+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-18.xml auth v928 ap=1 s=30735 n(v0 rc2024-05-09T08:27:52.913800+0000 b30735 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b4:[<0x1000000088d/article-18.xml v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b4:[<0x1000000088d/article-18.xml v928>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.451+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b6a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-5.xml auth v1071 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.941713+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b6a:[<0x1000000088d/comment-5.xml v1071>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b6a:[<0x1000000088d/comment-5.xml v1071>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.451+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000979 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-19.jpg auth v936 ap=1 s=54922 n(v0 rc2024-05-09T08:27:53.570781+0000 b54922 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000979:[<0x1000000088d/picture-19.jpg v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000979:[<0x1000000088d/picture-19.jpg v936>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.452+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000893 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-51.xml auth v914 ap=1 s=5360 n(v0 rc2024-05-09T08:27:52.816802+0000 b5360 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000893:[<0x1000000088d/article-51.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000893:[<0x1000000088d/article-51.xml v914>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.452+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b36 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-29.xml auth v1034 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.803717+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b36:[<0x1000000088d/comment-29.xml v1034>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b36:[<0x1000000088d/comment-29.xml v1034>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.453+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009a5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-4.jpg auth v944 ap=1 s=31783 n(v0 rc2024-05-09T08:27:53.736776+0000 b31783 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009a5:[<0x1000000088d/picture-4.jpg v944>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009a5:[<0x1000000088d/picture-4.jpg v944>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.454+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008e9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-82.jpg auth v924 ap=1 s=9615 n(v0 rc2024-05-09T08:27:53.085795+0000 b9615 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008e9:[<0x1000000088d/picture-82.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008e9:[<0x1000000088d/picture-82.jpg v924>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.454+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008ab [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-27.xml auth v920 ap=1 s=16363 n(v0 rc2024-05-09T08:27:52.889800+0000 b16363 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008ab:[<0x1000000088d/article-27.xml v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008ab:[<0x1000000088d/article-27.xml v920>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.454+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000094d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-35.jpg auth v930 ap=1 s=33355 n(v0 rc2024-05-09T08:27:53.418785+0000 b33355 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000094d:[<0x1000000088d/picture-35.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000094d:[<0x1000000088d/picture-35.jpg v930>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.455+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009ac [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-2.jpg auth v940 ap=1 s=54279 n(v0 rc2024-05-09T08:27:53.758776+0000 b54279 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009ac:[<0x1000000088d/picture-2.jpg v940>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009ac:[<0x1000000088d/picture-2.jpg v940>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.455+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008f2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-78.jpg auth v910 ap=1 s=35421 n(v0 rc2024-05-09T08:27:53.117794+0000 b35421 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59865f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008f2:[<0x1000000088d/picture-78.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008f2:[<0x1000000088d/picture-78.jpg v910>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.455+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008cf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-93.jpg auth v932 ap=1 s=10778 n(v0 rc2024-05-09T08:27:52.985798+0000 b10778 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008cf:[<0x1000000088d/picture-93.jpg v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008cf:[<0x1000000088d/picture-93.jpg v932>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.886 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.455+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008b2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-20.xml auth v938 ap=1 s=23568 n(v0 rc2024-05-09T08:27:52.907800+0000 b23568 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008b2:[<0x1000000088d/article-20.xml v938>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008b2:[<0x1000000088d/article-20.xml v938>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.456+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000983 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-14.jpg auth v926 ap=1 s=61632 n(v0 rc2024-05-09T08:27:53.604780+0000 b61632 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000983:[<0x1000000088d/picture-14.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000983:[<0x1000000088d/picture-14.jpg v926>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.456+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008d9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-89.jpg auth v904 ap=1 s=42959 n(v0 rc2024-05-09T08:27:53.015797+0000 b42959 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008d9:[<0x1000000088d/picture-89.jpg v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008d9:[<0x1000000088d/picture-89.jpg v904>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.456+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000089d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-41.xml auth v916 ap=1 s=4550 n(v0 rc2024-05-09T08:27:52.843802+0000 b4550 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000089d:[<0x1000000088d/article-41.xml v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000089d:[<0x1000000088d/article-41.xml v916>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.456+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000899 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-45.xml auth v906 ap=1 s=12887 n(v0 rc2024-05-09T08:27:52.832802+0000 b12887 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000899:[<0x1000000088d/article-45.xml v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000899:[<0x1000000088d/article-45.xml v906>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.457+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000914 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-61.jpg auth v912 ap=1 s=40267 n(v0 rc2024-05-09T08:27:53.215791+0000 b40267 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000914:[<0x1000000088d/picture-61.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000914:[<0x1000000088d/picture-61.jpg v912>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.457+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000099b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-7.jpg auth v922 ap=1 s=49878 n(v0 rc2024-05-09T08:27:53.706777+0000 b49878 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000099b:[<0x1000000088d/picture-7.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000099b:[<0x1000000088d/picture-7.jpg v922>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.457+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000008a7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-31.xml auth v942 ap=1 s=26041 n(v0 rc2024-05-09T08:27:52.877801+0000 b26041 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000008a7:[<0x1000000088d/article-31.xml v942>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000008a7:[<0x1000000088d/article-31.xml v942>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.457+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000993 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-9.jpg auth v918 ap=1 s=55847 n(v0 rc2024-05-09T08:27:53.686778+0000 b55847 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000993:[<0x1000000088d/picture-9.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000993:[<0x1000000088d/picture-9.jpg v918>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.457+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b52 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-11.xml auth v1052 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.879715+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59866f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b52:[<0x1000000088d/comment-11.xml v1052>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b52:[<0x1000000088d/comment-11.xml v1052>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:04.887 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:04.457+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b43 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-20.xml auth v1050 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.840716+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59867a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b43:[<0x1000000088d/comment-20.xml v1050>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b43:[<0x1000000088d/comment-20.xml v1050>,<0x10000000399/blog-9 v15883>,<0x10000000208/blogtest_in v327>,<0x10000000201/src v1986>,<0x10000000006/blogbench-1.0 v1761>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v43>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v77>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub queued for path: / 2024-05-09T08:28:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000207 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.in) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000205 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/blogbench.8) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000206 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.am) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.159 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: pgmap v56: 129 pgs: 129 active+clean; 48 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 567 op/s 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000203 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/m4/acx_pthread.m4) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b9f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-73.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000baa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-21.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003eb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-72.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000452 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-35.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-73.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c3b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-60.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.160 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000938 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-78.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-86.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-97.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000050d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-96.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000861 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-30.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000445 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-40.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003cd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-82.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000487 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-15.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000074d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-88.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c1b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-59.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d01 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-40.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-72.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008d0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-79.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000800 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-35.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.161 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d45 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-33.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-92.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000830 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-75.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c9a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-18.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000075b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-86.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d0b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-39.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000050b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-43.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-97.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-74.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007fb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003b8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-89.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000832 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-73.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000751 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-36.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000085e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-82.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.162 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000047c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-19.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007c5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-86.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bbd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-20.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007c9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-82.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d12 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-35.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c20 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-63.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000971 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c36 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-51.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d4d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-32.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c7a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-54.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c7e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-27.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000544 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-91.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000857 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000073f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-38.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bf9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-70.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000810 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-23.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c93 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-20.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000440 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-42.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000048b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-14.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c15 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-61.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-84.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bcf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-83.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000855 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000503 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-45.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c54 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-41.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-60.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009a4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.164 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c48 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-45.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000438 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-46.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007f9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-42.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bfd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-69.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-62.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007c2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-89.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004ca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-66.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c72 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-31.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000082c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-79.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000526 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-41.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.165 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003b2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-91.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000099a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-23.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d66 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-11.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-71.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bd9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-80.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b9a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-22.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000494 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-11.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c47 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-59.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000484 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-16.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.169 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cc9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-9.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000084e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-45.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000436 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-47.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000852 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-41.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000941 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-77.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c09 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-65.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000421 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-54.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c12 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-62.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000046a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-26.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bfb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-15.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c88 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-24.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003ac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-93.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c4a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-44.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004e1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-56.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.170 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000da4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-28.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-47.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-85.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000496 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-10.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bbf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-88.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000850 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-43.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c1d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-12.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-84.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c03 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-67.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ca2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-50.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000815 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000505 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-44.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ce6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-3.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000083f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-60.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.171 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000433 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-48.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cb6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-48.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cfb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-37.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-91.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-4.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cd3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-7.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000839 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-66.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000046d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-25.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c2a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-55.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007d4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-71.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004b8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-72.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ba5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-95.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000082b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-80.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.172 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004d9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-59.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ccf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-8.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000044a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-38.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000099e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-22.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000041f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-55.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000813 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-16.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000098b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-25.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007e8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000508 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-97.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000be7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-76.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000809 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-26.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003a7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-95.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d6c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-31.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000049d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-8.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009b6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-15.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bff (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-68.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cc0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-47.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-93.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000843 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-56.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cc8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-46.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-47.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000818 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-11.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000085d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-31.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003f7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-68.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bd1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-70.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007f2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003e0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-76.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000081f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.174 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c4d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-8.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000819 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d41 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-33.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000093d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-26.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009c8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-12.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dcb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-26.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c6f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-32.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004f3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-49.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c92 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-21.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000501 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-98.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000045a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-32.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c74 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-30.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007f3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-83.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ba8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-94.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000be0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-69.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.175 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cb1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-13.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000478 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-21.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000045f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-30.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003aa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-94.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cd8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-6.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000747 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-89.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c39 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-50.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009d0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-74.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-2.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-36.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bb7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-90.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000054b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-90.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d29 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-14.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000048e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-13.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d91 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-29.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.176 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c17 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-64.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c26 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-56.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c4e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-43.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000084f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-44.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000042d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-50.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000840 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-59.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c6b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-33.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bb4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-72.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c22 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-57.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000db1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-7.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004af (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-2.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bdd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-79.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004a3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-6.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dde (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-76.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.177 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000833 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-72.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dc8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-14.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d0f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-15.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bf5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-67.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c1e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-58.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c30 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-53.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000080a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-25.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c7b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-28.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004e7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-54.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d8d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-9.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-90.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-38.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c33 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-52.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000864 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-29.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000da3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-19.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.178 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000041b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000413 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-59.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c11 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dc5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-6.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c68 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003af (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-92.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cdc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-5.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bd8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003e6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-74.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bd5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-81.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000047f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-18.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000be1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-78.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bcd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000042a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.179 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d64 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-29.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000454 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-12.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c78 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-29.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000468 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-27.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c18 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-60.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c77 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d7c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-10.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-27.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000759 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000005ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-52.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c69 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-34.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c8b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c8f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-22.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003f1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-70.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000517 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-1.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.180 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000047a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d9d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-8.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-49.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c05 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006bc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-34.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dd4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-78.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000053c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-92.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-14.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000005ae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-53.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003ca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-83.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bcc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-84.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000040c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-61.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c45 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-46.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000442 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000043a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-45.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.181 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006f2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-16.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c42 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-47.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003f5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-69.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bf6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-71.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000523 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-94.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bb2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-91.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004d5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-61.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000045c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-31.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c59 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cc1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-11.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ca3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-16.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ddb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-11.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c2d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-54.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.182 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c41 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003d2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-23.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c81 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-26.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d99 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000411 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-17.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000704 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-93.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006c2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-32.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cc5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-10.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000605 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-48.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000866 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c3f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-48.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ce1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-4.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000710 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.183 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c5d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-38.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003ce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-58.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006e3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bf3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-72.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000488 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-9.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006c8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c85 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-25.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003bf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-60.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003c3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-24.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c66 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-35.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006ce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bc7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-86.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c56 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-40.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d7f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-30.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.184 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000068d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000689 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bd2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-82.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006ef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ca7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-15.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d04 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-36.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006e6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006da (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c98 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-19.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006bf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bbb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-89.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000578 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-7.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.185 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003a4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000422 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000649 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-42.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000670 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-39.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c51 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-42.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000063c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-43.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000963 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cf4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-38.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000719 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-3.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bc5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-87.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000591 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-55.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c0c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-64.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.186 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d3d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-13.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-5.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000701 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-11.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-75.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c60 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-37.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003de (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-57.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c5a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-39.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003b9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-61.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000db6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-27.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bf0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-73.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c63 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-36.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cb8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-12.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000066d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-40.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-1.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003da (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-22.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.187 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d2e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-34.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c06 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-66.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000713 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-5.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000be4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-77.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000005ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-50.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c9d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-17.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000056b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-58.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d56 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-32.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d51 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-12.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c8c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-23.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000005df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-51.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c0f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-63.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000061a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-46.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000530 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-62.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.188 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-25.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000497 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006c5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-31.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000005fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-49.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000070a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-8.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dd8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-77.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006d4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-26.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-21.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000967 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-13.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000404 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-64.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-13.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004b3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-87.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000509 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d92 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-21.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.189 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d75 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-26.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000059e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-54.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003e3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-75.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006cb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-29.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000044c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-37.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-27.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006fb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-13.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-73.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000402 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-18.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000447 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-39.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000044e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-36.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-18.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-52.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000491 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-12.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.190 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-19.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000039d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-27.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004b2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-1.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000716 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-4.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-35.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-67.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000071c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-2.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000da9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-18.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003f3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-20.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-14.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004ea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-53.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000540 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-61.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000075f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-34.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000070d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-7.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000707 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-9.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.191 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-16.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-66.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003dc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-77.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dd3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-12.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-49.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000755 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-87.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cf3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-42.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000039c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-98.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c0b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-65.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000473 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-23.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-100.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000482 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-17.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000085a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-32.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000406 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-63.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.192 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-48.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c99 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-51.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000865 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-81.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c7d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-3.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c84 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-53.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-55.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000763 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-85.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c29 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-11.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004bd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-70.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004a6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-5.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000455 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-34.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d33 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-35.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d6a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-28.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c5c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-57.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003d0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-81.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.193 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003d9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-78.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-71.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000461 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-29.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000040f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-60.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-63.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000536 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-40.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c35 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-61.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d1b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-38.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000476 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-22.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000be9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-68.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000430 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-49.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d8b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-22.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-80.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cda (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-44.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.194 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000043e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-43.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000458 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-33.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000052d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-93.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c6d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-5.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d70 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-27.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004f1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-50.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000098e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-76.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004db (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-58.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000418 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-57.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c91 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-52.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c01 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-66.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004cd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-65.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003d6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-79.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dc2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-15.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.195 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000415 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-58.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ce8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-43.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000424 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-53.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-17.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000464 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-28.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000427 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-52.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-69.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004ff (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-99.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004ef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-51.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cd2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-45.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-84.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-62.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-46.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.196 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004cf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-64.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003d4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-80.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d25 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-37.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bc2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-71.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c32 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-10.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000051c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-95.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c97 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-1.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000db8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-16.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000499 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-9.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c62 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-6.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000043c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-44.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000401 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-65.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c50 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-58.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000409 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-62.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.197 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c6e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-55.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000470 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-24.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d3b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-34.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d7b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-25.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.198 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008cc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-28.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003c5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-85.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cf9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-41.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.201 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-88.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003f9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-67.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000099c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-75.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000074b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-37.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000be3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-17.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004ab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-3.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003a3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-96.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000998 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-24.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000007ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-33.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-57.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-7.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000514 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-42.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000004c4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-68.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.202 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000546 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-39.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d87 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-23.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d82 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-24.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000974 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-73.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b4b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-13.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a9e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-73.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-35.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b42 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-21.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000acc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-60.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a8a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-78.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b3d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-26.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.203 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-75.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-37.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b7a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-1.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000964 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-27.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000089e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000089f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-39.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000095b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-36.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-12.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a86 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-79.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b11 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bb1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-52.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.204 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b29 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-33.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008c8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-1.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b14 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-39.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b33 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-30.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000094b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000950 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-34.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000093b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-43.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000924 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-53.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000908 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-67.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a7a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-82.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008d5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-90.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-32.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b72 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.205 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000976 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ac2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b45 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-18.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ae1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000980 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-16.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000089c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-42.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000947 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b4e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-12.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000096f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-23.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-74.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008f0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-79.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b0a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-42.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.206 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000abb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-65.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000935 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-45.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000940 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-41.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000af2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-49.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ac5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-62.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008af (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-23.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000916 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-60.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-17.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000898 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000090a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-66.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-76.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008ee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-80.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000897 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-91.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000955 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-32.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.207 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000972 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-22.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009f5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-63.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-11.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000092e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-48.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000acf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-59.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aeb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-16.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000090e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-64.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a8e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-77.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b1b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-37.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000890 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aa6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-71.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008ac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-26.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009b0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000981 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-15.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.208 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000988 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000930 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-47.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000091e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-56.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008bc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008bd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-9.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000985 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-12.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000089a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-44.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-50.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b47 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-16.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000939 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-44.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008e6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-83.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000896 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000099f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000af6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.209 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008c2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ad3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-58.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008ad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-25.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000add (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-55.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008fe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-72.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bb5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-24.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000918 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-59.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-38.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b25 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-34.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b2f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-31.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b3e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-25.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a7e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-81.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b56 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-92.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ae4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-53.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.210 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b0e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b66 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a96 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-75.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000920 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-55.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000961 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009a8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-3.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000949 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000987 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-11.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000097e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-5.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008e1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-85.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-34.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bbe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.211 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000902 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-70.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b22 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-35.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000096d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b3a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-28.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000953 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ad6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-57.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000091c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000900 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-71.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-29.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008bf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-7.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000942 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-40.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000093e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-42.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000910 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-63.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000906 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-68.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000945 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-39.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.212 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ac8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-61.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-23.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b3f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-24.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000984 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ada (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000097b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-18.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-22.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b6a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-5.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d68 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b18 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-38.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000979 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d4f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-18.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ab0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-68.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.213 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bd4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-18.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008de (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-87.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d4a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-19.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000893 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-84.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c0d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-82.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b36 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-29.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000895 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-49.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bb3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-27.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dbc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-4.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b5e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a82 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-80.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009a5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cdd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-35.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.214 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b41 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-22.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dc6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-2.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c19 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-78.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-82.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b76 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008ab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-27.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-20.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d62 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-14.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000094d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008e0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-86.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c4f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-61.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b03 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-44.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009ac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009f0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-94.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.215 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-7.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008f2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-78.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000089b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-43.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d7e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-9.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008cf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-93.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c46 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-64.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d78 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-10.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000983 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dcc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-1.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008d9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-89.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-33.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000089d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000db2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-6.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-81.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.216 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000899 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-45.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bf1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-9.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000914 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-61.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bc1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-24.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000099b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-31.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000993 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d22 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-28.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b52 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-11.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b43 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cbe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-37.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009a2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-5.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d54 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-17.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c04 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-3.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.217 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-27.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d35 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-24.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000092a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-50.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d02 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-33.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a92 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-76.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c5b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-57.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000091a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-58.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bf7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-7.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ae8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-52.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c8d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-42.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000090c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-65.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ca8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-39.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c3a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-68.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d31 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-25.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.218 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ab7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-66.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b62 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-7.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b48 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000891 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-53.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d84 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-8.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b07 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-43.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c73 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-49.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b46 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-17.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bf4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-8.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008aa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-28.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d2b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-26.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d17 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-31.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000892 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-52.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bcb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-21.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.219 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b5a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-9.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ba4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-30.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-77.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bfa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-6.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000904 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-69.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c96 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-13.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000928 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-51.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c07 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-2.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b00 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-45.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c64 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-54.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000912 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-62.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c10 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-81.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d1f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-29.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000db7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-5.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000933 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-46.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.220 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c70 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-50.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c57 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-58.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c43 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-65.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000baf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-28.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000088f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-55.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c1c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-77.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000096a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-25.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c37 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-69.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c6c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-51.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000af9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c4b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-62.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000afc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c7c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-46.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.221 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dc0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-3.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000958 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-31.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c61 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-55.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d73 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-11.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aa9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-70.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c34 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-70.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000092c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-49.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c31 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-71.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000968 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-26.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bfe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-5.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ba9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-29.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000997 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-8.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c49 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-63.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c9e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-40.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008dc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-88.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.222 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bc8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-22.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-32.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-69.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d39 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-23.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a76 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-83.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-79.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008cb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-95.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000be2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-14.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-21.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c7f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-45.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008a8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-30.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c94 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-41.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a9a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-74.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b1e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-36.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.223 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c52 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-60.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-66.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000abe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-64.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c13 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-80.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008b9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-13.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d3e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-22.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aa2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-72.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000beb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-11.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b6e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bb8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-26.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008c6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bda (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-16.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000894 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-50.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c5e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-56.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b49 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.224 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d5d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-15.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000008c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c79 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-47.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-10.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ab4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-67.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d6e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-12.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000922 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-54.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009e7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-90.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c83 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-44.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000095f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-29.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bbc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-25.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b44 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-56.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c00 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-4.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.225 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c2e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-72.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c55 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-59.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c2b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-73.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d42 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-21.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c23 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-75.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ce2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-100.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cf1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-97.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d1a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-30.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bde (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-15.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ccd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-36.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c8a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-43.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bd0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-19.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d47 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-20.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.226 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cb3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-38.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d27 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-27.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000be8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-12.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c6a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-52.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c0a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-1.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-34.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c67 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-53.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c3d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-67.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d58 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-16.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c27 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-74.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bc6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-23.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bd6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-17.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c1f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-76.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d15 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-32.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.227 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-63.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c75 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-48.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.228 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:05 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub queued for path: / 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000207 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.in) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000205 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/blogbench.8) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000206 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.am) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: pgmap v56: 129 pgs: 129 active+clean; 48 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 567 op/s 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000203 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/m4/acx_pthread.m4) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b9f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-73.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000baa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-21.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003eb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-72.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000452 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-35.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-73.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c3b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-60.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000938 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-78.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-86.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-97.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000050d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-96.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000861 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-30.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000445 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-40.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003cd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-82.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000487 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-15.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000074d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-88.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c1b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-59.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d01 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-40.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-72.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008d0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-79.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000800 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-35.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d45 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-33.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-92.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000830 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-75.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c9a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-18.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000075b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-86.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d0b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-39.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000050b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-43.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-97.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-74.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007fb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003b8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-89.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000832 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-73.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000751 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-36.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000085e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-82.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000047c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-19.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007c5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-86.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bbd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-20.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007c9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-82.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d12 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-35.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c20 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-63.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000971 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c36 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-51.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d4d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-32.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c7a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-54.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c7e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-27.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000544 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-91.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000857 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000073f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-38.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bf9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-70.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000810 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-23.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c93 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-20.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000440 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-42.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000048b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-14.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c15 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-61.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-84.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bcf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-83.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000855 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000503 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-45.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c54 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-41.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-60.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009a4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c48 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-45.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000438 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-46.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007f9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-42.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bfd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-69.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-62.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007c2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-89.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004ca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-66.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c72 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-31.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000082c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-79.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000526 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-41.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003b2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-91.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000099a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-23.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d66 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-11.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-71.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bd9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-80.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b9a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-22.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000494 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-11.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c47 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-59.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000484 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-16.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cc9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-9.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000084e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-45.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000436 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-47.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000852 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-41.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000941 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-77.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c09 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-65.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000421 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-54.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c12 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-62.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000046a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-26.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bfb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-15.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c88 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-24.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003ac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-93.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c4a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-44.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004e1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-56.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000da4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-28.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-47.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-85.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000496 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-10.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bbf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-88.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000850 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-43.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c1d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-12.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-84.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c03 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-67.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ca2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-50.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000815 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000505 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-44.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ce6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-3.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000083f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-60.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000433 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-48.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cb6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-48.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cfb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-37.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-91.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-4.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cd3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-7.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000839 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-66.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000046d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-25.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c2a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-55.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007d4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-71.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004b8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-72.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ba5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-95.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000082b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-80.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004d9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-59.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ccf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-8.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000044a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-38.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000099e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-22.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000041f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-55.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000813 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-16.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000098b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-25.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007e8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000508 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-97.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000be7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-76.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000809 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-26.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003a7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-95.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d6c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-31.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000049d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-8.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009b6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-15.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bff (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-68.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cc0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-47.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-93.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000843 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-56.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cc8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-46.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-47.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000818 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-11.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000085d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-31.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003f7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-68.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bd1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-70.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007f2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003e0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-76.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000081f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c4d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-8.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000819 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d41 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-33.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000093d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-26.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009c8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-12.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dcb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-26.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c6f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-32.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004f3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-49.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c92 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-21.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000501 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-98.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000045a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-32.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c74 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-30.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007f3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-83.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ba8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-94.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000be0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-69.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cb1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-13.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000478 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-21.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000045f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-30.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003aa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-94.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cd8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-6.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000747 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-89.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c39 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-50.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009d0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-74.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-2.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-36.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bb7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-90.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000054b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-90.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d29 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-14.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000048e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-13.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d91 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-29.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c17 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-64.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c26 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-56.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c4e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-43.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000084f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-44.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000042d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-50.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000840 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-59.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c6b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-33.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bb4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-72.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c22 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-57.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000db1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-7.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004af (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-2.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bdd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-79.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004a3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-6.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dde (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-76.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000833 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-72.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dc8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-14.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d0f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-15.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bf5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-67.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c1e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-58.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c30 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-53.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000080a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-25.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c7b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-28.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004e7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-54.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d8d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-9.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-90.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-38.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c33 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-52.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000864 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-29.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000da3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-19.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000041b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000413 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-59.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c11 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dc5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-6.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c68 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003af (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-92.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cdc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-5.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bd8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003e6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-74.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bd5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-81.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000047f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-18.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000be1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-78.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bcd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000042a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d64 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-29.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000454 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-12.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c78 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-29.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000468 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-27.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c18 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-60.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c77 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d7c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-10.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-27.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000759 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000005ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-52.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c69 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-34.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c8b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c8f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-22.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003f1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-70.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000517 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-1.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000047a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d9d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-8.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-49.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c05 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006bc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-34.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dd4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-78.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000053c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-92.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-14.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000005ae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-53.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003ca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-83.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bcc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-84.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000040c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-61.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c45 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-46.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000442 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000043a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-45.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006f2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-16.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c42 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-47.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003f5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-69.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bf6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-71.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000523 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-94.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bb2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-91.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004d5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-61.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000045c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-31.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c59 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cc1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-11.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ca3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-16.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ddb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-11.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c2d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-54.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c41 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003d2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-23.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c81 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-26.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d99 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000411 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-17.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000704 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-93.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006c2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-32.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cc5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-10.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000605 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-48.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000866 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c3f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-48.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ce1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-4.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000710 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c5d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-38.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003ce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-58.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006e3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bf3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-72.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000488 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-9.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006c8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c85 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-25.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003bf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-60.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003c3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-24.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c66 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-35.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006ce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bc7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-86.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c56 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-40.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d7f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-30.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000068d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000689 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bd2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-82.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006ef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ca7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-15.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d04 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-36.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006e6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006da (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c98 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-19.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006bf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bbb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-89.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000578 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-7.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003a4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000422 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000649 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-42.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000670 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-39.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c51 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-42.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000063c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-43.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000963 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cf4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-38.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000719 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-3.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bc5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-87.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000591 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-55.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c0c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-64.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d3d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-13.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-5.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000701 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-11.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-75.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c60 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-37.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003de (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-57.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c5a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-39.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003b9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-61.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000db6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-27.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bf0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-73.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c63 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-36.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cb8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-12.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000066d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-40.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-1.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003da (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-22.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d2e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-34.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c06 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-66.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000713 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-5.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000be4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-77.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000005ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-50.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c9d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-17.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000056b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-58.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d56 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-32.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d51 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-12.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c8c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-23.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000005df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-51.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c0f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-63.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000061a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-46.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000530 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-62.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-25.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000497 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006c5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-31.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000005fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-49.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000070a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-8.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dd8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-77.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006d4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-26.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-21.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000967 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-13.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000404 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-64.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-13.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004b3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-87.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000509 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d92 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-21.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d75 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-26.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000059e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-54.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003e3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-75.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006cb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-29.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000044c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-37.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-27.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006fb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-13.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-73.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000402 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-18.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000447 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-39.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000044e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-36.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-18.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-52.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000491 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-12.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-19.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000039d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-27.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004b2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-1.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000716 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-4.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-35.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-67.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000071c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-2.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000da9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-18.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003f3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-20.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-14.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004ea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-53.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000540 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-61.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000075f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-34.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000070d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-7.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000707 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-9.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-16.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-66.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003dc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-77.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dd3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-12.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-49.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000755 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-87.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cf3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-42.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000039c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-98.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c0b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-65.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000473 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-23.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-100.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000482 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-17.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000085a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-32.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000406 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-63.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-48.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c99 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-51.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000865 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-81.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c7d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-3.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c84 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-53.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-55.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000763 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-85.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c29 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-11.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004bd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-70.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004a6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-5.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000455 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-34.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d33 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-35.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d6a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-28.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c5c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-57.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003d0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-81.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003d9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-78.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-71.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000461 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-29.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000040f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-60.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-63.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000536 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-40.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c35 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-61.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d1b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-38.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000476 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-22.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000be9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-68.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000430 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-49.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d8b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-22.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-80.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cda (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-44.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000043e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-43.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000458 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-33.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000052d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-93.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c6d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-5.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d70 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-27.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004f1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-50.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000098e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-76.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004db (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-58.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000418 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-57.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c91 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-52.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c01 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-66.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004cd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-65.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003d6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-79.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dc2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-15.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000415 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-58.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ce8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-43.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000424 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-53.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-17.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000464 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-28.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000427 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-52.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-69.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004ff (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-99.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004ef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-51.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cd2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-45.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-84.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-62.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-46.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004cf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-64.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003d4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-80.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d25 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-37.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bc2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-71.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c32 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-10.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000051c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-95.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c97 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-1.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000db8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-16.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000499 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-9.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c62 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-6.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000043c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-44.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000401 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-65.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c50 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-58.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000409 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-62.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c6e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-55.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000470 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-24.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d3b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-34.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d7b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-25.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008cc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-28.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003c5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-85.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cf9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-41.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-88.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003f9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-67.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000099c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-75.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000074b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-37.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000be3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-17.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004ab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-3.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003a3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-96.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000998 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-24.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000007ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-33.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-57.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-7.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000514 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-42.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000004c4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-68.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000546 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-39.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d87 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-23.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d82 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-24.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000974 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-73.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b4b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-13.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a9e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-73.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-35.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b42 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-21.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000acc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-60.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a8a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-78.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b3d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-26.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-75.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-37.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b7a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-1.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000964 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-27.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000089e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000089f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-39.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000095b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-36.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-12.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a86 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-79.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b11 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bb1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-52.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b29 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-33.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008c8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-1.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b14 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-39.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b33 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-30.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000094b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000950 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-34.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000093b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-43.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000924 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-53.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000908 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-67.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a7a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-82.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008d5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-90.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-32.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b72 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000976 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ac2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b45 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-18.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ae1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000980 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-16.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000089c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-42.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000947 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b4e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-12.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000096f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-23.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-74.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008f0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-79.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b0a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-42.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000abb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-65.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000935 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-45.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000940 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-41.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000af2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-49.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ac5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-62.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008af (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-23.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000916 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-60.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-17.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000898 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000090a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-66.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-76.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008ee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-80.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000897 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-91.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000955 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-32.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000972 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-22.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009f5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-63.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-11.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000092e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-48.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000acf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-59.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aeb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-16.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000090e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-64.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a8e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-77.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b1b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-37.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000890 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aa6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-71.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008ac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-26.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009b0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000981 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-15.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000988 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000930 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-47.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000091e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-56.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008bc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008bd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-9.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000985 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-12.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000089a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-44.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-50.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b47 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-16.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000939 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-44.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008e6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-83.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000896 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000099f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000af6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008c2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ad3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-58.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008ad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-25.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000add (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-55.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008fe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-72.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bb5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-24.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000918 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-59.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-38.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b25 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-34.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b2f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-31.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b3e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-25.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a7e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-81.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b56 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-92.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ae4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-53.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b0e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b66 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a96 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-75.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000920 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-55.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000961 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009a8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-3.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000949 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000987 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-11.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000097e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-5.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008e1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-85.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-34.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bbe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000902 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-70.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b22 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-35.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000096d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b3a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-28.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000953 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ad6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-57.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000091c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000900 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-71.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-29.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008bf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-7.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000942 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-40.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000093e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-42.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000910 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-63.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000906 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-68.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000945 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-39.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ac8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-61.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-23.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b3f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-24.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000984 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ada (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000097b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-18.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-22.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b6a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-5.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d68 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b18 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-38.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000979 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d4f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-18.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ab0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-68.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bd4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-18.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008de (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-87.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d4a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-19.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000893 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-84.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c0d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-82.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b36 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-29.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000895 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-49.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bb3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-27.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dbc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-4.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b5e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a82 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-80.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009a5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cdd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-35.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b41 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-22.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dc6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-2.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c19 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-78.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-82.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b76 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008ab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-27.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-20.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d62 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-14.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000094d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008e0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-86.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c4f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-61.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b03 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-44.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009ac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009f0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-94.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-7.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008f2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-78.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000089b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-43.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d7e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-9.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008cf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-93.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c46 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-64.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d78 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-10.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000983 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dcc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-1.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008d9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-89.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-33.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000089d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000db2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-6.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-81.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000899 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-45.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bf1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-9.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000914 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-61.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bc1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-24.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000099b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-31.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000993 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d22 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-28.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b52 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-11.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b43 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cbe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-37.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009a2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-5.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d54 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-17.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c04 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-3.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-27.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d35 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-24.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000092a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-50.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d02 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-33.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a92 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-76.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c5b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-57.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000091a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-58.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bf7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-7.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ae8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-52.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c8d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-42.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000090c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-65.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ca8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-39.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c3a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-68.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d31 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-25.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ab7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-66.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b62 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-7.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b48 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000891 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-53.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d84 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-8.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b07 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-43.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c73 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-49.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b46 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-17.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bf4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-8.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008aa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-28.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d2b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-26.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d17 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-31.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000892 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-52.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bcb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-21.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b5a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-9.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ba4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-30.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-77.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bfa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-6.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000904 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-69.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c96 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-13.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000928 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-51.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c07 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-2.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b00 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-45.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c64 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-54.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000912 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-62.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c10 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-81.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d1f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-29.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000db7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-5.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000933 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-46.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c70 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-50.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c57 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-58.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c43 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-65.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000baf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-28.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000088f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-55.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c1c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-77.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000096a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-25.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c37 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-69.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c6c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-51.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000af9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c4b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-62.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000afc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c7c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-46.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dc0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-3.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000958 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-31.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c61 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-55.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d73 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-11.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aa9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-70.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c34 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-70.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000092c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-49.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c31 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-71.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000968 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-26.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bfe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-5.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ba9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-29.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000997 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-8.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c49 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-63.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c9e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-40.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008dc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-88.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bc8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-22.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-32.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-69.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d39 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-23.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a76 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-83.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-79.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008cb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-95.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000be2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-14.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-21.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c7f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-45.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008a8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-30.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c94 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-41.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a9a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-74.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b1e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-36.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c52 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-60.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-66.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000abe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-64.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c13 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-80.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008b9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-13.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d3e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-22.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aa2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-72.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000beb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-11.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b6e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bb8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-26.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008c6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bda (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-16.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000894 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-50.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c5e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-56.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b49 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d5d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-15.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000008c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c79 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-47.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-10.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ab4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-67.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d6e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-12.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000922 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-54.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009e7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-90.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c83 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-44.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000095f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-29.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bbc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-25.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b44 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-56.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c00 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-4.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c2e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-72.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c55 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-59.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c2b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-73.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d42 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-21.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c23 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-75.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ce2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-100.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cf1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-97.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d1a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-30.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bde (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-15.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ccd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-36.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c8a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-43.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bd0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-19.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d47 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-20.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cb3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-38.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d27 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-27.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000be8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-12.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c6a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-52.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c0a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-1.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-34.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c67 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-53.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c3d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-67.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d58 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-16.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c27 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-74.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bc6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-23.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bd6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-17.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c1f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-76.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d15 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-32.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-63.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c75 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-48.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:05 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub queued for path: / 2024-05-09T08:28:05.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000207 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.in) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000205 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/blogbench.8) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000206 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.am) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: pgmap v56: 129 pgs: 129 active+clean; 48 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 567 op/s 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000203 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/m4/acx_pthread.m4) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b9f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-73.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000baa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-21.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003eb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-72.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000452 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-35.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-73.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c3b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-60.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000938 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-78.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-86.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-97.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000050d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-96.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000861 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-30.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000445 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-40.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003cd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-82.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000487 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-15.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000074d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-88.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c1b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-59.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d01 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-40.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-72.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008d0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-79.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000800 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-35.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d45 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-33.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-92.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000830 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-75.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c9a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-18.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000075b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-86.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d0b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-39.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000050b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-43.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-97.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-74.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007fb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003b8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-89.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000832 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-73.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000751 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-36.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000085e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-82.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000047c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-19.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007c5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-86.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bbd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-20.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007c9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-82.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d12 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-35.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c20 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-63.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000971 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c36 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-51.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d4d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-32.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c7a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-54.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c7e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-27.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000544 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-91.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000857 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000073f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-38.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bf9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-70.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000810 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-23.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c93 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-20.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000440 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-42.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000048b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-14.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c15 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-61.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-84.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bcf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-83.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000855 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000503 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-45.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c54 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-41.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-60.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009a4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c48 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-45.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000438 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-46.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007f9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-42.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bfd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-69.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-62.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007c2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-89.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004ca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-66.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c72 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-31.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000082c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-79.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000526 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-41.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003b2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-91.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000099a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-23.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d66 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-11.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-71.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bd9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-80.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b9a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-22.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000494 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-11.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c47 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-59.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000484 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-16.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cc9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-9.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000084e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-45.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000436 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-47.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000852 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-41.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000941 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-77.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c09 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-65.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000421 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-54.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c12 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-62.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000046a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-26.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bfb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-15.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c88 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-24.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003ac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-93.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c4a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-44.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004e1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-56.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000da4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-28.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-47.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-85.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000496 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-10.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bbf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-88.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000850 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-43.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c1d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-12.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-84.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c03 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-67.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ca2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-50.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000815 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000505 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-44.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ce6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-3.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000083f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-60.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000433 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-48.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cb6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-48.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cfb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-37.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-91.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-4.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cd3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-7.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000839 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-66.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000046d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-25.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c2a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-55.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007d4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-71.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004b8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-72.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ba5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-95.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000082b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-80.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004d9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-59.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ccf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-8.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000044a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-38.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000099e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-22.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000041f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-55.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000813 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-16.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000098b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-25.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007e8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000508 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-97.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000be7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-76.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000809 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-26.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003a7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-95.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d6c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-31.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000049d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-8.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009b6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-15.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bff (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-68.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cc0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-47.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-93.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000843 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-56.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cc8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-46.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-47.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000818 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-11.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000085d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-31.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003f7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-68.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bd1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-70.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007f2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003e0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-76.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000081f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c4d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-8.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000819 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d41 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-33.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000093d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-26.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009c8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-12.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dcb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-26.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c6f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-32.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004f3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-49.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c92 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-21.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000501 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-98.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000045a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-32.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c74 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-30.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007f3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-83.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ba8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-94.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000be0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-69.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cb1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-13.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000478 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-21.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000045f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-30.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003aa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-94.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cd8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-6.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000747 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-89.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c39 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-50.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009d0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-74.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-2.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-36.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bb7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-90.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000054b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-90.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d29 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-14.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000048e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-13.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d91 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-29.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c17 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-64.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c26 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-56.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c4e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-43.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000084f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-44.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000042d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-50.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000840 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-59.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c6b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-33.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bb4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-72.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c22 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-57.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000db1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-7.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004af (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-2.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bdd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-79.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004a3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-6.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dde (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-76.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000833 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-72.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dc8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-14.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d0f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-15.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bf5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-67.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c1e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-58.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c30 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-53.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000080a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-25.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c7b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-28.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004e7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-54.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d8d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-9.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-90.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-38.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c33 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-52.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000864 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-29.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000da3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-19.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000041b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000413 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-59.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c11 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dc5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-6.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c68 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003af (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-92.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cdc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-5.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bd8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003e6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-74.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bd5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-81.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000047f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-18.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000be1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-78.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bcd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000042a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d64 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-29.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000454 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-12.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c78 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-29.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000468 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-27.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c18 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-60.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c77 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d7c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-10.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-27.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000759 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000005ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-52.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c69 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-34.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c8b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c8f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-22.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003f1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-70.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000517 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-1.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000047a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d9d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-8.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-49.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c05 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006bc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-34.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dd4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-78.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000053c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-92.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-14.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000005ae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-53.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003ca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-83.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bcc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-84.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000040c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-61.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c45 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-46.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000442 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000043a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-45.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006f2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-16.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c42 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-47.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003f5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-69.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bf6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-71.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000523 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-94.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bb2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-91.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004d5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-61.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000045c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-31.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c59 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cc1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-11.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ca3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-16.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ddb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-11.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c2d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-54.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c41 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003d2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-23.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c81 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-26.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d99 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000411 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-17.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000704 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-93.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006c2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-32.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cc5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-10.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000605 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-48.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000866 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c3f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-48.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ce1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-4.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000710 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c5d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-38.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.629 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003ce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-58.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006e3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bf3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-72.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000488 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-9.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006c8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c85 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-25.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003bf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-60.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003c3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-24.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c66 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-35.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006ce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bc7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-86.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c56 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-40.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d7f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-30.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000068d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.630 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000689 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bd2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-82.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006ef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ca7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-15.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d04 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-36.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006e6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006da (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c98 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-19.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006bf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bbb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-89.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000578 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-7.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003a4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000422 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000649 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-42.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000670 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-39.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c51 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-42.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000063c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-43.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000963 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cf4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-38.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000719 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-3.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bc5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-87.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000591 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-55.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c0c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-64.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d3d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-13.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-5.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000701 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-11.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-75.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c60 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-37.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003de (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-57.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c5a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-39.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003b9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-61.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000db6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-27.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bf0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-73.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c63 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-36.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cb8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-12.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000066d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-40.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-1.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003da (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-22.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d2e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-34.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.633 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c06 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-66.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000713 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-5.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000be4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-77.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000005ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-50.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c9d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-17.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000056b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-58.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d56 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-32.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d51 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-12.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c8c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-23.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000005df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-51.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c0f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-63.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000061a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-46.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.634 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000530 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-62.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-25.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000497 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006c5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-31.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000005fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-49.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000070a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-8.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dd8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-77.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006d4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-26.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-21.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000967 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-13.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000404 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-64.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-13.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004b3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-87.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000509 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.635 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d92 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-21.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d75 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-26.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000059e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-54.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003e3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-75.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006cb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-29.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000044c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-37.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008d7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-27.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006fb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-13.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-73.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000402 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-18.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000447 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-39.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000044e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-36.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-18.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-52.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.636 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000491 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-12.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-19.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000039d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-27.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004b2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-1.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000716 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-4.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-35.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-67.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000071c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-2.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000da9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-18.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003f3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-20.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-14.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004ea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-53.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000540 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-61.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000075f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-34.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.637 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000070d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-7.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000707 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-9.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-16.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-66.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003dc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-77.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dd3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-12.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-49.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000755 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-87.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cf3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-42.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000039c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-98.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c0b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-65.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000473 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-23.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-100.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000482 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-17.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000085a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-32.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000406 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-63.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-48.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c99 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-51.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000865 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-81.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c7d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-3.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c84 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-53.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-55.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000763 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-85.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c29 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-11.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004bd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-70.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004a6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-5.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000455 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-34.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d33 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-35.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.640 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d6a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-28.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.640 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c5c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-57.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.640 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003d0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-81.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.640 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003d9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-78.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.640 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-71.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.640 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000461 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-29.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000040f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-60.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-63.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000536 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-40.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c35 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-61.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d1b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-38.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000476 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-22.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000be9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-68.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000430 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-49.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d8b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-22.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-80.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cda (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-44.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000043e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-43.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000458 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-33.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000052d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-93.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c6d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-5.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d70 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-27.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004f1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-50.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000098e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-76.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004db (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-58.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000418 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-57.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c91 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-52.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c01 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-66.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004cd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-65.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003d6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-79.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dc2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-15.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000415 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-58.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ce8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-43.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000424 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-53.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-17.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000464 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-28.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000427 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-52.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-69.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004ff (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-99.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004ef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-51.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cd2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-45.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-84.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-62.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-46.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004cf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-64.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003d4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-80.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d25 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-37.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bc2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-71.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c32 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-10.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000051c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-95.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c97 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-1.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000db8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-16.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000499 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-9.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c62 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-6.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000043c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-44.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000401 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-65.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c50 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-58.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000409 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-62.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c6e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-55.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000470 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-24.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d3b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-34.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d7b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-25.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008cc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-28.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003c5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-85.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cf9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-41.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-88.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003f9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-67.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000099c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-75.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000074b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-37.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000be3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-17.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004ab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-3.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003a3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-96.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000998 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-24.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000007ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-33.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-57.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-7.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000514 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-42.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000004c4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-68.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000546 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-39.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d87 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-23.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d82 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-24.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000974 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-73.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b4b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-13.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a9e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-73.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-35.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b42 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-21.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000acc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-60.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a8a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-78.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b3d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-26.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-75.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-37.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b7a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-1.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000964 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-27.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000089e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000089f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-39.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000095b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-36.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-12.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a86 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-79.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b11 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bb1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-52.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b29 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-33.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008c8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-1.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b14 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-39.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b33 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-30.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000094b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000950 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-34.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000093b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-43.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000924 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-53.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000908 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-67.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a7a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-82.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008d5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-90.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-32.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b72 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000976 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ac2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b45 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-18.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ae1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000980 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-16.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000089c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-42.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000947 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b4e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-12.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000096f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-23.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-74.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008f0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-79.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b0a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-42.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000abb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-65.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000935 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-45.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000940 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-41.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000af2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-49.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ac5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-62.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008af (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-23.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000916 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-60.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-17.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000898 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000090a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-66.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-76.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008ee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-80.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000897 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-91.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000955 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-32.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000972 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-22.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009f5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-63.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-11.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000092e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-48.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000acf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-59.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aeb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-16.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000090e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-64.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a8e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-77.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b1b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-37.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000890 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aa6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-71.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008ac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-26.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009b0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000981 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-15.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000988 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000930 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-47.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000091e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-56.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008bc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008bd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-9.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000985 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-12.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000089a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-44.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-50.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b47 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-16.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000939 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-44.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008e6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-83.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000896 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000099f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000af6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008c2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ad3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-58.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008ad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-25.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000add (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-55.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008fe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-72.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bb5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-24.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000918 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-59.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-38.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b25 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-34.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b2f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-31.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b3e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-25.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a7e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-81.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b56 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-92.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ae4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-53.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b0e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b66 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a96 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-75.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000920 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-55.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000961 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009a8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-3.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000949 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000987 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-11.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000097e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-5.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008e1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-85.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-34.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bbe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000902 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-70.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b22 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-35.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000096d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b3a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-28.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000953 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ad6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-57.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000091c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000900 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-71.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-29.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008bf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-7.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000942 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-40.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000093e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-42.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000910 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-63.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000906 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-68.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000945 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-39.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ac8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-61.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-23.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b3f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-24.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000984 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ada (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000097b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-18.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-22.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b6a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-5.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d68 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b18 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-38.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000979 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d4f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-18.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ab0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-68.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bd4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-18.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008de (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-87.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d4a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-19.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000893 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-84.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c0d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-82.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b36 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-29.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000895 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-49.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bb3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-27.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dbc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-4.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b5e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a82 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-80.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009a5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cdd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-35.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b41 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-22.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dc6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-2.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c19 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-78.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008e9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-82.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b76 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008ab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-27.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-20.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d62 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-14.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000094d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008e0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-86.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c4f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-61.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b03 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-44.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009ac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009f0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-94.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dac (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-7.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008f2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-78.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000089b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-43.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d7e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-9.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008cf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-93.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c46 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-64.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d78 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-10.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000983 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dcc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-1.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008d9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-89.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-33.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000089d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000db2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-6.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-81.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000899 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-45.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bf1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-9.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000914 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-61.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bc1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-24.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000099b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-31.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000993 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d22 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-28.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b52 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-11.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b43 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-20.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cbe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-37.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009a2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-5.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d54 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-17.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c04 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-3.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-27.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d35 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-24.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000092a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-50.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d02 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-33.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a92 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-76.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c5b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-57.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000091a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-58.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bf7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-7.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ae8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-52.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c8d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-42.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000090c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-65.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ca8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-39.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c3a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-68.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d31 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-25.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ab7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-66.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b62 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-7.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b48 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000891 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-53.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d84 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-8.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b07 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-43.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c73 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-49.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b46 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-17.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bf4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-8.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008aa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-28.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d2b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-26.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d17 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-31.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000892 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-52.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bcb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-21.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b5a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-9.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ba4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-30.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-77.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bfa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-6.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000904 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-69.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c96 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-13.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000928 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-51.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c07 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-2.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b00 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-45.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c64 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-54.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000912 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-62.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.665 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c10 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-81.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d1f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-29.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000db7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-5.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000933 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-46.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c70 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-50.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c57 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-58.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c43 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-65.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000baf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-28.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000088f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-55.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c1c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-77.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000096a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-25.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c37 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-69.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008be (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c6c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-51.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000af9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c4b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-62.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000afc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c7c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-46.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dc0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-3.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000958 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-31.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c61 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-55.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d73 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-11.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aa9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-70.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c34 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-70.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000092c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-49.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c31 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-71.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000968 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-26.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bfe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-5.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ba9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-29.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000997 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-8.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c49 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-63.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c9e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-40.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008dc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-88.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bc8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-22.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-32.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-69.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d39 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-23.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a76 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-83.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-79.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008cb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-95.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000be2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-14.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-21.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c7f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-45.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008a8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-30.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c94 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-41.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a9a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-74.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b1e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-36.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c52 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-60.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-66.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000abe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-64.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c13 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-80.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008b9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-13.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d3e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-22.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aa2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-72.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000beb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-11.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b6e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bb8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-26.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008c6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bda (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-16.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000894 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-50.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c5e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-56.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b49 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d5d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-15.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000008c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c79 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-47.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-10.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ab4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-67.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d6e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-12.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000922 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-54.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009e7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-90.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c83 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-44.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000095f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-29.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bbc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-25.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b44 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-19.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009d1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-56.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c00 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-4.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c2e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-72.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c55 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-59.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c2b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-73.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d42 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-21.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c23 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-75.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ce2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-100.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cf1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-97.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d1a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-30.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bde (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-15.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ccd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-36.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c8a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-43.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bd0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-19.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d47 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-20.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cb3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-38.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d27 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-27.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000be8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-12.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c6a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-52.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c0a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-1.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cef (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-34.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c67 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-53.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c3d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-67.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d58 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-16.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c27 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-74.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bc6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-23.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.672 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bd6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-17.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.673 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c1f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-76.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.673 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d15 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-32.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.673 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-63.xml) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.673 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c75 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-48.jpg) see mds.i log and `damage ls` output for details 2024-05-09T08:28:05.673 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:05 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:07.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:07 smithi028 ceph-mon[30119]: pgmap v57: 129 pgs: 129 active+clean; 40 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 835 op/s 2024-05-09T08:28:07.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:07 smithi165 ceph-mon[34540]: pgmap v57: 129 pgs: 129 active+clean; 40 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 835 op/s 2024-05-09T08:28:07.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:07 smithi184 ceph-mon[34509]: pgmap v57: 129 pgs: 129 active+clean; 40 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 835 op/s 2024-05-09T08:28:08.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:08 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:28:08.142+0000 7f8912d3a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T08:28:08.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:08 smithi028 ceph-mon[30119]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T08:28:08.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:08 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] up:active 2024-05-09T08:28:08.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:08 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:08.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:08 smithi165 ceph-mon[34540]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T08:28:08.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:08 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] up:active 2024-05-09T08:28:08.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:08 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:08.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:08 smithi184 ceph-mon[34509]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T08:28:08.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:08 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] up:active 2024-05-09T08:28:08.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:08 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:08.704 INFO:tasks.workunit.client.0.smithi028.stdout: 30 122109 785 92802 873 40991 356 2024-05-09T08:28:09.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:09 smithi165 ceph-mon[34540]: pgmap v58: 129 pgs: 129 active+clean; 40 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 751 op/s 2024-05-09T08:28:09.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:09 smithi184 ceph-mon[34509]: pgmap v58: 129 pgs: 129 active+clean; 40 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 751 op/s 2024-05-09T08:28:09.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:09 smithi028 ceph-mon[30119]: pgmap v58: 129 pgs: 129 active+clean; 40 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 751 op/s 2024-05-09T08:28:10.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:10 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] up:active 2024-05-09T08:28:10.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:10 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:10.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:10 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] up:active 2024-05-09T08:28:10.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:10 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:10.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:10 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] up:active 2024-05-09T08:28:10.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:10 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:11.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:11 smithi165 ceph-mon[34540]: pgmap v59: 129 pgs: 129 active+clean; 47 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 932 op/s 2024-05-09T08:28:11.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:11 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] up:active 2024-05-09T08:28:11.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:11 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:11.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:11 smithi184 ceph-mon[34509]: pgmap v59: 129 pgs: 129 active+clean; 47 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 932 op/s 2024-05-09T08:28:11.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:11 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] up:active 2024-05-09T08:28:11.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:11 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:11.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:11 smithi028 ceph-mon[30119]: pgmap v59: 129 pgs: 129 active+clean; 47 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 932 op/s 2024-05-09T08:28:11.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:11 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] up:active 2024-05-09T08:28:11.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:11 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] up:active 2024-05-09T08:28:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:12 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:12.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] up:active 2024-05-09T08:28:12.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:12.607 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.440+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000069f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-64.xml auth v934 ap=1 s=32359 n(v0 rc2024-05-09T08:27:51.237848+0000 b32359 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2024c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000069f:[<0x10000000675/article-64.xml v934>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000069f:[<0x10000000675/article-64.xml v934>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.607 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.441+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a0e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-26.xml auth v982 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.036739+0000 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27ce100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a0e:[<0x10000000675/comment-26.xml v982>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a0e:[<0x10000000675/comment-26.xml v982>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.607 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.442+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000692 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-75.xml auth v940 ap=1 s=17023 n(v0 rc2024-05-09T08:27:51.195849+0000 b17023 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27ce680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000692:[<0x10000000675/article-75.xml v940>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000692:[<0x10000000675/article-75.xml v940>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.442+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000765 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-27.jpg auth v914 ap=1 s=26938 n(v0 rc2024-05-09T08:27:51.849830+0000 b26938 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27cf180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000765:[<0x10000000675/picture-27.jpg v914>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000765:[<0x10000000675/picture-27.jpg v914>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.443+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006c0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-37.xml auth v922 ap=1 s=3406 n(v0 rc2024-05-09T08:27:51.343845+0000 b3406 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27cec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006c0:[<0x10000000675/article-37.xml v922>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006c0:[<0x10000000675/article-37.xml v922>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.444+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000696 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-73.xml auth v938 ap=1 s=25279 n(v0 rc2024-05-09T08:27:51.210848+0000 b25279 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27cf700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000696:[<0x10000000675/article-73.xml v938>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000696:[<0x10000000675/article-73.xml v938>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.444+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006bb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-39.xml auth v924 ap=1 s=4660 n(v0 rc2024-05-09T08:27:51.323845+0000 b4660 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2258000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006bb:[<0x10000000675/article-39.xml v924>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006bb:[<0x10000000675/article-39.xml v924>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.444+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006c3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-36.xml auth v920 ap=1 s=14465 n(v0 rc2024-05-09T08:27:51.353844+0000 b14465 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2258580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006c3:[<0x10000000675/article-36.xml v920>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006c3:[<0x10000000675/article-36.xml v920>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.445+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000070b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-12.xml auth v906 ap=1 s=4451 n(v0 rc2024-05-09T08:27:51.570838+0000 b4451 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2258b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000070b:[<0x10000000675/article-12.xml v906>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000070b:[<0x10000000675/article-12.xml v906>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.445+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000733 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-52.jpg auth v928 ap=1 s=25641 n(v0 rc2024-05-09T08:27:51.693835+0000 b25641 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2259080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000733:[<0x10000000675/picture-52.jpg v928>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000733:[<0x10000000675/picture-52.jpg v928>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.446+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000727 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-1.xml auth{0=1} v904 ap=2 s=19010 n(v0 rc2024-05-09T08:27:51.657836+0000 b19010 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x555da2259600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000727:[<0x10000000675/article-1.xml v904>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000727:[<0x10000000675/article-1.xml v904>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.446+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009f4 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-30.xml auth{0=1} v970 pv1177 ap=5 s=0 n(v0 rc2024-05-09T08:27:54.950741+0000 1=1+0)->n(v0 rc2024-05-09T08:27:54.950741+0000 b8924 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x555da2259b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009f4:[<0x10000000675/comment-30.xml v970>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009f4:[<0x10000000675/comment-30.xml v970>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000758 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-34.jpg auth v918 ap=1 s=29669 n(v0 rc2024-05-09T08:27:51.813831+0000 b29669 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da225a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000758:[<0x10000000675/picture-34.jpg v918>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000758:[<0x10000000675/picture-34.jpg v918>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.608 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.447+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000731 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-53.jpg auth v930 ap=1 s=23960 n(v0 rc2024-05-09T08:27:51.687835+0000 b23960 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da225a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000731:[<0x10000000675/picture-53.jpg v930>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000731:[<0x10000000675/picture-53.jpg v930>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.610 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.448+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000069d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-66.xml auth v936 ap=1 s=1183 n(v0 rc2024-05-09T08:27:51.232848+0000 b1183 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da225b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000069d:[<0x10000000675/article-66.xml v936>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000069d:[<0x10000000675/article-66.xml v936>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.449+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000770 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-16.jpg auth v908 ap=1 s=59258 n(v0 rc2024-05-09T08:27:51.881829+0000 b59258 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27ba000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000770:[<0x10000000675/picture-16.jpg v908>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000770:[<0x10000000675/picture-16.jpg v908>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.449+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000068e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-77.xml auth v942 ap=1 s=27027 n(v0 rc2024-05-09T08:27:51.180849+0000 b27027 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27ba580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000068e:[<0x10000000675/article-77.xml v942>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000068e:[<0x10000000675/article-77.xml v942>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.451+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006ea [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-23.xml auth v912 ap=1 s=13104 n(v0 rc2024-05-09T08:27:51.470841+0000 b13104 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27bc680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006ea:[<0x10000000675/article-23.xml v912>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006ea:[<0x10000000675/article-23.xml v912>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.451+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006fc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-17.xml auth v910 ap=1 s=9804 n(v0 rc2024-05-09T08:27:51.524839+0000 b9804 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27bcc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006fc:[<0x10000000675/article-17.xml v910>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006fc:[<0x10000000675/article-17.xml v910>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.451+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006a0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-63.xml auth v932 ap=1 s=21323 n(v0 rc2024-05-09T08:27:51.240847+0000 b21323 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27c2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006a0:[<0x10000000675/article-63.xml v932>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006a0:[<0x10000000675/article-63.xml v932>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.451+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000075c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-32.jpg auth v916 ap=1 s=20769 n(v0 rc2024-05-09T08:27:51.825831+0000 b20769 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27bd700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000075c:[<0x10000000675/picture-32.jpg v916>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000075c:[<0x10000000675/picture-32.jpg v916>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.451+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000073b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-48.jpg auth v926 ap=1 s=21604 n(v0 rc2024-05-09T08:27:51.720834+0000 b21604 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27c2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000073b:[<0x10000000675/picture-48.jpg v926>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000073b:[<0x10000000675/picture-48.jpg v926>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.452+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009fd [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-64.jpg auth v1160 ap=1 s=13109 n(v0 rc2024-05-09T08:27:54.987740+0000 b13109 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da1eb4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009fd:[<0x10000000675/picture-64.jpg v973>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009fd:[<0x10000000675/picture-64.jpg v1160>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.452+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000685 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-80.xml auth v940 ap=1 s=16103 n(v0 rc2024-05-09T08:27:51.150850+0000 b16103 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da1eb5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000685:[<0x10000000675/article-80.xml v940>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000685:[<0x10000000675/article-80.xml v940>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.452+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000776 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-10.jpg auth v912 ap=1 s=64501 n(v0 rc2024-05-09T08:27:51.898829+0000 b64501 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22f2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000776:[<0x10000000675/picture-10.jpg v912>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000776:[<0x10000000675/picture-10.jpg v912>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.611 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.452+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000077f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-1.jpg auth v944 ap=1 s=36812 n(v0 rc2024-05-09T08:27:51.924828+0000 b36812 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22f2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000077f:[<0x10000000675/picture-1.jpg v944>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000077f:[<0x10000000675/picture-1.jpg v944>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.452+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000714 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-9.xml auth v910 ap=1 s=27632 n(v0 rc2024-05-09T08:27:51.603837+0000 b27632 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22f3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000714:[<0x10000000675/article-9.xml v910>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000714:[<0x10000000675/article-9.xml v910>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.452+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006b5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-44.xml auth v930 ap=1 s=11629 n(v0 rc2024-05-09T08:27:51.302846+0000 b11629 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22f3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006b5:[<0x10000000675/article-44.xml v930>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006b5:[<0x10000000675/article-44.xml v930>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.453+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000077a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-6.jpg auth v906 ap=1 s=64874 n(v0 rc2024-05-09T08:27:51.910828+0000 b64874 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22f4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000077a:[<0x10000000675/picture-6.jpg v906>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000077a:[<0x10000000675/picture-6.jpg v906>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.453+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000069e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-65.xml auth v936 ap=1 s=6846 n(v0 rc2024-05-09T08:27:51.234848+0000 b6846 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22f4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000069e:[<0x10000000675/article-65.xml v936>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000069e:[<0x10000000675/article-65.xml v936>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.453+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006a1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-62.xml auth v934 ap=1 s=11770 n(v0 rc2024-05-09T08:27:51.243847+0000 b11770 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22f5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006a1:[<0x10000000675/article-62.xml v934>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006a1:[<0x10000000675/article-62.xml v934>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.453+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000006e7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-24.xml auth v918 ap=1 s=19307 n(v0 rc2024-05-09T08:27:51.461841+0000 b19307 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22fc000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000006e7:[<0x10000000675/article-24.xml v918>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000006e7:[<0x10000000675/article-24.xml v918>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a12 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-25.xml auth v1168 ap=1 s=5952 n(v0 rc2024-05-09T08:27:55.051739+0000 b5952 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da22fcb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a12:[<0x10000000675/comment-25.xml v986>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a12:[<0x10000000675/comment-25.xml v1168>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000067a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-85.xml auth v942 ap=1 s=5336 n(v0 rc2024-05-09T08:27:51.100851+0000 b5336 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22fd600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000067a:[<0x10000000675/article-85.xml v942>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000067a:[<0x10000000675/article-85.xml v942>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000764 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-28.jpg auth v920 ap=1 s=15861 n(v0 rc2024-05-09T08:27:51.846830+0000 b15861 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22fd080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000764:[<0x10000000675/picture-28.jpg v920>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000764:[<0x10000000675/picture-28.jpg v920>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000069c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-67.xml auth v938 ap=1 s=21236 n(v0 rc2024-05-09T08:27:51.229848+0000 b21236 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22fe100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000069c:[<0x10000000675/article-67.xml v938>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000069c:[<0x10000000675/article-67.xml v938>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000752 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-37.jpg auth v924 ap=1 s=61458 n(v0 rc2024-05-09T08:27:51.797832+0000 b61458 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22fe680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000752:[<0x10000000675/picture-37.jpg v924>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000752:[<0x10000000675/picture-37.jpg v924>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.612 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000076f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-17.jpg auth v914 ap=1 s=55426 n(v0 rc2024-05-09T08:27:51.878829+0000 b55426 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22fec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000076f:[<0x10000000675/picture-17.jpg v914>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000076f:[<0x10000000675/picture-17.jpg v914>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.613 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.454+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000723 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-3.xml auth v904 ap=1 s=20938 n(v0 rc2024-05-09T08:27:51.646836+0000 b20938 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22ff180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000723:[<0x10000000675/article-3.xml v904>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000723:[<0x10000000675/article-3.xml v904>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.613 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.455+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000768 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-24.jpg auth v916 ap=1 s=49038 n(v0 rc2024-05-09T08:27:51.858830+0000 b49038 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da22ff700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000768:[<0x10000000675/picture-24.jpg v916>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000768:[<0x10000000675/picture-24.jpg v916>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.613 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.455+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000075a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-33.jpg auth v922 ap=1 s=36828 n(v0 rc2024-05-09T08:27:51.819831+0000 b36828 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2306000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000075a:[<0x10000000675/picture-33.jpg v922>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000075a:[<0x10000000675/picture-33.jpg v922>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.613 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.455+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000729 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-57.jpg auth v932 ap=1 s=18341 n(v0 rc2024-05-09T08:27:51.663835+0000 b18341 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2306580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000729:[<0x10000000675/picture-57.jpg v932>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000729:[<0x10000000675/picture-57.jpg v932>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.613 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.455+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000071a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-7.xml auth v908 ap=1 s=18940 n(v0 rc2024-05-09T08:27:51.620837+0000 b18940 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2306b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000071a:[<0x10000000675/article-7.xml v908>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000071a:[<0x10000000675/article-7.xml v908>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.613 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.455+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000748 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-42.jpg auth v928 ap=1 s=44046 n(v0 rc2024-05-09T08:27:51.769832+0000 b44046 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2307080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000748:[<0x10000000675/picture-42.jpg v928>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000748:[<0x10000000675/picture-42.jpg v928>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.613 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.455+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000074e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-39.jpg auth v926 ap=1 s=59144 n(v0 rc2024-05-09T08:27:51.785832+0000 b59144 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2307600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000074e:[<0x10000000675/picture-39.jpg v926>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000074e:[<0x10000000675/picture-39.jpg v926>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.613 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.455+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a16 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-24.xml auth v1164 ap=1 s=3921 n(v0 rc2024-05-09T08:27:55.065738+0000 b3921 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da2308100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a16:[<0x10000000675/comment-24.xml v990>,<0x10000000399/blog-5 v13794>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a16:[<0x10000000675/comment-24.xml v1164>,<0x10000000399/blog-5 v31994>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:12.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] up:active 2024-05-09T08:28:12.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:12 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:28:13.104 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.712+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-27.jpg auth v1529 ap=1 s=37759 n(v0 rc2024-05-09T08:27:48.850916+0000 b37759 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a81080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f4:[<0x1000000039e/picture-27.jpg v1529>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003f4:[<0x1000000039e/picture-27.jpg v1529>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.104 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.713+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000529 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-75.xml auth v1591 ap=1 s=9336 n(v0 rc2024-05-09T08:27:49.805888+0000 b9336 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a80b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000529:[<0x1000000039e/article-75.xml v1591>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000529:[<0x1000000039e/article-75.xml v1591>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.104 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.713+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c9c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-73.xml auth v1890 ap=1 s=27824 n(v0 rc2024-05-09T08:27:57.565667+0000 b27824 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c598a81600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000c9c:[<0x1000000039e/article-73.xml v1634>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000c9c:[<0x1000000039e/article-73.xml v1890>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.104 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.713+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003cf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-1.xml auth{0=1} v1341 ap=2 s=3538 n(v0 rc2024-05-09T08:27:48.793917+0000 b3538 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x55c598a82100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003cf:[<0x1000000039e/article-1.xml v1341>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003cf:[<0x1000000039e/article-1.xml v1341>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.104 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.713+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ae [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-12.xml auth v1385 ap=1 s=6281 n(v0 rc2024-05-09T08:27:48.737919+0000 b6281 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a81b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ae:[<0x1000000039e/article-12.xml v1385>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003ae:[<0x1000000039e/article-12.xml v1385>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.104 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.713+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003df [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-34.jpg auth v1543 ap=1 s=52618 n(v0 rc2024-05-09T08:27:48.817917+0000 b52618 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a82c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003df:[<0x1000000039e/picture-34.jpg v1543>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003df:[<0x1000000039e/picture-34.jpg v1543>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.104 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.714+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000414 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-16.jpg auth v1399 ap=1 s=52683 n(v0 rc2024-05-09T08:27:48.903914+0000 b52683 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5986c8580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000414:[<0x1000000039e/picture-16.jpg v1399>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000414:[<0x1000000039e/picture-16.jpg v1399>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.104 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.714+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000da0 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-49.xml auth v1870 ap=1 s=2649 n(v0 rc2024-05-09T08:27:58.268647+0000 b2649 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c5986c9080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000da0:[<0x1000000039e/comment-49.xml v1708>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000da0:[<0x1000000039e/comment-49.xml v1870>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.104 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.715+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cfa [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-87.xml auth v1872 ap=1 s=6722 n(v0 rc2024-05-09T08:27:57.805660+0000 b6722 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c5986c9600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cfa:[<0x1000000039e/comment-87.xml v1648>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cfa:[<0x1000000039e/comment-87.xml v1872>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.715+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dc1 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-42.xml auth v1874 ap=1 s=3754 n(v0 rc2024-05-09T08:27:58.419642+0000 b3754 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c5986c9b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dc1:[<0x1000000039e/comment-42.xml v1717>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dc1:[<0x1000000039e/comment-42.xml v1874>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.715+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d53 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-65.xml auth v1876 ap=1 s=4933 n(v0 rc2024-05-09T08:27:58.019654+0000 b4933 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c5986ca100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d53:[<0x1000000039e/comment-65.xml v1669>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d53:[<0x1000000039e/comment-65.xml v1876>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.715+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cbf [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-100.xml auth v1878 ap=1 s=2339 n(v0 rc2024-05-09T08:27:57.678664+0000 b2339 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c5986ca680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cbf:[<0x1000000039e/comment-100.xml v1638>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cbf:[<0x1000000039e/comment-100.xml v1878>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.716+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000039f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-17.xml auth v1405 ap=1 s=23976 n(v0 rc2024-05-09T08:27:48.710920+0000 b23976 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5986cac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000039f:[<0x1000000039e/article-17.xml v1405>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000039f:[<0x1000000039e/article-17.xml v1405>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.716+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-32.jpg auth v1561 ap=1 s=20134 n(v0 rc2024-05-09T08:27:48.827916+0000 b20134 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c5986cb700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e5:[<0x1000000039e/picture-32.jpg v1561>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003e5:[<0x1000000039e/picture-32.jpg v1561>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.716+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d98 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-51.xml auth v1884 ap=1 s=6879 n(v0 rc2024-05-09T08:27:58.236648+0000 b6879 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c598a40000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d98:[<0x1000000039e/comment-51.xml v1703>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d98:[<0x1000000039e/comment-51.xml v1884>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.716+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d10 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-81.xml auth v1814 ap=1 s=2880 n(v0 rc2024-05-09T08:27:57.867658+0000 b2880 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c598560000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d10:[<0x1000000039e/comment-81.xml v1666>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d10:[<0x1000000039e/comment-81.xml v1814>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.717+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000439 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-3.jpg auth v1347 ap=1 s=13995 n(v0 rc2024-05-09T08:27:48.966912+0000 b13995 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598560b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000439:[<0x1000000039e/picture-3.jpg v1347>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000439:[<0x1000000039e/picture-3.jpg v1347>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.717+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d90 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-53.xml auth v1834 ap=1 s=5001 n(v0 rc2024-05-09T08:27:58.203649+0000 b5001 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c598561080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d90:[<0x1000000039e/comment-53.xml v1702>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d90:[<0x1000000039e/comment-53.xml v1834>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.717+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d03 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-85.xml auth v1826 ap=1 s=7271 n(v0 rc2024-05-09T08:27:57.828659+0000 b7271 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c598561b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d03:[<0x1000000039e/comment-85.xml v1662>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d03:[<0x1000000039e/comment-85.xml v1826>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.717+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000423 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-11.jpg auth v1379 ap=1 s=38956 n(v0 rc2024-05-09T08:27:48.927914+0000 b38956 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598562100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000423:[<0x1000000039e/picture-11.jpg v1379>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000423:[<0x1000000039e/picture-11.jpg v1379>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.718+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-5.xml auth v1357 ap=1 s=3671 n(v0 rc2024-05-09T08:27:48.773918+0000 b3671 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598562c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c4:[<0x1000000039e/article-5.xml v1357>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003c4:[<0x1000000039e/article-5.xml v1357>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.105 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.718+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000050f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-70.jpg auth v1505 ap=1 s=33777 n(v0 rc2024-05-09T08:27:49.725891+0000 b33777 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598562680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000050f:[<0x1000000039e/picture-70.jpg v1505>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000050f:[<0x1000000039e/picture-70.jpg v1505>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.718+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d7d [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-57.xml auth v1850 ap=1 s=8543 n(v0 rc2024-05-09T08:27:58.121651+0000 b8543 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59856a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d7d:[<0x1000000039e/comment-57.xml v1697>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d7d:[<0x1000000039e/comment-57.xml v1850>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.719+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d67 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-61.xml auth v1842 ap=1 s=6522 n(v0 rc2024-05-09T08:27:58.069652+0000 b6522 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59856a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d67:[<0x1000000039e/comment-61.xml v1694>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d67:[<0x1000000039e/comment-61.xml v1842>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.719+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dd7 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-38.xml auth v1846 ap=1 s=9779 n(v0 rc2024-05-09T08:27:58.515640+0000 b9779 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59856ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dd7:[<0x1000000039e/comment-38.xml v1717>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dd7:[<0x1000000039e/comment-38.xml v1846>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.719+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d46 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-68.xml auth v1830 ap=1 s=5546 n(v0 rc2024-05-09T08:27:57.989655+0000 b5546 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59856b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d46:[<0x1000000039e/comment-68.xml v1687>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d46:[<0x1000000039e/comment-68.xml v1830>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.720+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d14 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-80.xml auth v1854 ap=1 s=5230 n(v0 rc2024-05-09T08:27:57.876658+0000 b5230 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59856bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d14:[<0x1000000039e/comment-80.xml v1669>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d14:[<0x1000000039e/comment-80.xml v1854>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.720+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000db9 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-44.xml auth v1886 ap=1 s=9376 n(v0 rc2024-05-09T08:27:58.368644+0000 b9376 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59841eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000db9:[<0x1000000039e/comment-44.xml v1712>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000db9:[<0x1000000039e/comment-44.xml v1886>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.720+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000435 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-5.jpg auth v1355 ap=1 s=33819 n(v0 rc2024-05-09T08:27:48.955913+0000 b33819 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c597f7b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000435:[<0x1000000039e/picture-5.jpg v1355>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000435:[<0x1000000039e/picture-5.jpg v1355>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.721+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cdf [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-93.xml auth v1884 ap=1 s=2522 n(v0 rc2024-05-09T08:27:57.749662+0000 b2522 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59841e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cdf:[<0x1000000039e/comment-93.xml v1645>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cdf:[<0x1000000039e/comment-93.xml v1884>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.721+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d21 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-76.xml auth v1882 ap=1 s=2198 n(v0 rc2024-05-09T08:27:57.911657+0000 b2198 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c5983d6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d21:[<0x1000000039e/comment-76.xml v1658>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d21:[<0x1000000039e/comment-76.xml v1882>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.721+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d95 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-52.xml auth v1892 ap=1 s=9162 n(v0 rc2024-05-09T08:27:58.218648+0000 b9162 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59883c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d95:[<0x1000000039e/comment-52.xml v1706>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d95:[<0x1000000039e/comment-52.xml v1892>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.721+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d4e [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-66.xml auth v1894 ap=1 s=4487 n(v0 rc2024-05-09T08:27:58.010654+0000 b4487 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59883c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d4e:[<0x1000000039e/comment-66.xml v1663>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d4e:[<0x1000000039e/comment-66.xml v1894>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.106 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.722+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dbd [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-43.xml auth v1900 ap=1 s=3307 n(v0 rc2024-05-09T08:27:58.395643+0000 b3307 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59883d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dbd:[<0x1000000039e/comment-43.xml v1715>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dbd:[<0x1000000039e/comment-43.xml v1900>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.722+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cc4 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-99.xml auth v1906 ap=1 s=2678 n(v0 rc2024-05-09T08:27:57.687663+0000 b2678 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59883e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cc4:[<0x1000000039e/comment-99.xml v1635>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cc4:[<0x1000000039e/comment-99.xml v1906>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.722+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000db3 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-45.xml auth v1908 ap=1 s=8509 n(v0 rc2024-05-09T08:27:58.344645+0000 b8509 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59883ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000db3:[<0x1000000039e/comment-45.xml v1709>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000db3:[<0x1000000039e/comment-45.xml v1908>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.723+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d08 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-84.xml auth v1910 ap=1 s=4514 n(v0 rc2024-05-09T08:27:57.847659+0000 b4514 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59883f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d08:[<0x1000000039e/comment-84.xml v1653>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d08:[<0x1000000039e/comment-84.xml v1910>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.723+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ddd [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-37.xml auth v1804 ap=1 s=2305 n(v0 rc2024-05-09T08:27:58.535639+0000 b2305 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59856c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ddd:[<0x1000000039e/comment-37.xml v1720>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ddd:[<0x1000000039e/comment-37.xml v1804>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.723+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d38 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-71.xml auth v1820 ap=1 s=4805 n(v0 rc2024-05-09T08:27:57.960656+0000 b4805 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59856c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d38:[<0x1000000039e/comment-71.xml v1684>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d38:[<0x1000000039e/comment-71.xml v1820>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.723+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000426 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-10.jpg auth v1375 ap=1 s=9916 n(v0 rc2024-05-09T08:27:48.931913+0000 b9916 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59856cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000426:[<0x1000000039e/picture-10.jpg v1375>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000426:[<0x1000000039e/picture-10.jpg v1375>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.723+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cd7 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-95.xml auth v1824 ap=1 s=8805 n(v0 rc2024-05-09T08:27:57.729662+0000 b8805 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c59856d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cd7:[<0x1000000039e/comment-95.xml v1634>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cd7:[<0x1000000039e/comment-95.xml v1824>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.724+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000043d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-1.jpg auth v1339 ap=1 s=48219 n(v0 rc2024-05-09T08:27:48.980912+0000 b48219 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15549 | request=0 lock=0 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c59856d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000043d:[<0x1000000039e/picture-1.jpg v1339>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000043d:[<0x1000000039e/picture-1.jpg v1339>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.724+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-9.xml auth v1373 ap=1 s=8758 n(v0 rc2024-05-09T08:27:48.752919+0000 b8758 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598576580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b7:[<0x1000000039e/article-9.xml v1373>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003b7:[<0x1000000039e/article-9.xml v1373>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.724+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000432 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-6.jpg auth v1359 ap=1 s=35724 n(v0 rc2024-05-09T08:27:48.950913+0000 b35724 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598576b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000432:[<0x1000000039e/picture-6.jpg v1359>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000432:[<0x1000000039e/picture-6.jpg v1359>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.724+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d77 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-58.xml auth v1840 ap=1 s=6239 n(v0 rc2024-05-09T08:27:58.106651+0000 b6239 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c598577080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d77:[<0x1000000039e/comment-58.xml v1691>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d77:[<0x1000000039e/comment-58.xml v1840>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.725+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d88 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-55.xml auth v1816 ap=1 s=3446 n(v0 rc2024-05-09T08:27:58.165650+0000 b3446 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c5984e3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d88:[<0x1000000039e/comment-55.xml v1698>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d88:[<0x1000000039e/comment-55.xml v1816>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.107 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.725+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-28.jpg auth v1495 ap=1 s=17122 n(v0 rc2024-05-09T08:27:48.846916+0000 b17122 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598578100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f0:[<0x1000000039e/picture-28.jpg v1495>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003f0:[<0x1000000039e/picture-28.jpg v1495>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.108 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.725+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dc7 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-41.xml auth v1836 ap=1 s=3806 n(v0 rc2024-05-09T08:27:58.447642+0000 b3806 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c598578680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dc7:[<0x1000000039e/comment-41.xml v1713>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dc7:[<0x1000000039e/comment-41.xml v1836>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.108 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.725+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d26 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-75.xml auth v1812 ap=1 s=7231 n(v0 rc2024-05-09T08:27:57.920657+0000 b7231 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c598578c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d26:[<0x1000000039e/comment-75.xml v1681>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d26:[<0x1000000039e/comment-75.xml v1812>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.108 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.725+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d5 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-37.jpg auth v1537 ap=1 s=40982 n(v0 rc2024-05-09T08:27:48.802917+0000 b40982 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598579180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d5:[<0x1000000039e/picture-37.jpg v1537>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003d5:[<0x1000000039e/picture-37.jpg v1537>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.109 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.725+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000412 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-17.jpg auth v1403 ap=1 s=60648 n(v0 rc2024-05-09T08:27:48.898914+0000 b60648 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598579700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000412:[<0x1000000039e/picture-17.jpg v1403>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000412:[<0x1000000039e/picture-17.jpg v1403>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.109 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.726+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-3.xml auth v1349 ap=1 s=7061 n(v0 rc2024-05-09T08:27:48.783918+0000 b7061 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598586000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c9:[<0x1000000039e/article-3.xml v1349>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003c9:[<0x1000000039e/article-3.xml v1349>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.109 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.726+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003fc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-24.jpg auth v1501 ap=1 s=59713 n(v0 rc2024-05-09T08:27:48.865915+0000 b59713 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598586580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003fc:[<0x1000000039e/picture-24.jpg v1501>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003fc:[<0x1000000039e/picture-24.jpg v1501>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.109 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.726+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-33.jpg auth v1541 ap=1 s=14305 n(v0 rc2024-05-09T08:27:48.822917+0000 b14305 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598586b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e2:[<0x1000000039e/picture-33.jpg v1541>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003e2:[<0x1000000039e/picture-33.jpg v1541>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.109 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.726+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000051f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-57.jpg auth v1551 ap=1 s=33791 n(v0 rc2024-05-09T08:27:49.773889+0000 b33791 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598587080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000051f:[<0x1000000039e/picture-57.jpg v1551>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000051f:[<0x1000000039e/picture-57.jpg v1551>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.109 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.727+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003bd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-7.xml auth v1365 ap=1 s=19893 n(v0 rc2024-05-09T08:27:48.763918+0000 b19893 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598587600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003bd:[<0x1000000039e/article-7.xml v1365>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003bd:[<0x1000000039e/article-7.xml v1365>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.109 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.728+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000041e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-13.jpg auth v1387 ap=1 s=20453 n(v0 rc2024-05-09T08:27:48.917914+0000 b20453 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a40580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000041e:[<0x1000000039e/picture-13.jpg v1387>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000041e:[<0x1000000039e/picture-13.jpg v1387>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.728+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d83 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-56.xml auth v1695 pv1845 ap=4 s=0 n(v0 rc2024-05-09T08:27:58.143650+0000 1=1+0)->n(v0 rc2024-05-09T08:27:58.143650+0000 b7838 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a40b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d83:[<0x1000000039e/comment-56.xml v1695>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d83:[<0x1000000039e/comment-56.xml v1695>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.729+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000040e [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-18.jpg auth v1407 ap=1 s=51862 n(v0 rc2024-05-09T08:27:48.894914+0000 b51862 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a41080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000040e:[<0x1000000039e/picture-18.jpg v1407>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000040e:[<0x1000000039e/picture-18.jpg v1407>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.730+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000040b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-19.jpg auth v1601 ap=1 s=42261 n(v0 rc2024-05-09T08:27:48.889915+0000 b42261 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a42100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000040b:[<0x1000000039e/picture-19.jpg v1601>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000040b:[<0x1000000039e/picture-19.jpg v1601>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.732+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000437 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-4.jpg auth v1351 ap=1 s=39925 n(v0 rc2024-05-09T08:27:48.959913+0000 b39925 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a42c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000437:[<0x1000000039e/picture-4.jpg v1351>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000437:[<0x1000000039e/picture-4.jpg v1351>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.732+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003db [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-35.jpg auth v1527 ap=1 s=42165 n(v0 rc2024-05-09T08:27:48.812917+0000 b42165 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a43180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003db:[<0x1000000039e/picture-35.jpg v1527>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003db:[<0x1000000039e/picture-35.jpg v1527>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.732+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000043b [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-2.jpg auth v1343 ap=1 s=46567 n(v0 rc2024-05-09T08:27:48.973912+0000 b46567 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a43700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000043b:[<0x1000000039e/picture-2.jpg v1343>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000043b:[<0x1000000039e/picture-2.jpg v1343>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.733+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000041a [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-14.jpg auth v1391 ap=1 s=21062 n(v0 rc2024-05-09T08:27:48.912914+0000 b21062 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a2a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000041a:[<0x1000000039e/picture-14.jpg v1391>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000041a:[<0x1000000039e/picture-14.jpg v1391>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.733+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ce4 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-92.xml auth v1653 pv1837 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.759661+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:57.759661+0000 b5239 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x55c598a2ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ce4:[<0x1000000039e/comment-92.xml v1653>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ce4:[<0x1000000039e/comment-92.xml v1653>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.734+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cdb [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-94.xml auth v1649 pv1839 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.738662+0000 1=1+0)->n(v0 rc2024-05-09T08:27:57.738662+0000 b9436 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a2b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cdb:[<0x1000000039e/comment-94.xml v1649>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cdb:[<0x1000000039e/comment-94.xml v1649>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.735+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000042f [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-7.jpg auth v1363 ap=1 s=10474 n(v0 rc2024-05-09T08:27:48.945913+0000 b10474 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a2b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000042f:[<0x1000000039e/picture-7.jpg v1363>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000042f:[<0x1000000039e/picture-7.jpg v1363>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.110 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:28:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:28:12.735+0000 7fc14dc74700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000429 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-9.jpg auth v1371 ap=1 s=38638 n(v0 rc2024-05-09T08:27:48.936913+0000 b38638 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55c598a2bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000429:[<0x1000000039e/picture-9.jpg v1371>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000429:[<0x1000000039e/picture-9.jpg v1371>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.112 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.687+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000405 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-21.jpg auth v1491 ap=1 s=40098 n(v0 rc2024-05-09T08:27:48.879915+0000 b40098 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27ad180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000405:[<0x1000000039e/picture-21.jpg v1491>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000405:[<0x1000000039e/picture-21.jpg v1491>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.112 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.687+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d30 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-73.xml auth v1679 pv1900 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.940656+0000 1=1+0)->n(v0 rc2024-05-09T08:27:57.940656+0000 b4153 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27ad700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d30:[<0x1000000039e/comment-73.xml v1679>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d30:[<0x1000000039e/comment-73.xml v1679>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.112 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.687+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d6d [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-60.xml auth v1695 pv1872 ap=4 s=0 n(v0 rc2024-05-09T08:27:58.081652+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:58.081652+0000 b6765 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x555da2776000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d6d:[<0x1000000039e/comment-60.xml v1695>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d6d:[<0x1000000039e/comment-60.xml v1695>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.112 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.688+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d19 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-78.xml auth v1676 pv1874 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.891657+0000 1=1+0)->n(v0 rc2024-05-09T08:27:57.891657+0000 b6476 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2776580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d19:[<0x1000000039e/comment-78.xml v1676>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d19:[<0x1000000039e/comment-78.xml v1676>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.112 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.688+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cd1 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-96.xml auth v1646 pv1876 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.717662+0000 1=1+0)->n(v0 rc2024-05-09T08:27:57.717662+0000 b8977 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2777600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cd1:[<0x1000000039e/comment-96.xml v1646>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cd1:[<0x1000000039e/comment-96.xml v1646>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.112 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.688+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ea [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-30.jpg auth v1513 ap=1 s=36080 n(v0 rc2024-05-09T08:27:48.836916+0000 b36080 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2777b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ea:[<0x1000000039e/picture-30.jpg v1513>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003ea:[<0x1000000039e/picture-30.jpg v1513>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.112 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.688+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cf7 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-88.xml auth v1654 pv1878 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.795660+0000 1=1+0)->n(v0 rc2024-05-09T08:27:57.795660+0000 b7096 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2778100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cf7:[<0x1000000039e/comment-88.xml v1654>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cf7:[<0x1000000039e/comment-88.xml v1654>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.112 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.688+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-15.xml auth v1397 ap=1 s=32055 n(v0 rc2024-05-09T08:27:48.721919+0000 b32055 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2778680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003a6:[<0x1000000039e/article-15.xml v1397>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003a6:[<0x1000000039e/article-15.xml v1397>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.689+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dcd [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-40.xml auth v1719 pv1882 ap=4 s=0 n(v0 rc2024-05-09T08:27:58.470641+0000 1=1+0)->n(v0 rc2024-05-09T08:27:58.470641+0000 b7599 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2779180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dcd:[<0x1000000039e/comment-40.xml v1719>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dcd:[<0x1000000039e/comment-40.xml v1719>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.689+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d16 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-79.xml auth v1673 pv1880 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.881658+0000 1=1+0)->n(v0 rc2024-05-09T08:27:57.881658+0000 b9271 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2778c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d16:[<0x1000000039e/comment-79.xml v1673>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d16:[<0x1000000039e/comment-79.xml v1673>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.689+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dd2 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-39.xml auth v1723 pv1886 ap=4 s=0 n(v0 rc2024-05-09T08:27:58.492640+0000 1=1+0)->n(v0 rc2024-05-09T08:27:58.492640+0000 b3563 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da277e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dd2:[<0x1000000039e/comment-39.xml v1723>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dd2:[<0x1000000039e/comment-39.xml v1723>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.689+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-36.jpg auth v1507 ap=1 s=40703 n(v0 rc2024-05-09T08:27:48.807917+0000 b40703 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da277eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d8:[<0x1000000039e/picture-36.jpg v1507>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003d8:[<0x1000000039e/picture-36.jpg v1507>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.691+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cfe [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-86.xml auth v1657 pv1888 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.815660+0000 1=1+0)->n(v0 rc2024-05-09T08:27:57.815660+0000 b8217 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da277e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cfe:[<0x1000000039e/comment-86.xml v1657>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cfe:[<0x1000000039e/comment-86.xml v1657>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.691+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000408 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-20.jpg auth v1493 ap=1 s=44688 n(v0 rc2024-05-09T08:27:48.884915+0000 b44688 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da277f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000408:[<0x1000000039e/picture-20.jpg v1493>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000408:[<0x1000000039e/picture-20.jpg v1493>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.691+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d5c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-63.xml auth v1690 pv1894 ap=4 s=0 n(v0 rc2024-05-09T08:27:58.043653+0000 1=1+0)->n(v0 rc2024-05-09T08:27:58.043653+0000 b6670 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2780100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d5c:[<0x1000000039e/comment-63.xml v1690>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d5c:[<0x1000000039e/comment-63.xml v1690>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.691+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d0e [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-82.xml auth v1666 pv1890 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.861658+0000 1=1+0)->n(v0 rc2024-05-09T08:27:57.861658+0000 b9049 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da277f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d0e:[<0x1000000039e/comment-82.xml v1666>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d0e:[<0x1000000039e/comment-82.xml v1666>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.692+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d8c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-54.xml auth v1700 pv1896 ap=4 s=0 n(v0 rc2024-05-09T08:27:58.185649+0000 1=1+0)->n(v0 rc2024-05-09T08:27:58.185649+0000 b9382 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2780680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d8c:[<0x1000000039e/comment-54.xml v1700>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d8c:[<0x1000000039e/comment-54.xml v1700>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.692+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ce9 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-91.xml auth v1650 pv1898 ap=4 s=0 n(v0 rc2024-05-09T08:27:57.767661+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:57.767661+0000 b6678 1=1+0) (ifile excl w=2) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=3 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x555da2780c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ce9:[<0x1000000039e/comment-91.xml v1650>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ce9:[<0x1000000039e/comment-91.xml v1650>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.692+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003d3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-38.jpg auth v1547 ap=1 s=55776 n(v0 rc2024-05-09T08:27:48.797917+0000 b55776 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2781180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003d3:[<0x1000000039e/picture-38.jpg v1547>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003d3:[<0x1000000039e/picture-38.jpg v1547>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.692+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000400 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-23.jpg auth v1563 ap=1 s=36888 n(v0 rc2024-05-09T08:27:48.870915+0000 b36888 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000400:[<0x1000000039e/picture-23.jpg v1563>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000400:[<0x1000000039e/picture-23.jpg v1563>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.113 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.692+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-14.xml auth v1393 ap=1 s=31179 n(v0 rc2024-05-09T08:27:48.726919+0000 b31179 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003a9:[<0x1000000039e/article-14.xml v1393>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003a9:[<0x1000000039e/article-14.xml v1393>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.692+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d61 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-62.xml auth v1784 ap=1 s=9963 n(v0 rc2024-05-09T08:27:58.056653+0000 b9963 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da220c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d61:[<0x1000000039e/comment-62.xml v1684>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d61:[<0x1000000039e/comment-62.xml v1784>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.693+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000518 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-91.xml auth v1517 ap=1 s=26033 n(v0 rc2024-05-09T08:27:49.746890+0000 b26033 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000518:[<0x1000000039e/article-91.xml v1517>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000518:[<0x1000000039e/article-91.xml v1517>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.693+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000403 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-22.jpg auth v1545 ap=1 s=25419 n(v0 rc2024-05-09T08:27:48.874915+0000 b25419 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000403:[<0x1000000039e/picture-22.jpg v1545>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000403:[<0x1000000039e/picture-22.jpg v1545>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.693+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b1 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-11.xml auth v1381 ap=1 s=22618 n(v0 rc2024-05-09T08:27:48.742919+0000 b22618 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da220d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b1:[<0x1000000039e/article-11.xml v1381>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003b1:[<0x1000000039e/article-11.xml v1381>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.693+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003a2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-16.xml auth v1401 ap=1 s=23218 n(v0 rc2024-05-09T08:27:48.716920+0000 b23218 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2220580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003a2:[<0x1000000039e/article-16.xml v1401>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003a2:[<0x1000000039e/article-16.xml v1401>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.693+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d72 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-59.xml auth v1776 ap=1 s=5624 n(v0 rc2024-05-09T08:27:58.094652+0000 b5624 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da2220000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d72:[<0x1000000039e/comment-59.xml v1687>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d72:[<0x1000000039e/comment-59.xml v1776>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.693+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d1e [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-77.xml auth v1772 ap=1 s=6384 n(v0 rc2024-05-09T08:27:57.900657+0000 b6384 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da2220b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d1e:[<0x1000000039e/comment-77.xml v1677>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d1e:[<0x1000000039e/comment-77.xml v1772>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.694+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000417 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-15.jpg auth v1395 ap=1 s=19886 n(v0 rc2024-05-09T08:27:48.908914+0000 b19886 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2221080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000417:[<0x1000000039e/picture-15.jpg v1395>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000417:[<0x1000000039e/picture-15.jpg v1395>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.694+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003b4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-10.xml auth v1377 ap=1 s=25723 n(v0 rc2024-05-09T08:27:48.747919+0000 b25723 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2221600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003b4:[<0x1000000039e/article-10.xml v1377>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003b4:[<0x1000000039e/article-10.xml v1377>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.694+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000420 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-12.jpg auth v1383 ap=1 s=30605 n(v0 rc2024-05-09T08:27:48.922914+0000 b30605 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2221b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000420:[<0x1000000039e/picture-12.jpg v1383>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000420:[<0x1000000039e/picture-12.jpg v1383>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.694+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d9c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-50.xml auth v1780 ap=1 s=3942 n(v0 rc2024-05-09T08:27:58.253647+0000 b3942 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da2222100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d9c:[<0x1000000039e/comment-50.xml v1704>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d9c:[<0x1000000039e/comment-50.xml v1780>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.114 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.694+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000da5 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-48.xml auth v1764 ap=1 s=5495 n(v0 rc2024-05-09T08:27:58.283646+0000 b5495 1=1+0) caps={15549=pAsLsXsFsc/-@1},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da2222680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000da5:[<0x1000000039e/comment-48.xml v1707>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000da5:[<0x1000000039e/comment-48.xml v1764>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.695+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-4.xml auth v1353 ap=1 s=6852 n(v0 rc2024-05-09T08:27:48.778918+0000 b6852 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da2222c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c6:[<0x1000000039e/article-4.xml v1353>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003c6:[<0x1000000039e/article-4.xml v1353>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.695+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-25.jpg auth v1593 ap=1 s=23478 n(v0 rc2024-05-09T08:27:48.860915+0000 b23478 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27c3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f8:[<0x1000000039e/picture-25.jpg v1593>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003f8:[<0x1000000039e/picture-25.jpg v1593>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.695+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cce [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-97.xml auth v1893 ap=1 s=3841 n(v0 rc2024-05-09T08:27:57.706663+0000 b3841 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27c3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cce:[<0x1000000039e/comment-97.xml v1642>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cce:[<0x1000000039e/comment-97.xml v1893>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.695+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ba [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-8.xml auth v1369 ap=1 s=31323 n(v0 rc2024-05-09T08:27:48.758918+0000 b31323 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27c4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ba:[<0x1000000039e/article-8.xml v1369>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003ba:[<0x1000000039e/article-8.xml v1369>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.695+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000da8 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-47.xml auth v1895 ap=1 s=10094 n(v0 rc2024-05-09T08:27:58.304646+0000 b10094 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27c4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000da8:[<0x1000000039e/comment-47.xml v1710>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000da8:[<0x1000000039e/comment-47.xml v1895>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.696+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000dad [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-46.xml auth v1897 ap=1 s=3963 n(v0 rc2024-05-09T08:27:58.323645+0000 b3963 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27c4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000dad:[<0x1000000039e/comment-46.xml v1713>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000dad:[<0x1000000039e/comment-46.xml v1897>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.696+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003e8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-31.jpg auth v1549 ap=1 s=17979 n(v0 rc2024-05-09T08:27:48.831916+0000 b17979 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27c5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003e8:[<0x1000000039e/picture-31.jpg v1549>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003e8:[<0x1000000039e/picture-31.jpg v1549>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.696+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d3c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-70.xml auth v1899 ap=1 s=7577 n(v0 rc2024-05-09T08:27:57.970655+0000 b7577 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27c5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d3c:[<0x1000000039e/comment-70.xml v1686>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d3c:[<0x1000000039e/comment-70.xml v1899>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.696+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000042c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-8.jpg auth v1367 ap=1 s=45113 n(v0 rc2024-05-09T08:27:48.941913+0000 b45113 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27d0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000042c:[<0x1000000039e/picture-8.jpg v1367>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x1000000042c:[<0x1000000039e/picture-8.jpg v1367>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.696+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003f6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-26.jpg auth v1515 ap=1 s=31815 n(v0 rc2024-05-09T08:27:48.855916+0000 b31815 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27d0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003f6:[<0x1000000039e/picture-26.jpg v1515>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003f6:[<0x1000000039e/picture-26.jpg v1515>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.696+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cc7 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-98.xml auth v1901 ap=1 s=8100 n(v0 rc2024-05-09T08:27:57.698663+0000 b8100 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27d0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cc7:[<0x1000000039e/comment-98.xml v1634>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cc7:[<0x1000000039e/comment-98.xml v1901>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.115 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.697+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d0d [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-83.xml auth v1903 ap=1 s=8244 n(v0 rc2024-05-09T08:27:57.856658+0000 b8244 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27d1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d0d:[<0x1000000039e/comment-83.xml v1662>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d0d:[<0x1000000039e/comment-83.xml v1903>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.116 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.629+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b5d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-13.jpg auth v478 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.907714+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ea439080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b5d:[<0x100000009b9/picture-13.jpg v478>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b5d:[<0x100000009b9/picture-13.jpg v478>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.116 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.630+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b38 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-18.jpg auth v448 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.861715+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacd7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b38:[<0x100000009b9/picture-18.jpg v448>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b38:[<0x100000009b9/picture-18.jpg v448>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.116 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.631+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b31 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-19.jpg auth v442 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.789718+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacfab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b31:[<0x100000009b9/picture-19.jpg v442>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b31:[<0x100000009b9/picture-19.jpg v442>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.116 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.631+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a60 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-18.xml auth v232 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.246733+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacd7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a60:[<0x100000009b9/article-18.xml v232>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a60:[<0x100000009b9/article-18.xml v232>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.697+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000cf2 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-89.xml auth v1907 ap=1 s=3250 n(v0 rc2024-05-09T08:27:57.788660+0000 b3250 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27d1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000cf2:[<0x1000000039e/comment-89.xml v1660>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000cf2:[<0x1000000039e/comment-89.xml v1907>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.697+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d40 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-69.xml auth v1905 ap=1 s=8272 n(v0 rc2024-05-09T08:27:57.981655+0000 b8272 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27d1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d40:[<0x1000000039e/comment-69.xml v1689>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d40:[<0x1000000039e/comment-69.xml v1905>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.697+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d2a [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-74.xml auth v1909 ap=1 s=2400 n(v0 rc2024-05-09T08:27:57.931656+0000 b2400 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27d2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d2a:[<0x1000000039e/comment-74.xml v1669>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d2a:[<0x1000000039e/comment-74.xml v1909>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.697+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ced [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-90.xml auth v1911 ap=1 s=4108 n(v0 rc2024-05-09T08:27:57.776661+0000 b4108 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27d2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ced:[<0x1000000039e/comment-90.xml v1658>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ced:[<0x1000000039e/comment-90.xml v1911>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.698+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d57 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-64.xml auth v1915 ap=1 s=7316 n(v0 rc2024-05-09T08:27:58.030654+0000 b7316 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27d3700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d57:[<0x1000000039e/comment-64.xml v1695>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d57:[<0x1000000039e/comment-64.xml v1915>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.698+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ab [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-13.xml auth v1389 ap=1 s=8414 n(v0 rc2024-05-09T08:27:48.731919+0000 b8414 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27d3180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ab:[<0x1000000039e/article-13.xml v1389>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003ab:[<0x1000000039e/article-13.xml v1389>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.698+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d34 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-72.xml auth v1883 ap=1 s=9154 n(v0 rc2024-05-09T08:27:57.950656+0000 b9154 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27aa000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d34:[<0x1000000039e/comment-72.xml v1683>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d34:[<0x1000000039e/comment-72.xml v1883>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.698+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003c0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-6.xml auth v1361 ap=1 s=26706 n(v0 rc2024-05-09T08:27:48.768918+0000 b26706 1=1+0) (ifile excl) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pFscr@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27ab080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003c0:[<0x1000000039e/article-6.xml v1361>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003c0:[<0x1000000039e/article-6.xml v1361>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.699+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003cc [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-2.xml auth v1345 ap=1 s=10258 n(v0 rc2024-05-09T08:27:48.788917+0000 b10258 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15549 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27ab600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003cc:[<0x1000000039e/article-2.xml v1345>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003cc:[<0x1000000039e/article-2.xml v1345>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.699+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000d49 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-67.xml auth v1889 ap=1 s=2670 n(v0 rc2024-05-09T08:27:58.000654+0000 b2670 1=1+0) caps={15549=pAsLsXsFsc/-@2},l=15549(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555da27ac100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000d49:[<0x1000000039e/comment-67.xml v1692>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000d49:[<0x1000000039e/comment-67.xml v1889>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.117 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:28:12.699+0000 7f9ab5c98700 -1 mds.4.scrubstack _validate_inode_done scrub error on inode [inode 0x100000003ec [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-29.jpg auth v1497 ap=1 s=35074 n(v0 rc2024-05-09T08:27:48.841916+0000 b35074 1=1+0) caps={15549=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x555da27ac680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000003ec:[<0x1000000039e/picture-29.jpg v1497>,<0x10000000399/blog-2 v20297>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000003ec:[<0x1000000039e/picture-29.jpg v1497>,<0x10000000399/blog-2 v32744>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.121 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.631+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009d8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-51.xml auth v34 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.842745+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacfb080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009d8:[<0x100000009b9/article-51.xml v34>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009d8:[<0x100000009b9/article-51.xml v34>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.632+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a3c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-27.xml auth v178 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.163735+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacfb600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a3c:[<0x100000009b9/article-27.xml v178>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a3c:[<0x100000009b9/article-27.xml v178>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.633+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b80 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-4.jpg auth v532 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.018711+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacfbb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b80:[<0x100000009b9/picture-4.jpg v532>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b80:[<0x100000009b9/picture-4.jpg v532>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.634+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ab2 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-35.jpg auth v346 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.459727+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacfc100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ab2:[<0x100000009b9/picture-35.jpg v346>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ab2:[<0x100000009b9/picture-35.jpg v346>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.634+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b85 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-2.jpg auth v544 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.033711+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacfc680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b85:[<0x100000009b9/picture-2.jpg v544>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b85:[<0x100000009b9/picture-2.jpg v544>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.635+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a58 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-20.xml auth v220 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.226734+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacfcc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a58:[<0x100000009b9/article-20.xml v220>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a58:[<0x100000009b9/article-20.xml v220>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.635+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b59 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-14.jpg auth v472 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.897714+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacfd180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b59:[<0x100000009b9/picture-14.jpg v472>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b59:[<0x100000009b9/picture-14.jpg v472>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.635+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a02 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-41.xml auth v94 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.995740+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacfd700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a02:[<0x100000009b9/article-41.xml v94>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a02:[<0x100000009b9/article-41.xml v94>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.636+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009f1 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-45.xml auth v70 pv731 ap=5 s=0 n(v0 rc2024-05-09T08:27:54.935742+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:54.935742+0000 b26869 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea69b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009f1:[<0x100000009b9/article-45.xml v70>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009f1:[<0x100000009b9/article-45.xml v70>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.637+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b75 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-7.jpg auth v514 pv715 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.979712+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.979712+0000 b13947 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea362100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b75:[<0x100000009b9/picture-7.jpg v514>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b75:[<0x100000009b9/picture-7.jpg v514>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.122 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.637+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a2c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-31.xml auth v154 pv719 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.127736+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.127736+0000 b12326 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea362c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a2c:[<0x100000009b9/article-31.xml v154>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a2c:[<0x100000009b9/article-31.xml v154>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.638+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b6d [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-9.jpg auth v502 pv711 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.950713+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.950713+0000 b30823 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea2ef700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b6d:[<0x100000009b9/picture-9.jpg v502>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b6d:[<0x100000009b9/picture-9.jpg v502>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.639+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b69 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-10.jpg auth v496 pv739 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.940713+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.940713+0000 b46053 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea73c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b69:[<0x100000009b9/picture-10.jpg v496>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b69:[<0x100000009b9/picture-10.jpg v496>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.640+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b87 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-1.jpg auth{0=1} v550 pv735 ap=6 s=0 n(v0 rc2024-05-09T08:27:56.037710+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:56.037710+0000 b9844 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@3},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=1 authpin=1 scrubqueue=0 0x5619ea714c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b87:[<0x100000009b9/picture-1.jpg v550>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b87:[<0x100000009b9/picture-1.jpg v550>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.640+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a84 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-9.xml auth v286 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.334731+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead18000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a84:[<0x100000009b9/article-9.xml v286>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a84:[<0x100000009b9/article-9.xml v286>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.641+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009f6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-44.xml auth v76 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.958741+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead18580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009f6:[<0x100000009b9/article-44.xml v76>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009f6:[<0x100000009b9/article-44.xml v76>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.641+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b79 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-6.jpg auth v520 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.993712+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead18b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b79:[<0x100000009b9/picture-6.jpg v520>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b79:[<0x100000009b9/picture-6.jpg v520>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.642+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bc9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-99.xml auth v562 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.875687+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead19080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bc9:[<0x100000009b9/article-99.xml v562>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bc9:[<0x100000009b9/article-99.xml v562>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.642+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a48 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-24.xml auth v196 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.190735+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead19600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a48:[<0x100000009b9/article-24.xml v196>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a48:[<0x100000009b9/article-24.xml v196>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.643+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ae6 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-28.jpg auth v388 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.596723+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead19b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ae6:[<0x100000009b9/picture-28.jpg v388>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ae6:[<0x100000009b9/picture-28.jpg v388>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.643+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ca9 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-85.xml auth v583 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.620665+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead1a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ca9:[<0x100000009b9/article-85.xml v583>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ca9:[<0x100000009b9/article-85.xml v583>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.643+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b4d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-17.jpg auth v454 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.870715+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead1a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b4d:[<0x100000009b9/picture-17.jpg v454>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b4d:[<0x100000009b9/picture-17.jpg v454>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.123 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.644+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a9c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-3.xml auth v322 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.389729+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead1ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a9c:[<0x100000009b9/article-3.xml v322>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a9c:[<0x100000009b9/article-3.xml v322>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.644+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b05 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-24.jpg auth v412 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.685721+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead1b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b05:[<0x100000009b9/picture-24.jpg v412>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b05:[<0x100000009b9/picture-24.jpg v412>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.645+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ac0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-33.jpg auth v358 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.496726+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619ead1b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ac0:[<0x100000009b9/picture-33.jpg v358>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ac0:[<0x100000009b9/picture-33.jpg v358>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.645+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a8c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-7.xml auth v298 pv743 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.352730+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.352730+0000 b4381 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9b8000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a8c:[<0x100000009b9/article-7.xml v298>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a8c:[<0x100000009b9/article-7.xml v298>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.646+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009c1 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-55.xml auth v10 pv747 ap=5 s=0 n(v0 rc2024-05-09T08:27:54.796746+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:54.796746+0000 b10688 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9b8580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009c1:[<0x100000009b9/article-55.xml v10>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009c1:[<0x100000009b9/article-55.xml v10>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.646+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000afe [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-25.jpg auth v406 pv751 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.660721+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.660721+0000 b31490 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9b8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000afe:[<0x100000009b9/picture-25.jpg v406>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000afe:[<0x100000009b9/picture-25.jpg v406>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.647+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ca0 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-8.xml auth v570 pv755 ap=5 s=0 n(v0 rc2024-05-09T08:27:57.583666+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:57.583666+0000 b8549 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9b9080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ca0:[<0x100000009b9/article-8.xml v570>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ca0:[<0x100000009b9/article-8.xml v570>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.647+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ad1 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-31.jpg auth v370 pv759 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.540725+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.540725+0000 b42311 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9b9600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ad1:[<0x100000009b9/picture-31.jpg v370>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ad1:[<0x100000009b9/picture-31.jpg v370>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.648+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b71 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-8.jpg auth v508 pv763 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.964713+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.964713+0000 b20724 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9b9b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b71:[<0x100000009b9/picture-8.jpg v508>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b71:[<0x100000009b9/picture-8.jpg v508>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.648+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000af5 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-26.jpg auth v400 pv767 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.641722+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.641722+0000 b57279 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9ba100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000af5:[<0x100000009b9/picture-26.jpg v400>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000af5:[<0x100000009b9/picture-26.jpg v400>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.649+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a28 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-32.xml auth v148 pv771 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.118737+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.118737+0000 b31269 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9ba680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a28:[<0x100000009b9/article-32.xml v148>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a28:[<0x100000009b9/article-32.xml v148>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.649+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a54 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-21.xml auth v214 pv775 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.217734+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.217734+0000 b17989 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9bac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a54:[<0x100000009b9/article-21.xml v214>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a54:[<0x100000009b9/article-21.xml v214>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.649+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a30 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-30.xml auth v160 pv779 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.136736+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.136736+0000 b19040 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9bb180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a30:[<0x100000009b9/article-30.xml v160>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a30:[<0x100000009b9/article-30.xml v160>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.124 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.650+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a74 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-13.xml auth v262 pv783 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.296732+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.296732+0000 b28552 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619ea9bb700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a74:[<0x100000009b9/article-13.xml v262>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a74:[<0x100000009b9/article-13.xml v262>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.650+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009db [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-50.xml auth v40 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.851744+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacba580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009db:[<0x100000009b9/article-50.xml v40>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009db:[<0x100000009b9/article-50.xml v40>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.651+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a90 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-6.xml auth v304 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.361730+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacbab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a90:[<0x100000009b9/article-6.xml v304>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a90:[<0x100000009b9/article-6.xml v304>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.651+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aa0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-2.xml auth v328 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.399729+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacbb080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aa0:[<0x100000009b9/article-2.xml v328>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aa0:[<0x100000009b9/article-2.xml v328>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.652+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000adf [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-29.jpg auth v382 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.577724+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacbbb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000adf:[<0x100000009b9/picture-29.jpg v382>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000adf:[<0x100000009b9/picture-29.jpg v382>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.653+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009bb [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-56.xml auth v4 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.784746+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacbc680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009bb:[<0x100000009b9/article-56.xml v4>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009bb:[<0x100000009b9/article-56.xml v4>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.654+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009fa [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-43.xml auth v82 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.977741+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacbd180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009fa:[<0x100000009b9/article-43.xml v82>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009fa:[<0x100000009b9/article-43.xml v82>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.654+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a24 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-33.xml auth v142 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.109737+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacbd700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a24:[<0x100000009b9/article-33.xml v142>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a24:[<0x100000009b9/article-33.xml v142>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.655+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b7d [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-5.jpg auth v526 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.008711+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacc0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b7d:[<0x100000009b9/picture-5.jpg v526>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b7d:[<0x100000009b9/picture-5.jpg v526>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.655+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ca4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-50.jpg auth v577 ap=1 s=0 n(v0 rc2024-05-09T08:27:57.602666+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacc0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ca4:[<0x100000009b9/picture-50.jpg v577>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ca4:[<0x100000009b9/picture-50.jpg v577>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.656+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009cd [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-53.xml auth v22 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.821745+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacc1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009cd:[<0x100000009b9/article-53.xml v22>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009cd:[<0x100000009b9/article-53.xml v22>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.657+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a38 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-28.xml auth v172 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.154736+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacc2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a38:[<0x100000009b9/article-28.xml v172>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a38:[<0x100000009b9/article-28.xml v172>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.125 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.658+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009d3 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-52.xml auth v28 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.834745+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacc3180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009d3:[<0x100000009b9/article-52.xml v28>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009d3:[<0x100000009b9/article-52.xml v28>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.126 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.659+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b83 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-3.jpg auth v538 ap=1 s=0 n(v0 rc2024-05-09T08:27:56.026711+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eaca0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b83:[<0x100000009b9/picture-3.jpg v538>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b83:[<0x100000009b9/picture-3.jpg v538>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.126 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.659+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b65 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-11.jpg auth v490 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.930714+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eaca0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b65:[<0x100000009b9/picture-11.jpg v490>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b65:[<0x100000009b9/picture-11.jpg v490>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.126 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.660+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a94 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-5.xml auth v310 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.371729+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eaca1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a94:[<0x100000009b9/article-5.xml v310>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a94:[<0x100000009b9/article-5.xml v310>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.126 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.660+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a20 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-34.xml auth v136 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.097737+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eaca1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a20:[<0x100000009b9/article-34.xml v136>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a20:[<0x100000009b9/article-34.xml v136>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.126 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.661+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a34 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-29.xml auth v166 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.145736+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eaca1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a34:[<0x100000009b9/article-29.xml v166>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a34:[<0x100000009b9/article-29.xml v166>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.126 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.661+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a50 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-22.xml auth v208 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.208734+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eaca2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a50:[<0x100000009b9/article-22.xml v208>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a50:[<0x100000009b9/article-22.xml v208>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.126 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.661+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009e0 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-49.xml auth v46 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.864744+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eaca2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009e0:[<0x100000009b9/article-49.xml v46>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009e0:[<0x100000009b9/article-49.xml v46>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.126 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.662+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b0c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-23.jpg auth v418 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.704720+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eaca3180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b0c:[<0x100000009b9/picture-23.jpg v418>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b0c:[<0x100000009b9/picture-23.jpg v418>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.663+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a70 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-14.xml auth v256 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.286732+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eaca3700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a70:[<0x100000009b9/article-14.xml v256>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a70:[<0x100000009b9/article-14.xml v256>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.663+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009ed [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-46.xml auth v64 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.913743+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009ed:[<0x100000009b9/article-46.xml v64>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009ed:[<0x100000009b9/article-46.xml v64>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.664+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009e8 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-47.xml auth v58 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.896743+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009e8:[<0x100000009b9/article-47.xml v58>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009e8:[<0x100000009b9/article-47.xml v58>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.665+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b16 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-22.jpg auth v424 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.725719+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b16:[<0x100000009b9/picture-22.jpg v424>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b16:[<0x100000009b9/picture-22.jpg v424>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.665+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a7c [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-11.xml auth v274 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.316731+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a7c:[<0x100000009b9/article-11.xml v274>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a7c:[<0x100000009b9/article-11.xml v274>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.666+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a68 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-16.xml auth v244 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.266732+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a68:[<0x100000009b9/article-16.xml v244>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a68:[<0x100000009b9/article-16.xml v244>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.667+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009c7 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-54.xml auth v16 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.808746+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009c7:[<0x100000009b9/article-54.xml v16>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009c7:[<0x100000009b9/article-54.xml v16>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.667+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a40 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-26.xml auth v184 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.172735+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a40:[<0x100000009b9/article-26.xml v184>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a40:[<0x100000009b9/article-26.xml v184>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.668+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b55 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-15.jpg auth v466 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.887715+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b55:[<0x100000009b9/picture-15.jpg v466>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b55:[<0x100000009b9/picture-15.jpg v466>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.669+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a80 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-10.xml auth v280 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.325731+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a80:[<0x100000009b9/article-10.xml v280>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a80:[<0x100000009b9/article-10.xml v280>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.669+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b61 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-12.jpg auth v484 ap=1 s=0 n(v0 rc2024-05-09T08:27:55.919714+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b61:[<0x100000009b9/picture-12.jpg v484>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b61:[<0x100000009b9/picture-12.jpg v484>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.670+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009e4 [3,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-48.xml auth v52 ap=1 s=0 n(v0 rc2024-05-09T08:27:54.878744+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15549=0-4194304@2} caps={15549=pAsxLsXsxFsxcrwb/pAsxXsxFxcwb@1},l=15549 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x5619eacb5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009e4:[<0x100000009b9/article-48.xml v52>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009e4:[<0x100000009b9/article-48.xml v52>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.127 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.670+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a98 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-4.xml auth v316 pv817 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.380729+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.380729+0000 b18022 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac84000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a98:[<0x100000009b9/article-4.xml v316>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a98:[<0x100000009b9/article-4.xml v316>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.671+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a44 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-25.xml auth v190 pv821 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.181735+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.181735+0000 b27321 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac84580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a44:[<0x100000009b9/article-25.xml v190>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a44:[<0x100000009b9/article-25.xml v190>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.671+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a10 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-38.xml auth v112 pv825 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.041739+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.041739+0000 b22535 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac84b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a10:[<0x100000009b9/article-38.xml v112>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a10:[<0x100000009b9/article-38.xml v112>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.672+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b20 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-21.jpg auth v430 pv829 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.750719+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.750719+0000 b62839 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac85080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b20:[<0x100000009b9/picture-21.jpg v430>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b20:[<0x100000009b9/picture-21.jpg v430>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.672+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a1c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-35.xml auth v130 pv833 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.083738+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.083738+0000 b10201 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac85600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a1c:[<0x100000009b9/article-35.xml v130>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a1c:[<0x100000009b9/article-35.xml v130>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.673+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a06 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-40.xml auth v100 pv837 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.007740+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.007740+0000 b27186 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac86100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a06:[<0x100000009b9/article-40.xml v100>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a06:[<0x100000009b9/article-40.xml v100>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.673+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ad8 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-30.jpg auth v376 pv841 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.559724+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.559724+0000 b22851 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac86680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ad8:[<0x100000009b9/picture-30.jpg v376>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ad8:[<0x100000009b9/picture-30.jpg v376>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.674+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a6c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-15.xml auth v250 pv845 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.276732+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.276732+0000 b22534 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac86c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a6c:[<0x100000009b9/article-15.xml v250>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a6c:[<0x100000009b9/article-15.xml v250>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.675+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aab [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-36.jpg auth v340 pv849 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.442727+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.442727+0000 b47093 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac87180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aab:[<0x100000009b9/picture-36.jpg v340>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aab:[<0x100000009b9/picture-36.jpg v340>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.675+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a5c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-19.xml auth v226 pv853 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.236733+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.236733+0000 b1396 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac87700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a5c:[<0x100000009b9/article-19.xml v226>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a5c:[<0x100000009b9/article-19.xml v226>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.676+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b27 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-20.jpg auth v436 pv857 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.771718+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.771718+0000 b51695 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac8a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b27:[<0x100000009b9/picture-20.jpg v436>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b27:[<0x100000009b9/picture-20.jpg v436>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.676+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000009fe [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-42.xml auth v88 pv861 ap=5 s=0 n(v0 rc2024-05-09T08:27:54.986740+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:54.986740+0000 b17194 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac8a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000009fe:[<0x100000009b9/article-42.xml v88>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x100000009fe:[<0x100000009b9/article-42.xml v88>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.128 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.677+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a14 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-37.xml auth v118 pv869 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.056738+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.056738+0000 b10576 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac8b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a14:[<0x100000009b9/article-37.xml v118>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a14:[<0x100000009b9/article-37.xml v118>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.678+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aee [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-27.jpg auth v394 pv873 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.618722+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.618722+0000 b44127 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac8b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aee:[<0x100000009b9/picture-27.jpg v394>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aee:[<0x100000009b9/picture-27.jpg v394>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.678+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a0b [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-39.xml auth v106 pv877 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.024739+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.024739+0000 b32108 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac8bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a0b:[<0x100000009b9/article-39.xml v106>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a0b:[<0x100000009b9/article-39.xml v106>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.679+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a18 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-36.xml auth v124 pv881 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.069738+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.069738+0000 b19041 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac8c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a18:[<0x100000009b9/article-36.xml v124>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a18:[<0x100000009b9/article-36.xml v124>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.679+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a78 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-12.xml auth v268 pv885 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.306731+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.306731+0000 b14007 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac8c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a78:[<0x100000009b9/article-12.xml v268>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a78:[<0x100000009b9/article-12.xml v268>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.680+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aa4 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-1.xml auth v334 pv889 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.425728+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.425728+0000 b17125 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac8cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aa4:[<0x100000009b9/article-1.xml v334>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aa4:[<0x100000009b9/article-1.xml v334>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.681+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000ab9 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-34.jpg auth v352 pv893 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.478726+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.478726+0000 b36949 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac8d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000ab9:[<0x100000009b9/picture-34.jpg v352>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000ab9:[<0x100000009b9/picture-34.jpg v352>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.682+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000b51 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-16.jpg auth v460 pv791 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.878715+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.878715+0000 b64183 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac6c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000b51:[<0x100000009b9/picture-16.jpg v460>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000b51:[<0x100000009b9/picture-16.jpg v460>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.683+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000bc3 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-74.jpg auth v556 pv799 ap=5 s=0 n(v0 rc2024-05-09T08:27:56.866687+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:56.866687+0000 b50834 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac6d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000bc3:[<0x100000009b9/picture-74.jpg v556>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000bc3:[<0x100000009b9/picture-74.jpg v556>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.684+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a4c [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-23.xml auth v202 pv803 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.199734+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.199734+0000 b12450 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac6d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a4c:[<0x100000009b9/article-23.xml v202>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a4c:[<0x100000009b9/article-23.xml v202>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.684+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000a64 [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-17.xml auth v238 pv807 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.255733+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.255733+0000 b21337 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac6db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000a64:[<0x100000009b9/article-17.xml v238>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000a64:[<0x100000009b9/article-17.xml v238>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.129 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:28:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:28:12.685+0000 7fe7d41bb700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000aca [4,head] /volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-32.jpg auth v364 pv813 ap=5 s=0 n(v0 rc2024-05-09T08:27:55.522725+0000 1=1+0)/n(v0 1=1+0)->n(v0 rc2024-05-09T08:27:55.522725+0000 b53102 1=1+0) (iauth excl->sync) (ifile excl w=1) (ixattr excl->sync) cr={15549=0-4194304@2} caps={15549=pAsLsXsFsxcrwb/pAsxLsXsxFsxcrwb/Fc@2},l=15549(-1) | ptrwaiter=2 lock=1 importingcaps=0 caps=1 dirtyrstat=0 replicated=0 authpin=1 scrubqueue=0 0x5619eac6e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000aca:[<0x100000009b9/picture-32.jpg v364>,<0x10000000399/blog-10 v17381>,<0x10000000208/blogtest_in v315>,<0x10000000201/src v1990>,<0x10000000006/blogbench-1.0 v1757>,<0x10000000005/tmp v46>,<0x100000001fb/client.0 v39>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v52>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v59>,<0x1/volumes v57>]//[3]","memoryvalue":"(4)0x10000000aca:[<0x100000009b9/picture-32.jpg v364>,<0x10000000399/blog-10 v22967>,<0x10000000208/blogtest_in v358>,<0x10000000201/src v2005>,<0x10000000006/blogbench-1.0 v1781>,<0x10000000005/tmp v32>,<0x100000001fb/client.0 v53>,<0x100000001fa/78711e5e-0ac4-4ad0-adea-054a648e2e00 v48>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v63>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T08:28:13.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: pgmap v60: 129 pgs: 129 active+clean; 47 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 866 op/s 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000069f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-64.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a0e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-26.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000692 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-75.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000765 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-27.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-37.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000696 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-73.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-39.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006c3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-36.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000070b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-12.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000733 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-52.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000727 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-1.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-30.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000758 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-34.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000731 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-53.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000069d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-66.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000770 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-16.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000068e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-77.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006ea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-23.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-17.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000075c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-32.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000073b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-48.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-64.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.604 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000685 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-80.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000776 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-10.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000077f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-1.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000714 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-9.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-44.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000077a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-6.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000069e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-65.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-62.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000006e7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-24.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a12 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-25.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000067a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-85.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000764 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-28.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000069c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-67.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.605 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000752 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-37.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000076f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-17.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000723 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-3.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000768 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-24.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000075a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-33.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000729 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-57.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000071a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-7.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000748 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-42.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000074e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-39.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-24.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b5d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-13.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b38 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-18.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b31 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-19.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a60 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-18.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009d8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-51.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-27.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b80 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-4.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ab2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-35.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b85 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-2.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a58 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-20.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b59 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-14.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a02 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-41.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009f1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-45.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b75 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-7.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-31.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b6d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-9.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b69 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-10.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b87 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-1.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a84 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-9.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-44.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b79 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-6.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bc9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-99.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a48 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-24.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ae6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-28.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ca9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-85.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b4d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-17.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a9c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-3.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b05 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-24.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ac0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-33.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a8c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-7.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-55.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000afe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-25.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ca0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-8.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ad1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-31.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b71 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-8.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000af5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-26.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a28 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-32.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a54 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-21.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a30 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-30.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a74 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-13.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009db (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-50.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a90 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-6.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aa0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-2.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000adf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-29.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-56.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-43.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a24 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-33.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b7d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-5.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ca4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-50.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009cd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-53.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a38 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-28.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-52.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b83 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-3.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b65 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-11.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a94 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-5.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a20 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-34.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a34 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-29.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a50 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-22.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009e0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-49.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b0c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-23.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a70 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-14.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-46.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009e8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-47.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-22.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a7c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-11.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a68 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-16.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-54.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-26.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b55 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-15.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a80 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-10.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b61 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-12.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-48.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a98 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-4.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a44 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-25.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a10 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-38.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b20 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-21.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a1c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-35.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a06 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-40.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ad8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-30.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a6c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-15.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-36.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a5c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-19.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b27 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-20.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000009fe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-42.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a14 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-37.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-27.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a0b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-39.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a18 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-36.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a78 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-12.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aa4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-1.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ab9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-34.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000b51 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-16.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000bc3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-74.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a4c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-23.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000a64 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-17.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000aca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-32.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000405 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d30 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-73.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d6d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-60.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d19 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-78.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cd1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-96.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003ea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cf7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-88.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003a6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dcd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-79.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dd2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-39.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003d8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cfe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-86.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000408 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d5c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d0e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-82.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d8c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ce9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-91.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000400 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-23.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d61 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-62.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000518 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-91.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000403 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-22.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003b1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-11.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003a2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-16.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d72 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-59.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d1e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-77.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000417 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-15.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003b4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000420 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-12.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d9c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-50.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000da5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003c6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-25.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.619 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-97.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000da8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003e8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-31.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-70.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000042c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-8.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-26.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cc7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-98.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d0d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-83.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cf2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-89.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-69.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d2a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-74.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ced (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-90.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d57 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-64.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003ab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-13.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d34 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-72.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003cc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d49 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-67.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-29.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-27.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000529 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-75.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000c9c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-73.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003cf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-1.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003ae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-12.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-34.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000414 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-16.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000da0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-49.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cfa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-87.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dc1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-42.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d53 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-65.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cbf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-100.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000039f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-17.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003e5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-32.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d98 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d10 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-81.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000439 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-3.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d90 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-53.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d03 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-85.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000423 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-11.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003c4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-5.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000050f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-70.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d7d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-57.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d67 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-61.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dd7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-38.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d46 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-68.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d14 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-80.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000db9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-44.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000435 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-5.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cdf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-93.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d21 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-76.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d95 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-52.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d4e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-66.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dbd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-43.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cc4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-99.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000db3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-45.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d08 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-84.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ddd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-37.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d38 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-71.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000426 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cd7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-95.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000043d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003b7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-9.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000432 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d77 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-58.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d88 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-55.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003f0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000dc7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d26 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-75.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003d5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000412 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003c9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003e2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000051f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003bd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-7.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000041e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000d83 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000040e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000040b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000437 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x100000003db (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000043b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000041a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000ce4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-92.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000cdb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-94.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x1000000042f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: Scrub error on inode 0x10000000429 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: scrub summary: active paths [/] 2024-05-09T08:28:13.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:13 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: pgmap v60: 129 pgs: 129 active+clean; 47 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 866 op/s 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000069f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-64.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a0e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-26.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000692 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-75.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000765 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-27.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-37.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000696 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-73.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-39.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.628 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006c3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-36.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000070b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-12.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000733 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-52.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000727 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-1.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-30.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000758 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-34.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000731 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-53.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000069d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-66.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000770 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-16.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000068e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-77.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006ea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-23.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-17.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000075c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-32.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000073b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-48.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-64.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.629 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000685 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-80.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000776 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-10.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000077f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-1.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000714 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-9.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-44.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000077a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-6.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000069e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-65.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-62.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000006e7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-24.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a12 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-25.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000067a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-85.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000764 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-28.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.630 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000069c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-67.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000752 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-37.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000076f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-17.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000723 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-3.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000768 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-24.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000075a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-33.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000729 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-57.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000071a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-7.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000748 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-42.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000074e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-39.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-24.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:13.631 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b5d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-13.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b38 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-18.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b31 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-19.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a60 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-18.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009d8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-51.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-27.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b80 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-4.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ab2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-35.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b85 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-2.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a58 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-20.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b59 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-14.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a02 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-41.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009f1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-45.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b75 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-7.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-31.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b6d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-9.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.632 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b69 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-10.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b87 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-1.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a84 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-9.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-44.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b79 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-6.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bc9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-99.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a48 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-24.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ae6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-28.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ca9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-85.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b4d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-17.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a9c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-3.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b05 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-24.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ac0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-33.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a8c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-7.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-55.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.633 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000afe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-25.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ca0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-8.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ad1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-31.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b71 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-8.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000af5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-26.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a28 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-32.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a54 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-21.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a30 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-30.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a74 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-13.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009db (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-50.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a90 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-6.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aa0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-2.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000adf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-29.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-56.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.634 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-43.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a24 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-33.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b7d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-5.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ca4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-50.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009cd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-53.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a38 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-28.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-52.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b83 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-3.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b65 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-11.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a94 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-5.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a20 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-34.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a34 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-29.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a50 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-22.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009e0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-49.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.635 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b0c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-23.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a70 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-14.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-46.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009e8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-47.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-22.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a7c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-11.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a68 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-16.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-54.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-26.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b55 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-15.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a80 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-10.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b61 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-12.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-48.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a98 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-4.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a44 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-25.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a10 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-38.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.636 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b20 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-21.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a1c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-35.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a06 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-40.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ad8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-30.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a6c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-15.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-36.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a5c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-19.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b27 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-20.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000009fe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-42.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a14 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-37.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-27.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a0b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-39.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a18 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-36.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.637 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a78 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-12.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aa4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-1.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ab9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-34.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000b51 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-16.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000bc3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-74.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a4c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-23.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000a64 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-17.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000aca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-32.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000405 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d30 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-73.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.638 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d6d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-60.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d19 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-78.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cd1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-96.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003ea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cf7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-88.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003a6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dcd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-79.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dd2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-39.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003d8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cfe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-86.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000408 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d5c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d0e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-82.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d8c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.639 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ce9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-91.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000400 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-23.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d61 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-62.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000518 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-91.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000403 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-22.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003b1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-11.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003a2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-16.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d72 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-59.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d1e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-77.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000417 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-15.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003b4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000420 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-12.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.640 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d9c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-50.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000da5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003c6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-25.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-97.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000da8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003e8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-31.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-70.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000042c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-8.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-26.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cc7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-98.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d0d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-83.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cf2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-89.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-69.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.641 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d2a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-74.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ced (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-90.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d57 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-64.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003ab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-13.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d34 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-72.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003cc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d49 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-67.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-29.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-27.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000529 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-75.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000c9c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-73.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003cf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-1.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.642 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003ae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-12.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-34.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000414 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-16.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000da0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-49.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cfa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-87.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dc1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-42.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d53 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-65.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cbf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-100.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000039f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-17.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003e5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-32.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d98 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d10 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-81.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000439 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-3.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d90 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-53.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d03 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-85.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.643 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000423 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-11.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003c4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-5.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000050f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-70.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d7d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-57.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d67 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-61.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dd7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-38.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d46 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-68.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d14 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-80.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000db9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-44.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000435 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-5.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cdf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-93.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d21 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-76.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d95 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-52.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d4e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-66.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.644 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dbd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-43.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cc4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-99.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000db3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-45.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d08 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-84.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ddd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-37.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d38 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-71.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000426 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cd7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-95.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000043d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003b7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-9.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000432 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d77 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-58.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d88 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-55.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003f0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000dc7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.645 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d26 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-75.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003d5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000412 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003c9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003e2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000051f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003bd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-7.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000041e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000d83 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000040e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000040b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000437 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x100000003db (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.646 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000043b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000041a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000ce4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-92.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000cdb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-94.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x1000000042f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: Scrub error on inode 0x10000000429 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: scrub summary: active paths [/] 2024-05-09T08:28:13.647 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:13 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: pgmap v60: 129 pgs: 129 active+clean; 47 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 866 op/s 2024-05-09T08:28:13.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:13.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000069f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-64.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a0e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-26.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000692 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-75.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000765 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-27.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-37.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000696 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-73.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-39.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006c3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-36.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000070b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-12.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000733 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-52.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000727 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-1.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-30.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000758 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-34.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000731 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-53.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000069d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-66.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000770 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-16.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000068e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-77.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006ea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-23.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-17.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006a0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000075c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-32.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000073b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-48.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009fd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-64.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000685 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-80.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000776 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-10.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000077f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-1.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000714 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-9.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006b5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-44.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000077a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-6.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000069e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-65.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006a1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-62.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000006e7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-24.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a12 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-25.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000067a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-85.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000764 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-28.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000069c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-67.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000752 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-37.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000076f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-17.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000723 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-3.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000768 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-24.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000075a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-33.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000729 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-57.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000071a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-7.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000748 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-42.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000074e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-39.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-24.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b5d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-13.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b38 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-18.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b31 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-19.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a60 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-18.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009d8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-51.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-27.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b80 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-4.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ab2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-35.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b85 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-2.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a58 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-20.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b59 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-14.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a02 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-41.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009f1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-45.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b75 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-7.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a2c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-31.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b6d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-9.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b69 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-10.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b87 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-1.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a84 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-9.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-44.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b79 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-6.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bc9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-99.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a48 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-24.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ae6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-28.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ca9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-85.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b4d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-17.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a9c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-3.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b05 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-24.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ac0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-33.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a8c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-7.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009c1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-55.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000afe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-25.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ca0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-8.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ad1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-31.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b71 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-8.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000af5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-26.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a28 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-32.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a54 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-21.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a30 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-30.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a74 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-13.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009db (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-50.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a90 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-6.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aa0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-2.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000adf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-29.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009bb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-56.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009fa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-43.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a24 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-33.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b7d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-5.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ca4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-50.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009cd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-53.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a38 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-28.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-52.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b83 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-3.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b65 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-11.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a94 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-5.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a20 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-34.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a34 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-29.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a50 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-22.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009e0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-49.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b0c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-23.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a70 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-14.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009ed (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-46.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009e8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-47.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-22.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a7c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-11.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a68 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-16.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009c7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-54.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-26.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b55 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-15.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a80 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-10.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b61 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-12.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009e4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-48.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a98 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-4.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a44 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-25.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a10 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-38.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b20 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-21.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a1c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-35.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a06 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-40.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ad8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-30.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a6c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-15.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-36.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a5c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-19.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b27 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-20.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000009fe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-42.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a14 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-37.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aee (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-27.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a0b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-39.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a18 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-36.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a78 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-12.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aa4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-1.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ab9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-34.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000b51 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-16.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000bc3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-74.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a4c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-23.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000a64 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-17.xml) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000aca (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-32.jpg) see mds.c log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000405 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-21.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d30 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-73.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d6d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-60.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d19 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-78.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cd1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-96.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003ea (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-30.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cf7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-88.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003a6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-15.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dcd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-40.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d16 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-79.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dd2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-39.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003d8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-36.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cfe (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-86.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000408 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-20.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d5c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-63.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d0e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-82.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d8c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-54.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ce9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-91.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003d3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-38.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000400 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-23.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003a9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-14.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d61 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-62.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000518 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-91.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000403 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-22.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003b1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-11.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003a2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-16.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d72 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-59.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d1e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-77.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000417 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-15.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003b4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-10.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000420 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-12.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d9c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-50.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000da5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-48.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003c6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-4.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003f8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-25.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cce (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-97.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003ba (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-8.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000da8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-47.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dad (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-46.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003e8 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-31.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d3c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-70.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000042c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-8.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003f6 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-26.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cc7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-98.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d0d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-83.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cf2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-89.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d40 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-69.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d2a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-74.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ced (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-90.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d57 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-64.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003ab (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-13.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d34 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-72.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003c0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-6.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003cc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-2.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d49 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-67.xml) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003ec (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-29.jpg) see mds.f log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003f4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-27.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000529 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-75.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000c9c (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-73.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003cf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-1.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003ae (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-12.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003df (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-34.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000414 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-16.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000da0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-49.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cfa (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-87.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dc1 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-42.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d53 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-65.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cbf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-100.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000039f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-17.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003e5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-32.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d98 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-51.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d10 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-81.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000439 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-3.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d90 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-53.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d03 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-85.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000423 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-11.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003c4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-5.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000050f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-70.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d7d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-57.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d67 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-61.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dd7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-38.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d46 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-68.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d14 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-80.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000db9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-44.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000435 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-5.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cdf (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-93.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d21 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-76.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d95 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-52.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d4e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-66.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dbd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-43.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cc4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-99.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000db3 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-45.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d08 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-84.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ddd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-37.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d38 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-71.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000426 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cd7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-95.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000043d (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003b7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-9.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000432 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d77 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-58.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d88 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-55.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003f0 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000dc7 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-41.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d26 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-75.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003d5 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000412 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003c9 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-3.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003fc (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003e2 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000051f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003bd (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-7.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000041e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000d83 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-56.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000040e (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000040b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000437 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x100000003db (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000043b (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000041a (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000ce4 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-92.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000cdb (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-94.xml) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x1000000042f (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: Scrub error on inode 0x10000000429 (/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-05-09T08:28:13.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: scrub summary: active paths [/] 2024-05-09T08:28:13.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:13 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:14.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.529+0000 7fedf8b29640 1 -- 172.21.15.28:0/1385440112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4100780 msgr2=0x7fedf4100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:14.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.529+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1385440112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4100780 0x7fedf4100c00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fede8009930 tx=0x7fede802a420 comp rx=0 tx=0).stop 2024-05-09T08:28:14.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.529+0000 7fedf8b29640 1 -- 172.21.15.28:0/1385440112 shutdown_connections 2024-05-09T08:28:14.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.529+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1385440112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf4101140 0x7fedf410d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.529+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1385440112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4100780 0x7fedf4100c00 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.529+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1385440112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf4106740 0x7fedf4106b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.529+0000 7fedf8b29640 1 -- 172.21.15.28:0/1385440112 >> 172.21.15.28:0/1385440112 conn(0x7fedf4077880 msgr2=0x7fedf4077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:14.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.529+0000 7fedf8b29640 1 -- 172.21.15.28:0/1385440112 shutdown_connections 2024-05-09T08:28:14.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.529+0000 7fedf8b29640 1 -- 172.21.15.28:0/1385440112 wait complete. 2024-05-09T08:28:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf8b29640 1 Processor -- start 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf8b29640 1 -- start start 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf8b29640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf4100780 0x7fedf40fd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf8b29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4101140 0x7fedf40fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf8b29640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf4106740 0x7fedf40fe160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf8b29640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedf4110b50 con 0x7fedf4101140 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf8b29640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fedf41109d0 con 0x7fedf4106740 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf8b29640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedf4110cd0 con 0x7fedf4100780 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4101140 0x7fedf40fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4101140 0x7fedf40fdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46614/0 (socket says 172.21.15.28:46614) 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf1d9b640 1 -- 172.21.15.28:0/1499957793 learned_addr learned my addr 172.21.15.28:0/1499957793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.530+0000 7fedf259c640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf4100780 0x7fedf40fd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fedf2d9d640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf4106740 0x7fedf40fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fedf1d9b640 1 -- 172.21.15.28:0/1499957793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf4100780 msgr2=0x7fedf40fd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fedf1d9b640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf4100780 0x7fedf40fd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fedf1d9b640 1 -- 172.21.15.28:0/1499957793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf4106740 msgr2=0x7fedf40fe160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:14.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fedf1d9b640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf4106740 0x7fedf40fe160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fedf1d9b640 1 -- 172.21.15.28:0/1499957793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf41a6580 con 0x7fedf4101140 2024-05-09T08:28:14.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fedf1d9b640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4101140 0x7fedf40fdc20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fede8009230 tx=0x7fede8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:14.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fede37fe640 1 -- 172.21.15.28:0/1499957793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede803f030 con 0x7fedf4101140 2024-05-09T08:28:14.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fedf8b29640 1 -- 172.21.15.28:0/1499957793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf41a6810 con 0x7fedf4101140 2024-05-09T08:28:14.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fede37fe640 1 -- 172.21.15.28:0/1499957793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fede8004440 con 0x7fedf4101140 2024-05-09T08:28:14.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fede37fe640 1 -- 172.21.15.28:0/1499957793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8035070 con 0x7fedf4101140 2024-05-09T08:28:14.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.531+0000 7fedf8b29640 1 -- 172.21.15.28:0/1499957793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf41a6d50 con 0x7fedf4101140 2024-05-09T08:28:14.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.534+0000 7fede37fe640 1 -- 172.21.15.28:0/1499957793 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fede8042030 con 0x7fedf4101140 2024-05-09T08:28:14.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.534+0000 7fedfb83b740 1 Processor -- start 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.534+0000 7fedfb83b740 1 -- start start 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.534+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd87f7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.534+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 0x563fd88020f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.534+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd8804a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.534+0000 7fedfb83b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563fd875c340 con 0x563fd8802630 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.534+0000 7fedfb83b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x563fd8704290 con 0x563fd87f7d70 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.534+0000 7fedfb83b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563fd8758300 con 0x563fd87f7450 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 0x563fd88020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf2d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd87f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd8804a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf2d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd87f7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:56722/0 (socket says 172.21.15.28:56722) 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf2d9d640 1 -- 172.21.15.28:0/498356588 learned_addr learned my addr 172.21.15.28:0/498356588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf2d9d640 1 -- 172.21.15.28:0/498356588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 msgr2=0x563fd88020f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf2d9d640 1 --2- 172.21.15.28:0/498356588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 0x563fd88020f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf2d9d640 1 -- 172.21.15.28:0/498356588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 msgr2=0x563fd8804a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf2d9d640 1 --2- 172.21.15.28:0/498356588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd8804a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf2d9d640 1 -- 172.21.15.28:0/498356588 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fd87288e0 con 0x563fd87f7450 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf259c640 1 --2- 172.21.15.28:0/498356588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 0x563fd88020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fede37fe640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7feddc0a2da0 0x7feddc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf259c640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7feddc0a2da0 0x7feddc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fedf2d9d640 1 --2- 172.21.15.28:0/498356588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd87f7830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fedec009fd0 tx=0x7fedec0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f33183ec22d5b1cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fede37fe640 1 -- 172.21.15.28:0/1499957793 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fede80efb60 con 0x7fedf4101140 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fede0ff9640 1 -- 172.21.15.28:0/498356588 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec004d20 con 0x563fd87f7450 2024-05-09T08:28:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fede0ff9640 1 -- 172.21.15.28:0/498356588 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedec004050 con 0x563fd87f7450 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.535+0000 7fede0ff9640 1 -- 172.21.15.28:0/498356588 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec005240 con 0x563fd87f7450 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedfb83b740 1 -- 172.21.15.28:0/498356588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 msgr2=0x563fd87f7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedf259c640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7feddc0a2da0 0x7feddc0a5260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x563fd87f89f0 tx=0x7fede4005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedfb83b740 1 --2- 172.21.15.28:0/498356588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd87f7830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fedec009fd0 tx=0x7fedec0118d0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedfb83b740 1 -- 172.21.15.28:0/498356588 shutdown_connections 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedfb83b740 1 --2- 172.21.15.28:0/498356588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd8804a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedfb83b740 1 --2- 172.21.15.28:0/498356588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 0x563fd88020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedfb83b740 1 --2- 172.21.15.28:0/498356588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd87f7830 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedfb83b740 1 -- 172.21.15.28:0/498356588 >> 172.21.15.28:0/498356588 conn(0x563fd87f3960 msgr2=0x563fd87f3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedfb83b740 1 -- 172.21.15.28:0/498356588 shutdown_connections 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.536+0000 7fedfb83b740 1 -- 172.21.15.28:0/498356588 wait complete. 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedfb83b740 1 -- start start 2024-05-09T08:28:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedfb83b740 1 Processor -- start 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd87f7450 0x563fd87ff570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 0x563fd87ffab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd8802630 0x563fd87ffff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedfb83b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563fd8805260 con 0x563fd87f7450 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedfb83b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x563fd8717790 con 0x563fd8802630 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedfb83b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563fd88053e0 con 0x563fd87f7d70 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedfb83b740 10 client.0 _ll_register_callbacks cb 0x563fd8752a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedf259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd87f7450 0x563fd87ff570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedf259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd87f7450 0x563fd87ff570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46632/0 (socket says 172.21.15.28:46632) 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedf259c640 1 -- 172.21.15.28:0/1476833542 learned_addr learned my addr 172.21.15.28:0/1476833542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedf2d9d640 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd8802630 0x563fd87ffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedf1d9b640 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 0x563fd87ffab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedf259c640 1 -- 172.21.15.28:0/1476833542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 msgr2=0x563fd87ffab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedf259c640 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 0x563fd87ffab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedf259c640 1 -- 172.21.15.28:0/1476833542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd8802630 msgr2=0x563fd87ffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.537+0000 7fedf259c640 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd8802630 0x563fd87ffff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedf259c640 1 -- 172.21.15.28:0/1476833542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fd8470210 con 0x563fd87f7450 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedf259c640 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd87f7450 0x563fd87ff570 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fede4009cd0 tx=0x7fede4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e031bec04309920 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedd9ffb640 1 -- 172.21.15.28:0/1476833542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede401f640 con 0x563fd87f7450 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedfb83b740 1 -- 172.21.15.28:0/1476833542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedec004050 con 0x563fd87f7450 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedd9ffb640 1 -- 172.21.15.28:0/1476833542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fede401f7e0 con 0x563fd87f7450 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedd9ffb640 1 -- 172.21.15.28:0/1476833542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fede401f9e0 con 0x563fd87f7450 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedfb83b740 10 client.16374 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedfb83b740 1 -- 172.21.15.28:0/1476833542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563fd853c620 con 0x563fd87f7450 2024-05-09T08:28:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.538+0000 7fedd9ffb640 1 -- 172.21.15.28:0/1476833542 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedec004050 con 0x563fd87f7450 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedd9ffb640 1 -- 172.21.15.28:0/1476833542 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fede4027070 con 0x563fd87f7450 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedfb83b740 10 client.16374 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedfb83b740 10 client.16374 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedfb83b740 20 client.16374 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563fd8812500 0x563fd8814950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedfb83b740 4 client.16374 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedfb83b740 1 -- 172.21.15.28:0/1476833542 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fd8769a70 con 0x563fd8812500 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedf1d9b640 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563fd8812500 0x563fd8814950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedf1d9b640 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563fd8812500 0x563fd8814950 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ca9da2293ba0ab5 server_cookie=a6d8d3572ed5547b in_seq=0 out_seq=0 2024-05-09T08:28:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.539+0000 7fedd9ffb640 10 client.16374 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:14.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.541+0000 7fedd9ffb640 1 -- 172.21.15.28:0/1476833542 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563fd8769a70 con 0x563fd8812500 2024-05-09T08:28:14.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.541+0000 7fedd9ffb640 10 client.16374 handle_command_reply: tid=0 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.541+0000 7fedfb83b740 1 client.16374 shutdown 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 -- 172.21.15.28:0/1476833542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd87f7450 msgr2=0x563fd87ff570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd87f7450 0x563fd87ff570 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fede4009cd0 tx=0x7fede4004730 comp rx=0 tx=0).stop 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 -- 172.21.15.28:0/1476833542 shutdown_connections 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563fd8812500 0x563fd8814950 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd8802630 0x563fd87ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 0x563fd87ffab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedd9ffb640 0 client.16374 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1476833542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd87f7450 0x563fd87ff570 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 -- 172.21.15.28:0/1476833542 >> 172.21.15.28:0/1476833542 conn(0x563fd87f3960 msgr2=0x563fd87f9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 -- 172.21.15.28:0/1476833542 shutdown_connections 2024-05-09T08:28:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 1 -- 172.21.15.28:0/1476833542 wait complete. 2024-05-09T08:28:14.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.542+0000 7fedfb83b740 20 client.16374 trim_cache size 0 max 16384 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.560+0000 7fedfb83b740 1 Processor -- start 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.560+0000 7fedfb83b740 1 -- start start 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.560+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd880e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.560+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 0x563fd87f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.560+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedfb83b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedf41b0fb0 con 0x563fd8802630 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedfb83b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fedf41b2bb0 con 0x563fd87f7d70 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedfb83b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedf41b1680 con 0x563fd87f7450 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd880e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd880e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:56746/0 (socket says 172.21.15.28:56746) 2024-05-09T08:28:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf259c640 1 -- 172.21.15.28:0/1761218973 learned_addr learned my addr 172.21.15.28:0/1761218973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf1d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 0x563fd87f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf259c640 1 -- 172.21.15.28:0/1761218973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 msgr2=0x563fd87f9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf259c640 1 --2- 172.21.15.28:0/1761218973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 0x563fd87f9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf259c640 1 -- 172.21.15.28:0/1761218973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 msgr2=0x563fd87f3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf259c640 1 --2- 172.21.15.28:0/1761218973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87f3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf259c640 1 -- 172.21.15.28:0/1761218973 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fd87f1f00 con 0x563fd87f7450 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf2d9d640 1 --2- 172.21.15.28:0/1761218973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7fedf259c640 1 --2- 172.21.15.28:0/1761218973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd880e120 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fede4009a10 tx=0x7fede4004690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b957a361ebf32ba2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7feddbfff640 1 -- 172.21.15.28:0/1761218973 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede401f840 con 0x563fd87f7450 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7feddbfff640 1 -- 172.21.15.28:0/1761218973 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fede4005500 con 0x563fd87f7450 2024-05-09T08:28:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.561+0000 7feddbfff640 1 -- 172.21.15.28:0/1761218973 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede4004de0 con 0x563fd87f7450 2024-05-09T08:28:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.562+0000 7fedfb83b740 1 -- 172.21.15.28:0/1761218973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 msgr2=0x563fd880e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.562+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1761218973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd880e120 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fede4009a10 tx=0x7fede4004690 comp rx=0 tx=0).stop 2024-05-09T08:28:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.562+0000 7fedfb83b740 1 -- 172.21.15.28:0/1761218973 shutdown_connections 2024-05-09T08:28:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.562+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1761218973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.562+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1761218973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7d70 0x563fd87f9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.562+0000 7fedfb83b740 1 --2- 172.21.15.28:0/1761218973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7450 0x563fd880e120 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.562+0000 7fedfb83b740 1 -- 172.21.15.28:0/1761218973 >> 172.21.15.28:0/1761218973 conn(0x563fd87f3960 msgr2=0x563fd87f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.562+0000 7fedfb83b740 1 -- 172.21.15.28:0/1761218973 shutdown_connections 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.562+0000 7fedfb83b740 1 -- 172.21.15.28:0/1761218973 wait complete. 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedfb83b740 1 -- start start 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedfb83b740 1 Processor -- start 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7450 0x563fd880a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 0x563fd880ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedfb83b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87fbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedfb83b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedf41b2420 con 0x563fd8802630 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedfb83b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fedf41b1d50 con 0x563fd87f7450 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedfb83b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedf41aef90 con 0x563fd87f7d70 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedfb83b740 10 client.0 _ll_register_callbacks cb 0x7fede4009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87fbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87fbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46652/0 (socket says 172.21.15.28:46652) 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf2d9d640 1 -- 172.21.15.28:0/559276197 learned_addr learned my addr 172.21.15.28:0/559276197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf259c640 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7450 0x563fd880a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf1d9b640 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 0x563fd880ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf2d9d640 1 -- 172.21.15.28:0/559276197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 msgr2=0x563fd880ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf2d9d640 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 0x563fd880ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf2d9d640 1 -- 172.21.15.28:0/559276197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7450 msgr2=0x563fd880a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf2d9d640 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7450 0x563fd880a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf2d9d640 1 -- 172.21.15.28:0/559276197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fd87fc600 con 0x563fd8802630 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.563+0000 7fedf1d9b640 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 0x563fd880ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedf2d9d640 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87fbe00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fedec009fd0 tx=0x7fedec005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e073578dea7941 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedd9ffb640 1 -- 172.21.15.28:0/559276197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec009c50 con 0x563fd8802630 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedfb83b740 1 -- 172.21.15.28:0/559276197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fede4005500 con 0x563fd8802630 2024-05-09T08:28:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedd9ffb640 1 -- 172.21.15.28:0/559276197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fedec0055f0 con 0x563fd8802630 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedd9ffb640 1 -- 172.21.15.28:0/559276197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fedec036050 con 0x563fd8802630 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedfb83b740 10 client.16380 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedfb83b740 1 -- 172.21.15.28:0/559276197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563fd88786a0 con 0x563fd8802630 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedd9ffb640 1 -- 172.21.15.28:0/559276197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fede4005500 con 0x563fd8802630 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedd9ffb640 1 -- 172.21.15.28:0/559276197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fedec02d470 con 0x563fd8802630 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.564+0000 7fedfb83b740 10 client.16380 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.565+0000 7fedfb83b740 10 client.16380 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.565+0000 7fedfb83b740 20 client.16380 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.565+0000 7fedfb83b740 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563fd8878ce0 0x563fd887b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.565+0000 7fedfb83b740 4 client.16380 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.565+0000 7fedfb83b740 1 -- 172.21.15.28:0/559276197 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563fd8763dc0 con 0x563fd8878ce0 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.565+0000 7fedf259c640 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563fd8878ce0 0x563fd887b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.565+0000 7fedd9ffb640 10 client.16380 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:14.565+0000 7fedf259c640 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563fd8878ce0 0x563fd887b100 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12959c546e3ec2a4 server_cookie=8feb55f00703068b in_seq=0 out_seq=0 2024-05-09T08:28:14.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:14 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:14.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:14 smithi165 ceph-mon[34540]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:14.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:14 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:14.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:14 smithi184 ceph-mon[34509]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:14.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:14 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:14.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:14 smithi028 ceph-mon[30119]: scrub summary: idle+waiting paths [/] 2024-05-09T08:28:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.545+0000 7fedd9ffb640 1 -- 172.21.15.28:0/559276197 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563fd8763dc0 con 0x563fd8878ce0 2024-05-09T08:28:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.545+0000 7fedd9ffb640 10 client.16380 handle_command_reply: tid=0 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 client.16380 shutdown 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 -- 172.21.15.28:0/559276197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 msgr2=0x563fd87fbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87fbe00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fedec009fd0 tx=0x7fedec005270 comp rx=0 tx=0).stop 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 -- 172.21.15.28:0/559276197 shutdown_connections 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563fd8878ce0 0x563fd887b100 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fd8802630 0x563fd87fbe00 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedd9ffb640 0 client.16380 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563fd87f7d70 0x563fd880ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 --2- 172.21.15.28:0/559276197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563fd87f7450 0x563fd880a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 -- 172.21.15.28:0/559276197 >> 172.21.15.28:0/559276197 conn(0x563fd87f3960 msgr2=0x563fd87f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 -- 172.21.15.28:0/559276197 shutdown_connections 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.546+0000 7fedfb83b740 1 -- 172.21.15.28:0/559276197 wait complete. 2024-05-09T08:28:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedfb83b740 20 client.16380 trim_cache size 0 max 16384 2024-05-09T08:28:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:28:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:28:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:28:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:28:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 -- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7feddc0a2da0 msgr2=0x7feddc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7feddc0a2da0 0x7feddc0a5260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x563fd87f89f0 tx=0x7fede4005f90 comp rx=0 tx=0).stop 2024-05-09T08:28:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 -- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4101140 msgr2=0x7fedf40fdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4101140 0x7fedf40fdc20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fede8009230 tx=0x7fede8004040 comp rx=0 tx=0).stop 2024-05-09T08:28:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 -- 172.21.15.28:0/1499957793 shutdown_connections 2024-05-09T08:28:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf4106740 0x7fedf40fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf4101140 0x7fedf40fdc20 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7feddc0a2da0 0x7feddc0a5260 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 --2- 172.21.15.28:0/1499957793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf4100780 0x7fedf40fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 -- 172.21.15.28:0/1499957793 >> 172.21.15.28:0/1499957793 conn(0x7fedf4077880 msgr2=0x7fedf4069d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 -- 172.21.15.28:0/1499957793 shutdown_connections 2024-05-09T08:28:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:15.547+0000 7fedf8b29640 1 -- 172.21.15.28:0/1499957793 wait complete. 2024-05-09T08:28:15.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:15 smithi165 ceph-mon[34540]: pgmap v61: 129 pgs: 129 active+clean; 89 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 1.05k op/s 2024-05-09T08:28:15.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:15 smithi184 ceph-mon[34509]: pgmap v61: 129 pgs: 129 active+clean; 89 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 1.05k op/s 2024-05-09T08:28:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:15 smithi028 ceph-mon[30119]: pgmap v61: 129 pgs: 129 active+clean; 89 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 1.05k op/s 2024-05-09T08:28:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:28:15.419+0000 7f8912d3a700 -1 log_channel(cluster) log [ERR] : Health check update: 4 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T08:28:16.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:16 smithi165 ceph-mon[34540]: Health check update: 4 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T08:28:16.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:16 smithi184 ceph-mon[34509]: Health check update: 4 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T08:28:16.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:16 smithi028 ceph-mon[30119]: Health check update: 4 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T08:28:17.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:17 smithi165 ceph-mon[34540]: pgmap v62: 129 pgs: 129 active+clean; 128 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 7.8 MiB/s wr, 855 op/s 2024-05-09T08:28:17.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:17 smithi184 ceph-mon[34509]: pgmap v62: 129 pgs: 129 active+clean; 128 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 7.8 MiB/s wr, 855 op/s 2024-05-09T08:28:17.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:17 smithi028 ceph-mon[30119]: pgmap v62: 129 pgs: 129 active+clean; 128 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 7.8 MiB/s wr, 855 op/s 2024-05-09T08:28:18.705 INFO:tasks.workunit.client.0.smithi028.stdout: 42 187533 580 144198 566 46050 540 2024-05-09T08:28:19.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:19 smithi165 ceph-mon[34540]: pgmap v63: 129 pgs: 129 active+clean; 128 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.7 MiB/s wr, 545 op/s 2024-05-09T08:28:19.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:19 smithi184 ceph-mon[34509]: pgmap v63: 129 pgs: 129 active+clean; 128 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.7 MiB/s wr, 545 op/s 2024-05-09T08:28:19.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:19 smithi028 ceph-mon[30119]: pgmap v63: 129 pgs: 129 active+clean; 128 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.7 MiB/s wr, 545 op/s 2024-05-09T08:28:21.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:21 smithi165 ceph-mon[34540]: pgmap v64: 129 pgs: 129 active+clean; 112 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 8.5 MiB/s wr, 1.01k op/s 2024-05-09T08:28:21.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:21 smithi184 ceph-mon[34509]: pgmap v64: 129 pgs: 129 active+clean; 112 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 8.5 MiB/s wr, 1.01k op/s 2024-05-09T08:28:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:21 smithi028 ceph-mon[30119]: pgmap v64: 129 pgs: 129 active+clean; 112 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 8.5 MiB/s wr, 1.01k op/s 2024-05-09T08:28:23.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:23 smithi165 ceph-mon[34540]: pgmap v65: 129 pgs: 129 active+clean; 112 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.9 MiB/s wr, 832 op/s 2024-05-09T08:28:23.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:23 smithi184 ceph-mon[34509]: pgmap v65: 129 pgs: 129 active+clean; 112 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.9 MiB/s wr, 832 op/s 2024-05-09T08:28:23.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:23 smithi028 ceph-mon[30119]: pgmap v65: 129 pgs: 129 active+clean; 112 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.9 MiB/s wr, 832 op/s 2024-05-09T08:28:25.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:25 smithi165 ceph-mon[34540]: pgmap v66: 129 pgs: 129 active+clean; 117 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.4 MiB/s wr, 858 op/s 2024-05-09T08:28:25.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:25 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:25.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:25 smithi184 ceph-mon[34509]: pgmap v66: 129 pgs: 129 active+clean; 117 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.4 MiB/s wr, 858 op/s 2024-05-09T08:28:25.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:25 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:25.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:25 smithi028 ceph-mon[30119]: pgmap v66: 129 pgs: 129 active+clean; 117 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.4 MiB/s wr, 858 op/s 2024-05-09T08:28:25.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:25 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:26 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:26 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:26 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:28:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:26 smithi165 ceph-mon[34540]: scrub summary: idle 2024-05-09T08:28:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:26 smithi165 ceph-mon[34540]: scrub summary: idle 2024-05-09T08:28:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:26 smithi165 ceph-mon[34540]: scrub summary: idle 2024-05-09T08:28:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:26 smithi165 ceph-mon[34540]: scrub summary: idle 2024-05-09T08:28:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:26 smithi165 ceph-mon[34540]: scrub summary: idle 2024-05-09T08:28:26.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:26 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:26.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:26 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:26.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:26 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:28:26.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:26 smithi184 ceph-mon[34509]: scrub summary: idle 2024-05-09T08:28:26.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:26 smithi184 ceph-mon[34509]: scrub summary: idle 2024-05-09T08:28:26.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:26 smithi184 ceph-mon[34509]: scrub summary: idle 2024-05-09T08:28:26.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:26 smithi184 ceph-mon[34509]: scrub summary: idle 2024-05-09T08:28:26.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:26 smithi184 ceph-mon[34509]: scrub summary: idle 2024-05-09T08:28:26.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:26 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:26 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:26 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:28:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:26 smithi028 ceph-mon[30119]: scrub summary: idle 2024-05-09T08:28:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:26 smithi028 ceph-mon[30119]: scrub summary: idle 2024-05-09T08:28:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:26 smithi028 ceph-mon[30119]: scrub summary: idle 2024-05-09T08:28:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:26 smithi028 ceph-mon[30119]: scrub summary: idle 2024-05-09T08:28:26.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:26 smithi028 ceph-mon[30119]: scrub summary: idle 2024-05-09T08:28:27.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:27 smithi028 ceph-mon[30119]: pgmap v67: 129 pgs: 129 active+clean; 120 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 699 op/s 2024-05-09T08:28:27.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:27 smithi165 ceph-mon[34540]: pgmap v67: 129 pgs: 129 active+clean; 120 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 699 op/s 2024-05-09T08:28:27.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:27 smithi184 ceph-mon[34509]: pgmap v67: 129 pgs: 129 active+clean; 120 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 699 op/s 2024-05-09T08:28:28.705 INFO:tasks.workunit.client.0.smithi028.stdout: 69 497242 1505 368211 1436 100224 44 2024-05-09T08:28:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.707+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3522677811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 msgr2=0x7f2b9c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.707+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3522677811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 0x7f2b9c1006d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009900 tx=0x7f2b9002a410 comp rx=0 tx=0).stop 2024-05-09T08:28:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.708+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3522677811 shutdown_connections 2024-05-09T08:28:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.708+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3522677811 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b9c10e4b0 0x7f2b9c1108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.708+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3522677811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 0x7f2b9c1006d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.708+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3522677811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b9c102930 0x7f2b9c0ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.708+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3522677811 >> 172.21.15.28:0/3522677811 conn(0x7f2b9c076cc0 msgr2=0x7f2b9c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.708+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3522677811 shutdown_connections 2024-05-09T08:28:28.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.708+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3522677811 wait complete. 2024-05-09T08:28:28.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.709+0000 7f2ba0ac3640 1 Processor -- start 2024-05-09T08:28:28.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.709+0000 7f2ba0ac3640 1 -- start start 2024-05-09T08:28:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.709+0000 7f2ba0ac3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 0x7f2b9c19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.709+0000 7f2ba0ac3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b9c102930 0x7f2b9c19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.709+0000 7f2ba0ac3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b9c10e4b0 0x7f2b9c19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.709+0000 7f2ba0ac3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c114010 con 0x7f2b9c100270 2024-05-09T08:28:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.709+0000 7f2ba0ac3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c113e90 con 0x7f2b9c10e4b0 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.709+0000 7f2ba0ac3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c114190 con 0x7f2b9c102930 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 0x7f2b9c19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 0x7f2b9c19a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56888/0 (socket says 172.21.15.28:56888) 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9ad9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b9c10e4b0 0x7f2b9c19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9a59c640 1 -- 172.21.15.28:0/3943461655 learned_addr learned my addr 172.21.15.28:0/3943461655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b9c102930 0x7f2b9c19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9ad9d640 1 -- 172.21.15.28:0/3943461655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b9c102930 msgr2=0x7f2b9c19a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9ad9d640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b9c102930 0x7f2b9c19a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9ad9d640 1 -- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 msgr2=0x7f2b9c19a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9ad9d640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 0x7f2b9c19a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9ad9d640 1 -- 172.21.15.28:0/3943461655 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9c19f400 con 0x7f2b9c10e4b0 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9a59c640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 0x7f2b9c19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b9ad9d640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b9c10e4b0 0x7f2b9c19ecd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400e940 tx=0x7f2b9400ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.710+0000 7f2b8b7fe640 1 -- 172.21.15.28:0/3943461655 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b94004d20 con 0x7f2b9c10e4b0 2024-05-09T08:28:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.711+0000 7f2b8b7fe640 1 -- 172.21.15.28:0/3943461655 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b94005020 con 0x7f2b9c10e4b0 2024-05-09T08:28:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.711+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3943461655 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b9c19f6f0 con 0x7f2b9c10e4b0 2024-05-09T08:28:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.711+0000 7f2b8b7fe640 1 -- 172.21.15.28:0/3943461655 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b940116d0 con 0x7f2b9c10e4b0 2024-05-09T08:28:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.711+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3943461655 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b9c1a6f80 con 0x7f2b9c10e4b0 2024-05-09T08:28:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.713+0000 7f2b8b7fe640 1 -- 172.21.15.28:0/3943461655 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2b940054d0 con 0x7f2b9c10e4b0 2024-05-09T08:28:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.713+0000 7f2ba37cc740 1 Processor -- start 2024-05-09T08:28:28.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2ba37cc740 1 -- start start 2024-05-09T08:28:28.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a07770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a1a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a1d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed6296c340 con 0x55ed62a07390 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ed62914290 con 0x55ed62a07cb0 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ed62968300 con 0x55ed62a1ae40 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b9a59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a1a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b99d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a07770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b9a59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a1a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:55602/0 (socket says 172.21.15.28:55602) 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b9a59c640 1 -- 172.21.15.28:0/1633643164 learned_addr learned my addr 172.21.15.28:0/1633643164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/1633643164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a1d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b9a59c640 1 -- 172.21.15.28:0/1633643164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 msgr2=0x55ed62a1d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b9a59c640 1 --2- 172.21.15.28:0/1633643164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a1d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b8b7fe640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2b840a2da0 0x7f2b840a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b9a59c640 1 -- 172.21.15.28:0/1633643164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 msgr2=0x55ed62a07770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b9a59c640 1 --2- 172.21.15.28:0/1633643164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a07770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.714+0000 7f2b9a59c640 1 -- 172.21.15.28:0/1633643164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed629388e0 con 0x55ed62a07cb0 2024-05-09T08:28:28.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.715+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/1633643164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a1d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:28:28.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.715+0000 7f2b9ad9d640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2b840a2da0 0x7f2b840a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.715+0000 7f2b8b7fe640 1 -- 172.21.15.28:0/3943461655 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f2b940cfb80 con 0x7f2b9c10e4b0 2024-05-09T08:28:28.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.715+0000 7f2b9a59c640 1 --2- 172.21.15.28:0/1633643164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a1a900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c011650 tx=0x7f2b8c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94dd8685e3e1d8fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:28.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.715+0000 7f2b88ff9640 1 -- 172.21.15.28:0/1633643164 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c01e070 con 0x55ed62a07cb0 2024-05-09T08:28:28.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.715+0000 7f2b88ff9640 1 -- 172.21.15.28:0/1633643164 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b8c00fba0 con 0x55ed62a07cb0 2024-05-09T08:28:28.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.715+0000 7f2b88ff9640 1 -- 172.21.15.28:0/1633643164 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c019c70 con 0x55ed62a07cb0 2024-05-09T08:28:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 -- 172.21.15.28:0/1633643164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 msgr2=0x55ed62a1a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/1633643164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a1a900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c011650 tx=0x7f2b8c009f90 comp rx=0 tx=0).stop 2024-05-09T08:28:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2b9ad9d640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2b840a2da0 0x7f2b840a5260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2b940a43b0 tx=0x7f2b9400ef90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 -- 172.21.15.28:0/1633643164 shutdown_connections 2024-05-09T08:28:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/1633643164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a1d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/1633643164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a1a900 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/1633643164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a07770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 -- 172.21.15.28:0/1633643164 >> 172.21.15.28:0/1633643164 conn(0x55ed62a03960 msgr2=0x55ed62a03d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 -- 172.21.15.28:0/1633643164 shutdown_connections 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 -- 172.21.15.28:0/1633643164 wait complete. 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.716+0000 7f2ba37cc740 1 -- start start 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2ba37cc740 1 Processor -- start 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a17d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07cb0 0x55ed62a18290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a187d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed62a1da10 con 0x55ed62a07cb0 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ed62927790 con 0x55ed62a07390 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ed62a1db90 con 0x55ed62a1ae40 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2ba37cc740 10 client.0 _ll_register_callbacks cb 0x55ed62962a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b9a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07cb0 0x55ed62a18290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b99d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a17d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b99d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a17d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:55618/0 (socket says 172.21.15.28:55618) 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b99d9b640 1 -- 172.21.15.28:0/3714889629 learned_addr learned my addr 172.21.15.28:0/3714889629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a187d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b99d9b640 1 -- 172.21.15.28:0/3714889629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 msgr2=0x55ed62a187d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a187d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b99d9b640 1 -- 172.21.15.28:0/3714889629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07cb0 msgr2=0x55ed62a18290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07cb0 0x55ed62a18290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.717+0000 7f2b99d9b640 1 -- 172.21.15.28:0/3714889629 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed62680210 con 0x55ed62a07390 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a17d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b90000c00 tx=0x7f2b900058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50981daaa3282c0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2b81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:28:28.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2b81ffb640 1 -- 172.21.15.28:0/3714889629 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90035040 con 0x55ed62a07390 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3714889629 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b8c00fba0 con 0x55ed62a07390 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2b81ffb640 1 -- 172.21.15.28:0/3714889629 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f2b90031db0 con 0x55ed62a07390 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2b81ffb640 1 -- 172.21.15.28:0/3714889629 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2b90053050 con 0x55ed62a07390 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2ba37cc740 10 client.25865 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3714889629 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ed6274c620 con 0x55ed62a07390 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2b81ffb640 1 -- 172.21.15.28:0/3714889629 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b8c00fba0 con 0x55ed62a07390 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2b81ffb640 1 -- 172.21.15.28:0/3714889629 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f2b9003d6b0 con 0x55ed62a07390 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2ba37cc740 10 client.25865 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2ba37cc740 10 client.25865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.718+0000 7f2ba37cc740 20 client.25865 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.719+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ed62a2ab00 0x55ed62a2cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.719+0000 7f2ba37cc740 4 client.25865 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:28.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.719+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3714889629 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed62979a70 con 0x55ed62a2ab00 2024-05-09T08:28:28.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.719+0000 7f2b9a59c640 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ed62a2ab00 0x55ed62a2cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.719+0000 7f2b81ffb640 10 client.25865 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:28.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.719+0000 7f2b9a59c640 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ed62a2ab00 0x55ed62a2cf60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac95083ea64f1bc8 server_cookie=ca0afaa585728413 in_seq=0 out_seq=0 2024-05-09T08:28:28.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.723+0000 7f2b81ffb640 1 -- 172.21.15.28:0/3714889629 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ed62979a70 con 0x55ed62a2ab00 2024-05-09T08:28:28.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.723+0000 7f2b81ffb640 10 client.25865 handle_command_reply: tid=0 2024-05-09T08:28:28.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.723+0000 7f2ba37cc740 1 client.25865 shutdown 2024-05-09T08:28:28.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3714889629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 msgr2=0x55ed62a17d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:28.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a17d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b90000c00 tx=0x7f2b900058d0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3714889629 shutdown_connections 2024-05-09T08:28:28.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ed62a2ab00 0x55ed62a2cf60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a187d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2b81ffb640 0 client.25865 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:28.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07cb0 0x55ed62a18290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3714889629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a17d50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3714889629 >> 172.21.15.28:0/3714889629 conn(0x55ed62a03960 msgr2=0x55ed62a096e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:28.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3714889629 shutdown_connections 2024-05-09T08:28:28.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3714889629 wait complete. 2024-05-09T08:28:28.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.724+0000 7f2ba37cc740 20 client.25865 trim_cache size 0 max 16384 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.739+0000 7f2ba37cc740 1 Processor -- start 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2ba37cc740 1 -- start start 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a26910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a02170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a17310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1acba0 con 0x55ed62a07390 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1ae7a0 con 0x55ed62a07cb0 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1ad270 con 0x55ed62a1ae40 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b99d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a26910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b9a59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a02170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b99d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a26910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56912/0 (socket says 172.21.15.28:56912) 2024-05-09T08:28:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b99d9b640 1 -- 172.21.15.28:0/3758730321 learned_addr learned my addr 172.21.15.28:0/3758730321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:28.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b99d9b640 1 -- 172.21.15.28:0/3758730321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 msgr2=0x55ed62a17310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:28.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/3758730321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a17310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b99d9b640 1 -- 172.21.15.28:0/3758730321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 msgr2=0x55ed62a02170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:28.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/3758730321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a02170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b99d9b640 1 -- 172.21.15.28:0/3758730321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed62a07ae0 con 0x55ed62a07390 2024-05-09T08:28:28.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.740+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/3758730321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a26910 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006010 tx=0x7f2b90009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd6c069f88d426 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.741+0000 7f2b83fff640 1 -- 172.21.15.28:0/3758730321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90035070 con 0x55ed62a07390 2024-05-09T08:28:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.741+0000 7f2b83fff640 1 -- 172.21.15.28:0/3758730321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b90005020 con 0x55ed62a07390 2024-05-09T08:28:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.741+0000 7f2b83fff640 1 -- 172.21.15.28:0/3758730321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b900049b0 con 0x55ed62a07390 2024-05-09T08:28:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.742+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3758730321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 msgr2=0x55ed62a26910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.742+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3758730321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a26910 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006010 tx=0x7f2b90009630 comp rx=0 tx=0).stop 2024-05-09T08:28:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.742+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3758730321 shutdown_connections 2024-05-09T08:28:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.742+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3758730321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a1ae40 0x55ed62a17310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.742+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3758730321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07cb0 0x55ed62a02170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.742+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/3758730321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a07390 0x55ed62a26910 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.742+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3758730321 >> 172.21.15.28:0/3758730321 conn(0x55ed62a03960 msgr2=0x55ed62a185a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.742+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3758730321 shutdown_connections 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.742+0000 7f2ba37cc740 1 -- 172.21.15.28:0/3758730321 wait complete. 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2ba37cc740 1 -- start start 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2ba37cc740 1 Processor -- start 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a23270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a07cb0 0x55ed62a14810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2ba37cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a1ae40 0x55ed62a14d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1ae010 con 0x55ed62a1ae40 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1ad940 con 0x55ed62a07390 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2ba37cc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1aab80 con 0x55ed62a07cb0 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2ba37cc740 10 client.0 _ll_register_callbacks cb 0x7f2b90005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2b99d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a23270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2b99d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a1ae40 0x55ed62a14d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2b99d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a1ae40 0x55ed62a14d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56916/0 (socket says 172.21.15.28:56916) 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2b9a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a07cb0 0x55ed62a14810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.743+0000 7f2b99d9b640 1 -- 172.21.15.28:0/2000997615 learned_addr learned my addr 172.21.15.28:0/2000997615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b99d9b640 1 -- 172.21.15.28:0/2000997615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a07cb0 msgr2=0x55ed62a14810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a07cb0 0x55ed62a14810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b99d9b640 1 -- 172.21.15.28:0/2000997615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 msgr2=0x55ed62a23270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a23270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b99d9b640 1 -- 172.21.15.28:0/2000997615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed62a155a0 con 0x55ed62a1ae40 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b99d9b640 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a1ae40 0x55ed62a14d70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2b9003dc20 tx=0x7f2b9002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02030fda26347c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b81ffb640 1 -- 172.21.15.28:0/2000997615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90035070 con 0x55ed62a1ae40 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b81ffb640 1 -- 172.21.15.28:0/2000997615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f2b90004050 con 0x55ed62a1ae40 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2ba37cc740 1 -- 172.21.15.28:0/2000997615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b90005020 con 0x55ed62a1ae40 2024-05-09T08:28:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2b81ffb640 1 -- 172.21.15.28:0/2000997615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2b90066050 con 0x55ed62a1ae40 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2ba37cc740 10 client.16398 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.744+0000 7f2ba37cc740 1 -- 172.21.15.28:0/2000997615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ed62a91140 con 0x55ed62a1ae40 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2b81ffb640 1 -- 172.21.15.28:0/2000997615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b90005020 con 0x55ed62a1ae40 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2b81ffb640 1 -- 172.21.15.28:0/2000997615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f2b900035f0 con 0x55ed62a1ae40 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2ba37cc740 10 client.16398 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2ba37cc740 10 client.16398 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2ba37cc740 20 client.16398 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ed62a91860 0x55ed62a93c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2ba37cc740 4 client.16398 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2ba37cc740 1 -- 172.21.15.28:0/2000997615 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ed62973dc0 con 0x55ed62a91860 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2b9a59c640 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ed62a91860 0x55ed62a93c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.745+0000 7f2b81ffb640 10 client.16398 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:28.746+0000 7f2b9a59c640 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ed62a91860 0x55ed62a93c80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eae0e3147938e47d server_cookie=a309fb183c62915d in_seq=0 out_seq=0 2024-05-09T08:28:29.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:29 smithi028 ceph-mon[30119]: pgmap v68: 129 pgs: 129 active+clean; 120 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 515 op/s 2024-05-09T08:28:29.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:29 smithi165 ceph-mon[34540]: pgmap v68: 129 pgs: 129 active+clean; 120 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 515 op/s 2024-05-09T08:28:29.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:29 smithi184 ceph-mon[34509]: pgmap v68: 129 pgs: 129 active+clean; 120 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 515 op/s 2024-05-09T08:28:31.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:31 smithi028 ceph-mon[30119]: pgmap v69: 129 pgs: 129 active+clean; 127 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 544 op/s 2024-05-09T08:28:31.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:31 smithi165 ceph-mon[34540]: pgmap v69: 129 pgs: 129 active+clean; 127 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 544 op/s 2024-05-09T08:28:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:31 smithi184 ceph-mon[34509]: pgmap v69: 129 pgs: 129 active+clean; 127 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 544 op/s 2024-05-09T08:28:33.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:33 smithi165 ceph-mon[34540]: pgmap v70: 129 pgs: 129 active+clean; 127 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-05-09T08:28:33.659 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:33 smithi184 ceph-mon[34509]: pgmap v70: 129 pgs: 129 active+clean; 127 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-05-09T08:28:33.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:33 smithi028 ceph-mon[30119]: pgmap v70: 129 pgs: 129 active+clean; 127 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-05-09T08:28:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2b81ffb640 1 -- 172.21.15.28:0/2000997615 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ed62973dc0 con 0x55ed62a91860 2024-05-09T08:28:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2b81ffb640 10 client.16398 handle_command_reply: tid=0 2024-05-09T08:28:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 client.16398 shutdown 2024-05-09T08:28:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 -- 172.21.15.28:0/2000997615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a1ae40 msgr2=0x55ed62a14d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a1ae40 0x55ed62a14d70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2b9003dc20 tx=0x7f2b9002a410 comp rx=0 tx=0).stop 2024-05-09T08:28:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 -- 172.21.15.28:0/2000997615 shutdown_connections 2024-05-09T08:28:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ed62a91860 0x55ed62a93c80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed62a1ae40 0x55ed62a14d70 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ed62a07cb0 0x55ed62a14810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2b81ffb640 0 client.16398 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 --2- 172.21.15.28:0/2000997615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ed62a07390 0x55ed62a23270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 -- 172.21.15.28:0/2000997615 >> 172.21.15.28:0/2000997615 conn(0x55ed62a03960 msgr2=0x55ed62a08e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 -- 172.21.15.28:0/2000997615 shutdown_connections 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 1 -- 172.21.15.28:0/2000997615 wait complete. 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.740+0000 7f2ba37cc740 20 client.16398 trim_cache size 0 max 16384 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:28:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:28:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2b840a2da0 msgr2=0x7f2b840a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2b840a2da0 0x7f2b840a5260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2b940a43b0 tx=0x7f2b9400ef90 comp rx=0 tx=0).stop 2024-05-09T08:28:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3943461655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b9c10e4b0 msgr2=0x7f2b9c19ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b9c10e4b0 0x7f2b9c19ecd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400e940 tx=0x7f2b9400ee00 comp rx=0 tx=0).stop 2024-05-09T08:28:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3943461655 shutdown_connections 2024-05-09T08:28:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b9c10e4b0 0x7f2b9c19ecd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2b840a2da0 0x7f2b840a5260 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2b9c102930 0x7f2b9c19a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 --2- 172.21.15.28:0/3943461655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9c100270 0x7f2b9c19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3943461655 >> 172.21.15.28:0/3943461655 conn(0x7f2b9c076cc0 msgr2=0x7f2b9c10f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3943461655 shutdown_connections 2024-05-09T08:28:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:33.741+0000 7f2ba0ac3640 1 -- 172.21.15.28:0/3943461655 wait complete. 2024-05-09T08:28:34.117 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:28:34.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.241+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/1132682213 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f48106710 msgr2=0x7f3f48106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.241+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/1132682213 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f48106710 0x7f3f48106af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009900 tx=0x7f3f3802a410 comp rx=0 tx=0).stop 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.241+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/1132682213 shutdown_connections 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.241+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/1132682213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f481010c0 0x7f3f4810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.241+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/1132682213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f48100700 0x7f3f48100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.241+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/1132682213 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f48106710 0x7f3f48106af0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.241+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/1132682213 >> 172.21.15.28:0/1132682213 conn(0x7f3f48077830 msgr2=0x7f3f48077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.241+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/1132682213 shutdown_connections 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.241+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/1132682213 wait complete. 2024-05-09T08:28:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.242+0000 7f3f4d2b3640 1 Processor -- start 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.242+0000 7f3f4d2b3640 1 -- start start 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.242+0000 7f3f4d2b3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f48100700 0x7f3f4819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.242+0000 7f3f4d2b3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f481010c0 0x7f3f4819a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.242+0000 7f3f4d2b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48106710 0x7f3f4819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.242+0000 7f3f4d2b3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f48110b20 con 0x7f3f48106710 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.242+0000 7f3f4d2b3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481109a0 con 0x7f3f48100700 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.242+0000 7f3f4d2b3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f48110ca0 con 0x7f3f481010c0 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48106710 0x7f3f4819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f46ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f48100700 0x7f3f4819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48106710 0x7f3f4819ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56942/0 (socket says 172.21.15.28:56942) 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f46ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f48100700 0x7f3f4819a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:55666/0 (socket says 172.21.15.28:55666) 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f477fe640 1 -- 172.21.15.28:0/4267560283 learned_addr learned my addr 172.21.15.28:0/4267560283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f467fc640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f481010c0 0x7f3f4819a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f477fe640 1 -- 172.21.15.28:0/4267560283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f481010c0 msgr2=0x7f3f4819a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f477fe640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f481010c0 0x7f3f4819a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f477fe640 1 -- 172.21.15.28:0/4267560283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f48100700 msgr2=0x7f3f4819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f477fe640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f48100700 0x7f3f4819a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f477fe640 1 -- 172.21.15.28:0/4267560283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4819f2f0 con 0x7f3f48106710 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f477fe640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48106710 0x7f3f4819ebc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000efd0 tx=0x7f3f4000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f2ffff640 1 -- 172.21.15.28:0/4267560283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40003b70 con 0x7f3f48106710 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f2ffff640 1 -- 172.21.15.28:0/4267560283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f40003d10 con 0x7f3f48106710 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/4267560283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4819f5e0 con 0x7f3f48106710 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f2ffff640 1 -- 172.21.15.28:0/4267560283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f400109f0 con 0x7f3f48106710 2024-05-09T08:28:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.243+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/4267560283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f481a6e70 con 0x7f3f48106710 2024-05-09T08:28:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.247+0000 7f3f2ffff640 1 -- 172.21.15.28:0/4267560283 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3f40004020 con 0x7f3f48106710 2024-05-09T08:28:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.247+0000 7f3f4ffc4740 1 Processor -- start 2024-05-09T08:28:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f4ffc4740 1 -- start start 2024-05-09T08:28:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa49660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa53d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa56660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a76f9aed00 con 0x55a76fa54270 2024-05-09T08:28:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a76f9564c0 con 0x55a76fa49ba0 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a76f9aacc0 con 0x55a76fa49280 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f467fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa49660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f46ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa53d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f46ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa53d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:55672/0 (socket says 172.21.15.28:55672) 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f467fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa49660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:49260/0 (socket says 172.21.15.28:49260) 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f46ffd640 1 -- 172.21.15.28:0/3346406028 learned_addr learned my addr 172.21.15.28:0/3346406028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f467fc640 1 --2- 172.21.15.28:0/3346406028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa56660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f2ffff640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3f300a2d80 0x7f3f300a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f46ffd640 1 -- 172.21.15.28:0/3346406028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 msgr2=0x55a76fa49660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/3346406028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa49660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f46ffd640 1 -- 172.21.15.28:0/3346406028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 msgr2=0x55a76fa56660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/3346406028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa56660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f46ffd640 1 -- 172.21.15.28:0/3346406028 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a76f78c7d0 con 0x55a76fa49ba0 2024-05-09T08:28:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f477fe640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3f300a2d80 0x7f3f300a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.248+0000 7f3f467fc640 1 --2- 172.21.15.28:0/3346406028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa49660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.249+0000 7f3f2ffff640 1 -- 172.21.15.28:0/4267560283 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3f400cfb30 con 0x7f3f48106710 2024-05-09T08:28:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.249+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/3346406028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa53d30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3f3802a930 tx=0x7f3f38005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=376c1c6c57d5f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.249+0000 7f3f477fe640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3f300a2d80 0x7f3f300a5240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f40098650 tx=0x7f3f40084fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.249+0000 7f3f2d7fa640 1 -- 172.21.15.28:0/3346406028 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38035030 con 0x55a76fa49ba0 2024-05-09T08:28:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.249+0000 7f3f2d7fa640 1 -- 172.21.15.28:0/3346406028 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f38040070 con 0x55a76fa49ba0 2024-05-09T08:28:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.249+0000 7f3f2d7fa640 1 -- 172.21.15.28:0/3346406028 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38031860 con 0x55a76fa49ba0 2024-05-09T08:28:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3346406028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 msgr2=0x55a76fa53d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3346406028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa53d30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3f3802a930 tx=0x7f3f38005dd0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3346406028 shutdown_connections 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3346406028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa56660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3346406028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa53d30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3346406028 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa49660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3346406028 >> 172.21.15.28:0/3346406028 conn(0x55a76fa455a0 msgr2=0x55a76fa45980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3346406028 shutdown_connections 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3346406028 wait complete. 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 -- start start 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.250+0000 7f3f4ffc4740 1 Processor -- start 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa51210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 0x55a76fa51750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 0x55a76fa51c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a76fa57000 con 0x55a76fa49280 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a76f969b90 con 0x55a76fa54270 2024-05-09T08:28:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a76fa57180 con 0x55a76fa49ba0 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f4ffc4740 10 client.0 _ll_register_callbacks cb 0x55a76f9a5da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa51210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa51210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56970/0 (socket says 172.21.15.28:56970) 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 -- 172.21.15.28:0/1536579134 learned_addr learned my addr 172.21.15.28:0/1536579134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 0x55a76fa51750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 0x55a76fa51c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 -- 172.21.15.28:0/1536579134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 msgr2=0x55a76fa51750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 0x55a76fa51750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 -- 172.21.15.28:0/1536579134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 msgr2=0x55a76fa51c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 0x55a76fa51c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 -- 172.21.15.28:0/1536579134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a76f6c23f0 con 0x55a76fa49280 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f467fc640 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa51210 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c004470 tx=0x7f3f3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54340e7cfa77841a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f2aff5640 1 -- 172.21.15.28:0/1536579134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c005220 con 0x55a76fa49280 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f2aff5640 1 -- 172.21.15.28:0/1536579134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3f3c0053c0 con 0x55a76fa49280 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.251+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/1536579134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f38040070 con 0x55a76fa49280 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f2aff5640 1 -- 172.21.15.28:0/1536579134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3f3c02a050 con 0x55a76fa49280 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f4ffc4740 10 client.16410 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/1536579134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a76f6820f0 con 0x55a76fa49280 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f2aff5640 1 -- 172.21.15.28:0/1536579134 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f38040070 con 0x55a76fa49280 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f2aff5640 1 -- 172.21.15.28:0/1536579134 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f3f3c0118d0 con 0x55a76fa49280 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f4ffc4740 10 client.16410 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f4ffc4740 10 client.16410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f4ffc4740 20 client.16410 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a76fa646e0 0x55a76fa66b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f4ffc4740 4 client.16410 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/1536579134 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a76f9bc430 con 0x55a76fa646e0 2024-05-09T08:28:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.252+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a76fa646e0 0x55a76fa66b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.253+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a76fa646e0 0x55a76fa66b40 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f051c411cefbabd9 server_cookie=84cb292b58cd3b2f in_seq=0 out_seq=0 2024-05-09T08:28:34.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.253+0000 7f3f2aff5640 10 client.16410 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f2aff5640 1 -- 172.21.15.28:0/1536579134 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a76f9bc430 con 0x55a76fa646e0 2024-05-09T08:28:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f2aff5640 10 client.16410 handle_command_reply: tid=0 2024-05-09T08:28:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 client.16410 shutdown 2024-05-09T08:28:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/1536579134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 msgr2=0x55a76fa51210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa51210 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c004470 tx=0x7f3f3c004730 comp rx=0 tx=0).stop 2024-05-09T08:28:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/1536579134 shutdown_connections 2024-05-09T08:28:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a76fa646e0 0x55a76fa66b40 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 0x55a76fa51c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 0x55a76fa51750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/1536579134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa51210 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f2aff5640 0 client.16410 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/1536579134 >> 172.21.15.28:0/1536579134 conn(0x55a76fa455a0 msgr2=0x55a76fa4b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/1536579134 shutdown_connections 2024-05-09T08:28:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/1536579134 wait complete. 2024-05-09T08:28:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.255+0000 7f3f4ffc4740 20 client.16410 trim_cache size 0 max 16384 2024-05-09T08:28:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f4ffc4740 1 Processor -- start 2024-05-09T08:28:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f4ffc4740 1 -- start start 2024-05-09T08:28:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa5fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa4ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa44660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481b0ff0 con 0x55a76fa54270 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481b2bf0 con 0x55a76fa49ba0 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481b16c0 con 0x55a76fa49280 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f467fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa44660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f467fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa44660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56972/0 (socket says 172.21.15.28:56972) 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f467fc640 1 -- 172.21.15.28:0/2915035477 learned_addr learned my addr 172.21.15.28:0/2915035477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/2915035477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa4ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f467fc640 1 --2- 172.21.15.28:0/2915035477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa5fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f467fc640 1 -- 172.21.15.28:0/2915035477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 msgr2=0x55a76fa5fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f467fc640 1 --2- 172.21.15.28:0/2915035477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa5fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f467fc640 1 -- 172.21.15.28:0/2915035477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 msgr2=0x55a76fa4ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f467fc640 1 --2- 172.21.15.28:0/2915035477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa4ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.276+0000 7f3f467fc640 1 -- 172.21.15.28:0/2915035477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a76fa453d0 con 0x55a76fa54270 2024-05-09T08:28:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.277+0000 7f3f467fc640 1 --2- 172.21.15.28:0/2915035477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa44660 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c001eb0 tx=0x7f3f3c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b8ce4858fb1264 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.277+0000 7f3f2cff9640 1 -- 172.21.15.28:0/2915035477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c011030 con 0x55a76fa54270 2024-05-09T08:28:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.277+0000 7f3f2cff9640 1 -- 172.21.15.28:0/2915035477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f3c0111d0 con 0x55a76fa54270 2024-05-09T08:28:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.277+0000 7f3f2cff9640 1 -- 172.21.15.28:0/2915035477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c010430 con 0x55a76fa54270 2024-05-09T08:28:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/2915035477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 msgr2=0x55a76fa44660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/2915035477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa44660 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c001eb0 tx=0x7f3f3c005f90 comp rx=0 tx=0).stop 2024-05-09T08:28:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/2915035477 shutdown_connections 2024-05-09T08:28:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/2915035477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa54270 0x55a76fa44660 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/2915035477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa49ba0 0x55a76fa4ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/2915035477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49280 0x55a76fa5fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/2915035477 >> 172.21.15.28:0/2915035477 conn(0x55a76fa455a0 msgr2=0x55a76fa43f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/2915035477 shutdown_connections 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/2915035477 wait complete. 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 -- start start 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 Processor -- start 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa5c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.278+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 0x55a76fa5c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f4ffc4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 0x55a76fa57b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481b2460 con 0x55a76fa49280 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481b1d90 con 0x55a76fa54270 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f4ffc4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481aefd0 con 0x55a76fa49ba0 2024-05-09T08:28:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f4ffc4740 10 client.0 _ll_register_callbacks cb 0x7f3f3c004260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa5c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa5c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56978/0 (socket says 172.21.15.28:56978) 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 -- 172.21.15.28:0/3858234275 learned_addr learned my addr 172.21.15.28:0/3858234275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 0x55a76fa5c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 0x55a76fa57b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 -- 172.21.15.28:0/3858234275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 msgr2=0x55a76fa5c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 0x55a76fa5c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 -- 172.21.15.28:0/3858234275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 msgr2=0x55a76fa57b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 0x55a76fa57b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 -- 172.21.15.28:0/3858234275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a76fa58330 con 0x55a76fa49280 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 0x55a76fa5c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f467fc640 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa5c450 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c005bf0 tx=0x7f3f3c021b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e82e9eeef8ea5b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f2aff5640 1 -- 172.21.15.28:0/3858234275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c021dc0 con 0x55a76fa49280 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f2aff5640 1 -- 172.21.15.28:0/3858234275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f3f3c011740 con 0x55a76fa49280 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3858234275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f3c0111d0 con 0x55a76fa49280 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f2aff5640 1 -- 172.21.15.28:0/3858234275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3f3c011940 con 0x55a76fa49280 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f4ffc4740 10 client.16416 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.279+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3858234275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a76faca920 con 0x55a76fa49280 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f2aff5640 1 -- 172.21.15.28:0/3858234275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f3c0111d0 con 0x55a76fa49280 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f2aff5640 1 -- 172.21.15.28:0/3858234275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f3f3c030ba0 con 0x55a76fa49280 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f4ffc4740 10 client.16416 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f4ffc4740 10 client.16416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f4ffc4740 20 client.16416 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a76facb080 0x55a76facd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f4ffc4740 4 client.16416 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3858234275 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a76f9b6780 con 0x55a76facb080 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a76facb080 0x55a76facd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f2aff5640 10 client.16416 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.280+0000 7f3f46ffd640 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a76facb080 0x55a76facd4a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f6d918e8a76a server_cookie=d475930e9440581c in_seq=0 out_seq=0 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f2aff5640 1 -- 172.21.15.28:0/3858234275 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a76f9b6780 con 0x55a76facb080 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f2aff5640 10 client.16416 handle_command_reply: tid=0 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 client.16416 shutdown 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3858234275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 msgr2=0x55a76fa5c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa5c450 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c005bf0 tx=0x7f3f3c021b20 comp rx=0 tx=0).stop 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3858234275 shutdown_connections 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a76facb080 0x55a76facd4a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a76fa54270 0x55a76fa57b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a76fa49ba0 0x55a76fa5c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 --2- 172.21.15.28:0/3858234275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a76fa49280 0x55a76fa5c450 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f2aff5640 0 client.16416 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3858234275 >> 172.21.15.28:0/3858234275 conn(0x55a76fa455a0 msgr2=0x55a76fa51a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3858234275 shutdown_connections 2024-05-09T08:28:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 1 -- 172.21.15.28:0/3858234275 wait complete. 2024-05-09T08:28:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.288+0000 7f3f4ffc4740 20 client.16416 trim_cache size 0 max 16384 2024-05-09T08:28:34.290 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:28:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-09T08:28:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-09T08:28:34.290 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:28:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3f300a2d80 msgr2=0x7f3f300a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3f300a2d80 0x7f3f300a5240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f40098650 tx=0x7f3f40084fd0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48106710 msgr2=0x7f3f4819ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48106710 0x7f3f4819ebc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000efd0 tx=0x7f3f4000c6a0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/4267560283 shutdown_connections 2024-05-09T08:28:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48106710 0x7f3f4819ebc0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f3f300a2d80 0x7f3f300a5240 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f481010c0 0x7f3f4819a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 --2- 172.21.15.28:0/4267560283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f48100700 0x7f3f4819a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/4267560283 >> 172.21.15.28:0/4267560283 conn(0x7f3f48077830 msgr2=0x7f3f480689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/4267560283 shutdown_connections 2024-05-09T08:28:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.289+0000 7f3f4d2b3640 1 -- 172.21.15.28:0/4267560283 wait complete. 2024-05-09T08:28:34.301 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:28:34.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:28:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.423+0000 7f3353c9a640 1 -- 172.21.15.28:0/1404267617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 msgr2=0x7f334c06a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.423+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1404267617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 0x7f334c06a060 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f333c009930 tx=0x7f333c02a420 comp rx=0 tx=0).stop 2024-05-09T08:28:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.423+0000 7f3353c9a640 1 -- 172.21.15.28:0/1404267617 shutdown_connections 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.423+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1404267617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f334c06a5a0 0x7f334c10e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.423+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1404267617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 0x7f334c06a060 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.423+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1404267617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f334c107820 0x7f334c107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.423+0000 7f3353c9a640 1 -- 172.21.15.28:0/1404267617 >> 172.21.15.28:0/1404267617 conn(0x7f334c076800 msgr2=0x7f334c076c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.423+0000 7f3353c9a640 1 -- 172.21.15.28:0/1404267617 shutdown_connections 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.423+0000 7f3353c9a640 1 -- 172.21.15.28:0/1404267617 wait complete. 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3353c9a640 1 Processor -- start 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3353c9a640 1 -- start start 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3353c9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 0x7f334c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3353c9a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f334c06a5a0 0x7f334c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3353c9a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f334c107820 0x7f334c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3353c9a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f334c111c60 con 0x7f334c069c00 2024-05-09T08:28:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3353c9a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f334c111ae0 con 0x7f334c107820 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3353c9a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f334c111de0 con 0x7f334c06a5a0 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3351a36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 0x7f334c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3351a36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 0x7f334c19a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57000/0 (socket says 172.21.15.28:57000) 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3351a36640 1 -- 172.21.15.28:0/1364260501 learned_addr learned my addr 172.21.15.28:0/1364260501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3351235640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f334c06a5a0 0x7f334c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3352237640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f334c107820 0x7f334c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3351a36640 1 -- 172.21.15.28:0/1364260501 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f334c06a5a0 msgr2=0x7f334c19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3351a36640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f334c06a5a0 0x7f334c19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3351a36640 1 -- 172.21.15.28:0/1364260501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f334c107820 msgr2=0x7f334c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3351a36640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f334c107820 0x7f334c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.424+0000 7f3351a36640 1 -- 172.21.15.28:0/1364260501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334c19f370 con 0x7f334c069c00 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.425+0000 7f3351a36640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 0x7f334c19a380 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f334800e9f0 tx=0x7f334800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.425+0000 7f3342ffd640 1 -- 172.21.15.28:0/1364260501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334800cd80 con 0x7f334c069c00 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.425+0000 7f3342ffd640 1 -- 172.21.15.28:0/1364260501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3348005500 con 0x7f334c069c00 2024-05-09T08:28:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.425+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f334c19f660 con 0x7f334c069c00 2024-05-09T08:28:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.425+0000 7f3342ffd640 1 -- 172.21.15.28:0/1364260501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3348010690 con 0x7f334c069c00 2024-05-09T08:28:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.425+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f334c1a6ef0 con 0x7f334c069c00 2024-05-09T08:28:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.426+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f334c067dc0 con 0x7f334c069c00 2024-05-09T08:28:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.427+0000 7f3342ffd640 1 -- 172.21.15.28:0/1364260501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f33480046c0 con 0x7f334c069c00 2024-05-09T08:28:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.428+0000 7f3342ffd640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f33380a2d00 0x7f33380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.428+0000 7f3351235640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f33380a2d00 0x7f33380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.428+0000 7f3342ffd640 1 -- 172.21.15.28:0/1364260501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f33480cfdb0 con 0x7f334c069c00 2024-05-09T08:28:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.429+0000 7f3351235640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f33380a2d00 0x7f33380a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f333c009900 tx=0x7f333c0058d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.433+0000 7f3342ffd640 1 -- 172.21.15.28:0/1364260501 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3348053740 con 0x7f334c069c00 2024-05-09T08:28:34.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.745+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f334c19b6d0 con 0x7f334c069c00 2024-05-09T08:28:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.752+0000 7f3342ffd640 1 -- 172.21.15.28:0/1364260501 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11939 (secure 0 0 0) 0x7f334c19b6d0 con 0x7f334c069c00 2024-05-09T08:28:34.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:28:34.754 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3978374239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3978374239},{"type":"v1","addr":"172.21.15.28:6837","nonce":3978374239}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:28:10.428132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/3736338591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":3736338591},{"type":"v1","addr":"172.21.15.165:6835","nonce":3736338591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":72,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":70,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":71,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":69,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24512,24463,24476,24499,24481]},"id":1}]} 2024-05-09T08:28:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-05-09T08:28:34.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.757+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f33380a2d00 msgr2=0x7f33380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.757+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f33380a2d00 0x7f33380a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f333c009900 tx=0x7f333c0058d0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.757+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 msgr2=0x7f334c19a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.757+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 0x7f334c19a380 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f334800e9f0 tx=0x7f334800edf0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.758+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 shutdown_connections 2024-05-09T08:28:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.758+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f334c107820 0x7f334c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.758+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f33380a2d00 0x7f33380a51c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.758+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f334c06a5a0 0x7f334c19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.758+0000 7f3353c9a640 1 --2- 172.21.15.28:0/1364260501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c069c00 0x7f334c19a380 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.758+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 >> 172.21.15.28:0/1364260501 conn(0x7f334c076800 msgr2=0x7f334c10cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.759+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 shutdown_connections 2024-05-09T08:28:34.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.760+0000 7f3353c9a640 1 -- 172.21.15.28:0/1364260501 wait complete. 2024-05-09T08:28:34.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-05-09T08:28:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.979+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/2608179258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d01033d0 msgr2=0x7fc1d010b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.979+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/2608179258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d01033d0 0x7fc1d010b8b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009900 tx=0x7fc1c002a560 comp rx=0 tx=0).stop 2024-05-09T08:28:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.980+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/2608179258 shutdown_connections 2024-05-09T08:28:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.980+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/2608179258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d010bdf0 0x7fc1d010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.980+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/2608179258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d01033d0 0x7fc1d010b8b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.980+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/2608179258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1d0102ab0 0x7fc1d0102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.980+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/2608179258 >> 172.21.15.28:0/2608179258 conn(0x7fc1d0077830 msgr2=0x7fc1d0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.981+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/2608179258 shutdown_connections 2024-05-09T08:28:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.981+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/2608179258 wait complete. 2024-05-09T08:28:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.982+0000 7fc1d4a5d640 1 Processor -- start 2024-05-09T08:28:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.982+0000 7fc1d4a5d640 1 -- start start 2024-05-09T08:28:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1d4a5d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d0102ab0 0x7fc1d019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1d4a5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1d01033d0 0x7fc1d019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1d4a5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d010bdf0 0x7fc1d0107b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d0102ab0 0x7fc1d019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d0102ab0 0x7fc1d019ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:53588/0 (socket says 172.21.15.28:53588) 2024-05-09T08:28:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1ce59c640 1 -- 172.21.15.28:0/10776471 learned_addr learned my addr 172.21.15.28:0/10776471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0111cd0 con 0x7fc1d01033d0 2024-05-09T08:28:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0111b50 con 0x7fc1d010bdf0 2024-05-09T08:28:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0111e50 con 0x7fc1d0102ab0 2024-05-09T08:28:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.983+0000 7fc1ced9d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d010bdf0 0x7fc1d0107b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.984+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1d01033d0 0x7fc1d019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.984+0000 7fc1ced9d640 1 -- 172.21.15.28:0/10776471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d0102ab0 msgr2=0x7fc1d019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.984+0000 7fc1ced9d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d0102ab0 0x7fc1d019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.984+0000 7fc1ced9d640 1 -- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1d01033d0 msgr2=0x7fc1d019f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.984+0000 7fc1ced9d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1d01033d0 0x7fc1d019f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.984+0000 7fc1ced9d640 1 -- 172.21.15.28:0/10776471 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d01083a0 con 0x7fc1d010bdf0 2024-05-09T08:28:34.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.984+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d0102ab0 0x7fc1d019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:34.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.984+0000 7fc1ced9d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d010bdf0 0x7fc1d0107b10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800be20 tx=0x7fc1c800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.985+0000 7fc1bf7fe640 1 -- 172.21.15.28:0/10776471 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8019070 con 0x7fc1d010bdf0 2024-05-09T08:28:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.985+0000 7fc1bf7fe640 1 -- 172.21.15.28:0/10776471 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c8014920 con 0x7fc1d010bdf0 2024-05-09T08:28:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.985+0000 7fc1bf7fe640 1 -- 172.21.15.28:0/10776471 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8012920 con 0x7fc1d010bdf0 2024-05-09T08:28:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.985+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d0108690 con 0x7fc1d010bdf0 2024-05-09T08:28:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.985+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d019aab0 con 0x7fc1d010bdf0 2024-05-09T08:28:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.989+0000 7fc1d776b740 1 Processor -- start 2024-05-09T08:28:34.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.991+0000 7fc1bf7fe640 1 -- 172.21.15.28:0/10776471 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc1c801c070 con 0x7fc1d010bdf0 2024-05-09T08:28:34.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.991+0000 7fc1d776b740 1 -- start start 2024-05-09T08:28:34.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.992+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007cb330 0x55a3007cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.992+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.992+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 0x55a3007d8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.992+0000 7fc1d776b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a300730c40 con 0x55a3007cbc50 2024-05-09T08:28:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.992+0000 7fc1d776b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a3006d8580 con 0x55a3007d6320 2024-05-09T08:28:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.992+0000 7fc1d776b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a30072cc00 con 0x55a3007cb330 2024-05-09T08:28:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.992+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007cb330 0x55a3007cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.993+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.993+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35666/0 (socket says 172.21.15.28:35666) 2024-05-09T08:28:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.993+0000 7fc1ce59c640 1 -- 172.21.15.28:0/555742499 learned_addr learned my addr 172.21.15.28:0/555742499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.993+0000 7fc1bf7fe640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b80a2ed0 0x7fc1b80a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.993+0000 7fc1ced9d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b80a2ed0 0x7fc1b80a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.993+0000 7fc1bf7fe640 1 -- 172.21.15.28:0/10776471 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fc1c80ceeb0 con 0x7fc1d010bdf0 2024-05-09T08:28:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.994+0000 7fc1ce59c640 1 -- 172.21.15.28:0/555742499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007cb330 msgr2=0x55a3007cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.994+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/555742499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007cb330 0x55a3007cb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.994+0000 7fc1ce59c640 1 -- 172.21.15.28:0/555742499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 msgr2=0x55a3007d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.994+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/555742499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 0x55a3007d8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.994+0000 7fc1ce59c640 1 -- 172.21.15.28:0/555742499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3004443f0 con 0x55a3007cbc50 2024-05-09T08:28:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.994+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/555742499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007cb330 0x55a3007cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:28:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.995+0000 7fc1ced9d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b80a2ed0 0x7fc1b80a5390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc1c809ae60 tx=0x7fc1c800f300 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.995+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/555742499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d5de0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009ed0 tx=0x7fc1c400e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efeda933f84b886 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.996+0000 7fc1bcff9640 1 -- 172.21.15.28:0/555742499 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4017070 con 0x55a3007cbc50 2024-05-09T08:28:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.996+0000 7fc1bcff9640 1 -- 172.21.15.28:0/555742499 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c4013950 con 0x55a3007cbc50 2024-05-09T08:28:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.996+0000 7fc1bcff9640 1 -- 172.21.15.28:0/555742499 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4012a70 con 0x55a3007cbc50 2024-05-09T08:28:34.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.997+0000 7fc1d776b740 1 -- 172.21.15.28:0/555742499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 msgr2=0x55a3007d5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:34.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.997+0000 7fc1d776b740 1 --2- 172.21.15.28:0/555742499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d5de0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009ed0 tx=0x7fc1c400e5a0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.997+0000 7fc1d776b740 1 -- 172.21.15.28:0/555742499 shutdown_connections 2024-05-09T08:28:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.997+0000 7fc1d776b740 1 --2- 172.21.15.28:0/555742499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 0x55a3007d8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.997+0000 7fc1d776b740 1 --2- 172.21.15.28:0/555742499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d5de0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.997+0000 7fc1d776b740 1 --2- 172.21.15.28:0/555742499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007cb330 0x55a3007cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.997+0000 7fc1d776b740 1 -- 172.21.15.28:0/555742499 >> 172.21.15.28:0/555742499 conn(0x55a3007c7650 msgr2=0x55a3007c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.998+0000 7fc1d776b740 1 -- 172.21.15.28:0/555742499 shutdown_connections 2024-05-09T08:28:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.998+0000 7fc1d776b740 1 -- 172.21.15.28:0/555742499 wait complete. 2024-05-09T08:28:34.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.998+0000 7fc1d776b740 1 -- start start 2024-05-09T08:28:34.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.998+0000 7fc1d776b740 1 Processor -- start 2024-05-09T08:28:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.999+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cb330 0x55a3007d3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.999+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 0x55a3007d37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.999+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cb330 0x55a3007d3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:34.999+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cb330 0x55a3007d3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35670/0 (socket says 172.21.15.28:35670) 2024-05-09T08:28:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.000+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/3083557654 learned_addr learned my addr 172.21.15.28:0/3083557654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.000+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 0x55a3007d3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.000+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a3007d9110 con 0x55a3007cb330 2024-05-09T08:28:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.000+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a3006ebc40 con 0x55a3007cbc50 2024-05-09T08:28:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.000+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a3007d9290 con 0x55a3007d6320 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.000+0000 7fc1d776b740 10 client.0 _ll_register_callbacks cb 0x55a300443be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.000+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 0x55a3007d3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.001+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/3083557654 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 msgr2=0x55a3007d3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.001+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 0x55a3007d3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.001+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/3083557654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 msgr2=0x55a3007d37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.001+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 0x55a3007d37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.001+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/3083557654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a30050e7d0 con 0x55a3007cb330 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.001+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cb330 0x55a3007d3290 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0005c40 tx=0x7fc1c0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7a7da887b9e7c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.001+0000 7fc1b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.002+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/3083557654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0035020 con 0x55a3007cb330 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.002+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/3083557654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fc1c0004250 con 0x55a3007cb330 2024-05-09T08:28:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.002+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1c4013950 con 0x55a3007cb330 2024-05-09T08:28:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.002+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/3083557654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc1c0004450 con 0x55a3007cb330 2024-05-09T08:28:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.002+0000 7fc1d776b740 10 client.16431 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.002+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a3007e5250 con 0x55a3007cb330 2024-05-09T08:28:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.003+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/3083557654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c4013950 con 0x55a3007cb330 2024-05-09T08:28:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.003+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/3083557654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fc1c00035f0 con 0x55a3007cb330 2024-05-09T08:28:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.004+0000 7fc1d776b740 10 client.16431 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.004+0000 7fc1d776b740 10 client.16431 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-05-09T08:28:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.004+0000 7fc1d776b740 20 client.16431 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.004+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55a3007e6710 0x55a3007e8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.004+0000 7fc1d776b740 4 client.16431 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.004+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a30073e370 con 0x55a3007e6710 2024-05-09T08:28:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.004+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55a3007e6710 0x55a3007e8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.005+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55a3007e6710 0x55a3007e8b40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ba7899219d3c7cf server_cookie=9fdda2142003c6f4 in_seq=0 out_seq=0 2024-05-09T08:28:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.005+0000 7fc1b5ffb640 10 client.16431 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:28:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.010+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/3083557654 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a30073e370 con 0x55a3007e6710 2024-05-09T08:28:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.010+0000 7fc1b5ffb640 10 client.16431 handle_command_reply: tid=0 2024-05-09T08:28:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.010+0000 7fc1d776b740 1 client.16431 shutdown 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.011+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cb330 msgr2=0x55a3007d3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.011+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cb330 0x55a3007d3290 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0005c40 tx=0x7fc1c0009580 comp rx=0 tx=0).stop 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.011+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 shutdown_connections 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.011+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55a3007e6710 0x55a3007e8b40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.011+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 0x55a3007d3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.011+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 0x55a3007d37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.011+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3083557654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cb330 0x55a3007d3290 secure :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0005c40 tx=0x7fc1c0009580 comp rx=0 tx=0).stop 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.011+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 >> 172.21.15.28:0/3083557654 conn(0x55a3007c7650 msgr2=0x55a3007cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.011+0000 7fc1b5ffb640 0 client.16431 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.012+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 shutdown_connections 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.012+0000 7fc1d776b740 1 -- 172.21.15.28:0/3083557654 wait complete. 2024-05-09T08:28:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.012+0000 7fc1d776b740 20 client.16431 trim_cache size 0 max 16384 2024-05-09T08:28:35.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.038+0000 7fc1d776b740 1 Processor -- start 2024-05-09T08:28:35.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.038+0000 7fc1d776b740 1 -- start start 2024-05-09T08:28:35.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.038+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 0x55a3007e1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.038+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 0x55a3007ee410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.038+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007ee950 0x55a3007d49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.038+0000 7fc1d776b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d01bdbe0 con 0x55a3007ee950 2024-05-09T08:28:35.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.038+0000 7fc1d776b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d01bf7e0 con 0x55a3007cbc50 2024-05-09T08:28:35.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.038+0000 7fc1d776b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d01be2b0 con 0x55a3007d6320 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 0x55a3007e1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 0x55a3007e1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:50446/0 (socket says 172.21.15.28:50446) 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/3717768784 learned_addr learned my addr 172.21.15.28:0/3717768784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/3717768784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 0x55a3007ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/3717768784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007ee950 0x55a3007d49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/3717768784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 msgr2=0x55a3007ee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/3717768784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 0x55a3007ee410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/3717768784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007ee950 msgr2=0x55a3007d49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/3717768784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007ee950 0x55a3007d49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/3717768784 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3007cba80 con 0x55a3007cbc50 2024-05-09T08:28:35.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.039+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/3717768784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 0x55a3007e1fd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0003c00 tx=0x7fc1c002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=834c3b05763f3250 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.040+0000 7fc1b7fff640 1 -- 172.21.15.28:0/3717768784 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0035040 con 0x55a3007cbc50 2024-05-09T08:28:35.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.040+0000 7fc1b7fff640 1 -- 172.21.15.28:0/3717768784 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c0047070 con 0x55a3007cbc50 2024-05-09T08:28:35.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.040+0000 7fc1b7fff640 1 -- 172.21.15.28:0/3717768784 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00047a0 con 0x55a3007cbc50 2024-05-09T08:28:35.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.041+0000 7fc1d776b740 1 -- 172.21.15.28:0/3717768784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 msgr2=0x55a3007e1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.041+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3717768784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 0x55a3007e1fd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0003c00 tx=0x7fc1c002a560 comp rx=0 tx=0).stop 2024-05-09T08:28:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.041+0000 7fc1d776b740 1 -- 172.21.15.28:0/3717768784 shutdown_connections 2024-05-09T08:28:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.041+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3717768784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007ee950 0x55a3007d49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.041+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3717768784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007d6320 0x55a3007ee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.042+0000 7fc1d776b740 1 --2- 172.21.15.28:0/3717768784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007cbc50 0x55a3007e1fd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.042+0000 7fc1d776b740 1 -- 172.21.15.28:0/3717768784 >> 172.21.15.28:0/3717768784 conn(0x55a3007c7650 msgr2=0x55a3007c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.042+0000 7fc1d776b740 1 -- 172.21.15.28:0/3717768784 shutdown_connections 2024-05-09T08:28:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.042+0000 7fc1d776b740 1 -- 172.21.15.28:0/3717768784 wait complete. 2024-05-09T08:28:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.042+0000 7fc1d776b740 1 -- start start 2024-05-09T08:28:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.042+0000 7fc1d776b740 1 Processor -- start 2024-05-09T08:28:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 0x55a3007d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1d776b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007ee950 0x55a3007d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1d776b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d01bf050 con 0x55a3007cbc50 2024-05-09T08:28:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35688/0 (socket says 172.21.15.28:35688) 2024-05-09T08:28:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/1174409547 learned_addr learned my addr 172.21.15.28:0/1174409547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 0x55a3007d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d01be980 con 0x55a3007d6320 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.043+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d01bbbc0 con 0x55a3007ee950 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007ee950 0x55a3007d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1d776b740 10 client.0 _ll_register_callbacks cb 0x7fc1c0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/1174409547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007ee950 msgr2=0x55a3007d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007ee950 0x55a3007d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/1174409547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 msgr2=0x55a3007d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 0x55a3007d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1cdd9b640 1 -- 172.21.15.28:0/1174409547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a30084be30 con 0x55a3007cbc50 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 0x55a3007d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1cdd9b640 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d0110 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0031460 tx=0x7fc1c00045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c97dea85b03005 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/1174409547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0035040 con 0x55a3007cbc50 2024-05-09T08:28:35.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/1174409547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fc1c0031a80 con 0x55a3007cbc50 2024-05-09T08:28:35.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.044+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1c0047070 con 0x55a3007cbc50 2024-05-09T08:28:35.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.045+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/1174409547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc1c0072050 con 0x55a3007cbc50 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.046+0000 7fc1d776b740 10 client.16437 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.046+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a30084d0d0 con 0x55a3007cbc50 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.046+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/1174409547 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c0031c80 con 0x55a3007cbc50 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.046+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/1174409547 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fc1c003d450 con 0x55a3007cbc50 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.047+0000 7fc1d776b740 10 client.16437 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.047+0000 7fc1d776b740 10 client.16437 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.047+0000 7fc1d776b740 20 client.16437 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.047+0000 7fc1d776b740 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55a30084d780 0x55a30084fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.047+0000 7fc1d776b740 4 client.16437 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:35.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.047+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 --> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc1c0047070 con 0x55a30084d780 2024-05-09T08:28:35.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.047+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55a30084d780 0x55a30084fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.047+0000 7fc1ce59c640 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55a30084d780 0x55a30084fb70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86b7dc5206e5fb6f server_cookie=3ba26a95597ed53 in_seq=0 out_seq=0 2024-05-09T08:28:35.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.047+0000 7fc1b5ffb640 10 client.16437 ms_handle_connect on v2:172.21.15.184:6834/4284085086 2024-05-09T08:28:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.051+0000 7fc1b5ffb640 1 -- 172.21.15.28:0/1174409547 <== mds.4 v2:172.21.15.184:6834/4284085086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+86506 (crc 0 0 0) 0x7fc1c0047070 con 0x55a30084d780 2024-05-09T08:28:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.051+0000 7fc1b5ffb640 10 client.16437 handle_command_reply: tid=0 2024-05-09T08:28:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.051+0000 7fc1d776b740 1 client.16437 shutdown 2024-05-09T08:28:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 msgr2=0x55a3007d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d0110 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0031460 tx=0x7fc1c00045f0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 shutdown_connections 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.184:6834/4284085086,v1:172.21.15.184:6835/4284085086] conn(0x55a30084d780 0x55a30084fb70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a3007ee950 0x55a3007d0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a3007d6320 0x55a3007d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1b5ffb640 0 client.16437 ms_handle_reset on v2:172.21.15.184:6834/4284085086 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 --2- 172.21.15.28:0/1174409547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3007cbc50 0x55a3007d0110 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 >> 172.21.15.28:0/1174409547 conn(0x55a3007c7650 msgr2=0x55a3007c5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 shutdown_connections 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 1 -- 172.21.15.28:0/1174409547 wait complete. 2024-05-09T08:28:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.052+0000 7fc1d776b740 20 client.16437 trim_cache size 0 max 16384 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3122147, 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629769, 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-82.xml" 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 27028900, 2024-05-09T08:28:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630110, 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-56.jpg" 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 29559599, 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630490, 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-74.xml" 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 56572195, 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631244, 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-54.xml" 2024-05-09T08:28:35.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 80446330, 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630536, 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-61.xml" 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 84432500, 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629500, 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-34.jpg" 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.057 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 108127729, 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629868, 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-79.jpg" 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 121764118, 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630064, 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-79.jpg" 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 121845381, 2024-05-09T08:28:35.058 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630044, 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-88.jpg" 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 124586333, 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629454, 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-77.xml" 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 141694022, 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630062, 2024-05-09T08:28:35.059 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-80.jpg" 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 147798359, 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628987, 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-5.xml" 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 167247785, 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631277, 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-46.xml" 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.060 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 170720683, 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629617, 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-53.jpg" 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 172582060, 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628730, 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-8.xml" 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 179626721, 2024-05-09T08:28:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631134, 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-77.xml" 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 183259744, 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630582, 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-48.xml" 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 187255471, 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630196, 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-21.jpg" 2024-05-09T08:28:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 188869239, 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631272, 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-47.xml" 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 242246494, 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628776, 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-31.jpg" 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.063 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 246898109, 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629512, 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-30.jpg" 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 253491091, 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629909, 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-38.jpg" 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 272413643, 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630702, 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-4.xml" 2024-05-09T08:28:35.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 279319761, 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629470, 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-65.xml" 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 279786293, 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630084, 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-69.jpg" 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.065 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 300954480, 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630564, 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-53.xml" 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 313223878, 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629543, 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-24.xml" 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 328152279, 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629974, 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-48.xml" 2024-05-09T08:28:35.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 340579546, 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629080, 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-91.xml" 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 351432328, 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629469, 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-66.xml" 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.067 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 364331973, 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630144, 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-41.jpg" 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 374768270, 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629481, 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-35.jpg" 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 375010218, 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629969, 2024-05-09T08:28:35.068 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-53.xml" 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 411844219, 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630022, 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-2.xml" 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 430941029, 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629163, 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-58.jpg" 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 441777616, 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631152, 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-73.xml" 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 443925966, 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629627, 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-48.jpg" 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.070 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 460342460, 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629754, 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-97.xml" 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 461299164, 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629521, 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-27.jpg" 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 466905608, 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629656, 2024-05-09T08:28:35.071 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-34.jpg" 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 482382428, 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630575, 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-50.xml" 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 507972938, 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629767, 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-84.xml" 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 508149882, 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629515, 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-29.jpg" 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 508164407, 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629579, 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-12.xml" 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 518405757, 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629619, 2024-05-09T08:28:35.073 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-52.jpg" 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 533625926, 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630124, 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-49.jpg" 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 551253288, 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629891, 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-56.jpg" 2024-05-09T08:28:35.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 574879309, 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631269, 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-48.xml" 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 583415587, 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629992, 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-30.xml" 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.075 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 589797440, 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629780, 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-71.xml" 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 610291600, 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629279, 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-51.jpg" 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 614258803, 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628772, 2024-05-09T08:28:35.076 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-21.xml" 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 614455532, 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628735, 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-60.xml" 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 619055510, 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630657, 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-22.xml" 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 624135436, 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630179, 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-38.jpg" 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 637764426, 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628762, 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-22.xml" 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 685665276, 2024-05-09T08:28:35.078 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630003, 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-19.xml" 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 706081444, 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629445, 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-80.xml" 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 733807174, 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630295, 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-47.jpg" 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 765732149, 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629472, 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-63.xml" 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 795370629, 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630194, 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-22.jpg" 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 797922242, 2024-05-09T08:28:35.080 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628792, 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-25.jpg" 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 810754471, 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630011, 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-11.xml" 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 811378527, 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629104, 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-62.jpg" 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 816672474, 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628780, 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-29.jpg" 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 832358842, 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629308, 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-49.jpg" 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 840058854, 2024-05-09T08:28:35.082 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629979, 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-43.xml" 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 842369323, 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630017, 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-5.xml" 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 846533387, 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630046, 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-87.jpg" 2024-05-09T08:28:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 857320549, 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629817, 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-42.xml" 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 860130822, 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630350, 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-26.xml" 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.084 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 868542374, 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630540, 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-60.xml" 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 870159831, 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630530, 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-63.xml" 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 880006411, 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629982, 2024-05-09T08:28:35.085 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-40.xml" 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 883191082, 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630568, 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-52.xml" 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 886100342, 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629872, 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-75.jpg" 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 902365683, 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630133, 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-45.jpg" 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 912518450, 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629434, 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-85.xml" 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.087 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 924951415, 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629471, 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-64.xml" 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 925508200, 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631090, 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-89.xml" 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 931103195, 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629596, 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-2.jpg" 2024-05-09T08:28:35.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 944043241, 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630526, 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-64.xml" 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 947344115, 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630234, 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-23.jpg" 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 947875925, 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629994, 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-28.xml" 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 959626148, 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630052, 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-84.jpg" 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 965217451, 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629548, 2024-05-09T08:28:35.090 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-18.jpg" 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 972289300, 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629993, 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-29.xml" 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1011984630, 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628736, 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-6.xml" 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1040972856, 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630498, 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-72.xml" 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1043445676, 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629887, 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-60.jpg" 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.092 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1061100231, 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630482, 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-76.xml" 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1075847450, 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630478, 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-77.xml" 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1093625863, 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629660, 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-32.jpg" 2024-05-09T08:28:35.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1100914249, 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631177, 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-67.xml" 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1101617597, 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630018, 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-4.xml" 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1125776827, 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629686, 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-10.jpg" 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1127340685, 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629564, 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-17.xml" 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1129531548, 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629646, 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-39.jpg" 2024-05-09T08:28:35.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1133176098, 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629800, 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-54.xml" 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1135080555, 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630509, 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-69.xml" 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.096 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1136061747, 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629640, 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-42.jpg" 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1147436751, 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630215, 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-11.jpg" 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1160342260, 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630006, 2024-05-09T08:28:35.097 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-16.xml" 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1160348512, 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629468, 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-67.xml" 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1173382910, 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629588, 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-9.xml" 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1212529847, 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629679, 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-17.jpg" 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1214590019, 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630092, 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-65.jpg" 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1216838909, 2024-05-09T08:28:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629453, 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-36.jpg" 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1243560309, 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628790, 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-26.jpg" 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1246285623, 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629976, 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-46.xml" 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.100 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1255489180, 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630122, 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-50.jpg" 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1256793287, 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629758, 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-93.xml" 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1271455125, 2024-05-09T08:28:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629881, 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-66.jpg" 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1274283293, 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630358, 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-24.xml" 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1307303884, 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630155, 2024-05-09T08:28:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-36.jpg" 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1332277722, 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629824, 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-35.xml" 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1350945680, 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629902, 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-45.jpg" 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.103 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1357873514, 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630561, 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-54.xml" 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1365659254, 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629849, 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-10.xml" 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1399930251, 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630078, 2024-05-09T08:28:35.104 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-72.jpg" 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1405136290, 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628808, 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-20.jpg" 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1422166591, 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630131, 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-46.jpg" 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.105 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1456328096, 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629473, 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-62.xml" 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1463252641, 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629372, 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-43.jpg" 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1487842521, 2024-05-09T08:28:35.106 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629874, 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-73.jpg" 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1531720051, 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629545, 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-19.jpg" 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1543193934, 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629554, 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-16.jpg" 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1551077538, 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629230, 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-53.jpg" 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1552813890, 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629848, 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-11.xml" 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1560124363, 2024-05-09T08:28:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629201, 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-55.jpg" 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1583144829, 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630505, 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-70.xml" 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1591900568, 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629980, 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-42.xml" 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1594027182, 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628823, 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-15.jpg" 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1600509526, 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631218, 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-59.xml" 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.110 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1610157321, 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630137, 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-44.jpg" 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1617360485, 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630027, 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-95.jpg" 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1627432665, 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631196, 2024-05-09T08:28:35.111 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-63.xml" 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1627641614, 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629493, 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-44.xml" 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1628538960, 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629972, 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-50.xml" 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1631967835, 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630280, 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-12.jpg" 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1639448297, 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629984, 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-38.xml" 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1649270679, 2024-05-09T08:28:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629607, 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-1.xml" 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1658961694, 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628832, 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-12.jpg" 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1674500102, 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629812, 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-47.xml" 2024-05-09T08:28:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1683236602, 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630120, 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-51.jpg" 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1690540401, 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630244, 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-20.jpg" 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.115 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1691486740, 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630652, 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-27.xml" 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1705368413, 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630238, 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-22.jpg" 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1714757549, 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630333, 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-64.jpg" 2024-05-09T08:28:35.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1726906836, 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630516, 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-67.xml" 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1733504095, 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629546, 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-23.xml" 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.117 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1750203202, 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628766, 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-57.xml" 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1761278138, 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628800, 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-23.jpg" 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1784928763, 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631191, 2024-05-09T08:28:35.118 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-64.xml" 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1796132747, 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629242, 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-52.jpg" 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1878694543, 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629563, 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-13.jpg" 2024-05-09T08:28:35.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1878738871, 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630072, 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-75.jpg" 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1879546364, 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629650, 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-37.jpg" 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.120 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1880123805, 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628760, 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-36.jpg" 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1891573126, 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629507, 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-36.xml" 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1897145035, 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629079, 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-1.xml" 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1904108445, 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628742, 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-4.xml" 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1935478281, 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630650, 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-28.xml" 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.122 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1947883827, 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628951, 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-8.xml" 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1954758325, 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629609, 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-57.jpg" 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1970483140, 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630609, 2024-05-09T08:28:35.123 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-40.xml" 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1972905958, 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629065, 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-2.xml" 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1975637879, 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629527, 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-25.jpg" 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2005173328, 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629668, 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-28.jpg" 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2007624465, 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628787, 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-20.xml" 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2007944974, 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630470, 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-79.xml" 2024-05-09T08:28:35.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2010447913, 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630494, 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-73.xml" 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2025402201, 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630458, 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-82.xml" 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.126 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2037518510, 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628729, 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-61.xml" 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2058570073, 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630001, 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-21.xml" 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2062761170, 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628805, 2024-05-09T08:28:35.127 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-21.jpg" 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2091779224, 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630543, 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-59.xml" 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2112337326, 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629975, 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-47.xml" 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.128 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2116722821, 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629840, 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-19.xml" 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2118028475, 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630171, 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-30.jpg" 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2139475037, 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629524, 2024-05-09T08:28:35.129 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-26.jpg" 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2140219779, 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629765, 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-86.xml" 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2140221425, 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628748, 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-2.xml" 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2158120663, 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628755, 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-38.jpg" 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2167661080, 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630213, 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-12.jpg" 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2168515137, 2024-05-09T08:28:35.131 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630533, 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-62.xml" 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2172547307, 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630098, 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-62.jpg" 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2183141665, 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631054, 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-97.xml" 2024-05-09T08:28:35.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2229916635, 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630664, 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-15.xml" 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2232079141, 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630289, 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-56.xml" 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.133 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2241412779, 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629990, 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-32.xml" 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2255454851, 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629779, 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-72.xml" 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2327452399, 2024-05-09T08:28:35.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629973, 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-49.xml" 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2327816034, 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629587, 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-5.jpg" 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2328036313, 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629760, 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-91.xml" 2024-05-09T08:28:35.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2330466228, 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629904, 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-43.jpg" 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2338191346, 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630146, 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-40.jpg" 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.136 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2341196066, 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630512, 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-68.xml" 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2369686584, 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629539, 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-21.jpg" 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2386936331, 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628834, 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-15.xml" 2024-05-09T08:28:35.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2411114745, 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629867, 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-80.jpg" 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2434015290, 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630184, 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-26.jpg" 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2447681886, 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629669, 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-27.jpg" 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2459608972, 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630198, 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-20.jpg" 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2468800825, 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629578, 2024-05-09T08:28:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-8.jpg" 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2488072898, 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631081, 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-91.xml" 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2489737563, 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630639, 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-31.xml" 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.140 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2495330033, 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628721, 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-11.xml" 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2515580151, 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630096, 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-63.jpg" 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2520902202, 2024-05-09T08:28:35.141 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630660, 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-19.xml" 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2531540116, 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630616, 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-38.xml" 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2541857051, 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629120, 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-61.jpg" 2024-05-09T08:28:35.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2567462298, 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631095, 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-88.xml" 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2604668337, 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631201, 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-62.xml" 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2612666498, 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629996, 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-26.xml" 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2621020273, 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629997, 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-25.xml" 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2634059902, 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630612, 2024-05-09T08:28:35.144 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-39.xml" 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2657044444, 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630354, 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-25.xml" 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2657935218, 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629421, 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-40.jpg" 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.145 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2660005309, 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629680, 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-16.jpg" 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2663991349, 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631164, 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-70.xml" 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2669091484, 2024-05-09T08:28:35.146 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630008, 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-14.xml" 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2675546695, 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629695, 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-1.jpg" 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2723739917, 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629590, 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-4.jpg" 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2740674572, 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629888, 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-59.jpg" 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2752113521, 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630248, 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-3.jpg" 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.148 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2789258338, 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630262, 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-15.jpg" 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2789971088, 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628803, 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-22.jpg" 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2799167722, 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630191, 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-23.jpg" 2024-05-09T08:28:35.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2802738960, 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630016, 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-6.xml" 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2817647686, 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629569, 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-11.jpg" 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.150 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2830119417, 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631129, 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-78.xml" 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2830150965, 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629338, 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-46.jpg" 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2844812108, 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630048, 2024-05-09T08:28:35.151 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-86.jpg" 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2901496993, 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631118, 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-82.xml" 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2907787229, 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629506, 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-32.jpg" 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.152 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2914248246, 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630462, 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-81.xml" 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2916678144, 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629762, 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-89.xml" 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2917372064, 2024-05-09T08:28:35.153 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630060, 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-81.jpg" 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2919682196, 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631309, 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-40.xml" 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2978081903, 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630585, 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-47.xml" 2024-05-09T08:28:35.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3006753426, 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630303, 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-46.xml" 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3011624057, 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630622, 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-36.xml" 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3012663908, 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630678, 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-10.xml" 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3019142861, 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631126, 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-79.xml" 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3035689779, 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630112, 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-55.jpg" 2024-05-09T08:28:35.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3042896209, 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630662, 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-17.xml" 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3042966305, 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630599, 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-43.xml" 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.157 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3061509973, 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630183, 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-13.xml" 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3077682983, 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630080, 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-71.jpg" 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3093181774, 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629458, 2024-05-09T08:28:35.158 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-75.xml" 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3097048537, 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630193, 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-30.jpg" 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3100908460, 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630000, 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-22.xml" 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3101067520, 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629593, 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-3.jpg" 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3112198065, 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630241, 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-21.jpg" 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3141570329, 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629855, 2024-05-09T08:28:35.160 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-4.xml" 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3143632943, 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630139, 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-43.jpg" 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3151679873, 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629819, 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-40.xml" 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.161 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3152998776, 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629903, 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-44.jpg" 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3187102063, 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629968, 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-54.xml" 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3187240234, 2024-05-09T08:28:35.162 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629875, 2024-05-09T08:28:35.163 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-72.jpg" 2024-05-09T08:28:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3187898192, 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629214, 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-54.jpg" 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3188764577, 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631146, 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-74.xml" 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3199192670, 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628979, 2024-05-09T08:28:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-6.xml" 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3202133489, 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628706, 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-16.xml" 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3211657934, 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630175, 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-29.jpg" 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3214665290, 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630068, 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-77.jpg" 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3224272711, 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630690, 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-7.xml" 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3228604792, 2024-05-09T08:28:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630320, 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-94.jpg" 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3236175507, 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630682, 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-9.xml" 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3254469694, 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630106, 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-58.jpg" 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3277670222, 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629843, 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-16.xml" 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3280768504, 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629581, 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-7.jpg" 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3342091720, 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631057, 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-96.xml" 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3342511677, 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629594, 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-7.xml" 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3377095679, 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630633, 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-33.xml" 2024-05-09T08:28:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3392236125, 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629292, 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-50.jpg" 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3394885590, 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629988, 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-34.xml" 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3406948383, 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629499, 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-39.xml" 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3417374128, 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631213, 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-60.xml" 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3418975498, 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629911, 2024-05-09T08:28:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-36.jpg" 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3436038551, 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631047, 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-98.xml" 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3438306503, 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629542, 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-20.jpg" 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3443452511, 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628710, 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-15.xml" 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3445819183, 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630076, 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-73.jpg" 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3451193409, 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630168, 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-31.jpg" 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3456751974, 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628715, 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-13.xml" 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3490969090, 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629504, 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-37.xml" 2024-05-09T08:28:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3491723286, 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628708, 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-63.xml" 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3502405583, 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631085, 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-90.xml" 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3505324111, 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630595, 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-44.xml" 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3512890753, 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630626, 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-35.xml" 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3520072677, 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629833, 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-26.xml" 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3581789947, 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631168, 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-69.xml" 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3582628323, 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630114, 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-54.jpg" 2024-05-09T08:28:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3607225472, 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629560, 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-14.jpg" 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3608281099, 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630592, 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-45.xml" 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3615832891, 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629509, 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-31.jpg" 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3627099935, 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629575, 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-9.jpg" 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3630838973, 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631314, 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-39.xml" 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3635447008, 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630588, 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-46.xml" 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3640792740, 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630035, 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-91.jpg" 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3643809039, 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629989, 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-33.xml" 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3651914794, 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630454, 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-83.xml" 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3665133221, 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628884, 2024-05-09T08:28:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-12.xml" 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3669989646, 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628713, 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-14.xml" 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3678909295, 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629821, 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-38.xml" 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3728191131, 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630714, 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-1.xml" 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3737041425, 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629970, 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-52.xml" 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3750193451, 2024-05-09T08:28:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629603, 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-3.xml" 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3753675429, 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628794, 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-19.xml" 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3761056468, 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630636, 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-32.xml" 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3763033883, 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630242, 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-5.jpg" 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3772445864, 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629317, 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-48.jpg" 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3772761426, 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631117, 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-83.xml" 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3784971172, 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630694, 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-6.xml" 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3788588053, 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630466, 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-80.xml" 2024-05-09T08:28:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3802728626, 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630082, 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-70.jpg" 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3827696746, 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630550, 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-57.xml" 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3832404694, 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629987, 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-35.xml" 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3881175636, 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630186, 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-25.jpg" 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3891377885, 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629906, 2024-05-09T08:28:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-41.jpg" 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3900523764, 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630231, 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-8.jpg" 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3903399581, 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630014, 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-8.xml" 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.191 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3906367250, 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631156, 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-72.xml" 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3935571964, 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629845, 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-14.xml" 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3940439370, 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630665, 2024-05-09T08:28:35.192 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-14.xml" 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3942869314, 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630686, 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-8.xml" 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3946725809, 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629462, 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-73.xml" 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.193 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3980319307, 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630012, 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-10.xml" 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3998396182, 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630090, 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-66.jpg" 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4018407528, 2024-05-09T08:28:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628817, 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-17.xml" 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4029863417, 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631102, 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-86.xml" 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4043249444, 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629967, 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-55.xml" 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4044711679, 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628778, 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-30.jpg" 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4049830008, 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628754, 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-23.xml" 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4062828789, 2024-05-09T08:28:35.196 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628724, 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-10.xml" 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4073726047, 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630209, 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-15.jpg" 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4086200377, 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630104, 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-59.jpg" 2024-05-09T08:28:35.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4092899621, 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630519, 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-66.xml" 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4111130473, 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628802, 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-18.xml" 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4114820146, 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630474, 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-78.xml" 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4115873745, 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629672, 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-24.jpg" 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4122293720, 2024-05-09T08:28:35.199 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628701, 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-27.xml" 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4132495499, 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629834, 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-25.xml" 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4137923861, 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629690, 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-6.jpg" 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4164036708, 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630102, 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-60.jpg" 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4172700781, 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628844, 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-8.jpg" 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4202423232, 2024-05-09T08:28:35.201 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629658, 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-33.jpg" 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4203229509, 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630007, 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-15.xml" 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4204706891, 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630311, 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-90.xml" 2024-05-09T08:28:35.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4207944627, 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630324, 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-30.xml" 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4208455058, 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629810, 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-48.xml" 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.203 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4218052630, 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630009, 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-13.xml" 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4240288003, 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631260, 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-50.xml" 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4253048807, 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630710, 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-2.xml" 2024-05-09T08:28:35.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4253873012, 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630151, 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-38.jpg" 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4266319135, 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630128, 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-47.jpg" 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:28:35.205 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-05-09T08:28:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b80a2ed0 msgr2=0x7fc1b80a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b80a2ed0 0x7fc1b80a5390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc1c809ae60 tx=0x7fc1c800f300 comp rx=0 tx=0).stop 2024-05-09T08:28:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d010bdf0 msgr2=0x7fc1d0107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d010bdf0 0x7fc1d0107b10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800be20 tx=0x7fc1c800bf10 comp rx=0 tx=0).stop 2024-05-09T08:28:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 shutdown_connections 2024-05-09T08:28:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d010bdf0 0x7fc1d0107b10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1d01033d0 0x7fc1d019f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b80a2ed0 0x7fc1b80a5390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 --2- 172.21.15.28:0/10776471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1d0102ab0 0x7fc1d019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.053+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 >> 172.21.15.28:0/10776471 conn(0x7fc1d0077830 msgr2=0x7fc1d0109040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.054+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 shutdown_connections 2024-05-09T08:28:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.054+0000 7fc1d4a5d640 1 -- 172.21.15.28:0/10776471 wait complete. 2024-05-09T08:28:35.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-05-09T08:28:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.326+0000 7fa74a709640 1 -- 172.21.15.28:0/522228928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 msgr2=0x7fa74406a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.326+0000 7fa74a709640 1 --2- 172.21.15.28:0/522228928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 0x7fa74406a330 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa738009930 tx=0x7fa73802a420 comp rx=0 tx=0).stop 2024-05-09T08:28:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.326+0000 7fa74a709640 1 -- 172.21.15.28:0/522228928 shutdown_connections 2024-05-09T08:28:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.326+0000 7fa74a709640 1 --2- 172.21.15.28:0/522228928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74406a870 0x7fa74410e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.326+0000 7fa74a709640 1 --2- 172.21.15.28:0/522228928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 0x7fa74406a330 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.326+0000 7fa74a709640 1 --2- 172.21.15.28:0/522228928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa744107820 0x7fa744107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.326+0000 7fa74a709640 1 -- 172.21.15.28:0/522228928 >> 172.21.15.28:0/522228928 conn(0x7fa744077790 msgr2=0x7fa744077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.326+0000 7fa74a709640 1 -- 172.21.15.28:0/522228928 shutdown_connections 2024-05-09T08:28:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.327+0000 7fa74a709640 1 -- 172.21.15.28:0/522228928 wait complete. 2024-05-09T08:28:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.327+0000 7fa74a709640 1 Processor -- start 2024-05-09T08:28:35.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.327+0000 7fa74a709640 1 -- start start 2024-05-09T08:28:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa74a709640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 0x7fa744110b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa74a709640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74406a870 0x7fa7441110a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa74a709640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa744107820 0x7fa7441115e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa74a709640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa744101830 con 0x7fa744069eb0 2024-05-09T08:28:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa74a709640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441016b0 con 0x7fa744107820 2024-05-09T08:28:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa74a709640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441019b0 con 0x7fa74406a870 2024-05-09T08:28:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa743fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 0x7fa744110b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa743fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 0x7fa744110b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35710/0 (socket says 172.21.15.28:35710) 2024-05-09T08:28:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa743fff640 1 -- 172.21.15.28:0/1862673365 learned_addr learned my addr 172.21.15.28:0/1862673365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa7437fe640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74406a870 0x7fa7441110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa748ca6640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa744107820 0x7fa7441115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa743fff640 1 -- 172.21.15.28:0/1862673365 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74406a870 msgr2=0x7fa7441110a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa743fff640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74406a870 0x7fa7441110a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa743fff640 1 -- 172.21.15.28:0/1862673365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa744107820 msgr2=0x7fa7441115e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa743fff640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa744107820 0x7fa7441115e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa743fff640 1 -- 172.21.15.28:0/1862673365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7441aaa00 con 0x7fa744069eb0 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa743fff640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 0x7fa744110b60 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa73400ea20 tx=0x7fa73400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa7417fa640 1 -- 172.21.15.28:0/1862673365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73400cd80 con 0x7fa744069eb0 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa74a709640 1 -- 172.21.15.28:0/1862673365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7441aacf0 con 0x7fa744069eb0 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa7417fa640 1 -- 172.21.15.28:0/1862673365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa734010910 con 0x7fa744069eb0 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.328+0000 7fa7417fa640 1 -- 172.21.15.28:0/1862673365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa734018920 con 0x7fa744069eb0 2024-05-09T08:28:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.329+0000 7fa74a709640 1 -- 172.21.15.28:0/1862673365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7441ab1c0 con 0x7fa744069eb0 2024-05-09T08:28:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.331+0000 7fa7417fa640 1 -- 172.21.15.28:0/1862673365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa734010ac0 con 0x7fa744069eb0 2024-05-09T08:28:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.331+0000 7fa74d40f740 1 Processor -- start 2024-05-09T08:28:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.331+0000 7fa74d40f740 1 -- start start 2024-05-09T08:28:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.331+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 0x562160a81710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a81c50 0x562160a8bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a8c320 0x562160a8e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa74d40f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621609e6c40 con 0x562160a81c50 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa74d40f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56216098e580 con 0x562160a81330 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa74d40f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5621609e2c00 con 0x562160a8c320 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa743fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a81c50 0x562160a8bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa748ca6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 0x562160a81710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa7437fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a8c320 0x562160a8e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa743fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a81c50 0x562160a8bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35712/0 (socket says 172.21.15.28:35712) 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa743fff640 1 -- 172.21.15.28:0/2103775832 learned_addr learned my addr 172.21.15.28:0/2103775832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa7417fa640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa72c0a2d00 0x7fa72c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa7437fe640 1 -- 172.21.15.28:0/2103775832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 msgr2=0x562160a81710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa748ca6640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa72c0a2d00 0x7fa72c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa7437fe640 1 --2- 172.21.15.28:0/2103775832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 0x562160a81710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa7437fe640 1 -- 172.21.15.28:0/2103775832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a81c50 msgr2=0x562160a8bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa7437fe640 1 --2- 172.21.15.28:0/2103775832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a81c50 0x562160a8bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa7437fe640 1 -- 172.21.15.28:0/2103775832 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621606fa3f0 con 0x562160a8c320 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa743fff640 1 --2- 172.21.15.28:0/2103775832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a81c50 0x562160a8bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa7417fa640 1 -- 172.21.15.28:0/1862673365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa7340cfb10 con 0x7fa744069eb0 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.332+0000 7fa7437fe640 1 --2- 172.21.15.28:0/2103775832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a8c320 0x562160a8e710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa738005940 tx=0x7fa738005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e360d80648a0fe1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.333+0000 7fa748ca6640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa72c0a2d00 0x7fa72c0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa73c005c40 tx=0x7fa73c004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.333+0000 7fa72a7fc640 1 -- 172.21.15.28:0/2103775832 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738040070 con 0x562160a8c320 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.333+0000 7fa72a7fc640 1 -- 172.21.15.28:0/2103775832 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa738031870 con 0x562160a8c320 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.333+0000 7fa72a7fc640 1 -- 172.21.15.28:0/2103775832 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738035040 con 0x562160a8c320 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 -- 172.21.15.28:0/2103775832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a8c320 msgr2=0x562160a8e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2103775832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a8c320 0x562160a8e710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa738005940 tx=0x7fa738005bd0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 -- 172.21.15.28:0/2103775832 shutdown_connections 2024-05-09T08:28:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2103775832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a8c320 0x562160a8e710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2103775832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a81c50 0x562160a8bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2103775832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 0x562160a81710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 -- 172.21.15.28:0/2103775832 >> 172.21.15.28:0/2103775832 conn(0x562160a7d650 msgr2=0x562160a7da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 -- 172.21.15.28:0/2103775832 shutdown_connections 2024-05-09T08:28:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 -- 172.21.15.28:0/2103775832 wait complete. 2024-05-09T08:28:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 -- start start 2024-05-09T08:28:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.334+0000 7fa74d40f740 1 Processor -- start 2024-05-09T08:28:35.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 0x562160a89240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81c50 0x562160a89780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a89cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa74d40f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562160a8f0b0 con 0x562160a8c320 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa74d40f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5621609a1c40 con 0x562160a81330 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa74d40f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562160a8f230 con 0x562160a81c50 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa74d40f740 10 client.0 _ll_register_callbacks cb 0x5621606f9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa748ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a89cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa7437fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81c50 0x562160a89780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa748ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a89cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35722/0 (socket says 172.21.15.28:35722) 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa748ca6640 1 -- 172.21.15.28:0/209740715 learned_addr learned my addr 172.21.15.28:0/209740715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa743fff640 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 0x562160a89240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa7437fe640 1 -- 172.21.15.28:0/209740715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 msgr2=0x562160a89240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa7437fe640 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 0x562160a89240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa7437fe640 1 -- 172.21.15.28:0/209740715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 msgr2=0x562160a89cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa7437fe640 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a89cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa7437fe640 1 -- 172.21.15.28:0/209740715 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621607c47d0 con 0x562160a81c50 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa748ca6640 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a89cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa7437fe640 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81c50 0x562160a89780 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x562160a7e7e0 tx=0x7fa738000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd6b8d71ea7b9c27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.335+0000 7fa7237fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa7237fe640 1 -- 172.21.15.28:0/209740715 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738040070 con 0x562160a81c50 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa7237fe640 1 -- 172.21.15.28:0/209740715 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa738031670 con 0x562160a81c50 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa74d40f740 1 -- 172.21.15.28:0/209740715 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa738031870 con 0x562160a81c50 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa7237fe640 1 -- 172.21.15.28:0/209740715 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa738056050 con 0x562160a81c50 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa74d40f740 10 client.25822 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa74d40f740 1 -- 172.21.15.28:0/209740715 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562160a8f0b0 con 0x562160a81c50 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa7237fe640 1 -- 172.21.15.28:0/209740715 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa738031870 con 0x562160a81c50 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa7237fe640 1 -- 172.21.15.28:0/209740715 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fa7380035f0 con 0x562160a81c50 2024-05-09T08:28:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa74d40f740 10 client.25822 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa74d40f740 10 client.25822 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa74d40f740 20 client.25822 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa74d40f740 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x562160a9c410 0x562160a9e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa74d40f740 4 client.25822 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.336+0000 7fa74d40f740 1 -- 172.21.15.28:0/209740715 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621609f4370 con 0x562160a9c410 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.337+0000 7fa743fff640 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x562160a9c410 0x562160a9e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.337+0000 7fa743fff640 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x562160a9c410 0x562160a9e800 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e198f3d9fb02c48f server_cookie=dbe7b867245bb67a in_seq=0 out_seq=0 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.337+0000 7fa7237fe640 10 client.25822 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa7237fe640 1 -- 172.21.15.28:0/209740715 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621609f4370 con 0x562160a9c410 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa7237fe640 10 client.25822 handle_command_reply: tid=0 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 client.25822 shutdown 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 -- 172.21.15.28:0/209740715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81c50 msgr2=0x562160a89780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81c50 0x562160a89780 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x562160a7e7e0 tx=0x7fa738000bc0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 -- 172.21.15.28:0/209740715 shutdown_connections 2024-05-09T08:28:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x562160a9c410 0x562160a9e800 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a89cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81c50 0x562160a89780 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 --2- 172.21.15.28:0/209740715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81330 0x562160a89240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 -- 172.21.15.28:0/209740715 >> 172.21.15.28:0/209740715 conn(0x562160a7d650 msgr2=0x562160a7c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa7237fe640 0 client.25822 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:28:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 -- 172.21.15.28:0/209740715 shutdown_connections 2024-05-09T08:28:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.338+0000 7fa74d40f740 1 -- 172.21.15.28:0/209740715 wait complete. 2024-05-09T08:28:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.339+0000 7fa74d40f740 20 client.25822 trim_cache size 0 max 16384 2024-05-09T08:28:35.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.354+0000 7fa74d40f740 1 Processor -- start 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.354+0000 7fa74d40f740 1 -- start start 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.354+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 0x562160a97f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.354+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 0x562160a83ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.354+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a88220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.354+0000 7fa74d40f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441b5460 con 0x562160a8c320 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.354+0000 7fa74d40f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441b7060 con 0x562160a81c50 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.354+0000 7fa74d40f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441b5b30 con 0x562160a81330 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.354+0000 7fa748ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a88220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa748ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a88220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35738/0 (socket says 172.21.15.28:35738) 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa743fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 0x562160a97f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa748ca6640 1 -- 172.21.15.28:0/3724904321 learned_addr learned my addr 172.21.15.28:0/3724904321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa7437fe640 1 --2- 172.21.15.28:0/3724904321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 0x562160a83ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa748ca6640 1 -- 172.21.15.28:0/3724904321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 msgr2=0x562160a97f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa748ca6640 1 --2- 172.21.15.28:0/3724904321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 0x562160a97f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa748ca6640 1 -- 172.21.15.28:0/3724904321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 msgr2=0x562160a83ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa748ca6640 1 --2- 172.21.15.28:0/3724904321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 0x562160a83ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa748ca6640 1 -- 172.21.15.28:0/3724904321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562160a6ed20 con 0x562160a8c320 2024-05-09T08:28:35.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa748ca6640 1 --2- 172.21.15.28:0/3724904321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a88220 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa73c003b80 tx=0x7fa73c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da670059d6f81be server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa729ffb640 1 -- 172.21.15.28:0/3724904321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c004050 con 0x562160a8c320 2024-05-09T08:28:35.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa729ffb640 1 -- 172.21.15.28:0/3724904321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa73c0046d0 con 0x562160a8c320 2024-05-09T08:28:35.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.355+0000 7fa729ffb640 1 -- 172.21.15.28:0/3724904321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c010430 con 0x562160a8c320 2024-05-09T08:28:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 -- 172.21.15.28:0/3724904321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 msgr2=0x562160a88220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 --2- 172.21.15.28:0/3724904321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a88220 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa73c003b80 tx=0x7fa73c005bd0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 -- 172.21.15.28:0/3724904321 shutdown_connections 2024-05-09T08:28:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 --2- 172.21.15.28:0/3724904321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a88220 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 --2- 172.21.15.28:0/3724904321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 0x562160a83ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 --2- 172.21.15.28:0/3724904321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 0x562160a97f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 -- 172.21.15.28:0/3724904321 >> 172.21.15.28:0/3724904321 conn(0x562160a7d650 msgr2=0x562160a87b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 -- 172.21.15.28:0/3724904321 shutdown_connections 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 -- 172.21.15.28:0/3724904321 wait complete. 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.356+0000 7fa74d40f740 1 -- start start 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa74d40f740 1 Processor -- start 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 0x562160afab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 0x562160afb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa74d40f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a8f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa74d40f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441b68d0 con 0x562160a8c320 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa74d40f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441b6200 con 0x562160a81c50 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa74d40f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441b3440 con 0x562160a81330 2024-05-09T08:28:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa74d40f740 10 client.0 _ll_register_callbacks cb 0x7fa734097fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa743fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 0x562160afab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa748ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a8f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa7437fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 0x562160afb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa748ca6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a8f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35748/0 (socket says 172.21.15.28:35748) 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa748ca6640 1 -- 172.21.15.28:0/2846788556 learned_addr learned my addr 172.21.15.28:0/2846788556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa748ca6640 1 -- 172.21.15.28:0/2846788556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 msgr2=0x562160afab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa748ca6640 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 0x562160afab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa748ca6640 1 -- 172.21.15.28:0/2846788556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 msgr2=0x562160afb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa748ca6640 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 0x562160afb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa748ca6640 1 -- 172.21.15.28:0/2846788556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562160a8fd60 con 0x562160a8c320 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.357+0000 7fa7437fe640 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 0x562160afb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa748ca6640 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a8f620 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa73c022a90 tx=0x7fa73c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b59c05e959eb8c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa7237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa7237fe640 1 -- 172.21.15.28:0/2846788556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c030040 con 0x562160a8c320 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa7237fe640 1 -- 172.21.15.28:0/2846788556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa73c00fdb0 con 0x562160a8c320 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa74d40f740 1 -- 172.21.15.28:0/2846788556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa73c0046d0 con 0x562160a8c320 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa7237fe640 1 -- 172.21.15.28:0/2846788556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa73c041050 con 0x562160a8c320 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa74d40f740 10 client.16455 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa74d40f740 1 -- 172.21.15.28:0/2846788556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562160b02300 con 0x562160a8c320 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa7237fe640 1 -- 172.21.15.28:0/2846788556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa73c0046d0 con 0x562160a8c320 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa7237fe640 1 -- 172.21.15.28:0/2846788556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fa73c033070 con 0x562160a8c320 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa74d40f740 10 client.16455 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa74d40f740 10 client.16455 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa74d40f740 20 client.16455 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x562160b02960 0x562160b04d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa74d40f740 4 client.16455 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.358+0000 7fa74d40f740 1 -- 172.21.15.28:0/2846788556 --> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621609ee6c0 con 0x562160b02960 2024-05-09T08:28:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.359+0000 7fa743fff640 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x562160b02960 0x562160b04d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.359+0000 7fa7237fe640 10 client.16455 ms_handle_connect on v2:172.21.15.165:6836/1615628763 2024-05-09T08:28:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.359+0000 7fa743fff640 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x562160b02960 0x562160b04d80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c1abac7954e51d1f server_cookie=430d1286924379c1 in_seq=0 out_seq=0 2024-05-09T08:28:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.362+0000 7fa7237fe640 1 -- 172.21.15.28:0/2846788556 <== mds.3 v2:172.21.15.165:6836/1615628763 1 ==== command_reply(tid 0: 0 ) ==== 8+0+73939 (crc 0 0 0) 0x5621609ee6c0 con 0x562160b02960 2024-05-09T08:28:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.362+0000 7fa7237fe640 10 client.16455 handle_command_reply: tid=0 2024-05-09T08:28:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.362+0000 7fa74d40f740 1 client.16455 shutdown 2024-05-09T08:28:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 -- 172.21.15.28:0/2846788556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 msgr2=0x562160a8f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a8f620 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa73c022a90 tx=0x7fa73c003970 comp rx=0 tx=0).stop 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 -- 172.21.15.28:0/2846788556 shutdown_connections 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.165:6836/1615628763,v1:172.21.15.165:6837/1615628763] conn(0x562160b02960 0x562160b04d80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562160a8c320 0x562160a8f620 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562160a81c50 0x562160afb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa7237fe640 0 client.16455 ms_handle_reset on v2:172.21.15.165:6836/1615628763 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 --2- 172.21.15.28:0/2846788556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562160a81330 0x562160afab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 -- 172.21.15.28:0/2846788556 >> 172.21.15.28:0/2846788556 conn(0x562160a7d650 msgr2=0x562160a7bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 -- 172.21.15.28:0/2846788556 shutdown_connections 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 1 -- 172.21.15.28:0/2846788556 wait complete. 2024-05-09T08:28:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74d40f740 20 client.16455 trim_cache size 0 max 16384 2024-05-09T08:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-05-09T08:28:35.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.369 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.369 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1691056, 2024-05-09T08:28:35.369 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628784, 2024-05-09T08:28:35.369 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-28.jpg" 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3075453, 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630970, 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-54.xml" 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7912273, 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630646, 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-29.xml" 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 19206521, 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628728, 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-89.xml" 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 35556203, 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628773, 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-32.jpg" 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 37064789, 2024-05-09T08:28:35.371 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631297, 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-42.xml" 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 49628498, 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628737, 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-86.xml" 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 53223335, 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630801, 2024-05-09T08:28:35.372 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-70.xml" 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 68998011, 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629013, 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-61.jpg" 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 100013233, 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631044, 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-99.xml" 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 142985928, 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630160, 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-34.jpg" 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 144999770, 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630066, 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-78.jpg" 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.374 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 202861153, 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628909, 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-25.xml" 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 239223632, 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628858, 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-45.xml" 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 244316040, 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630086, 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-68.jpg" 2024-05-09T08:28:35.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 255627609, 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628750, 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-58.xml" 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 287890724, 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631223, 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-58.xml" 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.376 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 289972258, 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628781, 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-71.xml" 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 293438506, 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628811, 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-19.jpg" 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 314289078, 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630547, 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-58.xml" 2024-05-09T08:28:35.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 316678561, 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628818, 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-17.jpg" 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 322148415, 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628969, 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-4.xml" 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.378 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 324928162, 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629043, 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-49.jpg" 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 325858341, 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628845, 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-50.xml" 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 327666284, 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630004, 2024-05-09T08:28:35.379 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-18.xml" 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 350175992, 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631160, 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-71.xml" 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 366431824, 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631148, 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-36.xml" 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 392403956, 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628830, 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-13.jpg" 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 415393201, 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631240, 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-55.xml" 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 433483280, 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631264, 2024-05-09T08:28:35.381 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-49.xml" 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 434108997, 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628767, 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-34.jpg" 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 443967667, 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630037, 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-90.jpg" 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 486932057, 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628743, 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-84.xml" 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 497102498, 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629385, 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-42.jpg" 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.383 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 500420415, 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628740, 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-5.xml" 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 501069725, 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630165, 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-32.jpg" 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 510706781, 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629064, 2024-05-09T08:28:35.384 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-97.xml" 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 511463612, 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630658, 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-21.xml" 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 529821141, 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628942, 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-13.xml" 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.385 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 537895592, 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628904, 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-27.xml" 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 576399834, 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628847, 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-7.jpg" 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 641104003, 2024-05-09T08:28:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629995, 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-27.xml" 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 649071471, 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630024, 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-1.xml" 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 657679186, 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630937, 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-7.jpg" 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 659311636, 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630208, 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-16.jpg" 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 664062766, 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629986, 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-36.xml" 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 695539810, 2024-05-09T08:28:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630808, 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-18.jpg" 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 733663014, 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628722, 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-91.xml" 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 734498331, 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628727, 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-9.xml" 2024-05-09T08:28:35.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 741829620, 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630206, 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-17.jpg" 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 749351313, 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630136, 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-78.xml" 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.390 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 800465188, 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630967, 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-4.jpg" 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 811525216, 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630643, 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-30.xml" 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 820308641, 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628841, 2024-05-09T08:28:35.391 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-9.jpg" 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 821387302, 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630659, 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-20.xml" 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 828659938, 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631063, 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-95.xml" 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 894847959, 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629124, 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-91.xml" 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 895259661, 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631071, 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-93.xml" 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.393 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 896737240, 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628939, 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-14.xml" 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 905815176, 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629649, 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-36.jpg" 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 919187694, 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630670, 2024-05-09T08:28:35.394 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-12.xml" 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 921114147, 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631256, 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-51.xml" 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 957453590, 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628906, 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-26.xml" 2024-05-09T08:28:35.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 961462280, 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628714, 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-94.xml" 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 967634153, 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630907, 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-60.xml" 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.396 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 972523860, 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630054, 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-83.jpg" 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1001020152, 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629176, 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-57.jpg" 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1017199308, 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630094, 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-64.jpg" 2024-05-09T08:28:35.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1044080981, 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631248, 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-53.xml" 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1049236503, 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630762, 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-21.jpg" 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1062125969, 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631187, 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-65.xml" 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1085602239, 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630663, 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-16.xml" 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.399 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1089233160, 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628854, 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-47.xml" 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1091565722, 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628827, 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-56.xml" 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1097657290, 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628719, 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-92.xml" 2024-05-09T08:28:35.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1127653550, 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629097, 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-75.xml" 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1148858153, 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629657, 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-35.jpg" 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.401 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1157290303, 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629985, 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-37.xml" 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1168790224, 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630656, 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-23.xml" 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1169465874, 2024-05-09T08:28:35.402 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629572, 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-10.jpg" 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1191164962, 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630772, 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-72.xml" 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1194843907, 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628789, 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-69.xml" 2024-05-09T08:28:35.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1201308634, 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630189, 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-24.jpg" 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1201325473, 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630010, 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-12.xml" 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.404 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1207757936, 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631235, 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-56.xml" 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1214994307, 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631323, 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-11.xml" 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1226376208, 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631173, 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-33.xml" 2024-05-09T08:28:35.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1238011462, 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630653, 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-26.xml" 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1260621263, 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629999, 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-23.xml" 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.406 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1272152249, 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630239, 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-6.jpg" 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1295638581, 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629983, 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-39.xml" 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1345210540, 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631030, 2024-05-09T08:28:35.407 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-48.xml" 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1353214759, 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630773, 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-24.xml" 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1369436936, 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630013, 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-9.xml" 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.408 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1371495524, 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630571, 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-51.xml" 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1376612431, 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630149, 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-39.jpg" 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1383406021, 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630219, 2024-05-09T08:28:35.409 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-25.jpg" 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1389858451, 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629811, 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-83.xml" 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1404741709, 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628833, 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-54.xml" 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1404938867, 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630554, 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-56.xml" 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1418746224, 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628950, 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-10.xml" 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1481765360, 2024-05-09T08:28:35.411 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630769, 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-52.jpg" 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1488997603, 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629924, 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-29.jpg" 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1492121554, 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631293, 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-43.xml" 2024-05-09T08:28:35.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1504816375, 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629050, 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-47.jpg" 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1509037501, 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630667, 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-13.xml" 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1569761346, 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628826, 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-14.jpg" 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1573911725, 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630288, 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-74.xml" 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1604549516, 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630049, 2024-05-09T08:28:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-85.jpg" 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1612352806, 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630216, 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-10.jpg" 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1616630931, 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630005, 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-17.xml" 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1631495822, 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629069, 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-96.xml" 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1668083825, 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630629, 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-34.xml" 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1680395066, 2024-05-09T08:28:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631112, 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-84.xml" 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1687689284, 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630235, 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-7.jpg" 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1738857850, 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629031, 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-54.jpg" 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1752119688, 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629978, 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-44.xml" 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1766799772, 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628716, 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-93.xml" 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1772949685, 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628890, 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-32.xml" 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1792411863, 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629091, 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-94.xml" 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1795845676, 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630116, 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-53.jpg" 2024-05-09T08:28:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1813529763, 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628948, 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-11.xml" 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1822616730, 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629926, 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-1.jpg" 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1823408244, 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629918, 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-82.xml" 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1860670429, 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629057, 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-98.xml" 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1861324653, 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630211, 2024-05-09T08:28:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-14.jpg" 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1865326124, 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628935, 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-15.xml" 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1876450698, 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628831, 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-55.xml" 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1883773261, 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629131, 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-90.xml" 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1899998272, 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628892, 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-31.xml" 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1907787573, 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628859, 2024-05-09T08:28:35.423 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-2.jpg" 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1925922062, 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629067, 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-43.jpg" 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1927160453, 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631303, 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-41.xml" 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.424 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1939037461, 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630502, 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-71.xml" 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1944394981, 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630270, 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-23.jpg" 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1948050872, 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628791, 2024-05-09T08:28:35.425 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-68.xml" 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1952986441, 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630654, 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-25.xml" 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1971062874, 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628749, 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-82.xml" 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1983262968, 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630163, 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-33.jpg" 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1987725090, 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630015, 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-7.xml" 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1994025039, 2024-05-09T08:28:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628856, 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-46.xml" 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2000409339, 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631142, 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-75.xml" 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2001777388, 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631124, 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-80.xml" 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2009599032, 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628866, 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-41.xml" 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2023983093, 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629033, 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-3.xml" 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.429 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2038470552, 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630126, 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-48.jpg" 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2068588029, 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629659, 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-86.xml" 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2076343252, 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630108, 2024-05-09T08:28:35.430 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-57.jpg" 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2077168423, 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631182, 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-66.xml" 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2085553459, 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628733, 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-7.xml" 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.431 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2098700022, 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631181, 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-32.xml" 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2119441230, 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628751, 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-1.xml" 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2120546907, 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631174, 2024-05-09T08:28:35.432 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-68.xml" 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2155008875, 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630041, 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-89.jpg" 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2168195926, 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630557, 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-55.xml" 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2169243449, 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630751, 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-73.xml" 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2171646185, 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630252, 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-2.jpg" 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.434 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2178890522, 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631107, 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-85.xml" 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2190584649, 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629639, 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-89.xml" 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2196803585, 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631319, 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-38.xml" 2024-05-09T08:28:35.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2231461834, 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630070, 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-76.jpg" 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2242404436, 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631048, 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-46.xml" 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.436 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2242495112, 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629017, 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-59.jpg" 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2245195649, 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628788, 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-27.jpg" 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2245472497, 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631115, 2024-05-09T08:28:35.437 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-39.xml" 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2254376886, 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628739, 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-24.xml" 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2267312302, 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630706, 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-3.xml" 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.438 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2292676533, 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629094, 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-41.jpg" 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2309304580, 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628927, 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-18.xml" 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2312936318, 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630057, 2024-05-09T08:28:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-82.jpg" 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2313711187, 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629002, 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-66.jpg" 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2320277316, 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628922, 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-20.xml" 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2321450964, 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631304, 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-14.xml" 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2323668827, 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631253, 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-52.xml" 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2326378844, 2024-05-09T08:28:35.441 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630100, 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-61.jpg" 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2339593201, 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630892, 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-62.xml" 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2370956116, 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631289, 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-44.xml" 2024-05-09T08:28:35.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2382256188, 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631207, 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-61.xml" 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2404226351, 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630781, 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-20.jpg" 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.443 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2427013856, 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630871, 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-64.xml" 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2433783974, 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629981, 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-41.xml" 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2471212844, 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630201, 2024-05-09T08:28:35.444 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-19.jpg" 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2497477784, 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629917, 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-31.jpg" 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2505711844, 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628855, 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-4.jpg" 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.445 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2519178280, 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630203, 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-18.jpg" 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2537402496, 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628882, 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-35.xml" 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2566611817, 2024-05-09T08:28:35.446 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628725, 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-90.xml" 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2574619235, 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629424, 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-39.jpg" 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2596203658, 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628864, 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-42.xml" 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2611049783, 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628981, 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-73.jpg" 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2612934612, 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629015, 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-60.jpg" 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2614384022, 2024-05-09T08:28:35.448 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628874, 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-38.xml" 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2646275824, 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629025, 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-56.jpg" 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2653232854, 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630661, 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-18.xml" 2024-05-09T08:28:35.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2673520015, 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630227, 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-9.jpg" 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2684595535, 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630033, 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-92.jpg" 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.450 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2702502505, 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628842, 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-51.xml" 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2710437174, 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631010, 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-50.xml" 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2712421549, 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628796, 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-24.jpg" 2024-05-09T08:28:35.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2717192708, 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630619, 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-37.xml" 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2747390815, 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628835, 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-11.jpg" 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.452 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2760424524, 2024-05-09T08:28:35.453 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628857, 2024-05-09T08:28:35.453 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-3.jpg" 2024-05-09T08:28:35.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.453 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.453 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2766985365, 2024-05-09T08:28:35.453 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630853, 2024-05-09T08:28:35.453 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-14.jpg" 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2783443461, 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631257, 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-20.xml" 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.454 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2788211772, 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630177, 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-28.jpg" 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2801297688, 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628774, 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-74.xml" 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2803344897, 2024-05-09T08:28:35.455 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628853, 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-5.jpg" 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2823979037, 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630602, 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-42.xml" 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2834394178, 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631325, 2024-05-09T08:28:35.456 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-37.xml" 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2857004487, 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628770, 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-33.jpg" 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2874795341, 2024-05-09T08:28:35.457 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631283, 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-45.xml" 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2883257357, 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629061, 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-44.jpg" 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2900083504, 2024-05-09T08:28:35.458 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630837, 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-67.xml" 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2918435494, 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631204, 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-29.xml" 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2930716047, 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628920, 2024-05-09T08:28:35.459 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-21.xml" 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2940210736, 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630141, 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-26.jpg" 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2948423890, 2024-05-09T08:28:35.460 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630877, 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-12.jpg" 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2959207404, 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630698, 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-5.xml" 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2964889244, 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631267, 2024-05-09T08:28:35.461 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-19.xml" 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2969592998, 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629631, 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-38.jpg" 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3015176136, 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629503, 2024-05-09T08:28:35.462 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-33.jpg" 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3023064105, 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628936, 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-9.xml" 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3027879308, 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628820, 2024-05-09T08:28:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-16.jpg" 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3054882650, 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628963, 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-6.xml" 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3057849478, 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629530, 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-24.jpg" 2024-05-09T08:28:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3063775713, 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628869, 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-40.xml" 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3067612133, 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629584, 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-6.jpg" 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3069425361, 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628924, 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-19.xml" 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3090137522, 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628957, 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-8.xml" 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3142933278, 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630325, 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-63.xml" 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3151219771, 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630987, 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-2.jpg" 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.467 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3190916067, 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630002, 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-20.xml" 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3194504054, 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628819, 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-59.xml" 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.468 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3196732154, 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629518, 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-28.jpg" 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3214258154, 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631120, 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-81.xml" 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.469 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3214820103, 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631076, 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-92.xml" 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3215492162, 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628984, 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-72.jpg" 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.470 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3218886991, 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628861, 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-1.jpg" 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3229858493, 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628895, 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-30.xml" 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3233470102, 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628932, 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-16.xml" 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3251106582, 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630245, 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-4.jpg" 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3321419444, 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628975, 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-2.xml" 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3334449824, 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630523, 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-65.xml" 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3337581020, 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630674, 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-11.xml" 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3364097015, 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629645, 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-88.xml" 2024-05-09T08:28:35.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3370023201, 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630142, 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-42.jpg" 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3375280213, 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630746, 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-22.jpg" 2024-05-09T08:28:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3380485576, 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630074, 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-74.jpg" 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3393146277, 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628838, 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-10.jpg" 2024-05-09T08:28:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3431332124, 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628745, 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-3.xml" 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3453508011, 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630816, 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-69.xml" 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3457173235, 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630843, 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-15.jpg" 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3482835605, 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629991, 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-31.xml" 2024-05-09T08:28:35.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3490024847, 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628718, 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-12.xml" 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3493298148, 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630153, 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-37.jpg" 2024-05-09T08:28:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3520706365, 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629059, 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-45.jpg" 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3533326829, 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628814, 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-18.jpg" 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3544061451, 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630032, 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-79.xml" 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3544136171, 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628851, 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-48.xml" 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3544292856, 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630913, 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-9.jpg" 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3555991547, 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630157, 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-35.jpg" 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3569811711, 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629071, 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-70.jpg" 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3576527437, 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630782, 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-3.xml" 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3576799226, 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629977, 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-45.xml" 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3584431990, 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630880, 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-63.xml" 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3586740279, 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630088, 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-67.jpg" 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3681632711, 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630952, 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-56.xml" 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3683371518, 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631105, 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-40.xml" 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3737938530, 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630865, 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-13.jpg" 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3752162032, 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628746, 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-83.xml" 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3765137158, 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631137, 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-76.xml" 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3834840612, 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629087, 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-57.jpg" 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3836276681, 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630578, 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-49.xml" 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3836698540, 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629449, 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-37.jpg" 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3852196762, 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629971, 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-51.xml" 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3871153241, 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628757, 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-37.jpg" 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3892000270, 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628961, 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-7.xml" 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3898944190, 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631229, 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-57.xml" 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3915324275, 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631039, 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-100.xml" 2024-05-09T08:28:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3924939659, 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630180, 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-27.jpg" 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4005398611, 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630797, 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-19.jpg" 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4028506300, 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630031, 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-93.jpg" 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4033444077, 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630256, 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-1.jpg" 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4052490565, 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630212, 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-13.jpg" 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4055218510, 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630919, 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-59.xml" 2024-05-09T08:28:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4058489515, 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631067, 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-94.xml" 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4059446553, 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631004, 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-73.xml" 2024-05-09T08:28:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4063592700, 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628812, 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-61.xml" 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4066110856, 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628785, 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-70.xml" 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4069568596, 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628768, 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-76.xml" 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4083994442, 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629921, 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-30.jpg" 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4085670590, 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628703, 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-17.xml" 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4098153106, 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630486, 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-75.xml" 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4132676385, 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630925, 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-8.jpg" 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4162897597, 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628711, 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-95.xml" 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4181086228, 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628763, 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-35.jpg" 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4220149583, 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631040, 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-47.xml" 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.500 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4248793657, 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628779, 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-72.xml" 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4251550757, 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631098, 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-87.xml" 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.501 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4265170918, 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628850, 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-6.jpg" 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4266600770, 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629551, 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-17.jpg" 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.502 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4267748662, 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628704, 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-97.xml" 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4269295579, 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630145, 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-77.xml" 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.503 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4286367932, 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630606, 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-41.xml" 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4294295330, 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629116, 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-92.xml" 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.504 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.505 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4294652910, 2024-05-09T08:28:35.505 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630655, 2024-05-09T08:28:35.505 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-24.xml" 2024-05-09T08:28:35.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:28:35.505 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-05-09T08:28:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.363+0000 7fa74a709640 1 -- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa72c0a2d00 msgr2=0x7fa72c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa72c0a2d00 0x7fa72c0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa73c005c40 tx=0x7fa73c004ab0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 -- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 msgr2=0x7fa744110b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 0x7fa744110b60 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa73400ea20 tx=0x7fa73400eee0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 -- 172.21.15.28:0/1862673365 shutdown_connections 2024-05-09T08:28:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa744107820 0x7fa7441115e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa72c0a2d00 0x7fa72c0a51c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa74406a870 0x7fa7441110a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 --2- 172.21.15.28:0/1862673365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa744069eb0 0x7fa744110b60 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 -- 172.21.15.28:0/1862673365 >> 172.21.15.28:0/1862673365 conn(0x7fa744077790 msgr2=0x7fa7440fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 -- 172.21.15.28:0/1862673365 shutdown_connections 2024-05-09T08:28:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.364+0000 7fa74a709640 1 -- 172.21.15.28:0/1862673365 wait complete. 2024-05-09T08:28:35.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:28:35.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.678+0000 7f20dff12640 1 -- 172.21.15.28:0/3955884620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d81010c0 msgr2=0x7f20d810d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.678+0000 7f20dff12640 1 --2- 172.21.15.28:0/3955884620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d81010c0 0x7f20d810d5d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009cd0 tx=0x7f20d002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.678+0000 7f20dff12640 1 -- 172.21.15.28:0/3955884620 shutdown_connections 2024-05-09T08:28:35.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.678+0000 7f20dff12640 1 --2- 172.21.15.28:0/3955884620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d81010c0 0x7f20d810d5d0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.678+0000 7f20dff12640 1 --2- 172.21.15.28:0/3955884620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20d8100700 0x7f20d8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.678+0000 7f20dff12640 1 --2- 172.21.15.28:0/3955884620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d8106710 0x7f20d8106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.678+0000 7f20dff12640 1 -- 172.21.15.28:0/3955884620 >> 172.21.15.28:0/3955884620 conn(0x7f20d8077830 msgr2=0x7f20d8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.679+0000 7f20dff12640 1 -- 172.21.15.28:0/3955884620 shutdown_connections 2024-05-09T08:28:35.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.679+0000 7f20dff12640 1 -- 172.21.15.28:0/3955884620 wait complete. 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.679+0000 7f20dff12640 1 Processor -- start 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.679+0000 7f20dff12640 1 -- start start 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20dff12640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20d8100700 0x7f20d810d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20dff12640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d81010c0 0x7f20d8109360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20dff12640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d8106710 0x7f20d81098a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20dff12640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20d810fa10 con 0x7f20d8106710 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20dff12640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f20d810f890 con 0x7f20d81010c0 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20dff12640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f20d810fb90 con 0x7f20d8100700 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20de4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d8106710 0x7f20d81098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20de4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d8106710 0x7f20d81098a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35766/0 (socket says 172.21.15.28:35766) 2024-05-09T08:28:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20de4af640 1 -- 172.21.15.28:0/3376358254 learned_addr learned my addr 172.21.15.28:0/3376358254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d81010c0 0x7f20d8109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20ddcae640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20d8100700 0x7f20d810d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20de4af640 1 -- 172.21.15.28:0/3376358254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20d8100700 msgr2=0x7f20d810d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20de4af640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20d8100700 0x7f20d810d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20de4af640 1 -- 172.21.15.28:0/3376358254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d81010c0 msgr2=0x7f20d8109360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20de4af640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d81010c0 0x7f20d8109360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20de4af640 1 -- 172.21.15.28:0/3376358254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d810a020 con 0x7f20d8106710 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20ddcae640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20d8100700 0x7f20d810d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.680+0000 7f20de4af640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d8106710 0x7f20d81098a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f20d0005cc0 tx=0x7f20d00093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.681+0000 7f20ceffd640 1 -- 172.21.15.28:0/3376358254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0003e00 con 0x7f20d8106710 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.681+0000 7f20dff12640 1 -- 172.21.15.28:0/3376358254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d81aadb0 con 0x7f20d8106710 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.681+0000 7f20ceffd640 1 -- 172.21.15.28:0/3376358254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d0004520 con 0x7f20d8106710 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.681+0000 7f20ceffd640 1 -- 172.21.15.28:0/3376358254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0030db0 con 0x7f20d8106710 2024-05-09T08:28:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.681+0000 7f20dff12640 1 -- 172.21.15.28:0/3376358254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d81ab270 con 0x7f20d8106710 2024-05-09T08:28:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:35 smithi028 ceph-mon[30119]: pgmap v71: 129 pgs: 129 active+clean; 160 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 706 op/s 2024-05-09T08:28:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:35 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1364260501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.683+0000 7f20ceffd640 1 -- 172.21.15.28:0/3376358254 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f20d00046d0 con 0x7f20d8106710 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.684+0000 7f20e2c16740 1 Processor -- start 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.684+0000 7f20e2c16740 1 -- start start 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.684+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fa6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.684+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fa6c50 0x5641f8fb0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fb1320 0x5641f8fb3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20e2c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641f8f0bc40 con 0x5641f8fb1320 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20e2c16740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5641f8eb3580 con 0x5641f8fa6330 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20e2c16740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5641f8f07c00 con 0x5641f8fa6c50 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ddcae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fa6c50 0x5641f8fb0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20dd4ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fa6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ddcae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fa6c50 0x5641f8fb0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:53682/0 (socket says 172.21.15.28:53682) 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ddcae640 1 -- 172.21.15.28:0/2013578890 learned_addr learned my addr 172.21.15.28:0/2013578890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2013578890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fb1320 0x5641f8fb3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ceffd640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f20c40a2d00 0x7f20c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ddcae640 1 -- 172.21.15.28:0/2013578890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 msgr2=0x5641f8fa6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ddcae640 1 --2- 172.21.15.28:0/2013578890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fa6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ddcae640 1 -- 172.21.15.28:0/2013578890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fb1320 msgr2=0x5641f8fb3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ddcae640 1 --2- 172.21.15.28:0/2013578890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fb1320 0x5641f8fb3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ddcae640 1 -- 172.21.15.28:0/2013578890 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641f8c1f3f0 con 0x5641f8fa6c50 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2013578890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fb1320 0x5641f8fb3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20de4af640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f20c40a2d00 0x7f20c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ddcae640 1 --2- 172.21.15.28:0/2013578890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fa6c50 0x5641f8fb0de0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f20d40114f0 tx=0x7f20d40119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b2d51e44c02598d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.685+0000 7f20ceffd640 1 -- 172.21.15.28:0/3376358254 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f20d002e030 con 0x7f20d8106710 2024-05-09T08:28:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.686+0000 7f20c3fff640 1 -- 172.21.15.28:0/2013578890 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d401e070 con 0x5641f8fa6c50 2024-05-09T08:28:35.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.686+0000 7f20c3fff640 1 -- 172.21.15.28:0/2013578890 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d400faa0 con 0x5641f8fa6c50 2024-05-09T08:28:35.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.686+0000 7f20c3fff640 1 -- 172.21.15.28:0/2013578890 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4019b50 con 0x5641f8fa6c50 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.686+0000 7f20de4af640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f20c40a2d00 0x7f20c40a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f20d004d000 tx=0x7f20d008cdc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.686+0000 7f20e2c16740 1 -- 172.21.15.28:0/2013578890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fa6c50 msgr2=0x5641f8fb0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.686+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2013578890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fa6c50 0x5641f8fb0de0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f20d40114f0 tx=0x7f20d40119b0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.686+0000 7f20e2c16740 1 -- 172.21.15.28:0/2013578890 shutdown_connections 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.686+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2013578890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fb1320 0x5641f8fb3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.687+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2013578890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fa6c50 0x5641f8fb0de0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.687+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2013578890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fa6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.687+0000 7f20e2c16740 1 -- 172.21.15.28:0/2013578890 >> 172.21.15.28:0/2013578890 conn(0x5641f8fa2650 msgr2=0x5641f8fa2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.687+0000 7f20e2c16740 1 -- 172.21.15.28:0/2013578890 shutdown_connections 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.687+0000 7f20e2c16740 1 -- 172.21.15.28:0/2013578890 wait complete. 2024-05-09T08:28:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.687+0000 7f20e2c16740 1 -- start start 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.687+0000 7f20e2c16740 1 Processor -- start 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.687+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fae210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.687+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6c50 0x5641f8fae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8faec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20e2c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641f8fb4110 con 0x5641f8fa6c50 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20e2c16740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5641f8ec6c40 con 0x5641f8fa6330 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20e2c16740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5641f8fb4290 con 0x5641f8fb1320 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20e2c16740 10 client.0 _ll_register_callbacks cb 0x5641f8c1ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20ddcae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6c50 0x5641f8fae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fae210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:50554/0 (socket says 172.21.15.28:50554) 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 -- 172.21.15.28:0/2931514606 learned_addr learned my addr 172.21.15.28:0/2931514606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8faec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 -- 172.21.15.28:0/2931514606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 msgr2=0x5641f8faec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8faec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 -- 172.21.15.28:0/2931514606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6c50 msgr2=0x5641f8fae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6c50 0x5641f8fae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 -- 172.21.15.28:0/2931514606 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641f8ce97d0 con 0x5641f8fa6330 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fae210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f20c8011ea0 tx=0x7f20c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34cd754a6669c07f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20c17fa640 1 -- 172.21.15.28:0/2931514606 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8011320 con 0x5641f8fa6330 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20c17fa640 1 -- 172.21.15.28:0/2931514606 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f20c8009040 con 0x5641f8fa6330 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.688+0000 7f20e2c16740 1 -- 172.21.15.28:0/2931514606 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20d400faa0 con 0x5641f8fa6330 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20c17fa640 1 -- 172.21.15.28:0/2931514606 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f20c8029050 con 0x5641f8fa6330 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20e2c16740 10 client.25934 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20e2c16740 1 -- 172.21.15.28:0/2931514606 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5641f8fc0290 con 0x5641f8fa6330 2024-05-09T08:28:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20c17fa640 1 -- 172.21.15.28:0/2931514606 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d400faa0 con 0x5641f8fa6330 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20c17fa640 1 -- 172.21.15.28:0/2931514606 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f20c8020040 con 0x5641f8fa6330 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20e2c16740 10 client.25934 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20e2c16740 10 client.25934 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20e2c16740 20 client.25934 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x5641f8fc16a0 0x5641f8fc3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20e2c16740 4 client.25934 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20e2c16740 1 -- 172.21.15.28:0/2931514606 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641f8f19370 con 0x5641f8fc16a0 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.689+0000 7f20ddcae640 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x5641f8fc16a0 0x5641f8fc3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.690+0000 7f20ddcae640 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x5641f8fc16a0 0x5641f8fc3ad0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=525029ebd0a13d40 server_cookie=3d02160f33e510dc in_seq=0 out_seq=0 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.690+0000 7f20c17fa640 10 client.25934 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.690+0000 7f20c17fa640 1 -- 172.21.15.28:0/2931514606 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5641f8f19370 con 0x5641f8fc16a0 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.690+0000 7f20c17fa640 10 client.25934 handle_command_reply: tid=0 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.690+0000 7f20e2c16740 1 client.25934 shutdown 2024-05-09T08:28:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.690+0000 7f20e2c16740 1 -- 172.21.15.28:0/2931514606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 msgr2=0x5641f8fae210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.690+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fae210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f20c8011ea0 tx=0x7f20c80046c0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20e2c16740 1 -- 172.21.15.28:0/2931514606 shutdown_connections 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x5641f8fc16a0 0x5641f8fc3ad0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8faec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6c50 0x5641f8fae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2931514606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6330 0x5641f8fae210 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20c17fa640 0 client.25934 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20e2c16740 1 -- 172.21.15.28:0/2931514606 >> 172.21.15.28:0/2931514606 conn(0x5641f8fa2650 msgr2=0x5641f8fa8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20e2c16740 1 -- 172.21.15.28:0/2931514606 shutdown_connections 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20e2c16740 1 -- 172.21.15.28:0/2931514606 wait complete. 2024-05-09T08:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.691+0000 7f20e2c16740 20 client.25934 trim_cache size 0 max 16384 2024-05-09T08:28:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.706+0000 7f20e2c16740 1 Processor -- start 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.706+0000 7f20e2c16740 1 -- start start 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.706+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fbd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.706+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 0x5641f8fc93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.706+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8fc98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.706+0000 7f20e2c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20d81b5450 con 0x5641f8fa6330 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.706+0000 7f20e2c16740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f20d81b7050 con 0x5641f8fa6c50 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.706+0000 7f20e2c16740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f20d81b5b20 con 0x5641f8fb1320 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fbd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fbd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35788/0 (socket says 172.21.15.28:35788) 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20ddcae640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 0x5641f8fc93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 -- 172.21.15.28:0/636494262 learned_addr learned my addr 172.21.15.28:0/636494262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/636494262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8fc98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 -- 172.21.15.28:0/636494262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 msgr2=0x5641f8fc98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/636494262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8fc98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 -- 172.21.15.28:0/636494262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 msgr2=0x5641f8fc93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/636494262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 0x5641f8fc93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 -- 172.21.15.28:0/636494262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641f8fa2480 con 0x5641f8fa6330 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20ddcae640 1 --2- 172.21.15.28:0/636494262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 0x5641f8fc93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/636494262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fbd000 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f20c8010010 tx=0x7f20c8009270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bab3861dd5bc36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20c37fe640 1 -- 172.21.15.28:0/636494262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8011320 con 0x5641f8fa6330 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20c37fe640 1 -- 172.21.15.28:0/636494262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20c8026070 con 0x5641f8fa6330 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.707+0000 7f20c37fe640 1 -- 172.21.15.28:0/636494262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8004360 con 0x5641f8fa6330 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 -- 172.21.15.28:0/636494262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 msgr2=0x5641f8fbd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 --2- 172.21.15.28:0/636494262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fbd000 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f20c8010010 tx=0x7f20c8009270 comp rx=0 tx=0).stop 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 -- 172.21.15.28:0/636494262 shutdown_connections 2024-05-09T08:28:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 --2- 172.21.15.28:0/636494262 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8fc98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 --2- 172.21.15.28:0/636494262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 0x5641f8fc93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 --2- 172.21.15.28:0/636494262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fbd000 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 -- 172.21.15.28:0/636494262 >> 172.21.15.28:0/636494262 conn(0x5641f8fa2650 msgr2=0x5641f8fa8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 -- 172.21.15.28:0/636494262 shutdown_connections 2024-05-09T08:28:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 -- 172.21.15.28:0/636494262 wait complete. 2024-05-09T08:28:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 -- start start 2024-05-09T08:28:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.708+0000 7f20e2c16740 1 Processor -- start 2024-05-09T08:28:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fb9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 0x5641f8fb9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20e2c16740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8fb64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20e2c16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20d81b68c0 con 0x5641f8fa6330 2024-05-09T08:28:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20e2c16740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f20d81b61f0 con 0x5641f8fa6c50 2024-05-09T08:28:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20e2c16740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f20d81b3430 con 0x5641f8fb1320 2024-05-09T08:28:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20e2c16740 10 client.0 _ll_register_callbacks cb 0x7f20c8009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20dd4ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fb9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20dd4ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fb9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35798/0 (socket says 172.21.15.28:35798) 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20dd4ad640 1 -- 172.21.15.28:0/2867589430 learned_addr learned my addr 172.21.15.28:0/2867589430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20ddcae640 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 0x5641f8fb9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8fb64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20dd4ad640 1 -- 172.21.15.28:0/2867589430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 msgr2=0x5641f8fb64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8fb64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20dd4ad640 1 -- 172.21.15.28:0/2867589430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 msgr2=0x5641f8fb9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 0x5641f8fb9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.709+0000 7f20dd4ad640 1 -- 172.21.15.28:0/2867589430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641f8fa2070 con 0x5641f8fa6330 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20dd4ad640 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fb9510 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f20c80116c0 tx=0x7f20c8003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6676dc84738ccc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20c17fa640 1 -- 172.21.15.28:0/2867589430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8003a00 con 0x5641f8fa6330 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20c17fa640 1 -- 172.21.15.28:0/2867589430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f20c80050c0 con 0x5641f8fa6330 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20e2c16740 1 -- 172.21.15.28:0/2867589430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20c8026070 con 0x5641f8fa6330 2024-05-09T08:28:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20c17fa640 1 -- 172.21.15.28:0/2867589430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f20c80052c0 con 0x5641f8fa6330 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20e2c16740 10 client.16473 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20e2c16740 1 -- 172.21.15.28:0/2867589430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5641f90284c0 con 0x5641f8fa6330 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20c17fa640 1 -- 172.21.15.28:0/2867589430 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20c8026070 con 0x5641f8fa6330 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20c17fa640 1 -- 172.21.15.28:0/2867589430 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f20c8005560 con 0x5641f8fa6330 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20e2c16740 10 client.16473 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20e2c16740 10 client.16473 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20e2c16740 20 client.16473 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x5641f9028b20 0x5641f902af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20e2c16740 4 client.16473 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.710+0000 7f20e2c16740 1 -- 172.21.15.28:0/2867589430 --> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641f8f136c0 con 0x5641f9028b20 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.711+0000 7f20ddcae640 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x5641f9028b20 0x5641f902af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.711+0000 7f20ddcae640 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x5641f9028b20 0x5641f902af10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fb81ea02ae5101fd server_cookie=c17c65e0961aab5a in_seq=0 out_seq=0 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.711+0000 7f20c17fa640 10 client.16473 ms_handle_connect on v2:172.21.15.184:6836/3795789755 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.712+0000 7f20c17fa640 1 -- 172.21.15.28:0/2867589430 <== mds.2 v2:172.21.15.184:6836/3795789755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+54231 (crc 0 0 0) 0x5641f8f136c0 con 0x5641f9028b20 2024-05-09T08:28:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.712+0000 7f20c17fa640 10 client.16473 handle_command_reply: tid=0 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.712+0000 7f20e2c16740 1 client.16473 shutdown 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 -- 172.21.15.28:0/2867589430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 msgr2=0x5641f8fb9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fb9510 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f20c80116c0 tx=0x7f20c8003f00 comp rx=0 tx=0).stop 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 -- 172.21.15.28:0/2867589430 shutdown_connections 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.184:6836/3795789755,v1:172.21.15.184:6837/3795789755] conn(0x5641f9028b20 0x5641f902af10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5641f8fb1320 0x5641f8fb64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5641f8fa6c50 0x5641f8fb9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20c17fa640 0 client.16473 ms_handle_reset on v2:172.21.15.184:6836/3795789755 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 --2- 172.21.15.28:0/2867589430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641f8fa6330 0x5641f8fb9510 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 -- 172.21.15.28:0/2867589430 >> 172.21.15.28:0/2867589430 conn(0x5641f8fa2650 msgr2=0x5641f8faf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 -- 172.21.15.28:0/2867589430 shutdown_connections 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 1 -- 172.21.15.28:0/2867589430 wait complete. 2024-05-09T08:28:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20e2c16740 20 client.16473 trim_cache size 0 max 16384 2024-05-09T08:28:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-05-09T08:28:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 35760778, 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628797, 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-66.xml" 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 54623227, 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631310, 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-13.xml" 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 65272418, 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630488, 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-4.xml" 2024-05-09T08:28:35.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 69406482, 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630460, 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-11.xml" 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 93575386, 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630279, 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-54.xml" 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.721 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 111493306, 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630713, 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-6.jpg" 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 147075460, 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631066, 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-44.xml" 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 156936873, 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630364, 2024-05-09T08:28:35.722 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-35.xml" 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 194355636, 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629653, 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-87.xml" 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 237996004, 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631315, 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-12.xml" 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.723 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 250235332, 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628918, 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-22.xml" 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 263481486, 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630831, 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-16.jpg" 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 266276969, 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631227, 2024-05-09T08:28:35.724 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-25.xml" 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 270662266, 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630456, 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-12.xml" 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 344390909, 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628752, 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-81.xml" 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.725 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 350068804, 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628809, 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-62.xml" 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 352486896, 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630412, 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-23.xml" 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 353286721, 2024-05-09T08:28:35.726 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630452, 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-13.xml" 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 357826965, 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630380, 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-31.xml" 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 423912074, 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629663, 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-34.jpg" 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 455105318, 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630528, 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-33.jpg" 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 474548961, 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628804, 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-64.xml" 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 475434354, 2024-05-09T08:28:35.728 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631001, 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-51.xml" 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 482116640, 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628996, 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-68.jpg" 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 502144801, 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628801, 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-65.xml" 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 504002431, 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631298, 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-15.xml" 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 512352290, 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630492, 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-3.xml" 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 551079241, 2024-05-09T08:28:35.730 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630440, 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-16.xml" 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 568326091, 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628971, 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-3.xml" 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 578508010, 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629101, 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-93.xml" 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 585490697, 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630993, 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-52.xml" 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 586253369, 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628885, 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-34.xml" 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 600108682, 2024-05-09T08:28:35.732 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630631, 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-20.jpg" 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 610020162, 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630342, 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-40.xml" 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 613897610, 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629019, 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-58.jpg" 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 650456428, 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628945, 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-12.xml" 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 690650604, 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630697, 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-10.jpg" 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.734 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 735204533, 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629053, 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-46.jpg" 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 761511694, 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630901, 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-61.xml" 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 825569894, 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630368, 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-34.xml" 2024-05-09T08:28:35.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 844645104, 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630849, 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-66.xml" 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 860054014, 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630291, 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-52.xml" 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.736 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 871903858, 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630999, 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-1.jpg" 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 879907343, 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630898, 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-10.jpg" 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 928122385, 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630039, 2024-05-09T08:28:35.737 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-27.jpg" 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 934156289, 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630709, 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-7.jpg" 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 947130980, 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630693, 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-11.jpg" 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.738 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 996032279, 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630308, 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-48.xml" 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1001085975, 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630304, 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-49.xml" 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1046886487, 2024-05-09T08:28:35.739 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630023, 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-80.xml" 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1051625495, 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631097, 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-41.xml" 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1062314364, 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629048, 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-100.xml" 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.740 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1082045085, 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630480, 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-6.xml" 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1091486068, 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628999, 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-67.jpg" 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1123052374, 2024-05-09T08:28:35.741 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628953, 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-9.xml" 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1134302960, 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630590, 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-25.jpg" 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1147205189, 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630444, 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-15.xml" 2024-05-09T08:28:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1197208736, 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631250, 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-21.xml" 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1198322143, 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628761, 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-78.xml" 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.743 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1204491445, 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630717, 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-5.jpg" 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1211013812, 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631216, 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-27.xml" 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1211179005, 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628915, 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-23.xml" 2024-05-09T08:28:35.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1226586275, 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629037, 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-52.jpg" 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1248283392, 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631155, 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-35.xml" 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.745 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1254458876, 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629126, 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-39.jpg" 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1256949360, 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630404, 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-25.xml" 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1265014645, 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628989, 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-70.jpg" 2024-05-09T08:28:35.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1266434386, 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630296, 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-51.xml" 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1267771524, 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628888, 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-33.xml" 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.747 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1278900048, 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630958, 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-55.xml" 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1285083143, 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630476, 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-7.xml" 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1290355374, 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630559, 2024-05-09T08:28:35.748 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-29.jpg" 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1291922669, 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630689, 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-12.jpg" 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1297372442, 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631091, 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-42.xml" 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.749 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1306885947, 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631012, 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-50.jpg" 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1308188440, 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628897, 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-29.xml" 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1309691372, 2024-05-09T08:28:35.750 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629110, 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-40.jpg" 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1374814927, 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630424, 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-20.xml" 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1398925088, 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629914, 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-32.jpg" 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1438690839, 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630720, 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-4.jpg" 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1442662600, 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628839, 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-52.xml" 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1458411165, 2024-05-09T08:28:35.752 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631326, 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-76.xml" 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1506775299, 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628912, 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-24.xml" 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1511418643, 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630545, 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-31.jpg" 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1511993454, 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630514, 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-35.jpg" 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1520591950, 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628960, 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-7.xml" 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1551873850, 2024-05-09T08:28:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630028, 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-28.jpg" 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1582192302, 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630428, 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-19.xml" 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1587813056, 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629667, 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-85.xml" 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1614985740, 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630673, 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-16.jpg" 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1671152772, 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630786, 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-71.xml" 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1678791002, 2024-05-09T08:28:35.756 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630566, 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-28.jpg" 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1713036954, 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630392, 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-28.xml" 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1739069595, 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630232, 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-24.jpg" 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.757 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1748512052, 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628756, 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-80.xml" 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1753956240, 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630723, 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-3.jpg" 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1779315948, 2024-05-09T08:28:35.758 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630376, 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-32.xml" 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1782575728, 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628862, 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-43.xml" 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1892380767, 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629643, 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-37.jpg" 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1900119677, 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628930, 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-17.xml" 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1941794947, 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631243, 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-22.xml" 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1988643797, 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628793, 2024-05-09T08:28:35.760 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-67.xml" 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1989727948, 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631080, 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-43.xml" 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1990297371, 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628758, 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-79.xml" 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.761 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1992618151, 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629076, 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-42.jpg" 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2007886677, 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630500, 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-1.xml" 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2014558499, 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630334, 2024-05-09T08:28:35.762 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-42.xml" 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2031274715, 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630388, 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-29.xml" 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2050663328, 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631288, 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-16.xml" 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.763 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2051501809, 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630285, 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-53.xml" 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2064675800, 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630727, 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-1.jpg" 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2069645392, 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630552, 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-30.jpg" 2024-05-09T08:28:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2070456912, 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630408, 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-24.xml" 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2102249167, 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631316, 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-78.xml" 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.765 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2118690436, 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630338, 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-41.xml" 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2155207996, 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628815, 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-60.xml" 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2168450852, 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630614, 2024-05-09T08:28:35.766 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-22.jpg" 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2210561452, 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630432, 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-18.xml" 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2244939576, 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629009, 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-63.jpg" 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2282399955, 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630889, 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-11.jpg" 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2296975012, 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630973, 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-3.jpg" 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2299385132, 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630326, 2024-05-09T08:28:35.768 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-44.xml" 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2302743836, 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628764, 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-77.xml" 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2322275793, 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629028, 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-55.jpg" 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.769 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2329182094, 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631021, 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-49.xml" 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2329739953, 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629805, 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-33.jpg" 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2348422284, 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629041, 2024-05-09T08:28:35.770 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-50.jpg" 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2350263242, 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630396, 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-27.xml" 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2356922213, 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630681, 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-14.jpg" 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.771 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2372345582, 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629925, 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-81.xml" 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2413893058, 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628821, 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-58.xml" 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2419265228, 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630222, 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-76.xml" 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2487716761, 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630538, 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-32.jpg" 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2500481837, 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629039, 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-51.jpg" 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2501724176, 2024-05-09T08:28:35.773 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631221, 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-26.xml" 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2555596776, 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630957, 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-5.jpg" 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2557128515, 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631163, 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-34.xml" 2024-05-09T08:28:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2568319086, 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630980, 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-53.xml" 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2615825468, 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630825, 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-68.xml" 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2660648004, 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630507, 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-36.jpg" 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2687710226, 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630372, 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-33.xml" 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2692468128, 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630416, 2024-05-09T08:28:35.776 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-22.xml" 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2692713310, 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630787, 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-74.jpg" 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2707818954, 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629055, 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-99.xml" 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.777 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2726609479, 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628993, 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-69.jpg" 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2734770244, 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630669, 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-17.jpg" 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2739722541, 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628876, 2024-05-09T08:28:35.778 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-37.xml" 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2756557460, 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630496, 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-2.xml" 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2757766092, 2024-05-09T08:28:35.779 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631320, 2024-05-09T08:28:35.781 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-77.xml" 2024-05-09T08:28:35.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2801900833, 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628986, 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-71.jpg" 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2803194865, 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631273, 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-18.xml" 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2810816703, 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630330, 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-43.xml" 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2840277356, 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630484, 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-5.xml" 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2855163432, 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631058, 2024-05-09T08:28:35.783 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-45.xml" 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2876165198, 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630420, 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-21.xml" 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2877999814, 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630624, 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-21.jpg" 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.784 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2881549383, 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630273, 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-55.xml" 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2915325761, 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631234, 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-24.xml" 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2948493922, 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631131, 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-38.xml" 2024-05-09T08:28:35.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2955860245, 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630946, 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-6.jpg" 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3021959999, 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630360, 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-36.xml" 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3035755689, 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630384, 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-30.xml" 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3045343735, 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628836, 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-53.xml" 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3045686266, 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630468, 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-9.xml" 2024-05-09T08:28:35.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3072234755, 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630400, 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-26.xml" 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3116956103, 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628871, 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-39.xml" 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.788 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3139966477, 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630705, 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-8.jpg" 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3148747583, 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630521, 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-34.jpg" 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3186364883, 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628700, 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-98.xml" 2024-05-09T08:28:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3202041410, 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631210, 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-28.xml" 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3202640401, 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630581, 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-26.jpg" 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.790 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3204814963, 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628848, 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-49.xml" 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3228849983, 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628966, 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-5.xml" 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3322986617, 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630819, 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-17.jpg" 2024-05-09T08:28:35.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3327231351, 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629007, 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-64.jpg" 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3342957466, 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629034, 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-53.jpg" 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.792 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3380534740, 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631008, 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-8.xml" 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3396907838, 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630685, 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-13.jpg" 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3419519430, 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630677, 2024-05-09T08:28:35.793 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-15.jpg" 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3419566771, 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628734, 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-87.xml" 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3431369979, 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630464, 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-10.xml" 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.794 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3480798335, 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629046, 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-48.jpg" 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3499237960, 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629023, 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-57.jpg" 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3533941930, 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630701, 2024-05-09T08:28:35.795 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-9.jpg" 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3547864074, 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630299, 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-50.xml" 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3560709864, 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630448, 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-14.xml" 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.796 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3580088222, 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629011, 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-62.jpg" 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3603286253, 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630641, 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-19.jpg" 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3648145472, 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628731, 2024-05-09T08:28:35.797 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-88.xml" 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3687584450, 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630347, 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-39.xml" 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3709348609, 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631278, 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-17.xml" 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.798 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3759584517, 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630597, 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-24.jpg" 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3766484848, 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630352, 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-38.xml" 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3774511662, 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630725, 2024-05-09T08:28:35.799 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-2.jpg" 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3801503953, 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628707, 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-96.xml" 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3822534895, 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628777, 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-73.xml" 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.800 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3826874770, 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630312, 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-47.xml" 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3833716520, 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628900, 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-28.xml" 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3846226173, 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628878, 2024-05-09T08:28:35.801 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-36.xml" 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3849377558, 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630356, 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-37.xml" 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3864999084, 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630648, 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-18.jpg" 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3866948246, 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629084, 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-95.xml" 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3887669704, 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630436, 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-17.xml" 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3944128765, 2024-05-09T08:28:35.803 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630928, 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-58.xml" 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3946554253, 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628741, 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-85.xml" 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3957937567, 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630321, 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-45.xml" 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3997532897, 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630859, 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-65.xml" 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4031472965, 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630940, 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-57.xml" 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4032227834, 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629005, 2024-05-09T08:28:35.805 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-65.jpg" 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4070785057, 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511629796, 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-84.xml" 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4074450533, 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628824, 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-57.xml" 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4094141632, 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630236, 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-75.xml" 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4119734159, 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628771, 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-75.xml" 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.807 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4150390513, 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631017, 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-85.xml" 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4189081402, 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628860, 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-44.xml" 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4207167395, 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630604, 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-23.jpg" 2024-05-09T08:28:35.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4231269062, 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630574, 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-27.jpg" 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4233346805, 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628978, 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-1.xml" 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.809 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4260774831, 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630267, 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-56.xml" 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4262845189, 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628806, 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-63.xml" 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4273427971, 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631239, 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-23.xml" 2024-05-09T08:28:35.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4280342626, 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630317, 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-46.xml" 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4280550513, 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631141, 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-37.xml" 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:35.811 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:35.812 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4285404534, 2024-05-09T08:28:35.812 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630793, 2024-05-09T08:28:35.812 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-99.xml" 2024-05-09T08:28:35.812 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:28:35.812 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-05-09T08:28:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20dff12640 1 -- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f20c40a2d00 msgr2=0x7f20c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20dff12640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f20c40a2d00 0x7f20c40a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f20d004d000 tx=0x7f20d008cdc0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20dff12640 1 -- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d8106710 msgr2=0x7f20d81098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20dff12640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d8106710 0x7f20d81098a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f20d0005cc0 tx=0x7f20d00093d0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20dff12640 1 -- 172.21.15.28:0/3376358254 shutdown_connections 2024-05-09T08:28:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20dff12640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20d8106710 0x7f20d81098a0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20dff12640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d81010c0 0x7f20d8109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20dff12640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f20c40a2d00 0x7f20c40a51c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.713+0000 7f20dff12640 1 --2- 172.21.15.28:0/3376358254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20d8100700 0x7f20d810d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.714+0000 7f20dff12640 1 -- 172.21.15.28:0/3376358254 >> 172.21.15.28:0/3376358254 conn(0x7f20d8077830 msgr2=0x7f20d8102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.714+0000 7f20dff12640 1 -- 172.21.15.28:0/3376358254 shutdown_connections 2024-05-09T08:28:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:35.714+0000 7f20dff12640 1 -- 172.21.15.28:0/3376358254 wait complete. 2024-05-09T08:28:35.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:28:35.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:35 smithi165 ceph-mon[34540]: pgmap v71: 129 pgs: 129 active+clean; 160 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 706 op/s 2024-05-09T08:28:35.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:35 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1364260501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:28:35.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:35 smithi184 ceph-mon[34509]: pgmap v71: 129 pgs: 129 active+clean; 160 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 706 op/s 2024-05-09T08:28:35.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:35 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1364260501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:28:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 -- 172.21.15.28:0/1572586655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 msgr2=0x7fb1b810e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1572586655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 0x7fb1b810e700 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009cd0 tx=0x7fb1b002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 -- 172.21.15.28:0/1572586655 shutdown_connections 2024-05-09T08:28:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1572586655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 0x7fb1b810e700 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1572586655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1b8101830 0x7fb1b8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1572586655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1b8107840 0x7fb1b8107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 -- 172.21.15.28:0/1572586655 >> 172.21.15.28:0/1572586655 conn(0x7fb1b8078cc0 msgr2=0x7fb1b80790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 -- 172.21.15.28:0/1572586655 shutdown_connections 2024-05-09T08:28:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 -- 172.21.15.28:0/1572586655 wait complete. 2024-05-09T08:28:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 Processor -- start 2024-05-09T08:28:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.096+0000 7fb1bf458640 1 -- start start 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bf458640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1b8101830 0x7fb1b8198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bf458640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 0x7fb1b8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bf458640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1b8107840 0x7fb1b819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bf458640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b8111c50 con 0x7fb1b81021f0 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bf458640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b8111ad0 con 0x7fb1b8107840 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bf458640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b8111dd0 con 0x7fb1b8101830 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bc9f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 0x7fb1b8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bc9f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 0x7fb1b8198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35816/0 (socket says 172.21.15.28:35816) 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bc9f3640 1 -- 172.21.15.28:0/1186291558 learned_addr learned my addr 172.21.15.28:0/1186291558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bd9f5640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1b8107840 0x7fb1b819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1b8101830 0x7fb1b8198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bc9f3640 1 -- 172.21.15.28:0/1186291558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1b8101830 msgr2=0x7fb1b8198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1b8101830 0x7fb1b8198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bc9f3640 1 -- 172.21.15.28:0/1186291558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1b8107840 msgr2=0x7fb1b819c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1b8107840 0x7fb1b819c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bc9f3640 1 -- 172.21.15.28:0/1186291558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b819d120 con 0x7fb1b81021f0 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 0x7fb1b8198670 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800e9f0 tx=0x7fb1a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/1186291558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a800cd80 con 0x7fb1b81021f0 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/1186291558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a8010910 con 0x7fb1b81021f0 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1bf458640 1 -- 172.21.15.28:0/1186291558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b819d410 con 0x7fb1b81021f0 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.097+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/1186291558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8018990 con 0x7fb1b81021f0 2024-05-09T08:28:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.098+0000 7fb1bf458640 1 -- 172.21.15.28:0/1186291558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b81a4ca0 con 0x7fb1b81021f0 2024-05-09T08:28:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.100+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/1186291558 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb1a8010ac0 con 0x7fb1b81021f0 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.100+0000 7fb1c2165740 1 Processor -- start 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.100+0000 7fb1c2165740 1 -- start start 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.100+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a1a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.100+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 0x5602b6a302f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1c2165740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602b697fc40 con 0x5602b6a1a000 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1c2165740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5602b6927580 con 0x5602b6a2df00 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1c2165740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5602b697bc00 con 0x5602b6a1a920 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1ae7fc640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fb1a40a2d00 0x7fb1a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd1f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd9f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a1a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd1f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:53746/0 (socket says 172.21.15.28:53746) 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd1f4640 1 -- 172.21.15.28:0/427810534 learned_addr learned my addr 172.21.15.28:0/427810534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd9f5640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fb1a40a2d00 0x7fb1a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/1186291558 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fb1a80cfb10 con 0x7fb1b81021f0 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/427810534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 0x5602b6a302f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd1f4640 1 -- 172.21.15.28:0/427810534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 msgr2=0x5602b6a302f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/427810534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 0x5602b6a302f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd1f4640 1 -- 172.21.15.28:0/427810534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 msgr2=0x5602b6a1a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/427810534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a1a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd1f4640 1 -- 172.21.15.28:0/427810534 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602b66933f0 con 0x5602b6a1a920 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd9f5640 1 --2- 172.21.15.28:0/427810534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a1a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/427810534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2d9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009ca0 tx=0x7fb1b4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6c8e6f72445f114 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1bd9f5640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fb1a40a2d00 0x7fb1a40a51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0004ab0 tx=0x7fb1b0004a40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.101+0000 7fb1a37fe640 1 -- 172.21.15.28:0/427810534 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b401e070 con 0x5602b6a1a920 2024-05-09T08:28:36.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1a37fe640 1 -- 172.21.15.28:0/427810534 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b4019b10 con 0x5602b6a1a920 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1a37fe640 1 -- 172.21.15.28:0/427810534 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b400ea70 con 0x5602b6a1a920 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1c2165740 1 -- 172.21.15.28:0/427810534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 msgr2=0x5602b6a2d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1c2165740 1 --2- 172.21.15.28:0/427810534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2d9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009ca0 tx=0x7fb1b4011810 comp rx=0 tx=0).stop 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1c2165740 1 -- 172.21.15.28:0/427810534 shutdown_connections 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1c2165740 1 --2- 172.21.15.28:0/427810534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 0x5602b6a302f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1c2165740 1 --2- 172.21.15.28:0/427810534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2d9c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1c2165740 1 --2- 172.21.15.28:0/427810534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a1a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1c2165740 1 -- 172.21.15.28:0/427810534 >> 172.21.15.28:0/427810534 conn(0x5602b6a16650 msgr2=0x5602b6a16a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1c2165740 1 -- 172.21.15.28:0/427810534 shutdown_connections 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.102+0000 7fb1c2165740 1 -- 172.21.15.28:0/427810534 wait complete. 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1c2165740 1 -- start start 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1c2165740 1 Processor -- start 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a2ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 0x5602b6a2b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1c2165740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602b6a30920 con 0x5602b6a1a000 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1c2165740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5602b693ac40 con 0x5602b6a2df00 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1c2165740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5602b6a30aa0 con 0x5602b6a1a920 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1c2165740 10 client.0 _ll_register_callbacks cb 0x5602b6692be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1bd1f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a2ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1bd1f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a2ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35830/0 (socket says 172.21.15.28:35830) 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1bd1f4640 1 -- 172.21.15.28:0/3870275908 learned_addr learned my addr 172.21.15.28:0/3870275908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1bd9f5640 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 0x5602b6a2b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.103+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1bd1f4640 1 -- 172.21.15.28:0/3870275908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 msgr2=0x5602b6a2b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1bd1f4640 1 -- 172.21.15.28:0/3870275908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 msgr2=0x5602b6a2b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 0x5602b6a2b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1bd1f4640 1 -- 172.21.15.28:0/3870275908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602b675d7d0 con 0x5602b6a1a000 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a2ae60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009c70 tx=0x7fb1b40198d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58110ceaae8b5ddd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3870275908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4026070 con 0x5602b6a1a000 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1c2165740 1 -- 172.21.15.28:0/3870275908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b4019b10 con 0x5602b6a1a000 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3870275908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fb1b4004220 con 0x5602b6a1a000 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3870275908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb1b4031050 con 0x5602b6a1a000 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1c2165740 10 client.16485 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1c2165740 1 -- 172.21.15.28:0/3870275908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5602b6a3cb30 con 0x5602b6a1a000 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.104+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3870275908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b4019b10 con 0x5602b6a1a000 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3870275908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fb1b400e770 con 0x5602b6a1a000 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1c2165740 10 client.16485 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1c2165740 10 client.16485 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1c2165740 20 client.16485 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5602b6a3dee0 0x5602b6a402d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1c2165740 4 client.16485 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1c2165740 1 -- 172.21.15.28:0/3870275908 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602b698d370 con 0x5602b6a3dee0 2024-05-09T08:28:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5602b6a3dee0 0x5602b6a402d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1a0ff9640 10 client.16485 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:28:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.105+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5602b6a3dee0 0x5602b6a402d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d76f0c1b194ebe46 server_cookie=baba621eda1906b0 in_seq=0 out_seq=0 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.106+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3870275908 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602b698d370 con 0x5602b6a3dee0 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.106+0000 7fb1a0ff9640 10 client.16485 handle_command_reply: tid=0 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.106+0000 7fb1c2165740 1 client.16485 shutdown 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 -- 172.21.15.28:0/3870275908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 msgr2=0x5602b6a2ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a2ae60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009c70 tx=0x7fb1b40198d0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 -- 172.21.15.28:0/3870275908 shutdown_connections 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5602b6a3dee0 0x5602b6a402d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a2df00 0x5602b6a2b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1a0ff9640 0 client.16485 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:28:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3870275908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a1a000 0x5602b6a2ae60 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 -- 172.21.15.28:0/3870275908 >> 172.21.15.28:0/3870275908 conn(0x5602b6a16650 msgr2=0x5602b6a1c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 -- 172.21.15.28:0/3870275908 shutdown_connections 2024-05-09T08:28:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 1 -- 172.21.15.28:0/3870275908 wait complete. 2024-05-09T08:28:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.107+0000 7fb1c2165740 20 client.16485 trim_cache size 0 max 16384 2024-05-09T08:28:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1c2165740 1 Processor -- start 2024-05-09T08:28:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1c2165740 1 -- start start 2024-05-09T08:28:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a000 0x5602b6a39820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a1cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1c2165740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81acc20 con 0x5602b6a2df00 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1c2165740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81ae820 con 0x5602b6a1a000 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1c2165740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81ad2f0 con 0x5602b6a1a920 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1bd9f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a1cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1bc9f3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1bd1f4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a000 0x5602b6a39820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1bd9f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a1cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35842/0 (socket says 172.21.15.28:35842) 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.121+0000 7fb1bd9f5640 1 -- 172.21.15.28:0/2916990657 learned_addr learned my addr 172.21.15.28:0/2916990657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.122+0000 7fb1bc9f3640 1 -- 172.21.15.28:0/2916990657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a000 msgr2=0x5602b6a39820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.122+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/2916990657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a000 0x5602b6a39820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.122+0000 7fb1bc9f3640 1 -- 172.21.15.28:0/2916990657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 msgr2=0x5602b6a1cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.122+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/2916990657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a1cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.122+0000 7fb1bc9f3640 1 -- 172.21.15.28:0/2916990657 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602b6a1a750 con 0x5602b6a1a920 2024-05-09T08:28:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.122+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/2916990657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2a010 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80cce90 tx=0x7fb1a8097fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=927b415eea16e442 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.122+0000 7fb1a2ffd640 1 -- 172.21.15.28:0/2916990657 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a80b4e60 con 0x5602b6a1a920 2024-05-09T08:28:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.122+0000 7fb1a2ffd640 1 -- 172.21.15.28:0/2916990657 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a80b1070 con 0x5602b6a1a920 2024-05-09T08:28:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.122+0000 7fb1a2ffd640 1 -- 172.21.15.28:0/2916990657 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8018d90 con 0x5602b6a1a920 2024-05-09T08:28:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 -- 172.21.15.28:0/2916990657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 msgr2=0x5602b6a2a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 --2- 172.21.15.28:0/2916990657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2a010 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80cce90 tx=0x7fb1a8097fb0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 -- 172.21.15.28:0/2916990657 shutdown_connections 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 --2- 172.21.15.28:0/2916990657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a1cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 --2- 172.21.15.28:0/2916990657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a920 0x5602b6a2a010 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 --2- 172.21.15.28:0/2916990657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a000 0x5602b6a39820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 -- 172.21.15.28:0/2916990657 >> 172.21.15.28:0/2916990657 conn(0x5602b6a16650 msgr2=0x5602b6a157b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 -- 172.21.15.28:0/2916990657 shutdown_connections 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 -- 172.21.15.28:0/2916990657 wait complete. 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 -- start start 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.123+0000 7fb1c2165740 1 Processor -- start 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a000 0x5602b6a31940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a920 0x5602b6a31e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1c2165740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a323c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1c2165740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81ae090 con 0x5602b6a2df00 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1c2165740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81ad9c0 con 0x5602b6a1a920 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1c2165740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81aac00 con 0x5602b6a1a000 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1c2165740 10 client.0 _ll_register_callbacks cb 0x7fb1b401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd1f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a000 0x5602b6a31940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd9f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bc9f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a920 0x5602b6a31e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd1f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a000 0x5602b6a31940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:53756/0 (socket says 172.21.15.28:53756) 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd1f4640 1 -- 172.21.15.28:0/3489361516 learned_addr learned my addr 172.21.15.28:0/3489361516 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bc9f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a920 0x5602b6a31e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:50654/0 (socket says 172.21.15.28:50654) 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd9f5640 1 -- 172.21.15.28:0/3489361516 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a000 msgr2=0x5602b6a31940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd9f5640 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a000 0x5602b6a31940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd9f5640 1 -- 172.21.15.28:0/3489361516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a920 msgr2=0x5602b6a31e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd9f5640 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a920 0x5602b6a31e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd9f5640 1 -- 172.21.15.28:0/3489361516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602b6aa3500 con 0x5602b6a2df00 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bc9f3640 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a920 0x5602b6a31e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.124+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a000 0x5602b6a31940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1bd9f5640 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a323c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0006010 tx=0x7fb1b00093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573656ad841136b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3489361516 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b00305a0 con 0x5602b6a2df00 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3489361516 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fb1b0030740 con 0x5602b6a2df00 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1c2165740 1 -- 172.21.15.28:0/3489361516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1a80b1070 con 0x5602b6a2df00 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3489361516 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb1b0030940 con 0x5602b6a2df00 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1c2165740 10 client.16491 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1c2165740 1 -- 172.21.15.28:0/3489361516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5602b6aa47e0 con 0x5602b6a2df00 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3489361516 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a80b1070 con 0x5602b6a2df00 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3489361516 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fb1b0030070 con 0x5602b6a2df00 2024-05-09T08:28:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1c2165740 10 client.16491 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1c2165740 10 client.16491 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T08:28:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1c2165740 20 client.16491 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.125+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5602b6aa4f30 0x5602b6aa7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.126+0000 7fb1c2165740 4 client.16491 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.126+0000 7fb1c2165740 1 -- 172.21.15.28:0/3489361516 --> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602b69876c0 con 0x5602b6aa4f30 2024-05-09T08:28:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.126+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5602b6aa4f30 0x5602b6aa7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.126+0000 7fb1a0ff9640 10 client.16491 ms_handle_connect on v2:172.21.15.184:6838/224925653 2024-05-09T08:28:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.126+0000 7fb1bd1f4640 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5602b6aa4f30 0x5602b6aa7350 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c9faa94da73e9dc server_cookie=56096fe2b2fcdf78 in_seq=0 out_seq=0 2024-05-09T08:28:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.128+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/3489361516 <== mds.1 v2:172.21.15.184:6838/224925653 1 ==== command_reply(tid 0: 0 ) ==== 8+0+57407 (crc 0 0 0) 0x5602b69876c0 con 0x5602b6aa4f30 2024-05-09T08:28:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.128+0000 7fb1a0ff9640 10 client.16491 handle_command_reply: tid=0 2024-05-09T08:28:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.128+0000 7fb1c2165740 1 client.16491 shutdown 2024-05-09T08:28:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 -- 172.21.15.28:0/3489361516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 msgr2=0x5602b6a323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a323c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0006010 tx=0x7fb1b00093a0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 -- 172.21.15.28:0/3489361516 shutdown_connections 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.184:6838/224925653,v1:172.21.15.184:6839/224925653] conn(0x5602b6aa4f30 0x5602b6aa7350 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602b6a2df00 0x5602b6a323c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5602b6a1a920 0x5602b6a31e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1a0ff9640 0 client.16491 ms_handle_reset on v2:172.21.15.184:6838/224925653 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 --2- 172.21.15.28:0/3489361516 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5602b6a1a000 0x5602b6a31940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 -- 172.21.15.28:0/3489361516 >> 172.21.15.28:0/3489361516 conn(0x5602b6a16650 msgr2=0x5602b6a2ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 -- 172.21.15.28:0/3489361516 shutdown_connections 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 1 -- 172.21.15.28:0/3489361516 wait complete. 2024-05-09T08:28:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1c2165740 20 client.16491 trim_cache size 0 max 16384 2024-05-09T08:28:36.134 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-05-09T08:28:36.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.134 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.134 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8943116, 2024-05-09T08:28:36.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631153, 2024-05-09T08:28:36.134 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-25.jpg" 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 11778878, 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630776, 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-26.xml" 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 13704377, 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630886, 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-56.xml" 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.135 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 43245315, 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630908, 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-49.xml" 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 54628180, 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630798, 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-20.xml" 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 96269479, 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630934, 2024-05-09T08:28:36.136 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-40.xml" 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 114007711, 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630789, 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-87.xml" 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 115446025, 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630930, 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-60.jpg" 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.137 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 120725893, 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631150, 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-34.jpg" 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 133881957, 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631276, 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-7.jpg" 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 139592900, 2024-05-09T08:28:36.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630991, 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-22.xml" 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 206732543, 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628295, 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.in" 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 248801395, 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630760, 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-94.xml" 2024-05-09T08:28:36.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 256571965, 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630975, 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-45.jpg" 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 259557527, 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630845, 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-69.xml" 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.140 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 274201124, 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631165, 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-13.xml" 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 279001455, 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631143, 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-27.jpg" 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 281589376, 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630968, 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-29.xml" 2024-05-09T08:28:36.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 292509880, 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630809, 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-80.xml" 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 307594014, 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631212, 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-31.jpg" 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.142 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 360564831, 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630891, 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-73.jpg" 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 374497860, 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630923, 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-62.jpg" 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 393094977, 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630984, 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-24.xml" 2024-05-09T08:28:36.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 395119492, 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630909, 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-67.jpg" 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 434266254, 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630917, 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-46.xml" 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 450276542, 2024-05-09T08:28:36.144 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630854, 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-66.xml" 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 478119976, 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630805, 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-81.xml" 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 488383482, 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630818, 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-14.xml" 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 499154671, 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631041, 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-11.xml" 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 499939241, 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630890, 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-55.xml" 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.146 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 508040015, 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630875, 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-59.xml" 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 528367303, 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630838, 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-71.xml" 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 541128304, 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630905, 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-50.xml" 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 560218702, 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631287, 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-5.jpg" 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 574973953, 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630860, 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-64.xml" 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.148 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 581809392, 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630813, 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-79.xml" 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 594796927, 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630869, 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-61.xml" 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 610241051, 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630959, 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-32.xml" 2024-05-09T08:28:36.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 621310278, 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630878, 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-58.xml" 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 627014123, 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630827, 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-11.xml" 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 637822482, 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630757, 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-95.xml" 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 651697087, 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630799, 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-83.xml" 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 680262708, 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631025, 2024-05-09T08:28:36.151 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-13.xml" 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 686241527, 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631286, 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-27.jpg" 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 714518857, 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630933, 2024-05-09T08:28:36.152 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-59.jpg" 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 733066404, 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631082, 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-2.xml" 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 762283623, 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630835, 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-72.xml" 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.153 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 776495413, 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631135, 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-29.jpg" 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 777171548, 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631145, 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-14.xml" 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 806500509, 2024-05-09T08:28:36.154 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630972, 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-46.jpg" 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 813287422, 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631178, 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-19.jpg" 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 823539219, 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631108, 2024-05-09T08:28:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-36.jpg" 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 827485018, 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630920, 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-45.xml" 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 851899159, 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630870, 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-79.jpg" 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.156 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 876615957, 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630800, 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-19.xml" 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 907071463, 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630927, 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-61.jpg" 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 922095159, 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630836, 2024-05-09T08:28:36.157 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-8.xml" 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 930764074, 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630756, 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-30.xml" 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 961961810, 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631000, 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-19.xml" 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.158 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 993534053, 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631092, 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-38.jpg" 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1008851181, 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631038, 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-37.jpg" 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1021692193, 2024-05-09T08:28:36.159 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630775, 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-90.xml" 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1024185954, 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630855, 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-2.xml" 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1035302394, 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631016, 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-39.jpg" 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1038461858, 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631214, 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-12.jpg" 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1063708593, 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630770, 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-91.xml" 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1098322284, 2024-05-09T08:28:36.161 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630918, 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-64.jpg" 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1099784531, 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630921, 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-63.jpg" 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1114482346, 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630820, 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-77.xml" 2024-05-09T08:28:36.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1117701220, 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630779, 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-89.xml" 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1124093012, 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631197, 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-15.jpg" 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1130191735, 2024-05-09T08:28:36.163 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630817, 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-78.xml" 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1152525556, 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630857, 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-65.xml" 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1173064378, 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630866, 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-62.xml" 2024-05-09T08:28:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1212150833, 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630944, 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-37.xml" 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1224061100, 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630926, 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-43.xml" 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.165 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1263478421, 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631074, 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-100.jpg" 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1272807307, 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630897, 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-71.jpg" 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1278666373, 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630832, 2024-05-09T08:28:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-73.xml" 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1291141109, 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630954, 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-52.jpg" 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1307823063, 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630947, 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-36.xml" 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1319017800, 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630953, 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-34.xml" 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1365984352, 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630785, 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-24.xml" 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1368883188, 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631301, 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-6.xml" 2024-05-09T08:28:36.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1405413318, 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630986, 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-43.jpg" 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1435607274, 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630974, 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-27.xml" 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1482260481, 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631228, 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-10.xml" 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1482764421, 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630810, 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-16.xml" 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1483538437, 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630882, 2024-05-09T08:28:36.170 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-57.xml" 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1500811244, 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631230, 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-9.jpg" 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1510015900, 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631302, 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-2.jpg" 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.171 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1517578577, 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630830, 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-10.xml" 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1522174032, 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631138, 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-28.jpg" 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1526184443, 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630981, 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-25.xml" 2024-05-09T08:28:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1561930030, 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630761, 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-29.xml" 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1570753925, 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630823, 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-76.xml" 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.173 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1602185210, 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630861, 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-82.jpg" 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1607214392, 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631183, 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-18.jpg" 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1625131374, 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631049, 2024-05-09T08:28:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-9.xml" 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1640154226, 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631170, 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-21.jpg" 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1659625515, 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630900, 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-70.jpg" 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.175 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1660750539, 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631006, 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-40.jpg" 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1666425410, 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630902, 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-51.xml" 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1682661947, 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630806, 2024-05-09T08:28:36.176 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-17.xml" 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1787976441, 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630879, 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-76.jpg" 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1816087138, 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631078, 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-3.xml" 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.177 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1826718532, 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630771, 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-27.xml" 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1881497274, 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631190, 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-32.jpg" 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1890531637, 2024-05-09T08:28:36.178 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631087, 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-34.jpg" 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1892767691, 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630863, 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-63.xml" 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1902454523, 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631027, 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-38.jpg" 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1931537498, 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631055, 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-8.xml" 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1948162057, 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630996, 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-41.jpg" 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.180 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1953596509, 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631224, 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-10.jpg" 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1956159489, 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631106, 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-33.jpg" 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1991721603, 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630963, 2024-05-09T08:28:36.181 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-49.jpg" 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2002177737, 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631161, 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-23.jpg" 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2003025707, 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631130, 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-30.jpg" 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2011874752, 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631157, 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-24.jpg" 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2040168068, 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628291, 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/m4/acx_pthread.m4" 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2041548884, 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630795, 2024-05-09T08:28:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-21.xml" 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2042030591, 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631185, 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-12.xml" 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2094129653, 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631296, 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-3.jpg" 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.184 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2120436309, 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630964, 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-30.xml" 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2132148010, 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630766, 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-92.xml" 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2155860064, 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630979, 2024-05-09T08:28:36.185 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-44.jpg" 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2204988411, 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630846, 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-5.xml" 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2205908140, 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630851, 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-67.xml" 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2226275066, 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631282, 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-6.jpg" 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2226798065, 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630935, 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-58.jpg" 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2238684077, 2024-05-09T08:28:36.187 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631015, 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-15.xml" 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2246050672, 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631188, 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-17.jpg" 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2281203079, 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630950, 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-35.xml" 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.188 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2295689093, 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630995, 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-20.xml" 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2295774629, 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630960, 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-50.jpg" 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2335786967, 2024-05-09T08:28:36.189 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630998, 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-13.xml" 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2357126611, 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630906, 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-68.jpg" 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2360137224, 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631202, 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-14.jpg" 2024-05-09T08:28:36.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2373754058, 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631219, 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-11.jpg" 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2407712590, 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631032, 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-12.xml" 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.191 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2424569036, 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630962, 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-31.xml" 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2457416004, 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630763, 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-93.xml" 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2460269510, 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630914, 2024-05-09T08:28:36.192 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-47.xml" 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2479931741, 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630826, 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-75.xml" 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2480003381, 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630887, 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-74.jpg" 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.193 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2494236234, 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631192, 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-16.jpg" 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2508253155, 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630948, 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-54.jpg" 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2521477954, 2024-05-09T08:28:36.194 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630951, 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-53.jpg" 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2532918798, 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630911, 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-48.xml" 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2577315679, 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630873, 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-78.jpg" 2024-05-09T08:28:36.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2585272124, 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630796, 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-84.xml" 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2598278841, 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630994, 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-21.xml" 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.196 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2607458058, 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631019, 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-14.xml" 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2620489274, 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631053, 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-36.jpg" 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2628991500, 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630814, 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-15.xml" 2024-05-09T08:28:36.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2631640260, 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631064, 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-6.xml" 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2648372595, 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630883, 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-75.jpg" 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.198 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2669315771, 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630833, 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-9.xml" 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2671019580, 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631169, 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-33.jpg" 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2702333695, 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630839, 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-7.xml" 2024-05-09T08:28:36.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2742068038, 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630767, 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-28.xml" 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2796470149, 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631281, 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-7.xml" 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.200 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2819228142, 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630894, 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-72.jpg" 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2849885880, 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630804, 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-18.xml" 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2852127139, 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630989, 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-42.jpg" 2024-05-09T08:28:36.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2876148261, 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631099, 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-37.jpg" 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2935180819, 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631166, 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-22.jpg" 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.202 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2952562154, 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630864, 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-81.jpg" 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2961757688, 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630847, 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-68.xml" 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2987041157, 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631292, 2024-05-09T08:28:36.203 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-4.jpg" 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3022068579, 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630942, 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-56.jpg" 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3030019791, 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630824, 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-12.xml" 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.204 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3033361981, 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630988, 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-23.xml" 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3039239612, 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630929, 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-42.xml" 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3046002880, 2024-05-09T08:28:36.205 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631122, 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-35.jpg" 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3061088480, 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631059, 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-7.xml" 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3068206700, 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630971, 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-28.xml" 2024-05-09T08:28:36.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3076846201, 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631125, 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-32.jpg" 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3077964145, 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630858, 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-1.xml" 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.207 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3088141567, 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631261, 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-8.xml" 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3126763208, 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630955, 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-33.xml" 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3133016251, 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631206, 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-11.xml" 2024-05-09T08:28:36.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3145833287, 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631231, 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-30.jpg" 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3184982412, 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630941, 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-38.xml" 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.209 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3210192486, 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630932, 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-41.xml" 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3213081426, 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631011, 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-16.xml" 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3217452053, 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631073, 2024-05-09T08:28:36.210 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-4.xml" 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3260192408, 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631268, 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-28.jpg" 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3291101529, 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630794, 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-85.xml" 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.211 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3357071724, 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631249, 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-29.jpg" 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3371257531, 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630939, 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-57.jpg" 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3389739121, 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628293, 2024-05-09T08:28:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/blogbench.8" 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3393273485, 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630791, 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-86.xml" 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3408893139, 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630790, 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-23.xml" 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.213 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3414767969, 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631236, 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-8.jpg" 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3426821685, 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630841, 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-70.xml" 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3429976806, 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630969, 2024-05-09T08:28:36.214 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-47.jpg" 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3453725544, 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630956, 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-51.jpg" 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3478099366, 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631002, 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-18.xml" 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.215 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3482737730, 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511628294, 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.am" 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3495717671, 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630780, 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-25.xml" 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3513547080, 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631069, 2024-05-09T08:28:36.216 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-35.jpg" 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3523400585, 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631245, 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-9.xml" 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3596016628, 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631208, 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg" 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3599479262, 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631119, 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-15.xml" 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3627462661, 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630792, 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-22.xml" 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3655450662, 2024-05-09T08:28:36.218 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630802, 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-82.xml" 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3669233711, 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630842, 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-6.xml" 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3683853032, 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630852, 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-3.xml" 2024-05-09T08:28:36.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3805714737, 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630945, 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-55.jpg" 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3822465223, 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631127, 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-31.jpg" 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.220 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3828750481, 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631175, 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-20.jpg" 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3873886506, 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631147, 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-26.jpg" 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3936304929, 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630893, 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-54.xml" 2024-05-09T08:28:36.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3943058417, 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631086, 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-1.xml" 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3970664374, 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631005, 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-17.xml" 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.222 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3972004974, 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631308, 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-1.jpg" 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3973432103, 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630896, 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-53.xml" 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4008393929, 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630872, 2024-05-09T08:28:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-60.xml" 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4009962745, 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630915, 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-65.jpg" 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4055701442, 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630848, 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-4.xml" 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4067492818, 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630977, 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-26.xml" 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4068754583, 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630903, 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-69.jpg" 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4087444034, 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631307, 2024-05-09T08:28:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-26.jpg" 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4096161404, 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631068, 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-5.xml" 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4101604561, 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630912, 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-66.jpg" 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4110528567, 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630965, 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-48.jpg" 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4118383068, 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630867, 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-80.jpg" 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4122453697, 2024-05-09T08:28:36.227 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630876, 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-77.jpg" 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4134376484, 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630938, 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-39.xml" 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4158186746, 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630922, 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-44.xml" 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4171117759, 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630783, 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-88.xml" 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4201151415, 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630899, 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-52.xml" 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.229 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4243568538, 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630829, 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-74.xml" 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4245261084, 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631045, 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-10.xml" 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4287446791, 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631089, 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-97.xml" 2024-05-09T08:28:36.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:28:36.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:28:36.231 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-05-09T08:28:36.231 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4293505871, 2024-05-09T08:28:36.231 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511631050, 2024-05-09T08:28:36.231 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-63.xml" 2024-05-09T08:28:36.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:28:36.231 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-05-09T08:28:36.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1bf458640 1 -- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fb1a40a2d00 msgr2=0x7fb1a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fb1a40a2d00 0x7fb1a40a51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0004ab0 tx=0x7fb1b0004a40 comp rx=0 tx=0).stop 2024-05-09T08:28:36.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1bf458640 1 -- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 msgr2=0x7fb1b8198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.129+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 0x7fb1b8198670 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800e9f0 tx=0x7fb1a800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.130+0000 7fb1bf458640 1 -- 172.21.15.28:0/1186291558 shutdown_connections 2024-05-09T08:28:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.130+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1b8107840 0x7fb1b819c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.130+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b81021f0 0x7fb1b8198670 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.130+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fb1a40a2d00 0x7fb1a40a51c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.130+0000 7fb1bf458640 1 --2- 172.21.15.28:0/1186291558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1b8101830 0x7fb1b8198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.130+0000 7fb1bf458640 1 -- 172.21.15.28:0/1186291558 >> 172.21.15.28:0/1186291558 conn(0x7fb1b8078cc0 msgr2=0x7fb1b80ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.130+0000 7fb1bf458640 1 -- 172.21.15.28:0/1186291558 shutdown_connections 2024-05-09T08:28:36.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.130+0000 7fb1bf458640 1 -- 172.21.15.28:0/1186291558 wait complete. 2024-05-09T08:28:36.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:28:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.347+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/1135939934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8100700 msgr2=0x7fa6d8100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.347+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/1135939934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8100700 0x7fa6d8100b80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc009cd0 tx=0x7fa6cc02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.347+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/1135939934 shutdown_connections 2024-05-09T08:28:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.347+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/1135939934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6d81010c0 0x7fa6d810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.347+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/1135939934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8100700 0x7fa6d8100b80 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.347+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/1135939934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6d8106710 0x7fa6d8106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.347+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/1135939934 >> 172.21.15.28:0/1135939934 conn(0x7fa6d8077830 msgr2=0x7fa6d8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.347+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/1135939934 shutdown_connections 2024-05-09T08:28:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.347+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/1135939934 wait complete. 2024-05-09T08:28:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6dd1b7640 1 Processor -- start 2024-05-09T08:28:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6dd1b7640 1 -- start start 2024-05-09T08:28:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6dd1b7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6d8100700 0x7fa6d8102ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6dd1b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d81010c0 0x7fa6d81031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6dd1b7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6d8106710 0x7fa6d8103720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6dd1b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d8110b20 con 0x7fa6d81010c0 2024-05-09T08:28:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6dd1b7640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81109a0 con 0x7fa6d8100700 2024-05-09T08:28:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6dd1b7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d8110ca0 con 0x7fa6d8106710 2024-05-09T08:28:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d81010c0 0x7fa6d81031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d81010c0 0x7fa6d81031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35870/0 (socket says 172.21.15.28:35870) 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d659c640 1 -- 172.21.15.28:0/3296467994 learned_addr learned my addr 172.21.15.28:0/3296467994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d759e640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6d8106710 0x7fa6d8103720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6d8100700 0x7fa6d8102ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d659c640 1 -- 172.21.15.28:0/3296467994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6d8106710 msgr2=0x7fa6d8103720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d659c640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6d8106710 0x7fa6d8103720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d659c640 1 -- 172.21.15.28:0/3296467994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6d8100700 msgr2=0x7fa6d8102ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d659c640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6d8100700 0x7fa6d8102ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.348+0000 7fa6d659c640 1 -- 172.21.15.28:0/3296467994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d81a6630 con 0x7fa6d81010c0 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.349+0000 7fa6d659c640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d81010c0 0x7fa6d81031e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc02ae00 tx=0x7fa6cc031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.349+0000 7fa6bffff640 1 -- 172.21.15.28:0/3296467994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc003e00 con 0x7fa6d81010c0 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.349+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/3296467994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d81a68c0 con 0x7fa6d81010c0 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.349+0000 7fa6bffff640 1 -- 172.21.15.28:0/3296467994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6cc004520 con 0x7fa6d81010c0 2024-05-09T08:28:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.349+0000 7fa6bffff640 1 -- 172.21.15.28:0/3296467994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc030de0 con 0x7fa6d81010c0 2024-05-09T08:28:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.349+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/3296467994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d81a6e60 con 0x7fa6d81010c0 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6bffff640 1 -- 172.21.15.28:0/3296467994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa6cc0046d0 con 0x7fa6d81010c0 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6dffa4740 1 Processor -- start 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6dffa4740 1 -- start start 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8de710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 0x562d9a8eb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d9a843c40 con 0x562d9a8dec50 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x562d9a7eb580 con 0x562d9a8e9320 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562d9a83fc00 con 0x562d9a8de330 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.352+0000 7fa6bffff640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa6c00a2d00 0x7fa6c00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d759e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8de710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d659c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 0x562d9a8eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d759e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8de710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:53780/0 (socket says 172.21.15.28:53780) 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35872/0 (socket says 172.21.15.28:35872) 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d759e640 1 -- 172.21.15.28:0/2022441119 learned_addr learned my addr 172.21.15.28:0/2022441119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d759e640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa6c00a2d00 0x7fa6c00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6bffff640 1 -- 172.21.15.28:0/3296467994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa6cc02e030 con 0x7fa6d81010c0 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d6d9d640 1 -- 172.21.15.28:0/2022441119 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 msgr2=0x562d9a8de710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/2022441119 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8de710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d6d9d640 1 -- 172.21.15.28:0/2022441119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 msgr2=0x562d9a8eb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/2022441119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 0x562d9a8eb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d6d9d640 1 -- 172.21.15.28:0/2022441119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d9a5573f0 con 0x562d9a8dec50 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d659c640 1 --2- 172.21.15.28:0/2022441119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 0x562d9a8eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/2022441119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e8de0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009c20 tx=0x7fa6c8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a91ca3a08c79c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6d759e640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa6c00a2d00 0x7fa6c00a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0004650 tx=0x7fa6d0005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6bd7fa640 1 -- 172.21.15.28:0/2022441119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8004050 con 0x562d9a8dec50 2024-05-09T08:28:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6bd7fa640 1 -- 172.21.15.28:0/2022441119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c8005500 con 0x562d9a8dec50 2024-05-09T08:28:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.353+0000 7fa6bd7fa640 1 -- 172.21.15.28:0/2022441119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8004d90 con 0x562d9a8dec50 2024-05-09T08:28:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.354+0000 7fa6dffa4740 1 -- 172.21.15.28:0/2022441119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 msgr2=0x562d9a8e8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.354+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/2022441119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e8de0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009c20 tx=0x7fa6c8011810 comp rx=0 tx=0).stop 2024-05-09T08:28:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.354+0000 7fa6dffa4740 1 -- 172.21.15.28:0/2022441119 shutdown_connections 2024-05-09T08:28:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.354+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/2022441119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 0x562d9a8eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.354+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/2022441119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e8de0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.354+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/2022441119 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8de710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.354+0000 7fa6dffa4740 1 -- 172.21.15.28:0/2022441119 >> 172.21.15.28:0/2022441119 conn(0x562d9a8da650 msgr2=0x562d9a8daa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.354+0000 7fa6dffa4740 1 -- 172.21.15.28:0/2022441119 shutdown_connections 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.354+0000 7fa6dffa4740 1 -- 172.21.15.28:0/2022441119 wait complete. 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6dffa4740 1 -- start start 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6dffa4740 1 Processor -- start 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8de330 0x562d9a8e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8dec50 0x562d9a8e6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a8e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d9a8ec110 con 0x562d9a8e9320 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x562d9a7fec40 con 0x562d9a8de330 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562d9a8ec290 con 0x562d9a8dec50 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6dffa4740 10 client.0 _ll_register_callbacks cb 0x562d9a556be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a8e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d6d9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8de330 0x562d9a8e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8dec50 0x562d9a8e6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a8e6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35880/0 (socket says 172.21.15.28:35880) 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d759e640 1 -- 172.21.15.28:0/114334789 learned_addr learned my addr 172.21.15.28:0/114334789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d759e640 1 -- 172.21.15.28:0/114334789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8dec50 msgr2=0x562d9a8e6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d759e640 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8dec50 0x562d9a8e6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d759e640 1 -- 172.21.15.28:0/114334789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8de330 msgr2=0x562d9a8e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d759e640 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8de330 0x562d9a8e6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d759e640 1 -- 172.21.15.28:0/114334789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d9a6217d0 con 0x562d9a8e9320 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d659c640 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8dec50 0x562d9a8e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.355+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8de330 0x562d9a8e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6d759e640 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a8e6cc0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000fd60 tx=0x7fa6d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce807f737276a5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6baff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6baff5640 1 -- 172.21.15.28:0/114334789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d000fe50 con 0x562d9a8e9320 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6baff5640 1 -- 172.21.15.28:0/114334789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa6d001ec50 con 0x562d9a8e9320 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6dffa4740 1 -- 172.21.15.28:0/114334789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6c8005500 con 0x562d9a8e9320 2024-05-09T08:28:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6baff5640 1 -- 172.21.15.28:0/114334789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa6d0034050 con 0x562d9a8e9320 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6dffa4740 10 client.16503 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6dffa4740 1 -- 172.21.15.28:0/114334789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562d9a8f82a0 con 0x562d9a8e9320 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6baff5640 1 -- 172.21.15.28:0/114334789 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c8005500 con 0x562d9a8e9320 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6baff5640 1 -- 172.21.15.28:0/114334789 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fa6d001d070 con 0x562d9a8e9320 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6dffa4740 10 client.16503 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6dffa4740 10 client.16503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6dffa4740 20 client.16503 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562d9a8f9830 0x562d9a8fbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6dffa4740 4 client.16503 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.356+0000 7fa6dffa4740 1 -- 172.21.15.28:0/114334789 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d9a851370 con 0x562d9a8f9830 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.357+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562d9a8f9830 0x562d9a8fbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.357+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562d9a8f9830 0x562d9a8fbc90 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=696a56c5e3261b9c server_cookie=e9f452cea7035d4e in_seq=0 out_seq=0 2024-05-09T08:28:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.357+0000 7fa6baff5640 10 client.16503 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6baff5640 1 -- 172.21.15.28:0/114334789 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d9a851370 con 0x562d9a8f9830 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6baff5640 10 client.16503 handle_command_reply: tid=0 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 client.16503 shutdown 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 -- 172.21.15.28:0/114334789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 msgr2=0x562d9a8e6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a8e6cc0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000fd60 tx=0x7fa6d00046c0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 -- 172.21.15.28:0/114334789 shutdown_connections 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562d9a8f9830 0x562d9a8fbc90 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a8e6cc0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8dec50 0x562d9a8e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/114334789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8de330 0x562d9a8e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 -- 172.21.15.28:0/114334789 >> 172.21.15.28:0/114334789 conn(0x562d9a8da650 msgr2=0x562d9a8e0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6baff5640 0 client.16503 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 -- 172.21.15.28:0/114334789 shutdown_connections 2024-05-09T08:28:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 1 -- 172.21.15.28:0/114334789 wait complete. 2024-05-09T08:28:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.361+0000 7fa6dffa4740 20 client.16503 trim_cache size 0 max 16384 2024-05-09T08:28:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6dffa4740 1 Processor -- start 2024-05-09T08:28:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6dffa4740 1 -- start start 2024-05-09T08:28:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8dec50 0x562d9a8e0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a901560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81b1000 con 0x562d9a8e9320 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81b2c00 con 0x562d9a8dec50 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81b16d0 con 0x562d9a8de330 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6d759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a901560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6d759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a901560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35888/0 (socket says 172.21.15.28:35888) 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6d759e640 1 -- 172.21.15.28:0/601291653 learned_addr learned my addr 172.21.15.28:0/601291653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.376+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/601291653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6d659c640 1 --2- 172.21.15.28:0/601291653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8dec50 0x562d9a8e0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6d759e640 1 -- 172.21.15.28:0/601291653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 msgr2=0x562d9a8f4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6d759e640 1 --2- 172.21.15.28:0/601291653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8f4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6d759e640 1 -- 172.21.15.28:0/601291653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8dec50 msgr2=0x562d9a8e0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6d759e640 1 --2- 172.21.15.28:0/601291653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8dec50 0x562d9a8e0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6d759e640 1 -- 172.21.15.28:0/601291653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d9a8da480 con 0x562d9a8e9320 2024-05-09T08:28:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6d759e640 1 --2- 172.21.15.28:0/601291653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a901560 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0005890 tx=0x7fa6d000fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc248a8c55b5248 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6bcff9640 1 -- 172.21.15.28:0/601291653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d00117c0 con 0x562d9a8e9320 2024-05-09T08:28:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6bcff9640 1 -- 172.21.15.28:0/601291653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6d00294a0 con 0x562d9a8e9320 2024-05-09T08:28:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.377+0000 7fa6bcff9640 1 -- 172.21.15.28:0/601291653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0029a70 con 0x562d9a8e9320 2024-05-09T08:28:36.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 -- 172.21.15.28:0/601291653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 msgr2=0x562d9a901560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/601291653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a901560 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0005890 tx=0x7fa6d000fb70 comp rx=0 tx=0).stop 2024-05-09T08:28:36.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 -- 172.21.15.28:0/601291653 shutdown_connections 2024-05-09T08:28:36.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/601291653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8e9320 0x562d9a901560 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/601291653 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8dec50 0x562d9a8e0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/601291653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 -- 172.21.15.28:0/601291653 >> 172.21.15.28:0/601291653 conn(0x562d9a8da650 msgr2=0x562d9a8e15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 -- 172.21.15.28:0/601291653 shutdown_connections 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 -- 172.21.15.28:0/601291653 wait complete. 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 -- start start 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.378+0000 7fa6dffa4740 1 Processor -- start 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8f1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6dffa4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 0x562d9a8e29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81b2470 con 0x562d9a8dec50 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81b1da0 con 0x562d9a8e9320 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6dffa4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81aefe0 con 0x562d9a8de330 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6dffa4740 10 client.0 _ll_register_callbacks cb 0x7fa6d000fc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35894/0 (socket says 172.21.15.28:35894) 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d659c640 1 -- 172.21.15.28:0/854271624 learned_addr learned my addr 172.21.15.28:0/854271624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:36.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d759e640 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 0x562d9a8e29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d659c640 1 -- 172.21.15.28:0/854271624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 msgr2=0x562d9a8f1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d659c640 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8f1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d659c640 1 -- 172.21.15.28:0/854271624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 msgr2=0x562d9a8e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d659c640 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 0x562d9a8e29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d659c640 1 -- 172.21.15.28:0/854271624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d9a8e3060 con 0x562d9a8dec50 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6d659c640 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e24b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0ee200 tx=0x7fa6cc08ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a400d35bd467a05d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6baff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6baff5640 1 -- 172.21.15.28:0/854271624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc0d7070 con 0x562d9a8dec50 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6dffa4740 1 -- 172.21.15.28:0/854271624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6d00294a0 con 0x562d9a8dec50 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.379+0000 7fa6baff5640 1 -- 172.21.15.28:0/854271624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7fa6cc0dbdb0 con 0x562d9a8dec50 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6baff5640 1 -- 172.21.15.28:0/854271624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa6cc0a4050 con 0x562d9a8dec50 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6dffa4740 10 client.16509 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6dffa4740 1 -- 172.21.15.28:0/854271624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562d9a95edd0 con 0x562d9a8dec50 2024-05-09T08:28:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6baff5640 1 -- 172.21.15.28:0/854271624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6d00294a0 con 0x562d9a8dec50 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6baff5640 1 -- 172.21.15.28:0/854271624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fa6cc04e070 con 0x562d9a8dec50 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6dffa4740 10 client.16509 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6dffa4740 10 client.16509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6dffa4740 20 client.16509 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562d9a95f500 0x562d9a961920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6dffa4740 4 client.16509 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6dffa4740 1 -- 172.21.15.28:0/854271624 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562d9a84b6c0 con 0x562d9a95f500 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562d9a95f500 0x562d9a961920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.380+0000 7fa6d6d9d640 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562d9a95f500 0x562d9a961920 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78d2e433d7be5736 server_cookie=5c804697613553c6 in_seq=0 out_seq=0 2024-05-09T08:28:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.381+0000 7fa6baff5640 10 client.16509 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6baff5640 1 -- 172.21.15.28:0/854271624 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562d9a84b6c0 con 0x562d9a95f500 2024-05-09T08:28:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6baff5640 10 client.16509 handle_command_reply: tid=0 2024-05-09T08:28:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 client.16509 shutdown 2024-05-09T08:28:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 -- 172.21.15.28:0/854271624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 msgr2=0x562d9a8e24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e24b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0ee200 tx=0x7fa6cc08ef30 comp rx=0 tx=0).stop 2024-05-09T08:28:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 -- 172.21.15.28:0/854271624 shutdown_connections 2024-05-09T08:28:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562d9a95f500 0x562d9a961920 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562d9a8e9320 0x562d9a8e29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6baff5640 0 client.16509 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9a8dec50 0x562d9a8e24b0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 --2- 172.21.15.28:0/854271624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562d9a8de330 0x562d9a8f1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 -- 172.21.15.28:0/854271624 >> 172.21.15.28:0/854271624 conn(0x562d9a8da650 msgr2=0x562d9a8e4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 -- 172.21.15.28:0/854271624 shutdown_connections 2024-05-09T08:28:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 1 -- 172.21.15.28:0/854271624 wait complete. 2024-05-09T08:28:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.386+0000 7fa6dffa4740 20 client.16509 trim_cache size 0 max 16384 2024-05-09T08:28:36.388 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa6c00a2d00 msgr2=0x7fa6c00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa6c00a2d00 0x7fa6c00a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0004650 tx=0x7fa6d0005f90 comp rx=0 tx=0).stop 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d81010c0 msgr2=0x7fa6d81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d81010c0 0x7fa6d81031e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc02ae00 tx=0x7fa6cc031e60 comp rx=0 tx=0).stop 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/3296467994 shutdown_connections 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa6c00a2d00 0x7fa6c00a51c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6d8106710 0x7fa6d8103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d81010c0 0x7fa6d81031e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 --2- 172.21.15.28:0/3296467994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6d8100700 0x7fa6d8102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/3296467994 >> 172.21.15.28:0/3296467994 conn(0x7fa6d8077830 msgr2=0x7fa6d8068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/3296467994 shutdown_connections 2024-05-09T08:28:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:28:36.387+0000 7fa6dd1b7640 1 -- 172.21.15.28:0/3296467994 wait complete. 2024-05-09T08:28:36.399 ERROR:tasks.fwd_scrub:rank 4 damaged: [{'damage_type': 'backtrace', 'id': 3122147, 'ino': 1099511629769, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-82.xml'}, {'damage_type': 'backtrace', 'id': 27028900, 'ino': 1099511630110, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-56.jpg'}, {'damage_type': 'backtrace', 'id': 29559599, 'ino': 1099511630490, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-74.xml'}, {'damage_type': 'backtrace', 'id': 56572195, 'ino': 1099511631244, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-54.xml'}, {'damage_type': 'backtrace', 'id': 80446330, 'ino': 1099511630536, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-61.xml'}, {'damage_type': 'backtrace', 'id': 84432500, 'ino': 1099511629500, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-34.jpg'}, {'damage_type': 'backtrace', 'id': 108127729, 'ino': 1099511629868, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-79.jpg'}, {'damage_type': 'backtrace', 'id': 121764118, 'ino': 1099511630064, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-79.jpg'}, {'damage_type': 'backtrace', 'id': 121845381, 'ino': 1099511630044, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-88.jpg'}, {'damage_type': 'backtrace', 'id': 124586333, 'ino': 1099511629454, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-77.xml'}, {'damage_type': 'backtrace', 'id': 141694022, 'ino': 1099511630062, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-80.jpg'}, {'damage_type': 'backtrace', 'id': 147798359, 'ino': 1099511628987, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-5.xml'}, {'damage_type': 'backtrace', 'id': 167247785, 'ino': 1099511631277, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-46.xml'}, {'damage_type': 'backtrace', 'id': 170720683, 'ino': 1099511629617, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-53.jpg'}, {'damage_type': 'backtrace', 'id': 172582060, 'ino': 1099511628730, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-8.xml'}, {'damage_type': 'backtrace', 'id': 179626721, 'ino': 1099511631134, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-77.xml'}, {'damage_type': 'backtrace', 'id': 183259744, 'ino': 1099511630582, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-48.xml'}, {'damage_type': 'backtrace', 'id': 187255471, 'ino': 1099511630196, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-21.jpg'}, {'damage_type': 'backtrace', 'id': 188869239, 'ino': 1099511631272, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-47.xml'}, {'damage_type': 'backtrace', 'id': 242246494, 'ino': 1099511628776, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-31.jpg'}, {'damage_type': 'backtrace', 'id': 246898109, 'ino': 1099511629512, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-30.jpg'}, {'damage_type': 'backtrace', 'id': 253491091, 'ino': 1099511629909, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-38.jpg'}, {'damage_type': 'backtrace', 'id': 272413643, 'ino': 1099511630702, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-4.xml'}, {'damage_type': 'backtrace', 'id': 279319761, 'ino': 1099511629470, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-65.xml'}, {'damage_type': 'backtrace', 'id': 279786293, 'ino': 1099511630084, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-69.jpg'}, {'damage_type': 'backtrace', 'id': 300954480, 'ino': 1099511630564, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-53.xml'}, {'damage_type': 'backtrace', 'id': 313223878, 'ino': 1099511629543, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-24.xml'}, {'damage_type': 'backtrace', 'id': 328152279, 'ino': 1099511629974, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-48.xml'}, {'damage_type': 'backtrace', 'id': 340579546, 'ino': 1099511629080, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-91.xml'}, {'damage_type': 'backtrace', 'id': 351432328, 'ino': 1099511629469, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-66.xml'}, {'damage_type': 'backtrace', 'id': 364331973, 'ino': 1099511630144, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-41.jpg'}, {'damage_type': 'backtrace', 'id': 374768270, 'ino': 1099511629481, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-35.jpg'}, {'damage_type': 'backtrace', 'id': 375010218, 'ino': 1099511629969, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-53.xml'}, {'damage_type': 'backtrace', 'id': 411844219, 'ino': 1099511630022, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-2.xml'}, {'damage_type': 'backtrace', 'id': 430941029, 'ino': 1099511629163, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-58.jpg'}, {'damage_type': 'backtrace', 'id': 441777616, 'ino': 1099511631152, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-73.xml'}, {'damage_type': 'backtrace', 'id': 443925966, 'ino': 1099511629627, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-48.jpg'}, {'damage_type': 'backtrace', 'id': 460342460, 'ino': 1099511629754, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-97.xml'}, {'damage_type': 'backtrace', 'id': 461299164, 'ino': 1099511629521, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-27.jpg'}, {'damage_type': 'backtrace', 'id': 466905608, 'ino': 1099511629656, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-34.jpg'}, {'damage_type': 'backtrace', 'id': 482382428, 'ino': 1099511630575, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-50.xml'}, {'damage_type': 'backtrace', 'id': 507972938, 'ino': 1099511629767, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-84.xml'}, {'damage_type': 'backtrace', 'id': 508149882, 'ino': 1099511629515, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-29.jpg'}, {'damage_type': 'backtrace', 'id': 508164407, 'ino': 1099511629579, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-12.xml'}, {'damage_type': 'backtrace', 'id': 518405757, 'ino': 1099511629619, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-52.jpg'}, {'damage_type': 'backtrace', 'id': 533625926, 'ino': 1099511630124, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-49.jpg'}, {'damage_type': 'backtrace', 'id': 551253288, 'ino': 1099511629891, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-56.jpg'}, {'damage_type': 'backtrace', 'id': 574879309, 'ino': 1099511631269, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-48.xml'}, {'damage_type': 'backtrace', 'id': 583415587, 'ino': 1099511629992, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-30.xml'}, {'damage_type': 'backtrace', 'id': 589797440, 'ino': 1099511629780, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-71.xml'}, {'damage_type': 'backtrace', 'id': 610291600, 'ino': 1099511629279, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-51.jpg'}, {'damage_type': 'backtrace', 'id': 614258803, 'ino': 1099511628772, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-21.xml'}, {'damage_type': 'backtrace', 'id': 614455532, 'ino': 1099511628735, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-60.xml'}, {'damage_type': 'backtrace', 'id': 619055510, 'ino': 1099511630657, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-22.xml'}, {'damage_type': 'backtrace', 'id': 624135436, 'ino': 1099511630179, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-38.jpg'}, {'damage_type': 'backtrace', 'id': 637764426, 'ino': 1099511628762, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-22.xml'}, {'damage_type': 'backtrace', 'id': 685665276, 'ino': 1099511630003, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-19.xml'}, {'damage_type': 'backtrace', 'id': 706081444, 'ino': 1099511629445, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-80.xml'}, {'damage_type': 'backtrace', 'id': 733807174, 'ino': 1099511630295, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-47.jpg'}, {'damage_type': 'backtrace', 'id': 765732149, 'ino': 1099511629472, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-63.xml'}, {'damage_type': 'backtrace', 'id': 795370629, 'ino': 1099511630194, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-22.jpg'}, {'damage_type': 'backtrace', 'id': 797922242, 'ino': 1099511628792, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-25.jpg'}, {'damage_type': 'backtrace', 'id': 810754471, 'ino': 1099511630011, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-11.xml'}, {'damage_type': 'backtrace', 'id': 811378527, 'ino': 1099511629104, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-62.jpg'}, {'damage_type': 'backtrace', 'id': 816672474, 'ino': 1099511628780, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-29.jpg'}, {'damage_type': 'backtrace', 'id': 832358842, 'ino': 1099511629308, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-49.jpg'}, {'damage_type': 'backtrace', 'id': 840058854, 'ino': 1099511629979, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-43.xml'}, {'damage_type': 'backtrace', 'id': 842369323, 'ino': 1099511630017, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-5.xml'}, {'damage_type': 'backtrace', 'id': 846533387, 'ino': 1099511630046, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-87.jpg'}, {'damage_type': 'backtrace', 'id': 857320549, 'ino': 1099511629817, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-42.xml'}, {'damage_type': 'backtrace', 'id': 860130822, 'ino': 1099511630350, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-26.xml'}, {'damage_type': 'backtrace', 'id': 868542374, 'ino': 1099511630540, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-60.xml'}, {'damage_type': 'backtrace', 'id': 870159831, 'ino': 1099511630530, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-63.xml'}, {'damage_type': 'backtrace', 'id': 880006411, 'ino': 1099511629982, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-40.xml'}, {'damage_type': 'backtrace', 'id': 883191082, 'ino': 1099511630568, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-52.xml'}, {'damage_type': 'backtrace', 'id': 886100342, 'ino': 1099511629872, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-75.jpg'}, {'damage_type': 'backtrace', 'id': 902365683, 'ino': 1099511630133, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-45.jpg'}, {'damage_type': 'backtrace', 'id': 912518450, 'ino': 1099511629434, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-85.xml'}, {'damage_type': 'backtrace', 'id': 924951415, 'ino': 1099511629471, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-64.xml'}, {'damage_type': 'backtrace', 'id': 925508200, 'ino': 1099511631090, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-89.xml'}, {'damage_type': 'backtrace', 'id': 931103195, 'ino': 1099511629596, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-2.jpg'}, {'damage_type': 'backtrace', 'id': 944043241, 'ino': 1099511630526, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-64.xml'}, {'damage_type': 'backtrace', 'id': 947344115, 'ino': 1099511630234, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-23.jpg'}, {'damage_type': 'backtrace', 'id': 947875925, 'ino': 1099511629994, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-28.xml'}, {'damage_type': 'backtrace', 'id': 959626148, 'ino': 1099511630052, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-84.jpg'}, {'damage_type': 'backtrace', 'id': 965217451, 'ino': 1099511629548, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-18.jpg'}, {'damage_type': 'backtrace', 'id': 972289300, 'ino': 1099511629993, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-29.xml'}, {'damage_type': 'backtrace', 'id': 1011984630, 'ino': 1099511628736, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-6.xml'}, {'damage_type': 'backtrace', 'id': 1040972856, 'ino': 1099511630498, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-72.xml'}, {'damage_type': 'backtrace', 'id': 1043445676, 'ino': 1099511629887, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-60.jpg'}, {'damage_type': 'backtrace', 'id': 1061100231, 'ino': 1099511630482, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-76.xml'}, {'damage_type': 'backtrace', 'id': 1075847450, 'ino': 1099511630478, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-77.xml'}, {'damage_type': 'backtrace', 'id': 1093625863, 'ino': 1099511629660, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-32.jpg'}, {'damage_type': 'backtrace', 'id': 1100914249, 'ino': 1099511631177, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-67.xml'}, {'damage_type': 'backtrace', 'id': 1101617597, 'ino': 1099511630018, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-4.xml'}, {'damage_type': 'backtrace', 'id': 1125776827, 'ino': 1099511629686, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-10.jpg'}, {'damage_type': 'backtrace', 'id': 1127340685, 'ino': 1099511629564, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-17.xml'}, {'damage_type': 'backtrace', 'id': 1129531548, 'ino': 1099511629646, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-39.jpg'}, {'damage_type': 'backtrace', 'id': 1133176098, 'ino': 1099511629800, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-54.xml'}, {'damage_type': 'backtrace', 'id': 1135080555, 'ino': 1099511630509, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-69.xml'}, {'damage_type': 'backtrace', 'id': 1136061747, 'ino': 1099511629640, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-42.jpg'}, {'damage_type': 'backtrace', 'id': 1147436751, 'ino': 1099511630215, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-11.jpg'}, {'damage_type': 'backtrace', 'id': 1160342260, 'ino': 1099511630006, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-16.xml'}, {'damage_type': 'backtrace', 'id': 1160348512, 'ino': 1099511629468, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-67.xml'}, {'damage_type': 'backtrace', 'id': 1173382910, 'ino': 1099511629588, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-9.xml'}, {'damage_type': 'backtrace', 'id': 1212529847, 'ino': 1099511629679, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-17.jpg'}, {'damage_type': 'backtrace', 'id': 1214590019, 'ino': 1099511630092, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-65.jpg'}, {'damage_type': 'backtrace', 'id': 1216838909, 'ino': 1099511629453, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-36.jpg'}, {'damage_type': 'backtrace', 'id': 1243560309, 'ino': 1099511628790, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-26.jpg'}, {'damage_type': 'backtrace', 'id': 1246285623, 'ino': 1099511629976, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-46.xml'}, {'damage_type': 'backtrace', 'id': 1255489180, 'ino': 1099511630122, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-50.jpg'}, {'damage_type': 'backtrace', 'id': 1256793287, 'ino': 1099511629758, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-93.xml'}, {'damage_type': 'backtrace', 'id': 1271455125, 'ino': 1099511629881, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-66.jpg'}, {'damage_type': 'backtrace', 'id': 1274283293, 'ino': 1099511630358, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-24.xml'}, {'damage_type': 'backtrace', 'id': 1307303884, 'ino': 1099511630155, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-36.jpg'}, {'damage_type': 'backtrace', 'id': 1332277722, 'ino': 1099511629824, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-35.xml'}, {'damage_type': 'backtrace', 'id': 1350945680, 'ino': 1099511629902, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-45.jpg'}, {'damage_type': 'backtrace', 'id': 1357873514, 'ino': 1099511630561, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-54.xml'}, {'damage_type': 'backtrace', 'id': 1365659254, 'ino': 1099511629849, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-10.xml'}, {'damage_type': 'backtrace', 'id': 1399930251, 'ino': 1099511630078, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-72.jpg'}, {'damage_type': 'backtrace', 'id': 1405136290, 'ino': 1099511628808, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-20.jpg'}, {'damage_type': 'backtrace', 'id': 1422166591, 'ino': 1099511630131, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-46.jpg'}, {'damage_type': 'backtrace', 'id': 1456328096, 'ino': 1099511629473, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-62.xml'}, {'damage_type': 'backtrace', 'id': 1463252641, 'ino': 1099511629372, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-43.jpg'}, {'damage_type': 'backtrace', 'id': 1487842521, 'ino': 1099511629874, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-73.jpg'}, {'damage_type': 'backtrace', 'id': 1531720051, 'ino': 1099511629545, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-19.jpg'}, {'damage_type': 'backtrace', 'id': 1543193934, 'ino': 1099511629554, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-16.jpg'}, {'damage_type': 'backtrace', 'id': 1551077538, 'ino': 1099511629230, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-53.jpg'}, {'damage_type': 'backtrace', 'id': 1552813890, 'ino': 1099511629848, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-11.xml'}, {'damage_type': 'backtrace', 'id': 1560124363, 'ino': 1099511629201, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-55.jpg'}, {'damage_type': 'backtrace', 'id': 1583144829, 'ino': 1099511630505, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-70.xml'}, {'damage_type': 'backtrace', 'id': 1591900568, 'ino': 1099511629980, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-42.xml'}, {'damage_type': 'backtrace', 'id': 1594027182, 'ino': 1099511628823, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-15.jpg'}, {'damage_type': 'backtrace', 'id': 1600509526, 'ino': 1099511631218, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-59.xml'}, {'damage_type': 'backtrace', 'id': 1610157321, 'ino': 1099511630137, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-44.jpg'}, {'damage_type': 'backtrace', 'id': 1617360485, 'ino': 1099511630027, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-95.jpg'}, {'damage_type': 'backtrace', 'id': 1627432665, 'ino': 1099511631196, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-63.xml'}, {'damage_type': 'backtrace', 'id': 1627641614, 'ino': 1099511629493, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-44.xml'}, {'damage_type': 'backtrace', 'id': 1628538960, 'ino': 1099511629972, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-50.xml'}, {'damage_type': 'backtrace', 'id': 1631967835, 'ino': 1099511630280, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-12.jpg'}, {'damage_type': 'backtrace', 'id': 1639448297, 'ino': 1099511629984, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-38.xml'}, {'damage_type': 'backtrace', 'id': 1649270679, 'ino': 1099511629607, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-1.xml'}, {'damage_type': 'backtrace', 'id': 1658961694, 'ino': 1099511628832, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-12.jpg'}, {'damage_type': 'backtrace', 'id': 1674500102, 'ino': 1099511629812, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-47.xml'}, {'damage_type': 'backtrace', 'id': 1683236602, 'ino': 1099511630120, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-51.jpg'}, {'damage_type': 'backtrace', 'id': 1690540401, 'ino': 1099511630244, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-20.jpg'}, {'damage_type': 'backtrace', 'id': 1691486740, 'ino': 1099511630652, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-27.xml'}, {'damage_type': 'backtrace', 'id': 1705368413, 'ino': 1099511630238, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-22.jpg'}, {'damage_type': 'backtrace', 'id': 1714757549, 'ino': 1099511630333, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-64.jpg'}, {'damage_type': 'backtrace', 'id': 1726906836, 'ino': 1099511630516, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-67.xml'}, {'damage_type': 'backtrace', 'id': 1733504095, 'ino': 1099511629546, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-23.xml'}, {'damage_type': 'backtrace', 'id': 1750203202, 'ino': 1099511628766, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-57.xml'}, {'damage_type': 'backtrace', 'id': 1761278138, 'ino': 1099511628800, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-23.jpg'}, {'damage_type': 'backtrace', 'id': 1784928763, 'ino': 1099511631191, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-64.xml'}, {'damage_type': 'backtrace', 'id': 1796132747, 'ino': 1099511629242, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-52.jpg'}, {'damage_type': 'backtrace', 'id': 1878694543, 'ino': 1099511629563, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-13.jpg'}, {'damage_type': 'backtrace', 'id': 1878738871, 'ino': 1099511630072, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-75.jpg'}, {'damage_type': 'backtrace', 'id': 1879546364, 'ino': 1099511629650, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-37.jpg'}, {'damage_type': 'backtrace', 'id': 1880123805, 'ino': 1099511628760, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-36.jpg'}, {'damage_type': 'backtrace', 'id': 1891573126, 'ino': 1099511629507, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-36.xml'}, {'damage_type': 'backtrace', 'id': 1897145035, 'ino': 1099511629079, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-1.xml'}, {'damage_type': 'backtrace', 'id': 1904108445, 'ino': 1099511628742, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-4.xml'}, {'damage_type': 'backtrace', 'id': 1935478281, 'ino': 1099511630650, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-28.xml'}, {'damage_type': 'backtrace', 'id': 1947883827, 'ino': 1099511628951, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-8.xml'}, {'damage_type': 'backtrace', 'id': 1954758325, 'ino': 1099511629609, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-57.jpg'}, {'damage_type': 'backtrace', 'id': 1970483140, 'ino': 1099511630609, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-40.xml'}, {'damage_type': 'backtrace', 'id': 1972905958, 'ino': 1099511629065, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-2.xml'}, {'damage_type': 'backtrace', 'id': 1975637879, 'ino': 1099511629527, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-25.jpg'}, {'damage_type': 'backtrace', 'id': 2005173328, 'ino': 1099511629668, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-28.jpg'}, {'damage_type': 'backtrace', 'id': 2007624465, 'ino': 1099511628787, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-20.xml'}, {'damage_type': 'backtrace', 'id': 2007944974, 'ino': 1099511630470, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-79.xml'}, {'damage_type': 'backtrace', 'id': 2010447913, 'ino': 1099511630494, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-73.xml'}, {'damage_type': 'backtrace', 'id': 2025402201, 'ino': 1099511630458, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-82.xml'}, {'damage_type': 'backtrace', 'id': 2037518510, 'ino': 1099511628729, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-61.xml'}, {'damage_type': 'backtrace', 'id': 2058570073, 'ino': 1099511630001, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-21.xml'}, {'damage_type': 'backtrace', 'id': 2062761170, 'ino': 1099511628805, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-21.jpg'}, {'damage_type': 'backtrace', 'id': 2091779224, 'ino': 1099511630543, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-59.xml'}, {'damage_type': 'backtrace', 'id': 2112337326, 'ino': 1099511629975, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-47.xml'}, {'damage_type': 'backtrace', 'id': 2116722821, 'ino': 1099511629840, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-19.xml'}, {'damage_type': 'backtrace', 'id': 2118028475, 'ino': 1099511630171, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-30.jpg'}, {'damage_type': 'backtrace', 'id': 2139475037, 'ino': 1099511629524, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-26.jpg'}, {'damage_type': 'backtrace', 'id': 2140219779, 'ino': 1099511629765, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-86.xml'}, {'damage_type': 'backtrace', 'id': 2140221425, 'ino': 1099511628748, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-2.xml'}, {'damage_type': 'backtrace', 'id': 2158120663, 'ino': 1099511628755, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-38.jpg'}, {'damage_type': 'backtrace', 'id': 2167661080, 'ino': 1099511630213, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-12.jpg'}, {'damage_type': 'backtrace', 'id': 2168515137, 'ino': 1099511630533, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-62.xml'}, {'damage_type': 'backtrace', 'id': 2172547307, 'ino': 1099511630098, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-62.jpg'}, {'damage_type': 'backtrace', 'id': 2183141665, 'ino': 1099511631054, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-97.xml'}, {'damage_type': 'backtrace', 'id': 2229916635, 'ino': 1099511630664, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-15.xml'}, {'damage_type': 'backtrace', 'id': 2232079141, 'ino': 1099511630289, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-56.xml'}, {'damage_type': 'backtrace', 'id': 2241412779, 'ino': 1099511629990, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-32.xml'}, {'damage_type': 'backtrace', 'id': 2255454851, 'ino': 1099511629779, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-72.xml'}, {'damage_type': 'backtrace', 'id': 2327452399, 'ino': 1099511629973, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-49.xml'}, {'damage_type': 'backtrace', 'id': 2327816034, 'ino': 1099511629587, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-5.jpg'}, {'damage_type': 'backtrace', 'id': 2328036313, 'ino': 1099511629760, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-91.xml'}, {'damage_type': 'backtrace', 'id': 2330466228, 'ino': 1099511629904, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-43.jpg'}, {'damage_type': 'backtrace', 'id': 2338191346, 'ino': 1099511630146, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-40.jpg'}, {'damage_type': 'backtrace', 'id': 2341196066, 'ino': 1099511630512, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-68.xml'}, {'damage_type': 'backtrace', 'id': 2369686584, 'ino': 1099511629539, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-21.jpg'}, {'damage_type': 'backtrace', 'id': 2386936331, 'ino': 1099511628834, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-15.xml'}, {'damage_type': 'backtrace', 'id': 2411114745, 'ino': 1099511629867, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-80.jpg'}, {'damage_type': 'backtrace', 'id': 2434015290, 'ino': 1099511630184, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-26.jpg'}, {'damage_type': 'backtrace', 'id': 2447681886, 'ino': 1099511629669, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-27.jpg'}, {'damage_type': 'backtrace', 'id': 2459608972, 'ino': 1099511630198, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-20.jpg'}, {'damage_type': 'backtrace', 'id': 2468800825, 'ino': 1099511629578, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-8.jpg'}, {'damage_type': 'backtrace', 'id': 2488072898, 'ino': 1099511631081, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-91.xml'}, {'damage_type': 'backtrace', 'id': 2489737563, 'ino': 1099511630639, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-31.xml'}, {'damage_type': 'backtrace', 'id': 2495330033, 'ino': 1099511628721, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-11.xml'}, {'damage_type': 'backtrace', 'id': 2515580151, 'ino': 1099511630096, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-63.jpg'}, {'damage_type': 'backtrace', 'id': 2520902202, 'ino': 1099511630660, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-19.xml'}, {'damage_type': 'backtrace', 'id': 2531540116, 'ino': 1099511630616, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-38.xml'}, {'damage_type': 'backtrace', 'id': 2541857051, 'ino': 1099511629120, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-61.jpg'}, {'damage_type': 'backtrace', 'id': 2567462298, 'ino': 1099511631095, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-88.xml'}, {'damage_type': 'backtrace', 'id': 2604668337, 'ino': 1099511631201, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-62.xml'}, {'damage_type': 'backtrace', 'id': 2612666498, 'ino': 1099511629996, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-26.xml'}, {'damage_type': 'backtrace', 'id': 2621020273, 'ino': 1099511629997, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-25.xml'}, {'damage_type': 'backtrace', 'id': 2634059902, 'ino': 1099511630612, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-39.xml'}, {'damage_type': 'backtrace', 'id': 2657044444, 'ino': 1099511630354, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-25.xml'}, {'damage_type': 'backtrace', 'id': 2657935218, 'ino': 1099511629421, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-40.jpg'}, {'damage_type': 'backtrace', 'id': 2660005309, 'ino': 1099511629680, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-16.jpg'}, {'damage_type': 'backtrace', 'id': 2663991349, 'ino': 1099511631164, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-70.xml'}, {'damage_type': 'backtrace', 'id': 2669091484, 'ino': 1099511630008, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-14.xml'}, {'damage_type': 'backtrace', 'id': 2675546695, 'ino': 1099511629695, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-1.jpg'}, {'damage_type': 'backtrace', 'id': 2723739917, 'ino': 1099511629590, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-4.jpg'}, {'damage_type': 'backtrace', 'id': 2740674572, 'ino': 1099511629888, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-59.jpg'}, {'damage_type': 'backtrace', 'id': 2752113521, 'ino': 1099511630248, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-3.jpg'}, {'damage_type': 'backtrace', 'id': 2789258338, 'ino': 1099511630262, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-15.jpg'}, {'damage_type': 'backtrace', 'id': 2789971088, 'ino': 1099511628803, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-22.jpg'}, {'damage_type': 'backtrace', 'id': 2799167722, 'ino': 1099511630191, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-23.jpg'}, {'damage_type': 'backtrace', 'id': 2802738960, 'ino': 1099511630016, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-6.xml'}, {'damage_type': 'backtrace', 'id': 2817647686, 'ino': 1099511629569, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-11.jpg'}, {'damage_type': 'backtrace', 'id': 2830119417, 'ino': 1099511631129, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-78.xml'}, {'damage_type': 'backtrace', 'id': 2830150965, 'ino': 1099511629338, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-46.jpg'}, {'damage_type': 'backtrace', 'id': 2844812108, 'ino': 1099511630048, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-86.jpg'}, {'damage_type': 'backtrace', 'id': 2901496993, 'ino': 1099511631118, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-82.xml'}, {'damage_type': 'backtrace', 'id': 2907787229, 'ino': 1099511629506, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-32.jpg'}, {'damage_type': 'backtrace', 'id': 2914248246, 'ino': 1099511630462, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-81.xml'}, {'damage_type': 'backtrace', 'id': 2916678144, 'ino': 1099511629762, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-89.xml'}, {'damage_type': 'backtrace', 'id': 2917372064, 'ino': 1099511630060, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-81.jpg'}, {'damage_type': 'backtrace', 'id': 2919682196, 'ino': 1099511631309, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-40.xml'}, {'damage_type': 'backtrace', 'id': 2978081903, 'ino': 1099511630585, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-47.xml'}, {'damage_type': 'backtrace', 'id': 3006753426, 'ino': 1099511630303, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-46.xml'}, {'damage_type': 'backtrace', 'id': 3011624057, 'ino': 1099511630622, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-36.xml'}, {'damage_type': 'backtrace', 'id': 3012663908, 'ino': 1099511630678, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-10.xml'}, {'damage_type': 'backtrace', 'id': 3019142861, 'ino': 1099511631126, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-79.xml'}, {'damage_type': 'backtrace', 'id': 3035689779, 'ino': 1099511630112, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-55.jpg'}, {'damage_type': 'backtrace', 'id': 3042896209, 'ino': 1099511630662, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-17.xml'}, {'damage_type': 'backtrace', 'id': 3042966305, 'ino': 1099511630599, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-43.xml'}, {'damage_type': 'backtrace', 'id': 3061509973, 'ino': 1099511630183, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-13.xml'}, {'damage_type': 'backtrace', 'id': 3077682983, 'ino': 1099511630080, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-71.jpg'}, {'damage_type': 'backtrace', 'id': 3093181774, 'ino': 1099511629458, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-75.xml'}, {'damage_type': 'backtrace', 'id': 3097048537, 'ino': 1099511630193, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-30.jpg'}, {'damage_type': 'backtrace', 'id': 3100908460, 'ino': 1099511630000, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-22.xml'}, {'damage_type': 'backtrace', 'id': 3101067520, 'ino': 1099511629593, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-3.jpg'}, {'damage_type': 'backtrace', 'id': 3112198065, 'ino': 1099511630241, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-21.jpg'}, {'damage_type': 'backtrace', 'id': 3141570329, 'ino': 1099511629855, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-4.xml'}, {'damage_type': 'backtrace', 'id': 3143632943, 'ino': 1099511630139, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-43.jpg'}, {'damage_type': 'backtrace', 'id': 3151679873, 'ino': 1099511629819, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-40.xml'}, {'damage_type': 'backtrace', 'id': 3152998776, 'ino': 1099511629903, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-44.jpg'}, {'damage_type': 'backtrace', 'id': 3187102063, 'ino': 1099511629968, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-54.xml'}, {'damage_type': 'backtrace', 'id': 3187240234, 'ino': 1099511629875, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-72.jpg'}, {'damage_type': 'backtrace', 'id': 3187898192, 'ino': 1099511629214, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-54.jpg'}, {'damage_type': 'backtrace', 'id': 3188764577, 'ino': 1099511631146, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-74.xml'}, {'damage_type': 'backtrace', 'id': 3199192670, 'ino': 1099511628979, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-6.xml'}, {'damage_type': 'backtrace', 'id': 3202133489, 'ino': 1099511628706, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-16.xml'}, {'damage_type': 'backtrace', 'id': 3211657934, 'ino': 1099511630175, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-29.jpg'}, {'damage_type': 'backtrace', 'id': 3214665290, 'ino': 1099511630068, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-77.jpg'}, {'damage_type': 'backtrace', 'id': 3224272711, 'ino': 1099511630690, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-7.xml'}, {'damage_type': 'backtrace', 'id': 3228604792, 'ino': 1099511630320, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-94.jpg'}, {'damage_type': 'backtrace', 'id': 3236175507, 'ino': 1099511630682, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-9.xml'}, {'damage_type': 'backtrace', 'id': 3254469694, 'ino': 1099511630106, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-58.jpg'}, {'damage_type': 'backtrace', 'id': 3277670222, 'ino': 1099511629843, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-16.xml'}, {'damage_type': 'backtrace', 'id': 3280768504, 'ino': 1099511629581, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-7.jpg'}, {'damage_type': 'backtrace', 'id': 3342091720, 'ino': 1099511631057, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-96.xml'}, {'damage_type': 'backtrace', 'id': 3342511677, 'ino': 1099511629594, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-7.xml'}, {'damage_type': 'backtrace', 'id': 3377095679, 'ino': 1099511630633, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-33.xml'}, {'damage_type': 'backtrace', 'id': 3392236125, 'ino': 1099511629292, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-50.jpg'}, {'damage_type': 'backtrace', 'id': 3394885590, 'ino': 1099511629988, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-34.xml'}, {'damage_type': 'backtrace', 'id': 3406948383, 'ino': 1099511629499, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-39.xml'}, {'damage_type': 'backtrace', 'id': 3417374128, 'ino': 1099511631213, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-60.xml'}, {'damage_type': 'backtrace', 'id': 3418975498, 'ino': 1099511629911, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-36.jpg'}, {'damage_type': 'backtrace', 'id': 3436038551, 'ino': 1099511631047, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-98.xml'}, {'damage_type': 'backtrace', 'id': 3438306503, 'ino': 1099511629542, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-20.jpg'}, {'damage_type': 'backtrace', 'id': 3443452511, 'ino': 1099511628710, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-15.xml'}, {'damage_type': 'backtrace', 'id': 3445819183, 'ino': 1099511630076, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-73.jpg'}, {'damage_type': 'backtrace', 'id': 3451193409, 'ino': 1099511630168, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-31.jpg'}, {'damage_type': 'backtrace', 'id': 3456751974, 'ino': 1099511628715, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-13.xml'}, {'damage_type': 'backtrace', 'id': 3490969090, 'ino': 1099511629504, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-37.xml'}, {'damage_type': 'backtrace', 'id': 3491723286, 'ino': 1099511628708, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-63.xml'}, {'damage_type': 'backtrace', 'id': 3502405583, 'ino': 1099511631085, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-90.xml'}, {'damage_type': 'backtrace', 'id': 3505324111, 'ino': 1099511630595, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-44.xml'}, {'damage_type': 'backtrace', 'id': 3512890753, 'ino': 1099511630626, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-35.xml'}, {'damage_type': 'backtrace', 'id': 3520072677, 'ino': 1099511629833, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-26.xml'}, {'damage_type': 'backtrace', 'id': 3581789947, 'ino': 1099511631168, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-69.xml'}, {'damage_type': 'backtrace', 'id': 3582628323, 'ino': 1099511630114, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-54.jpg'}, {'damage_type': 'backtrace', 'id': 3607225472, 'ino': 1099511629560, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-14.jpg'}, {'damage_type': 'backtrace', 'id': 3608281099, 'ino': 1099511630592, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-45.xml'}, {'damage_type': 'backtrace', 'id': 3615832891, 'ino': 1099511629509, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-31.jpg'}, {'damage_type': 'backtrace', 'id': 3627099935, 'ino': 1099511629575, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-9.jpg'}, {'damage_type': 'backtrace', 'id': 3630838973, 'ino': 1099511631314, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-39.xml'}, {'damage_type': 'backtrace', 'id': 3635447008, 'ino': 1099511630588, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-46.xml'}, {'damage_type': 'backtrace', 'id': 3640792740, 'ino': 1099511630035, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-91.jpg'}, {'damage_type': 'backtrace', 'id': 3643809039, 'ino': 1099511629989, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-33.xml'}, {'damage_type': 'backtrace', 'id': 3651914794, 'ino': 1099511630454, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-83.xml'}, {'damage_type': 'backtrace', 'id': 3665133221, 'ino': 1099511628884, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-12.xml'}, {'damage_type': 'backtrace', 'id': 3669989646, 'ino': 1099511628713, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-14.xml'}, {'damage_type': 'backtrace', 'id': 3678909295, 'ino': 1099511629821, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-38.xml'}, {'damage_type': 'backtrace', 'id': 3728191131, 'ino': 1099511630714, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-1.xml'}, {'damage_type': 'backtrace', 'id': 3737041425, 'ino': 1099511629970, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-52.xml'}, {'damage_type': 'backtrace', 'id': 3750193451, 'ino': 1099511629603, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-3.xml'}, {'damage_type': 'backtrace', 'id': 3753675429, 'ino': 1099511628794, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-19.xml'}, {'damage_type': 'backtrace', 'id': 3761056468, 'ino': 1099511630636, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-32.xml'}, {'damage_type': 'backtrace', 'id': 3763033883, 'ino': 1099511630242, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-5.jpg'}, {'damage_type': 'backtrace', 'id': 3772445864, 'ino': 1099511629317, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-48.jpg'}, {'damage_type': 'backtrace', 'id': 3772761426, 'ino': 1099511631117, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-83.xml'}, {'damage_type': 'backtrace', 'id': 3784971172, 'ino': 1099511630694, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-6.xml'}, {'damage_type': 'backtrace', 'id': 3788588053, 'ino': 1099511630466, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-80.xml'}, {'damage_type': 'backtrace', 'id': 3802728626, 'ino': 1099511630082, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-70.jpg'}, {'damage_type': 'backtrace', 'id': 3827696746, 'ino': 1099511630550, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-57.xml'}, {'damage_type': 'backtrace', 'id': 3832404694, 'ino': 1099511629987, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-35.xml'}, {'damage_type': 'backtrace', 'id': 3881175636, 'ino': 1099511630186, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-25.jpg'}, {'damage_type': 'backtrace', 'id': 3891377885, 'ino': 1099511629906, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-41.jpg'}, {'damage_type': 'backtrace', 'id': 3900523764, 'ino': 1099511630231, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-8.jpg'}, {'damage_type': 'backtrace', 'id': 3903399581, 'ino': 1099511630014, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-8.xml'}, {'damage_type': 'backtrace', 'id': 3906367250, 'ino': 1099511631156, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-72.xml'}, {'damage_type': 'backtrace', 'id': 3935571964, 'ino': 1099511629845, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-14.xml'}, {'damage_type': 'backtrace', 'id': 3940439370, 'ino': 1099511630665, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-14.xml'}, {'damage_type': 'backtrace', 'id': 3942869314, 'ino': 1099511630686, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-8.xml'}, {'damage_type': 'backtrace', 'id': 3946725809, 'ino': 1099511629462, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/article-73.xml'}, {'damage_type': 'backtrace', 'id': 3980319307, 'ino': 1099511630012, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-10.xml'}, {'damage_type': 'backtrace', 'id': 3998396182, 'ino': 1099511630090, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-66.jpg'}, {'damage_type': 'backtrace', 'id': 4018407528, 'ino': 1099511628817, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-17.xml'}, {'damage_type': 'backtrace', 'id': 4029863417, 'ino': 1099511631102, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-86.xml'}, {'damage_type': 'backtrace', 'id': 4043249444, 'ino': 1099511629967, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-55.xml'}, {'damage_type': 'backtrace', 'id': 4044711679, 'ino': 1099511628778, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-30.jpg'}, {'damage_type': 'backtrace', 'id': 4049830008, 'ino': 1099511628754, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-23.xml'}, {'damage_type': 'backtrace', 'id': 4062828789, 'ino': 1099511628724, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-10.xml'}, {'damage_type': 'backtrace', 'id': 4073726047, 'ino': 1099511630209, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-15.jpg'}, {'damage_type': 'backtrace', 'id': 4086200377, 'ino': 1099511630104, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-59.jpg'}, {'damage_type': 'backtrace', 'id': 4092899621, 'ino': 1099511630519, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-66.xml'}, {'damage_type': 'backtrace', 'id': 4111130473, 'ino': 1099511628802, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-18.xml'}, {'damage_type': 'backtrace', 'id': 4114820146, 'ino': 1099511630474, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-78.xml'}, {'damage_type': 'backtrace', 'id': 4115873745, 'ino': 1099511629672, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-24.jpg'}, {'damage_type': 'backtrace', 'id': 4122293720, 'ino': 1099511628701, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-27.xml'}, {'damage_type': 'backtrace', 'id': 4132495499, 'ino': 1099511629834, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-25.xml'}, {'damage_type': 'backtrace', 'id': 4137923861, 'ino': 1099511629690, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-6.jpg'}, {'damage_type': 'backtrace', 'id': 4164036708, 'ino': 1099511630102, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-60.jpg'}, {'damage_type': 'backtrace', 'id': 4172700781, 'ino': 1099511628844, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-8.jpg'}, {'damage_type': 'backtrace', 'id': 4202423232, 'ino': 1099511629658, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/picture-33.jpg'}, {'damage_type': 'backtrace', 'id': 4203229509, 'ino': 1099511630007, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-15.xml'}, {'damage_type': 'backtrace', 'id': 4204706891, 'ino': 1099511630311, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-90.xml'}, {'damage_type': 'backtrace', 'id': 4207944627, 'ino': 1099511630324, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5/comment-30.xml'}, {'damage_type': 'backtrace', 'id': 4208455058, 'ino': 1099511629810, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-48.xml'}, {'damage_type': 'backtrace', 'id': 4218052630, 'ino': 1099511630009, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-13.xml'}, {'damage_type': 'backtrace', 'id': 4240288003, 'ino': 1099511631260, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-50.xml'}, {'damage_type': 'backtrace', 'id': 4253048807, 'ino': 1099511630710, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-2.xml'}, {'damage_type': 'backtrace', 'id': 4253873012, 'ino': 1099511630151, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-38.jpg'}, {'damage_type': 'backtrace', 'id': 4266319135, 'ino': 1099511630128, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-47.jpg'}] 2024-05-09T08:28:36.400 ERROR:tasks.fwd_scrub:rank 3 damaged: [{'damage_type': 'backtrace', 'id': 1691056, 'ino': 1099511628784, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-28.jpg'}, {'damage_type': 'backtrace', 'id': 3075453, 'ino': 1099511630970, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-54.xml'}, {'damage_type': 'backtrace', 'id': 7912273, 'ino': 1099511630646, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-29.xml'}, {'damage_type': 'backtrace', 'id': 19206521, 'ino': 1099511628728, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-89.xml'}, {'damage_type': 'backtrace', 'id': 35556203, 'ino': 1099511628773, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-32.jpg'}, {'damage_type': 'backtrace', 'id': 37064789, 'ino': 1099511631297, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-42.xml'}, {'damage_type': 'backtrace', 'id': 49628498, 'ino': 1099511628737, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-86.xml'}, {'damage_type': 'backtrace', 'id': 53223335, 'ino': 1099511630801, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-70.xml'}, {'damage_type': 'backtrace', 'id': 68998011, 'ino': 1099511629013, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-61.jpg'}, {'damage_type': 'backtrace', 'id': 100013233, 'ino': 1099511631044, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-99.xml'}, {'damage_type': 'backtrace', 'id': 142985928, 'ino': 1099511630160, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-34.jpg'}, {'damage_type': 'backtrace', 'id': 144999770, 'ino': 1099511630066, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-78.jpg'}, {'damage_type': 'backtrace', 'id': 202861153, 'ino': 1099511628909, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-25.xml'}, {'damage_type': 'backtrace', 'id': 239223632, 'ino': 1099511628858, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-45.xml'}, {'damage_type': 'backtrace', 'id': 244316040, 'ino': 1099511630086, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-68.jpg'}, {'damage_type': 'backtrace', 'id': 255627609, 'ino': 1099511628750, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/comment-58.xml'}, {'damage_type': 'backtrace', 'id': 287890724, 'ino': 1099511631223, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-58.xml'}, {'damage_type': 'backtrace', 'id': 289972258, 'ino': 1099511628781, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-71.xml'}, {'damage_type': 'backtrace', 'id': 293438506, 'ino': 1099511628811, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-19.jpg'}, {'damage_type': 'backtrace', 'id': 314289078, 'ino': 1099511630547, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-58.xml'}, {'damage_type': 'backtrace', 'id': 316678561, 'ino': 1099511628818, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-17.jpg'}, {'damage_type': 'backtrace', 'id': 322148415, 'ino': 1099511628969, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-4.xml'}, {'damage_type': 'backtrace', 'id': 324928162, 'ino': 1099511629043, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-49.jpg'}, {'damage_type': 'backtrace', 'id': 325858341, 'ino': 1099511628845, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-50.xml'}, {'damage_type': 'backtrace', 'id': 327666284, 'ino': 1099511630004, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-18.xml'}, {'damage_type': 'backtrace', 'id': 350175992, 'ino': 1099511631160, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-71.xml'}, {'damage_type': 'backtrace', 'id': 366431824, 'ino': 1099511631148, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-36.xml'}, {'damage_type': 'backtrace', 'id': 392403956, 'ino': 1099511628830, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-13.jpg'}, {'damage_type': 'backtrace', 'id': 415393201, 'ino': 1099511631240, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-55.xml'}, {'damage_type': 'backtrace', 'id': 433483280, 'ino': 1099511631264, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-49.xml'}, {'damage_type': 'backtrace', 'id': 434108997, 'ino': 1099511628767, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-34.jpg'}, {'damage_type': 'backtrace', 'id': 443967667, 'ino': 1099511630037, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-90.jpg'}, {'damage_type': 'backtrace', 'id': 486932057, 'ino': 1099511628743, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-84.xml'}, {'damage_type': 'backtrace', 'id': 497102498, 'ino': 1099511629385, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-42.jpg'}, {'damage_type': 'backtrace', 'id': 500420415, 'ino': 1099511628740, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-5.xml'}, {'damage_type': 'backtrace', 'id': 501069725, 'ino': 1099511630165, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-32.jpg'}, {'damage_type': 'backtrace', 'id': 510706781, 'ino': 1099511629064, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-97.xml'}, {'damage_type': 'backtrace', 'id': 511463612, 'ino': 1099511630658, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-21.xml'}, {'damage_type': 'backtrace', 'id': 529821141, 'ino': 1099511628942, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-13.xml'}, {'damage_type': 'backtrace', 'id': 537895592, 'ino': 1099511628904, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-27.xml'}, {'damage_type': 'backtrace', 'id': 576399834, 'ino': 1099511628847, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-7.jpg'}, {'damage_type': 'backtrace', 'id': 641104003, 'ino': 1099511629995, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-27.xml'}, {'damage_type': 'backtrace', 'id': 649071471, 'ino': 1099511630024, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-1.xml'}, {'damage_type': 'backtrace', 'id': 657679186, 'ino': 1099511630937, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-7.jpg'}, {'damage_type': 'backtrace', 'id': 659311636, 'ino': 1099511630208, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-16.jpg'}, {'damage_type': 'backtrace', 'id': 664062766, 'ino': 1099511629986, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-36.xml'}, {'damage_type': 'backtrace', 'id': 695539810, 'ino': 1099511630808, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-18.jpg'}, {'damage_type': 'backtrace', 'id': 733663014, 'ino': 1099511628722, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-91.xml'}, {'damage_type': 'backtrace', 'id': 734498331, 'ino': 1099511628727, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-9.xml'}, {'damage_type': 'backtrace', 'id': 741829620, 'ino': 1099511630206, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-17.jpg'}, {'damage_type': 'backtrace', 'id': 749351313, 'ino': 1099511630136, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-78.xml'}, {'damage_type': 'backtrace', 'id': 800465188, 'ino': 1099511630967, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-4.jpg'}, {'damage_type': 'backtrace', 'id': 811525216, 'ino': 1099511630643, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-30.xml'}, {'damage_type': 'backtrace', 'id': 820308641, 'ino': 1099511628841, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-9.jpg'}, {'damage_type': 'backtrace', 'id': 821387302, 'ino': 1099511630659, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-20.xml'}, {'damage_type': 'backtrace', 'id': 828659938, 'ino': 1099511631063, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-95.xml'}, {'damage_type': 'backtrace', 'id': 894847959, 'ino': 1099511629124, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-91.xml'}, {'damage_type': 'backtrace', 'id': 895259661, 'ino': 1099511631071, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-93.xml'}, {'damage_type': 'backtrace', 'id': 896737240, 'ino': 1099511628939, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-14.xml'}, {'damage_type': 'backtrace', 'id': 905815176, 'ino': 1099511629649, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-36.jpg'}, {'damage_type': 'backtrace', 'id': 919187694, 'ino': 1099511630670, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-12.xml'}, {'damage_type': 'backtrace', 'id': 921114147, 'ino': 1099511631256, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-51.xml'}, {'damage_type': 'backtrace', 'id': 957453590, 'ino': 1099511628906, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-26.xml'}, {'damage_type': 'backtrace', 'id': 961462280, 'ino': 1099511628714, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-94.xml'}, {'damage_type': 'backtrace', 'id': 967634153, 'ino': 1099511630907, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-60.xml'}, {'damage_type': 'backtrace', 'id': 972523860, 'ino': 1099511630054, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-83.jpg'}, {'damage_type': 'backtrace', 'id': 1001020152, 'ino': 1099511629176, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-57.jpg'}, {'damage_type': 'backtrace', 'id': 1017199308, 'ino': 1099511630094, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-64.jpg'}, {'damage_type': 'backtrace', 'id': 1044080981, 'ino': 1099511631248, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-53.xml'}, {'damage_type': 'backtrace', 'id': 1049236503, 'ino': 1099511630762, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-21.jpg'}, {'damage_type': 'backtrace', 'id': 1062125969, 'ino': 1099511631187, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-65.xml'}, {'damage_type': 'backtrace', 'id': 1085602239, 'ino': 1099511630663, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-16.xml'}, {'damage_type': 'backtrace', 'id': 1089233160, 'ino': 1099511628854, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-47.xml'}, {'damage_type': 'backtrace', 'id': 1091565722, 'ino': 1099511628827, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-56.xml'}, {'damage_type': 'backtrace', 'id': 1097657290, 'ino': 1099511628719, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-92.xml'}, {'damage_type': 'backtrace', 'id': 1127653550, 'ino': 1099511629097, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-75.xml'}, {'damage_type': 'backtrace', 'id': 1148858153, 'ino': 1099511629657, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-35.jpg'}, {'damage_type': 'backtrace', 'id': 1157290303, 'ino': 1099511629985, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-37.xml'}, {'damage_type': 'backtrace', 'id': 1168790224, 'ino': 1099511630656, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-23.xml'}, {'damage_type': 'backtrace', 'id': 1169465874, 'ino': 1099511629572, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-10.jpg'}, {'damage_type': 'backtrace', 'id': 1191164962, 'ino': 1099511630772, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-72.xml'}, {'damage_type': 'backtrace', 'id': 1194843907, 'ino': 1099511628789, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-69.xml'}, {'damage_type': 'backtrace', 'id': 1201308634, 'ino': 1099511630189, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-24.jpg'}, {'damage_type': 'backtrace', 'id': 1201325473, 'ino': 1099511630010, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-12.xml'}, {'damage_type': 'backtrace', 'id': 1207757936, 'ino': 1099511631235, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-56.xml'}, {'damage_type': 'backtrace', 'id': 1214994307, 'ino': 1099511631323, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-11.xml'}, {'damage_type': 'backtrace', 'id': 1226376208, 'ino': 1099511631173, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-33.xml'}, {'damage_type': 'backtrace', 'id': 1238011462, 'ino': 1099511630653, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-26.xml'}, {'damage_type': 'backtrace', 'id': 1260621263, 'ino': 1099511629999, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-23.xml'}, {'damage_type': 'backtrace', 'id': 1272152249, 'ino': 1099511630239, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-6.jpg'}, {'damage_type': 'backtrace', 'id': 1295638581, 'ino': 1099511629983, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-39.xml'}, {'damage_type': 'backtrace', 'id': 1345210540, 'ino': 1099511631030, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-48.xml'}, {'damage_type': 'backtrace', 'id': 1353214759, 'ino': 1099511630773, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-24.xml'}, {'damage_type': 'backtrace', 'id': 1369436936, 'ino': 1099511630013, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-9.xml'}, {'damage_type': 'backtrace', 'id': 1371495524, 'ino': 1099511630571, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-51.xml'}, {'damage_type': 'backtrace', 'id': 1376612431, 'ino': 1099511630149, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-39.jpg'}, {'damage_type': 'backtrace', 'id': 1383406021, 'ino': 1099511630219, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-25.jpg'}, {'damage_type': 'backtrace', 'id': 1389858451, 'ino': 1099511629811, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-83.xml'}, {'damage_type': 'backtrace', 'id': 1404741709, 'ino': 1099511628833, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-54.xml'}, {'damage_type': 'backtrace', 'id': 1404938867, 'ino': 1099511630554, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-56.xml'}, {'damage_type': 'backtrace', 'id': 1418746224, 'ino': 1099511628950, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-10.xml'}, {'damage_type': 'backtrace', 'id': 1481765360, 'ino': 1099511630769, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-52.jpg'}, {'damage_type': 'backtrace', 'id': 1488997603, 'ino': 1099511629924, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-29.jpg'}, {'damage_type': 'backtrace', 'id': 1492121554, 'ino': 1099511631293, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-43.xml'}, {'damage_type': 'backtrace', 'id': 1504816375, 'ino': 1099511629050, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-47.jpg'}, {'damage_type': 'backtrace', 'id': 1509037501, 'ino': 1099511630667, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-13.xml'}, {'damage_type': 'backtrace', 'id': 1569761346, 'ino': 1099511628826, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-14.jpg'}, {'damage_type': 'backtrace', 'id': 1573911725, 'ino': 1099511630288, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-74.xml'}, {'damage_type': 'backtrace', 'id': 1604549516, 'ino': 1099511630049, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-85.jpg'}, {'damage_type': 'backtrace', 'id': 1612352806, 'ino': 1099511630216, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-10.jpg'}, {'damage_type': 'backtrace', 'id': 1616630931, 'ino': 1099511630005, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-17.xml'}, {'damage_type': 'backtrace', 'id': 1631495822, 'ino': 1099511629069, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-96.xml'}, {'damage_type': 'backtrace', 'id': 1668083825, 'ino': 1099511630629, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-34.xml'}, {'damage_type': 'backtrace', 'id': 1680395066, 'ino': 1099511631112, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-84.xml'}, {'damage_type': 'backtrace', 'id': 1687689284, 'ino': 1099511630235, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-7.jpg'}, {'damage_type': 'backtrace', 'id': 1738857850, 'ino': 1099511629031, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-54.jpg'}, {'damage_type': 'backtrace', 'id': 1752119688, 'ino': 1099511629978, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-44.xml'}, {'damage_type': 'backtrace', 'id': 1766799772, 'ino': 1099511628716, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-93.xml'}, {'damage_type': 'backtrace', 'id': 1772949685, 'ino': 1099511628890, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-32.xml'}, {'damage_type': 'backtrace', 'id': 1792411863, 'ino': 1099511629091, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-94.xml'}, {'damage_type': 'backtrace', 'id': 1795845676, 'ino': 1099511630116, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-53.jpg'}, {'damage_type': 'backtrace', 'id': 1813529763, 'ino': 1099511628948, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-11.xml'}, {'damage_type': 'backtrace', 'id': 1822616730, 'ino': 1099511629926, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-1.jpg'}, {'damage_type': 'backtrace', 'id': 1823408244, 'ino': 1099511629918, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-82.xml'}, {'damage_type': 'backtrace', 'id': 1860670429, 'ino': 1099511629057, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-98.xml'}, {'damage_type': 'backtrace', 'id': 1861324653, 'ino': 1099511630211, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-14.jpg'}, {'damage_type': 'backtrace', 'id': 1865326124, 'ino': 1099511628935, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-15.xml'}, {'damage_type': 'backtrace', 'id': 1876450698, 'ino': 1099511628831, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-55.xml'}, {'damage_type': 'backtrace', 'id': 1883773261, 'ino': 1099511629131, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-90.xml'}, {'damage_type': 'backtrace', 'id': 1899998272, 'ino': 1099511628892, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-31.xml'}, {'damage_type': 'backtrace', 'id': 1907787573, 'ino': 1099511628859, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-2.jpg'}, {'damage_type': 'backtrace', 'id': 1925922062, 'ino': 1099511629067, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-43.jpg'}, {'damage_type': 'backtrace', 'id': 1927160453, 'ino': 1099511631303, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-41.xml'}, {'damage_type': 'backtrace', 'id': 1939037461, 'ino': 1099511630502, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-71.xml'}, {'damage_type': 'backtrace', 'id': 1944394981, 'ino': 1099511630270, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-23.jpg'}, {'damage_type': 'backtrace', 'id': 1948050872, 'ino': 1099511628791, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-68.xml'}, {'damage_type': 'backtrace', 'id': 1952986441, 'ino': 1099511630654, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-25.xml'}, {'damage_type': 'backtrace', 'id': 1971062874, 'ino': 1099511628749, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-82.xml'}, {'damage_type': 'backtrace', 'id': 1983262968, 'ino': 1099511630163, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-33.jpg'}, {'damage_type': 'backtrace', 'id': 1987725090, 'ino': 1099511630015, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-7.xml'}, {'damage_type': 'backtrace', 'id': 1994025039, 'ino': 1099511628856, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-46.xml'}, {'damage_type': 'backtrace', 'id': 2000409339, 'ino': 1099511631142, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-75.xml'}, {'damage_type': 'backtrace', 'id': 2001777388, 'ino': 1099511631124, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-80.xml'}, {'damage_type': 'backtrace', 'id': 2009599032, 'ino': 1099511628866, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-41.xml'}, {'damage_type': 'backtrace', 'id': 2023983093, 'ino': 1099511629033, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-3.xml'}, {'damage_type': 'backtrace', 'id': 2038470552, 'ino': 1099511630126, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-48.jpg'}, {'damage_type': 'backtrace', 'id': 2068588029, 'ino': 1099511629659, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-86.xml'}, {'damage_type': 'backtrace', 'id': 2076343252, 'ino': 1099511630108, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-57.jpg'}, {'damage_type': 'backtrace', 'id': 2077168423, 'ino': 1099511631182, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-66.xml'}, {'damage_type': 'backtrace', 'id': 2085553459, 'ino': 1099511628733, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-7.xml'}, {'damage_type': 'backtrace', 'id': 2098700022, 'ino': 1099511631181, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-32.xml'}, {'damage_type': 'backtrace', 'id': 2119441230, 'ino': 1099511628751, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-1.xml'}, {'damage_type': 'backtrace', 'id': 2120546907, 'ino': 1099511631174, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-68.xml'}, {'damage_type': 'backtrace', 'id': 2155008875, 'ino': 1099511630041, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-89.jpg'}, {'damage_type': 'backtrace', 'id': 2168195926, 'ino': 1099511630557, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-55.xml'}, {'damage_type': 'backtrace', 'id': 2169243449, 'ino': 1099511630751, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-73.xml'}, {'damage_type': 'backtrace', 'id': 2171646185, 'ino': 1099511630252, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-2.jpg'}, {'damage_type': 'backtrace', 'id': 2178890522, 'ino': 1099511631107, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-85.xml'}, {'damage_type': 'backtrace', 'id': 2190584649, 'ino': 1099511629639, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-89.xml'}, {'damage_type': 'backtrace', 'id': 2196803585, 'ino': 1099511631319, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-38.xml'}, {'damage_type': 'backtrace', 'id': 2231461834, 'ino': 1099511630070, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-76.jpg'}, {'damage_type': 'backtrace', 'id': 2242404436, 'ino': 1099511631048, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-46.xml'}, {'damage_type': 'backtrace', 'id': 2242495112, 'ino': 1099511629017, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-59.jpg'}, {'damage_type': 'backtrace', 'id': 2245195649, 'ino': 1099511628788, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-27.jpg'}, {'damage_type': 'backtrace', 'id': 2245472497, 'ino': 1099511631115, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-39.xml'}, {'damage_type': 'backtrace', 'id': 2254376886, 'ino': 1099511628739, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-24.xml'}, {'damage_type': 'backtrace', 'id': 2267312302, 'ino': 1099511630706, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-3.xml'}, {'damage_type': 'backtrace', 'id': 2292676533, 'ino': 1099511629094, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-41.jpg'}, {'damage_type': 'backtrace', 'id': 2309304580, 'ino': 1099511628927, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-18.xml'}, {'damage_type': 'backtrace', 'id': 2312936318, 'ino': 1099511630057, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-82.jpg'}, {'damage_type': 'backtrace', 'id': 2313711187, 'ino': 1099511629002, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-66.jpg'}, {'damage_type': 'backtrace', 'id': 2320277316, 'ino': 1099511628922, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-20.xml'}, {'damage_type': 'backtrace', 'id': 2321450964, 'ino': 1099511631304, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-14.xml'}, {'damage_type': 'backtrace', 'id': 2323668827, 'ino': 1099511631253, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-52.xml'}, {'damage_type': 'backtrace', 'id': 2326378844, 'ino': 1099511630100, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-61.jpg'}, {'damage_type': 'backtrace', 'id': 2339593201, 'ino': 1099511630892, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-62.xml'}, {'damage_type': 'backtrace', 'id': 2370956116, 'ino': 1099511631289, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-44.xml'}, {'damage_type': 'backtrace', 'id': 2382256188, 'ino': 1099511631207, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-61.xml'}, {'damage_type': 'backtrace', 'id': 2404226351, 'ino': 1099511630781, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-20.jpg'}, {'damage_type': 'backtrace', 'id': 2427013856, 'ino': 1099511630871, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-64.xml'}, {'damage_type': 'backtrace', 'id': 2433783974, 'ino': 1099511629981, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-41.xml'}, {'damage_type': 'backtrace', 'id': 2471212844, 'ino': 1099511630201, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-19.jpg'}, {'damage_type': 'backtrace', 'id': 2497477784, 'ino': 1099511629917, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-31.jpg'}, {'damage_type': 'backtrace', 'id': 2505711844, 'ino': 1099511628855, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-4.jpg'}, {'damage_type': 'backtrace', 'id': 2519178280, 'ino': 1099511630203, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-18.jpg'}, {'damage_type': 'backtrace', 'id': 2537402496, 'ino': 1099511628882, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-35.xml'}, {'damage_type': 'backtrace', 'id': 2566611817, 'ino': 1099511628725, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-90.xml'}, {'damage_type': 'backtrace', 'id': 2574619235, 'ino': 1099511629424, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-39.jpg'}, {'damage_type': 'backtrace', 'id': 2596203658, 'ino': 1099511628864, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-42.xml'}, {'damage_type': 'backtrace', 'id': 2611049783, 'ino': 1099511628981, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-73.jpg'}, {'damage_type': 'backtrace', 'id': 2612934612, 'ino': 1099511629015, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-60.jpg'}, {'damage_type': 'backtrace', 'id': 2614384022, 'ino': 1099511628874, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-38.xml'}, {'damage_type': 'backtrace', 'id': 2646275824, 'ino': 1099511629025, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-56.jpg'}, {'damage_type': 'backtrace', 'id': 2653232854, 'ino': 1099511630661, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-18.xml'}, {'damage_type': 'backtrace', 'id': 2673520015, 'ino': 1099511630227, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-9.jpg'}, {'damage_type': 'backtrace', 'id': 2684595535, 'ino': 1099511630033, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-92.jpg'}, {'damage_type': 'backtrace', 'id': 2702502505, 'ino': 1099511628842, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-51.xml'}, {'damage_type': 'backtrace', 'id': 2710437174, 'ino': 1099511631010, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-50.xml'}, {'damage_type': 'backtrace', 'id': 2712421549, 'ino': 1099511628796, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-24.jpg'}, {'damage_type': 'backtrace', 'id': 2717192708, 'ino': 1099511630619, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-37.xml'}, {'damage_type': 'backtrace', 'id': 2747390815, 'ino': 1099511628835, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-11.jpg'}, {'damage_type': 'backtrace', 'id': 2760424524, 'ino': 1099511628857, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-3.jpg'}, {'damage_type': 'backtrace', 'id': 2766985365, 'ino': 1099511630853, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-14.jpg'}, {'damage_type': 'backtrace', 'id': 2783443461, 'ino': 1099511631257, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-20.xml'}, {'damage_type': 'backtrace', 'id': 2788211772, 'ino': 1099511630177, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-28.jpg'}, {'damage_type': 'backtrace', 'id': 2801297688, 'ino': 1099511628774, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-74.xml'}, {'damage_type': 'backtrace', 'id': 2803344897, 'ino': 1099511628853, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-5.jpg'}, {'damage_type': 'backtrace', 'id': 2823979037, 'ino': 1099511630602, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-42.xml'}, {'damage_type': 'backtrace', 'id': 2834394178, 'ino': 1099511631325, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-37.xml'}, {'damage_type': 'backtrace', 'id': 2857004487, 'ino': 1099511628770, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-33.jpg'}, {'damage_type': 'backtrace', 'id': 2874795341, 'ino': 1099511631283, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-45.xml'}, {'damage_type': 'backtrace', 'id': 2883257357, 'ino': 1099511629061, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-44.jpg'}, {'damage_type': 'backtrace', 'id': 2900083504, 'ino': 1099511630837, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-67.xml'}, {'damage_type': 'backtrace', 'id': 2918435494, 'ino': 1099511631204, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-29.xml'}, {'damage_type': 'backtrace', 'id': 2930716047, 'ino': 1099511628920, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-21.xml'}, {'damage_type': 'backtrace', 'id': 2940210736, 'ino': 1099511630141, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-26.jpg'}, {'damage_type': 'backtrace', 'id': 2948423890, 'ino': 1099511630877, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-12.jpg'}, {'damage_type': 'backtrace', 'id': 2959207404, 'ino': 1099511630698, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-5.xml'}, {'damage_type': 'backtrace', 'id': 2964889244, 'ino': 1099511631267, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-19.xml'}, {'damage_type': 'backtrace', 'id': 2969592998, 'ino': 1099511629631, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-38.jpg'}, {'damage_type': 'backtrace', 'id': 3015176136, 'ino': 1099511629503, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-33.jpg'}, {'damage_type': 'backtrace', 'id': 3023064105, 'ino': 1099511628936, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-9.xml'}, {'damage_type': 'backtrace', 'id': 3027879308, 'ino': 1099511628820, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-16.jpg'}, {'damage_type': 'backtrace', 'id': 3054882650, 'ino': 1099511628963, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-6.xml'}, {'damage_type': 'backtrace', 'id': 3057849478, 'ino': 1099511629530, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-24.jpg'}, {'damage_type': 'backtrace', 'id': 3063775713, 'ino': 1099511628869, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-40.xml'}, {'damage_type': 'backtrace', 'id': 3067612133, 'ino': 1099511629584, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-6.jpg'}, {'damage_type': 'backtrace', 'id': 3069425361, 'ino': 1099511628924, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-19.xml'}, {'damage_type': 'backtrace', 'id': 3090137522, 'ino': 1099511628957, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-8.xml'}, {'damage_type': 'backtrace', 'id': 3142933278, 'ino': 1099511630325, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-63.xml'}, {'damage_type': 'backtrace', 'id': 3151219771, 'ino': 1099511630987, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-2.jpg'}, {'damage_type': 'backtrace', 'id': 3190916067, 'ino': 1099511630002, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-20.xml'}, {'damage_type': 'backtrace', 'id': 3194504054, 'ino': 1099511628819, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-59.xml'}, {'damage_type': 'backtrace', 'id': 3196732154, 'ino': 1099511629518, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-28.jpg'}, {'damage_type': 'backtrace', 'id': 3214258154, 'ino': 1099511631120, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-81.xml'}, {'damage_type': 'backtrace', 'id': 3214820103, 'ino': 1099511631076, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-92.xml'}, {'damage_type': 'backtrace', 'id': 3215492162, 'ino': 1099511628984, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-72.jpg'}, {'damage_type': 'backtrace', 'id': 3218886991, 'ino': 1099511628861, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-1.jpg'}, {'damage_type': 'backtrace', 'id': 3229858493, 'ino': 1099511628895, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-30.xml'}, {'damage_type': 'backtrace', 'id': 3233470102, 'ino': 1099511628932, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-16.xml'}, {'damage_type': 'backtrace', 'id': 3251106582, 'ino': 1099511630245, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-4.jpg'}, {'damage_type': 'backtrace', 'id': 3321419444, 'ino': 1099511628975, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-2.xml'}, {'damage_type': 'backtrace', 'id': 3334449824, 'ino': 1099511630523, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-65.xml'}, {'damage_type': 'backtrace', 'id': 3337581020, 'ino': 1099511630674, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-11.xml'}, {'damage_type': 'backtrace', 'id': 3364097015, 'ino': 1099511629645, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-88.xml'}, {'damage_type': 'backtrace', 'id': 3370023201, 'ino': 1099511630142, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-42.jpg'}, {'damage_type': 'backtrace', 'id': 3375280213, 'ino': 1099511630746, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-22.jpg'}, {'damage_type': 'backtrace', 'id': 3380485576, 'ino': 1099511630074, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-74.jpg'}, {'damage_type': 'backtrace', 'id': 3393146277, 'ino': 1099511628838, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-10.jpg'}, {'damage_type': 'backtrace', 'id': 3431332124, 'ino': 1099511628745, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-3.xml'}, {'damage_type': 'backtrace', 'id': 3453508011, 'ino': 1099511630816, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-69.xml'}, {'damage_type': 'backtrace', 'id': 3457173235, 'ino': 1099511630843, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-15.jpg'}, {'damage_type': 'backtrace', 'id': 3482835605, 'ino': 1099511629991, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-31.xml'}, {'damage_type': 'backtrace', 'id': 3490024847, 'ino': 1099511628718, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-12.xml'}, {'damage_type': 'backtrace', 'id': 3493298148, 'ino': 1099511630153, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-37.jpg'}, {'damage_type': 'backtrace', 'id': 3520706365, 'ino': 1099511629059, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-45.jpg'}, {'damage_type': 'backtrace', 'id': 3533326829, 'ino': 1099511628814, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-18.jpg'}, {'damage_type': 'backtrace', 'id': 3544061451, 'ino': 1099511630032, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-79.xml'}, {'damage_type': 'backtrace', 'id': 3544136171, 'ino': 1099511628851, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-48.xml'}, {'damage_type': 'backtrace', 'id': 3544292856, 'ino': 1099511630913, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-9.jpg'}, {'damage_type': 'backtrace', 'id': 3555991547, 'ino': 1099511630157, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-35.jpg'}, {'damage_type': 'backtrace', 'id': 3569811711, 'ino': 1099511629071, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-70.jpg'}, {'damage_type': 'backtrace', 'id': 3576527437, 'ino': 1099511630782, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-3.xml'}, {'damage_type': 'backtrace', 'id': 3576799226, 'ino': 1099511629977, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-45.xml'}, {'damage_type': 'backtrace', 'id': 3584431990, 'ino': 1099511630880, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-63.xml'}, {'damage_type': 'backtrace', 'id': 3586740279, 'ino': 1099511630088, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-67.jpg'}, {'damage_type': 'backtrace', 'id': 3681632711, 'ino': 1099511630952, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-56.xml'}, {'damage_type': 'backtrace', 'id': 3683371518, 'ino': 1099511631105, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-40.xml'}, {'damage_type': 'backtrace', 'id': 3737938530, 'ino': 1099511630865, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-13.jpg'}, {'damage_type': 'backtrace', 'id': 3752162032, 'ino': 1099511628746, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-83.xml'}, {'damage_type': 'backtrace', 'id': 3765137158, 'ino': 1099511631137, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-76.xml'}, {'damage_type': 'backtrace', 'id': 3834840612, 'ino': 1099511629087, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-57.jpg'}, {'damage_type': 'backtrace', 'id': 3836276681, 'ino': 1099511630578, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-49.xml'}, {'damage_type': 'backtrace', 'id': 3836698540, 'ino': 1099511629449, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-37.jpg'}, {'damage_type': 'backtrace', 'id': 3852196762, 'ino': 1099511629971, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/article-51.xml'}, {'damage_type': 'backtrace', 'id': 3871153241, 'ino': 1099511628757, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-37.jpg'}, {'damage_type': 'backtrace', 'id': 3892000270, 'ino': 1099511628961, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/article-7.xml'}, {'damage_type': 'backtrace', 'id': 3898944190, 'ino': 1099511631229, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-57.xml'}, {'damage_type': 'backtrace', 'id': 3915324275, 'ino': 1099511631039, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-100.xml'}, {'damage_type': 'backtrace', 'id': 3924939659, 'ino': 1099511630180, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-27.jpg'}, {'damage_type': 'backtrace', 'id': 4005398611, 'ino': 1099511630797, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-19.jpg'}, {'damage_type': 'backtrace', 'id': 4028506300, 'ino': 1099511630031, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-93.jpg'}, {'damage_type': 'backtrace', 'id': 4033444077, 'ino': 1099511630256, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-1.jpg'}, {'damage_type': 'backtrace', 'id': 4052490565, 'ino': 1099511630212, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/picture-13.jpg'}, {'damage_type': 'backtrace', 'id': 4055218510, 'ino': 1099511630919, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-59.xml'}, {'damage_type': 'backtrace', 'id': 4058489515, 'ino': 1099511631067, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-94.xml'}, {'damage_type': 'backtrace', 'id': 4059446553, 'ino': 1099511631004, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-73.xml'}, {'damage_type': 'backtrace', 'id': 4063592700, 'ino': 1099511628812, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-61.xml'}, {'damage_type': 'backtrace', 'id': 4066110856, 'ino': 1099511628785, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-70.xml'}, {'damage_type': 'backtrace', 'id': 4069568596, 'ino': 1099511628768, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-76.xml'}, {'damage_type': 'backtrace', 'id': 4083994442, 'ino': 1099511629921, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-30.jpg'}, {'damage_type': 'backtrace', 'id': 4085670590, 'ino': 1099511628703, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/article-17.xml'}, {'damage_type': 'backtrace', 'id': 4098153106, 'ino': 1099511630486, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-75.xml'}, {'damage_type': 'backtrace', 'id': 4132676385, 'ino': 1099511630925, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-8.jpg'}, {'damage_type': 'backtrace', 'id': 4162897597, 'ino': 1099511628711, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-95.xml'}, {'damage_type': 'backtrace', 'id': 4181086228, 'ino': 1099511628763, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-35.jpg'}, {'damage_type': 'backtrace', 'id': 4220149583, 'ino': 1099511631040, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-47.xml'}, {'damage_type': 'backtrace', 'id': 4248793657, 'ino': 1099511628779, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-72.xml'}, {'damage_type': 'backtrace', 'id': 4251550757, 'ino': 1099511631098, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/comment-87.xml'}, {'damage_type': 'backtrace', 'id': 4265170918, 'ino': 1099511628850, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-2/picture-6.jpg'}, {'damage_type': 'backtrace', 'id': 4266600770, 'ino': 1099511629551, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-1/picture-17.jpg'}, {'damage_type': 'backtrace', 'id': 4267748662, 'ino': 1099511628704, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-97.xml'}, {'damage_type': 'backtrace', 'id': 4269295579, 'ino': 1099511630145, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-77.xml'}, {'damage_type': 'backtrace', 'id': 4286367932, 'ino': 1099511630606, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-41.xml'}, {'damage_type': 'backtrace', 'id': 4294295330, 'ino': 1099511629116, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-92.xml'}, {'damage_type': 'backtrace', 'id': 4294652910, 'ino': 1099511630655, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-9/comment-24.xml'}] 2024-05-09T08:28:36.401 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 35760778, 'ino': 1099511628797, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-66.xml'}, {'damage_type': 'backtrace', 'id': 54623227, 'ino': 1099511631310, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-13.xml'}, {'damage_type': 'backtrace', 'id': 65272418, 'ino': 1099511630488, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-4.xml'}, {'damage_type': 'backtrace', 'id': 69406482, 'ino': 1099511630460, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-11.xml'}, {'damage_type': 'backtrace', 'id': 93575386, 'ino': 1099511630279, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-54.xml'}, {'damage_type': 'backtrace', 'id': 111493306, 'ino': 1099511630713, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-6.jpg'}, {'damage_type': 'backtrace', 'id': 147075460, 'ino': 1099511631066, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-44.xml'}, {'damage_type': 'backtrace', 'id': 156936873, 'ino': 1099511630364, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-35.xml'}, {'damage_type': 'backtrace', 'id': 194355636, 'ino': 1099511629653, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-87.xml'}, {'damage_type': 'backtrace', 'id': 237996004, 'ino': 1099511631315, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-12.xml'}, {'damage_type': 'backtrace', 'id': 250235332, 'ino': 1099511628918, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-22.xml'}, {'damage_type': 'backtrace', 'id': 263481486, 'ino': 1099511630831, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-16.jpg'}, {'damage_type': 'backtrace', 'id': 266276969, 'ino': 1099511631227, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-25.xml'}, {'damage_type': 'backtrace', 'id': 270662266, 'ino': 1099511630456, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-12.xml'}, {'damage_type': 'backtrace', 'id': 344390909, 'ino': 1099511628752, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-81.xml'}, {'damage_type': 'backtrace', 'id': 350068804, 'ino': 1099511628809, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-62.xml'}, {'damage_type': 'backtrace', 'id': 352486896, 'ino': 1099511630412, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-23.xml'}, {'damage_type': 'backtrace', 'id': 353286721, 'ino': 1099511630452, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-13.xml'}, {'damage_type': 'backtrace', 'id': 357826965, 'ino': 1099511630380, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-31.xml'}, {'damage_type': 'backtrace', 'id': 423912074, 'ino': 1099511629663, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-34.jpg'}, {'damage_type': 'backtrace', 'id': 455105318, 'ino': 1099511630528, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-33.jpg'}, {'damage_type': 'backtrace', 'id': 474548961, 'ino': 1099511628804, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-64.xml'}, {'damage_type': 'backtrace', 'id': 475434354, 'ino': 1099511631001, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-51.xml'}, {'damage_type': 'backtrace', 'id': 482116640, 'ino': 1099511628996, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-68.jpg'}, {'damage_type': 'backtrace', 'id': 502144801, 'ino': 1099511628801, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-65.xml'}, {'damage_type': 'backtrace', 'id': 504002431, 'ino': 1099511631298, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-15.xml'}, {'damage_type': 'backtrace', 'id': 512352290, 'ino': 1099511630492, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-3.xml'}, {'damage_type': 'backtrace', 'id': 551079241, 'ino': 1099511630440, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-16.xml'}, {'damage_type': 'backtrace', 'id': 568326091, 'ino': 1099511628971, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-3.xml'}, {'damage_type': 'backtrace', 'id': 578508010, 'ino': 1099511629101, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-93.xml'}, {'damage_type': 'backtrace', 'id': 585490697, 'ino': 1099511630993, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-52.xml'}, {'damage_type': 'backtrace', 'id': 586253369, 'ino': 1099511628885, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-34.xml'}, {'damage_type': 'backtrace', 'id': 600108682, 'ino': 1099511630631, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-20.jpg'}, {'damage_type': 'backtrace', 'id': 610020162, 'ino': 1099511630342, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-40.xml'}, {'damage_type': 'backtrace', 'id': 613897610, 'ino': 1099511629019, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-58.jpg'}, {'damage_type': 'backtrace', 'id': 650456428, 'ino': 1099511628945, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-12.xml'}, {'damage_type': 'backtrace', 'id': 690650604, 'ino': 1099511630697, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-10.jpg'}, {'damage_type': 'backtrace', 'id': 735204533, 'ino': 1099511629053, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-46.jpg'}, {'damage_type': 'backtrace', 'id': 761511694, 'ino': 1099511630901, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-61.xml'}, {'damage_type': 'backtrace', 'id': 825569894, 'ino': 1099511630368, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-34.xml'}, {'damage_type': 'backtrace', 'id': 844645104, 'ino': 1099511630849, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-66.xml'}, {'damage_type': 'backtrace', 'id': 860054014, 'ino': 1099511630291, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-52.xml'}, {'damage_type': 'backtrace', 'id': 871903858, 'ino': 1099511630999, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-1.jpg'}, {'damage_type': 'backtrace', 'id': 879907343, 'ino': 1099511630898, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-10.jpg'}, {'damage_type': 'backtrace', 'id': 928122385, 'ino': 1099511630039, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-27.jpg'}, {'damage_type': 'backtrace', 'id': 934156289, 'ino': 1099511630709, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-7.jpg'}, {'damage_type': 'backtrace', 'id': 947130980, 'ino': 1099511630693, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-11.jpg'}, {'damage_type': 'backtrace', 'id': 996032279, 'ino': 1099511630308, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-48.xml'}, {'damage_type': 'backtrace', 'id': 1001085975, 'ino': 1099511630304, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-49.xml'}, {'damage_type': 'backtrace', 'id': 1046886487, 'ino': 1099511630023, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-80.xml'}, {'damage_type': 'backtrace', 'id': 1051625495, 'ino': 1099511631097, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-41.xml'}, {'damage_type': 'backtrace', 'id': 1062314364, 'ino': 1099511629048, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-100.xml'}, {'damage_type': 'backtrace', 'id': 1082045085, 'ino': 1099511630480, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-6.xml'}, {'damage_type': 'backtrace', 'id': 1091486068, 'ino': 1099511628999, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-67.jpg'}, {'damage_type': 'backtrace', 'id': 1123052374, 'ino': 1099511628953, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-9.xml'}, {'damage_type': 'backtrace', 'id': 1134302960, 'ino': 1099511630590, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-25.jpg'}, {'damage_type': 'backtrace', 'id': 1147205189, 'ino': 1099511630444, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-15.xml'}, {'damage_type': 'backtrace', 'id': 1197208736, 'ino': 1099511631250, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-21.xml'}, {'damage_type': 'backtrace', 'id': 1198322143, 'ino': 1099511628761, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-78.xml'}, {'damage_type': 'backtrace', 'id': 1204491445, 'ino': 1099511630717, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-5.jpg'}, {'damage_type': 'backtrace', 'id': 1211013812, 'ino': 1099511631216, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-27.xml'}, {'damage_type': 'backtrace', 'id': 1211179005, 'ino': 1099511628915, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-23.xml'}, {'damage_type': 'backtrace', 'id': 1226586275, 'ino': 1099511629037, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-52.jpg'}, {'damage_type': 'backtrace', 'id': 1248283392, 'ino': 1099511631155, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-35.xml'}, {'damage_type': 'backtrace', 'id': 1254458876, 'ino': 1099511629126, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-39.jpg'}, {'damage_type': 'backtrace', 'id': 1256949360, 'ino': 1099511630404, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-25.xml'}, {'damage_type': 'backtrace', 'id': 1265014645, 'ino': 1099511628989, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-70.jpg'}, {'damage_type': 'backtrace', 'id': 1266434386, 'ino': 1099511630296, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-51.xml'}, {'damage_type': 'backtrace', 'id': 1267771524, 'ino': 1099511628888, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-33.xml'}, {'damage_type': 'backtrace', 'id': 1278900048, 'ino': 1099511630958, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-55.xml'}, {'damage_type': 'backtrace', 'id': 1285083143, 'ino': 1099511630476, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-7.xml'}, {'damage_type': 'backtrace', 'id': 1290355374, 'ino': 1099511630559, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-29.jpg'}, {'damage_type': 'backtrace', 'id': 1291922669, 'ino': 1099511630689, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-12.jpg'}, {'damage_type': 'backtrace', 'id': 1297372442, 'ino': 1099511631091, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-42.xml'}, {'damage_type': 'backtrace', 'id': 1306885947, 'ino': 1099511631012, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-50.jpg'}, {'damage_type': 'backtrace', 'id': 1308188440, 'ino': 1099511628897, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-29.xml'}, {'damage_type': 'backtrace', 'id': 1309691372, 'ino': 1099511629110, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-40.jpg'}, {'damage_type': 'backtrace', 'id': 1374814927, 'ino': 1099511630424, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-20.xml'}, {'damage_type': 'backtrace', 'id': 1398925088, 'ino': 1099511629914, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-32.jpg'}, {'damage_type': 'backtrace', 'id': 1438690839, 'ino': 1099511630720, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-4.jpg'}, {'damage_type': 'backtrace', 'id': 1442662600, 'ino': 1099511628839, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-52.xml'}, {'damage_type': 'backtrace', 'id': 1458411165, 'ino': 1099511631326, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-76.xml'}, {'damage_type': 'backtrace', 'id': 1506775299, 'ino': 1099511628912, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-24.xml'}, {'damage_type': 'backtrace', 'id': 1511418643, 'ino': 1099511630545, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-31.jpg'}, {'damage_type': 'backtrace', 'id': 1511993454, 'ino': 1099511630514, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-35.jpg'}, {'damage_type': 'backtrace', 'id': 1520591950, 'ino': 1099511628960, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-7.xml'}, {'damage_type': 'backtrace', 'id': 1551873850, 'ino': 1099511630028, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-28.jpg'}, {'damage_type': 'backtrace', 'id': 1582192302, 'ino': 1099511630428, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-19.xml'}, {'damage_type': 'backtrace', 'id': 1587813056, 'ino': 1099511629667, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-85.xml'}, {'damage_type': 'backtrace', 'id': 1614985740, 'ino': 1099511630673, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-16.jpg'}, {'damage_type': 'backtrace', 'id': 1671152772, 'ino': 1099511630786, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-71.xml'}, {'damage_type': 'backtrace', 'id': 1678791002, 'ino': 1099511630566, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-28.jpg'}, {'damage_type': 'backtrace', 'id': 1713036954, 'ino': 1099511630392, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-28.xml'}, {'damage_type': 'backtrace', 'id': 1739069595, 'ino': 1099511630232, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-24.jpg'}, {'damage_type': 'backtrace', 'id': 1748512052, 'ino': 1099511628756, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-80.xml'}, {'damage_type': 'backtrace', 'id': 1753956240, 'ino': 1099511630723, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-3.jpg'}, {'damage_type': 'backtrace', 'id': 1779315948, 'ino': 1099511630376, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-32.xml'}, {'damage_type': 'backtrace', 'id': 1782575728, 'ino': 1099511628862, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-43.xml'}, {'damage_type': 'backtrace', 'id': 1892380767, 'ino': 1099511629643, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-37.jpg'}, {'damage_type': 'backtrace', 'id': 1900119677, 'ino': 1099511628930, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-17.xml'}, {'damage_type': 'backtrace', 'id': 1941794947, 'ino': 1099511631243, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-22.xml'}, {'damage_type': 'backtrace', 'id': 1988643797, 'ino': 1099511628793, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-67.xml'}, {'damage_type': 'backtrace', 'id': 1989727948, 'ino': 1099511631080, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-43.xml'}, {'damage_type': 'backtrace', 'id': 1990297371, 'ino': 1099511628758, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-79.xml'}, {'damage_type': 'backtrace', 'id': 1992618151, 'ino': 1099511629076, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-42.jpg'}, {'damage_type': 'backtrace', 'id': 2007886677, 'ino': 1099511630500, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-1.xml'}, {'damage_type': 'backtrace', 'id': 2014558499, 'ino': 1099511630334, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-42.xml'}, {'damage_type': 'backtrace', 'id': 2031274715, 'ino': 1099511630388, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-29.xml'}, {'damage_type': 'backtrace', 'id': 2050663328, 'ino': 1099511631288, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-16.xml'}, {'damage_type': 'backtrace', 'id': 2051501809, 'ino': 1099511630285, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-53.xml'}, {'damage_type': 'backtrace', 'id': 2064675800, 'ino': 1099511630727, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-1.jpg'}, {'damage_type': 'backtrace', 'id': 2069645392, 'ino': 1099511630552, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-30.jpg'}, {'damage_type': 'backtrace', 'id': 2070456912, 'ino': 1099511630408, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-24.xml'}, {'damage_type': 'backtrace', 'id': 2102249167, 'ino': 1099511631316, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-78.xml'}, {'damage_type': 'backtrace', 'id': 2118690436, 'ino': 1099511630338, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-41.xml'}, {'damage_type': 'backtrace', 'id': 2155207996, 'ino': 1099511628815, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-60.xml'}, {'damage_type': 'backtrace', 'id': 2168450852, 'ino': 1099511630614, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-22.jpg'}, {'damage_type': 'backtrace', 'id': 2210561452, 'ino': 1099511630432, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-18.xml'}, {'damage_type': 'backtrace', 'id': 2244939576, 'ino': 1099511629009, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-63.jpg'}, {'damage_type': 'backtrace', 'id': 2282399955, 'ino': 1099511630889, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-11.jpg'}, {'damage_type': 'backtrace', 'id': 2296975012, 'ino': 1099511630973, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-3.jpg'}, {'damage_type': 'backtrace', 'id': 2299385132, 'ino': 1099511630326, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-44.xml'}, {'damage_type': 'backtrace', 'id': 2302743836, 'ino': 1099511628764, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-77.xml'}, {'damage_type': 'backtrace', 'id': 2322275793, 'ino': 1099511629028, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-55.jpg'}, {'damage_type': 'backtrace', 'id': 2329182094, 'ino': 1099511631021, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-49.xml'}, {'damage_type': 'backtrace', 'id': 2329739953, 'ino': 1099511629805, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-33.jpg'}, {'damage_type': 'backtrace', 'id': 2348422284, 'ino': 1099511629041, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-50.jpg'}, {'damage_type': 'backtrace', 'id': 2350263242, 'ino': 1099511630396, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-27.xml'}, {'damage_type': 'backtrace', 'id': 2356922213, 'ino': 1099511630681, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-14.jpg'}, {'damage_type': 'backtrace', 'id': 2372345582, 'ino': 1099511629925, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-81.xml'}, {'damage_type': 'backtrace', 'id': 2413893058, 'ino': 1099511628821, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-58.xml'}, {'damage_type': 'backtrace', 'id': 2419265228, 'ino': 1099511630222, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-76.xml'}, {'damage_type': 'backtrace', 'id': 2487716761, 'ino': 1099511630538, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-32.jpg'}, {'damage_type': 'backtrace', 'id': 2500481837, 'ino': 1099511629039, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-51.jpg'}, {'damage_type': 'backtrace', 'id': 2501724176, 'ino': 1099511631221, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-26.xml'}, {'damage_type': 'backtrace', 'id': 2555596776, 'ino': 1099511630957, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-5.jpg'}, {'damage_type': 'backtrace', 'id': 2557128515, 'ino': 1099511631163, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-34.xml'}, {'damage_type': 'backtrace', 'id': 2568319086, 'ino': 1099511630980, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-53.xml'}, {'damage_type': 'backtrace', 'id': 2615825468, 'ino': 1099511630825, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-68.xml'}, {'damage_type': 'backtrace', 'id': 2660648004, 'ino': 1099511630507, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-36.jpg'}, {'damage_type': 'backtrace', 'id': 2687710226, 'ino': 1099511630372, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-33.xml'}, {'damage_type': 'backtrace', 'id': 2692468128, 'ino': 1099511630416, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-22.xml'}, {'damage_type': 'backtrace', 'id': 2692713310, 'ino': 1099511630787, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-74.jpg'}, {'damage_type': 'backtrace', 'id': 2707818954, 'ino': 1099511629055, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-99.xml'}, {'damage_type': 'backtrace', 'id': 2726609479, 'ino': 1099511628993, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-69.jpg'}, {'damage_type': 'backtrace', 'id': 2734770244, 'ino': 1099511630669, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-17.jpg'}, {'damage_type': 'backtrace', 'id': 2739722541, 'ino': 1099511628876, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-37.xml'}, {'damage_type': 'backtrace', 'id': 2756557460, 'ino': 1099511630496, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-2.xml'}, {'damage_type': 'backtrace', 'id': 2757766092, 'ino': 1099511631320, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-15/article-77.xml'}, {'damage_type': 'backtrace', 'id': 2801900833, 'ino': 1099511628986, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-71.jpg'}, {'damage_type': 'backtrace', 'id': 2803194865, 'ino': 1099511631273, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-18.xml'}, {'damage_type': 'backtrace', 'id': 2810816703, 'ino': 1099511630330, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-43.xml'}, {'damage_type': 'backtrace', 'id': 2840277356, 'ino': 1099511630484, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-5.xml'}, {'damage_type': 'backtrace', 'id': 2855163432, 'ino': 1099511631058, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-45.xml'}, {'damage_type': 'backtrace', 'id': 2876165198, 'ino': 1099511630420, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-21.xml'}, {'damage_type': 'backtrace', 'id': 2877999814, 'ino': 1099511630624, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-21.jpg'}, {'damage_type': 'backtrace', 'id': 2881549383, 'ino': 1099511630273, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-55.xml'}, {'damage_type': 'backtrace', 'id': 2915325761, 'ino': 1099511631234, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-24.xml'}, {'damage_type': 'backtrace', 'id': 2948493922, 'ino': 1099511631131, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-38.xml'}, {'damage_type': 'backtrace', 'id': 2955860245, 'ino': 1099511630946, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-6.jpg'}, {'damage_type': 'backtrace', 'id': 3021959999, 'ino': 1099511630360, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-36.xml'}, {'damage_type': 'backtrace', 'id': 3035755689, 'ino': 1099511630384, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-30.xml'}, {'damage_type': 'backtrace', 'id': 3045343735, 'ino': 1099511628836, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-53.xml'}, {'damage_type': 'backtrace', 'id': 3045686266, 'ino': 1099511630468, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-9.xml'}, {'damage_type': 'backtrace', 'id': 3072234755, 'ino': 1099511630400, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-26.xml'}, {'damage_type': 'backtrace', 'id': 3116956103, 'ino': 1099511628871, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-39.xml'}, {'damage_type': 'backtrace', 'id': 3139966477, 'ino': 1099511630705, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-8.jpg'}, {'damage_type': 'backtrace', 'id': 3148747583, 'ino': 1099511630521, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-34.jpg'}, {'damage_type': 'backtrace', 'id': 3186364883, 'ino': 1099511628700, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-98.xml'}, {'damage_type': 'backtrace', 'id': 3202041410, 'ino': 1099511631210, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-28.xml'}, {'damage_type': 'backtrace', 'id': 3202640401, 'ino': 1099511630581, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-26.jpg'}, {'damage_type': 'backtrace', 'id': 3204814963, 'ino': 1099511628848, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-49.xml'}, {'damage_type': 'backtrace', 'id': 3228849983, 'ino': 1099511628966, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-5.xml'}, {'damage_type': 'backtrace', 'id': 3322986617, 'ino': 1099511630819, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-17.jpg'}, {'damage_type': 'backtrace', 'id': 3327231351, 'ino': 1099511629007, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-64.jpg'}, {'damage_type': 'backtrace', 'id': 3342957466, 'ino': 1099511629034, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-53.jpg'}, {'damage_type': 'backtrace', 'id': 3380534740, 'ino': 1099511631008, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-8.xml'}, {'damage_type': 'backtrace', 'id': 3396907838, 'ino': 1099511630685, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-13.jpg'}, {'damage_type': 'backtrace', 'id': 3419519430, 'ino': 1099511630677, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-15.jpg'}, {'damage_type': 'backtrace', 'id': 3419566771, 'ino': 1099511628734, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-87.xml'}, {'damage_type': 'backtrace', 'id': 3431369979, 'ino': 1099511630464, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-10.xml'}, {'damage_type': 'backtrace', 'id': 3480798335, 'ino': 1099511629046, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-48.jpg'}, {'damage_type': 'backtrace', 'id': 3499237960, 'ino': 1099511629023, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-57.jpg'}, {'damage_type': 'backtrace', 'id': 3533941930, 'ino': 1099511630701, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-9.jpg'}, {'damage_type': 'backtrace', 'id': 3547864074, 'ino': 1099511630299, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-50.xml'}, {'damage_type': 'backtrace', 'id': 3560709864, 'ino': 1099511630448, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-14.xml'}, {'damage_type': 'backtrace', 'id': 3580088222, 'ino': 1099511629011, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-62.jpg'}, {'damage_type': 'backtrace', 'id': 3603286253, 'ino': 1099511630641, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-19.jpg'}, {'damage_type': 'backtrace', 'id': 3648145472, 'ino': 1099511628731, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-88.xml'}, {'damage_type': 'backtrace', 'id': 3687584450, 'ino': 1099511630347, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-39.xml'}, {'damage_type': 'backtrace', 'id': 3709348609, 'ino': 1099511631278, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-17.xml'}, {'damage_type': 'backtrace', 'id': 3759584517, 'ino': 1099511630597, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-24.jpg'}, {'damage_type': 'backtrace', 'id': 3766484848, 'ino': 1099511630352, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-38.xml'}, {'damage_type': 'backtrace', 'id': 3774511662, 'ino': 1099511630725, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-2.jpg'}, {'damage_type': 'backtrace', 'id': 3801503953, 'ino': 1099511628707, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-96.xml'}, {'damage_type': 'backtrace', 'id': 3822534895, 'ino': 1099511628777, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-73.xml'}, {'damage_type': 'backtrace', 'id': 3826874770, 'ino': 1099511630312, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-47.xml'}, {'damage_type': 'backtrace', 'id': 3833716520, 'ino': 1099511628900, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-28.xml'}, {'damage_type': 'backtrace', 'id': 3846226173, 'ino': 1099511628878, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-36.xml'}, {'damage_type': 'backtrace', 'id': 3849377558, 'ino': 1099511630356, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-37.xml'}, {'damage_type': 'backtrace', 'id': 3864999084, 'ino': 1099511630648, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-18.jpg'}, {'damage_type': 'backtrace', 'id': 3866948246, 'ino': 1099511629084, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-95.xml'}, {'damage_type': 'backtrace', 'id': 3887669704, 'ino': 1099511630436, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-17.xml'}, {'damage_type': 'backtrace', 'id': 3944128765, 'ino': 1099511630928, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-58.xml'}, {'damage_type': 'backtrace', 'id': 3946554253, 'ino': 1099511628741, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-85.xml'}, {'damage_type': 'backtrace', 'id': 3957937567, 'ino': 1099511630321, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-45.xml'}, {'damage_type': 'backtrace', 'id': 3997532897, 'ino': 1099511630859, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-65.xml'}, {'damage_type': 'backtrace', 'id': 4031472965, 'ino': 1099511630940, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-57.xml'}, {'damage_type': 'backtrace', 'id': 4032227834, 'ino': 1099511629005, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/picture-65.jpg'}, {'damage_type': 'backtrace', 'id': 4070785057, 'ino': 1099511629796, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-84.xml'}, {'damage_type': 'backtrace', 'id': 4074450533, 'ino': 1099511628824, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-57.xml'}, {'damage_type': 'backtrace', 'id': 4094141632, 'ino': 1099511630236, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-75.xml'}, {'damage_type': 'backtrace', 'id': 4119734159, 'ino': 1099511628771, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-75.xml'}, {'damage_type': 'backtrace', 'id': 4150390513, 'ino': 1099511631017, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-85.xml'}, {'damage_type': 'backtrace', 'id': 4189081402, 'ino': 1099511628860, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-44.xml'}, {'damage_type': 'backtrace', 'id': 4207167395, 'ino': 1099511630604, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-23.jpg'}, {'damage_type': 'backtrace', 'id': 4231269062, 'ino': 1099511630574, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/picture-27.jpg'}, {'damage_type': 'backtrace', 'id': 4233346805, 'ino': 1099511628978, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-1.xml'}, {'damage_type': 'backtrace', 'id': 4260774831, 'ino': 1099511630267, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-56.xml'}, {'damage_type': 'backtrace', 'id': 4262845189, 'ino': 1099511628806, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/article-63.xml'}, {'damage_type': 'backtrace', 'id': 4273427971, 'ino': 1099511631239, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-23.xml'}, {'damage_type': 'backtrace', 'id': 4280342626, 'ino': 1099511630317, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-46.xml'}, {'damage_type': 'backtrace', 'id': 4280550513, 'ino': 1099511631141, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-0/comment-37.xml'}, {'damage_type': 'backtrace', 'id': 4285404534, 'ino': 1099511630793, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-10/article-99.xml'}] 2024-05-09T08:28:36.401 ERROR:tasks.fwd_scrub:rank 1 damaged: [{'damage_type': 'backtrace', 'id': 8943116, 'ino': 1099511631153, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-25.jpg'}, {'damage_type': 'backtrace', 'id': 11778878, 'ino': 1099511630776, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-26.xml'}, {'damage_type': 'backtrace', 'id': 13704377, 'ino': 1099511630886, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-56.xml'}, {'damage_type': 'backtrace', 'id': 43245315, 'ino': 1099511630908, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-49.xml'}, {'damage_type': 'backtrace', 'id': 54628180, 'ino': 1099511630798, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-20.xml'}, {'damage_type': 'backtrace', 'id': 96269479, 'ino': 1099511630934, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-40.xml'}, {'damage_type': 'backtrace', 'id': 114007711, 'ino': 1099511630789, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-87.xml'}, {'damage_type': 'backtrace', 'id': 115446025, 'ino': 1099511630930, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-60.jpg'}, {'damage_type': 'backtrace', 'id': 120725893, 'ino': 1099511631150, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-34.jpg'}, {'damage_type': 'backtrace', 'id': 133881957, 'ino': 1099511631276, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-7.jpg'}, {'damage_type': 'backtrace', 'id': 139592900, 'ino': 1099511630991, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-22.xml'}, {'damage_type': 'backtrace', 'id': 206732543, 'ino': 1099511628295, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.in'}, {'damage_type': 'backtrace', 'id': 248801395, 'ino': 1099511630760, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-94.xml'}, {'damage_type': 'backtrace', 'id': 256571965, 'ino': 1099511630975, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-45.jpg'}, {'damage_type': 'backtrace', 'id': 259557527, 'ino': 1099511630845, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-69.xml'}, {'damage_type': 'backtrace', 'id': 274201124, 'ino': 1099511631165, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-13.xml'}, {'damage_type': 'backtrace', 'id': 279001455, 'ino': 1099511631143, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-27.jpg'}, {'damage_type': 'backtrace', 'id': 281589376, 'ino': 1099511630968, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-29.xml'}, {'damage_type': 'backtrace', 'id': 292509880, 'ino': 1099511630809, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-80.xml'}, {'damage_type': 'backtrace', 'id': 307594014, 'ino': 1099511631212, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-31.jpg'}, {'damage_type': 'backtrace', 'id': 360564831, 'ino': 1099511630891, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-73.jpg'}, {'damage_type': 'backtrace', 'id': 374497860, 'ino': 1099511630923, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-62.jpg'}, {'damage_type': 'backtrace', 'id': 393094977, 'ino': 1099511630984, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-24.xml'}, {'damage_type': 'backtrace', 'id': 395119492, 'ino': 1099511630909, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-67.jpg'}, {'damage_type': 'backtrace', 'id': 434266254, 'ino': 1099511630917, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-46.xml'}, {'damage_type': 'backtrace', 'id': 450276542, 'ino': 1099511630854, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-66.xml'}, {'damage_type': 'backtrace', 'id': 478119976, 'ino': 1099511630805, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-81.xml'}, {'damage_type': 'backtrace', 'id': 488383482, 'ino': 1099511630818, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-14.xml'}, {'damage_type': 'backtrace', 'id': 499154671, 'ino': 1099511631041, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-11.xml'}, {'damage_type': 'backtrace', 'id': 499939241, 'ino': 1099511630890, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-55.xml'}, {'damage_type': 'backtrace', 'id': 508040015, 'ino': 1099511630875, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-59.xml'}, {'damage_type': 'backtrace', 'id': 528367303, 'ino': 1099511630838, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-71.xml'}, {'damage_type': 'backtrace', 'id': 541128304, 'ino': 1099511630905, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-50.xml'}, {'damage_type': 'backtrace', 'id': 560218702, 'ino': 1099511631287, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-5.jpg'}, {'damage_type': 'backtrace', 'id': 574973953, 'ino': 1099511630860, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-64.xml'}, {'damage_type': 'backtrace', 'id': 581809392, 'ino': 1099511630813, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-79.xml'}, {'damage_type': 'backtrace', 'id': 594796927, 'ino': 1099511630869, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-61.xml'}, {'damage_type': 'backtrace', 'id': 610241051, 'ino': 1099511630959, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-32.xml'}, {'damage_type': 'backtrace', 'id': 621310278, 'ino': 1099511630878, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-58.xml'}, {'damage_type': 'backtrace', 'id': 627014123, 'ino': 1099511630827, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-11.xml'}, {'damage_type': 'backtrace', 'id': 637822482, 'ino': 1099511630757, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-95.xml'}, {'damage_type': 'backtrace', 'id': 651697087, 'ino': 1099511630799, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-83.xml'}, {'damage_type': 'backtrace', 'id': 680262708, 'ino': 1099511631025, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-13.xml'}, {'damage_type': 'backtrace', 'id': 686241527, 'ino': 1099511631286, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-27.jpg'}, {'damage_type': 'backtrace', 'id': 714518857, 'ino': 1099511630933, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-59.jpg'}, {'damage_type': 'backtrace', 'id': 733066404, 'ino': 1099511631082, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-2.xml'}, {'damage_type': 'backtrace', 'id': 762283623, 'ino': 1099511630835, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-72.xml'}, {'damage_type': 'backtrace', 'id': 776495413, 'ino': 1099511631135, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-29.jpg'}, {'damage_type': 'backtrace', 'id': 777171548, 'ino': 1099511631145, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-14.xml'}, {'damage_type': 'backtrace', 'id': 806500509, 'ino': 1099511630972, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-46.jpg'}, {'damage_type': 'backtrace', 'id': 813287422, 'ino': 1099511631178, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-19.jpg'}, {'damage_type': 'backtrace', 'id': 823539219, 'ino': 1099511631108, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-36.jpg'}, {'damage_type': 'backtrace', 'id': 827485018, 'ino': 1099511630920, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-45.xml'}, {'damage_type': 'backtrace', 'id': 851899159, 'ino': 1099511630870, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-79.jpg'}, {'damage_type': 'backtrace', 'id': 876615957, 'ino': 1099511630800, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-19.xml'}, {'damage_type': 'backtrace', 'id': 907071463, 'ino': 1099511630927, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-61.jpg'}, {'damage_type': 'backtrace', 'id': 922095159, 'ino': 1099511630836, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-8.xml'}, {'damage_type': 'backtrace', 'id': 930764074, 'ino': 1099511630756, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-30.xml'}, {'damage_type': 'backtrace', 'id': 961961810, 'ino': 1099511631000, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-19.xml'}, {'damage_type': 'backtrace', 'id': 993534053, 'ino': 1099511631092, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-38.jpg'}, {'damage_type': 'backtrace', 'id': 1008851181, 'ino': 1099511631038, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-37.jpg'}, {'damage_type': 'backtrace', 'id': 1021692193, 'ino': 1099511630775, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-90.xml'}, {'damage_type': 'backtrace', 'id': 1024185954, 'ino': 1099511630855, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-2.xml'}, {'damage_type': 'backtrace', 'id': 1035302394, 'ino': 1099511631016, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-39.jpg'}, {'damage_type': 'backtrace', 'id': 1038461858, 'ino': 1099511631214, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-12.jpg'}, {'damage_type': 'backtrace', 'id': 1063708593, 'ino': 1099511630770, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-91.xml'}, {'damage_type': 'backtrace', 'id': 1098322284, 'ino': 1099511630918, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-64.jpg'}, {'damage_type': 'backtrace', 'id': 1099784531, 'ino': 1099511630921, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-63.jpg'}, {'damage_type': 'backtrace', 'id': 1114482346, 'ino': 1099511630820, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-77.xml'}, {'damage_type': 'backtrace', 'id': 1117701220, 'ino': 1099511630779, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-89.xml'}, {'damage_type': 'backtrace', 'id': 1124093012, 'ino': 1099511631197, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-15.jpg'}, {'damage_type': 'backtrace', 'id': 1130191735, 'ino': 1099511630817, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-78.xml'}, {'damage_type': 'backtrace', 'id': 1152525556, 'ino': 1099511630857, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-65.xml'}, {'damage_type': 'backtrace', 'id': 1173064378, 'ino': 1099511630866, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-62.xml'}, {'damage_type': 'backtrace', 'id': 1212150833, 'ino': 1099511630944, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-37.xml'}, {'damage_type': 'backtrace', 'id': 1224061100, 'ino': 1099511630926, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-43.xml'}, {'damage_type': 'backtrace', 'id': 1263478421, 'ino': 1099511631074, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-100.jpg'}, {'damage_type': 'backtrace', 'id': 1272807307, 'ino': 1099511630897, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-71.jpg'}, {'damage_type': 'backtrace', 'id': 1278666373, 'ino': 1099511630832, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-73.xml'}, {'damage_type': 'backtrace', 'id': 1291141109, 'ino': 1099511630954, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-52.jpg'}, {'damage_type': 'backtrace', 'id': 1307823063, 'ino': 1099511630947, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-36.xml'}, {'damage_type': 'backtrace', 'id': 1319017800, 'ino': 1099511630953, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-34.xml'}, {'damage_type': 'backtrace', 'id': 1365984352, 'ino': 1099511630785, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-24.xml'}, {'damage_type': 'backtrace', 'id': 1368883188, 'ino': 1099511631301, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-6.xml'}, {'damage_type': 'backtrace', 'id': 1405413318, 'ino': 1099511630986, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-43.jpg'}, {'damage_type': 'backtrace', 'id': 1435607274, 'ino': 1099511630974, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-27.xml'}, {'damage_type': 'backtrace', 'id': 1482260481, 'ino': 1099511631228, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-10.xml'}, {'damage_type': 'backtrace', 'id': 1482764421, 'ino': 1099511630810, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-16.xml'}, {'damage_type': 'backtrace', 'id': 1483538437, 'ino': 1099511630882, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-57.xml'}, {'damage_type': 'backtrace', 'id': 1500811244, 'ino': 1099511631230, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-9.jpg'}, {'damage_type': 'backtrace', 'id': 1510015900, 'ino': 1099511631302, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-2.jpg'}, {'damage_type': 'backtrace', 'id': 1517578577, 'ino': 1099511630830, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-10.xml'}, {'damage_type': 'backtrace', 'id': 1522174032, 'ino': 1099511631138, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-28.jpg'}, {'damage_type': 'backtrace', 'id': 1526184443, 'ino': 1099511630981, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-25.xml'}, {'damage_type': 'backtrace', 'id': 1561930030, 'ino': 1099511630761, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-29.xml'}, {'damage_type': 'backtrace', 'id': 1570753925, 'ino': 1099511630823, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-76.xml'}, {'damage_type': 'backtrace', 'id': 1602185210, 'ino': 1099511630861, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-82.jpg'}, {'damage_type': 'backtrace', 'id': 1607214392, 'ino': 1099511631183, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-18.jpg'}, {'damage_type': 'backtrace', 'id': 1625131374, 'ino': 1099511631049, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-9.xml'}, {'damage_type': 'backtrace', 'id': 1640154226, 'ino': 1099511631170, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-21.jpg'}, {'damage_type': 'backtrace', 'id': 1659625515, 'ino': 1099511630900, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-70.jpg'}, {'damage_type': 'backtrace', 'id': 1660750539, 'ino': 1099511631006, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-40.jpg'}, {'damage_type': 'backtrace', 'id': 1666425410, 'ino': 1099511630902, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-51.xml'}, {'damage_type': 'backtrace', 'id': 1682661947, 'ino': 1099511630806, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-17.xml'}, {'damage_type': 'backtrace', 'id': 1787976441, 'ino': 1099511630879, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-76.jpg'}, {'damage_type': 'backtrace', 'id': 1816087138, 'ino': 1099511631078, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-3.xml'}, {'damage_type': 'backtrace', 'id': 1826718532, 'ino': 1099511630771, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-27.xml'}, {'damage_type': 'backtrace', 'id': 1881497274, 'ino': 1099511631190, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-32.jpg'}, {'damage_type': 'backtrace', 'id': 1890531637, 'ino': 1099511631087, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-34.jpg'}, {'damage_type': 'backtrace', 'id': 1892767691, 'ino': 1099511630863, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-63.xml'}, {'damage_type': 'backtrace', 'id': 1902454523, 'ino': 1099511631027, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-38.jpg'}, {'damage_type': 'backtrace', 'id': 1931537498, 'ino': 1099511631055, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-8.xml'}, {'damage_type': 'backtrace', 'id': 1948162057, 'ino': 1099511630996, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-41.jpg'}, {'damage_type': 'backtrace', 'id': 1953596509, 'ino': 1099511631224, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-10.jpg'}, {'damage_type': 'backtrace', 'id': 1956159489, 'ino': 1099511631106, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-33.jpg'}, {'damage_type': 'backtrace', 'id': 1991721603, 'ino': 1099511630963, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-49.jpg'}, {'damage_type': 'backtrace', 'id': 2002177737, 'ino': 1099511631161, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-23.jpg'}, {'damage_type': 'backtrace', 'id': 2003025707, 'ino': 1099511631130, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-30.jpg'}, {'damage_type': 'backtrace', 'id': 2011874752, 'ino': 1099511631157, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-24.jpg'}, {'damage_type': 'backtrace', 'id': 2040168068, 'ino': 1099511628291, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/m4/acx_pthread.m4'}, {'damage_type': 'backtrace', 'id': 2041548884, 'ino': 1099511630795, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-21.xml'}, {'damage_type': 'backtrace', 'id': 2042030591, 'ino': 1099511631185, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-12.xml'}, {'damage_type': 'backtrace', 'id': 2094129653, 'ino': 1099511631296, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-3.jpg'}, {'damage_type': 'backtrace', 'id': 2120436309, 'ino': 1099511630964, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-30.xml'}, {'damage_type': 'backtrace', 'id': 2132148010, 'ino': 1099511630766, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-92.xml'}, {'damage_type': 'backtrace', 'id': 2155860064, 'ino': 1099511630979, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-44.jpg'}, {'damage_type': 'backtrace', 'id': 2204988411, 'ino': 1099511630846, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-5.xml'}, {'damage_type': 'backtrace', 'id': 2205908140, 'ino': 1099511630851, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-67.xml'}, {'damage_type': 'backtrace', 'id': 2226275066, 'ino': 1099511631282, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-6.jpg'}, {'damage_type': 'backtrace', 'id': 2226798065, 'ino': 1099511630935, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-58.jpg'}, {'damage_type': 'backtrace', 'id': 2238684077, 'ino': 1099511631015, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-15.xml'}, {'damage_type': 'backtrace', 'id': 2246050672, 'ino': 1099511631188, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-17.jpg'}, {'damage_type': 'backtrace', 'id': 2281203079, 'ino': 1099511630950, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-35.xml'}, {'damage_type': 'backtrace', 'id': 2295689093, 'ino': 1099511630995, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-20.xml'}, {'damage_type': 'backtrace', 'id': 2295774629, 'ino': 1099511630960, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-50.jpg'}, {'damage_type': 'backtrace', 'id': 2335786967, 'ino': 1099511630998, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-13.xml'}, {'damage_type': 'backtrace', 'id': 2357126611, 'ino': 1099511630906, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-68.jpg'}, {'damage_type': 'backtrace', 'id': 2360137224, 'ino': 1099511631202, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-14.jpg'}, {'damage_type': 'backtrace', 'id': 2373754058, 'ino': 1099511631219, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-11.jpg'}, {'damage_type': 'backtrace', 'id': 2407712590, 'ino': 1099511631032, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-12.xml'}, {'damage_type': 'backtrace', 'id': 2424569036, 'ino': 1099511630962, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-31.xml'}, {'damage_type': 'backtrace', 'id': 2457416004, 'ino': 1099511630763, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-93.xml'}, {'damage_type': 'backtrace', 'id': 2460269510, 'ino': 1099511630914, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-47.xml'}, {'damage_type': 'backtrace', 'id': 2479931741, 'ino': 1099511630826, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-75.xml'}, {'damage_type': 'backtrace', 'id': 2480003381, 'ino': 1099511630887, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-74.jpg'}, {'damage_type': 'backtrace', 'id': 2494236234, 'ino': 1099511631192, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-16.jpg'}, {'damage_type': 'backtrace', 'id': 2508253155, 'ino': 1099511630948, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-54.jpg'}, {'damage_type': 'backtrace', 'id': 2521477954, 'ino': 1099511630951, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-53.jpg'}, {'damage_type': 'backtrace', 'id': 2532918798, 'ino': 1099511630911, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-48.xml'}, {'damage_type': 'backtrace', 'id': 2577315679, 'ino': 1099511630873, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-78.jpg'}, {'damage_type': 'backtrace', 'id': 2585272124, 'ino': 1099511630796, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-84.xml'}, {'damage_type': 'backtrace', 'id': 2598278841, 'ino': 1099511630994, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-21.xml'}, {'damage_type': 'backtrace', 'id': 2607458058, 'ino': 1099511631019, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-14.xml'}, {'damage_type': 'backtrace', 'id': 2620489274, 'ino': 1099511631053, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-36.jpg'}, {'damage_type': 'backtrace', 'id': 2628991500, 'ino': 1099511630814, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-15.xml'}, {'damage_type': 'backtrace', 'id': 2631640260, 'ino': 1099511631064, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-6.xml'}, {'damage_type': 'backtrace', 'id': 2648372595, 'ino': 1099511630883, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-75.jpg'}, {'damage_type': 'backtrace', 'id': 2669315771, 'ino': 1099511630833, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-9.xml'}, {'damage_type': 'backtrace', 'id': 2671019580, 'ino': 1099511631169, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-33.jpg'}, {'damage_type': 'backtrace', 'id': 2702333695, 'ino': 1099511630839, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-7.xml'}, {'damage_type': 'backtrace', 'id': 2742068038, 'ino': 1099511630767, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-28.xml'}, {'damage_type': 'backtrace', 'id': 2796470149, 'ino': 1099511631281, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-7.xml'}, {'damage_type': 'backtrace', 'id': 2819228142, 'ino': 1099511630894, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-72.jpg'}, {'damage_type': 'backtrace', 'id': 2849885880, 'ino': 1099511630804, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-18.xml'}, {'damage_type': 'backtrace', 'id': 2852127139, 'ino': 1099511630989, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-42.jpg'}, {'damage_type': 'backtrace', 'id': 2876148261, 'ino': 1099511631099, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-37.jpg'}, {'damage_type': 'backtrace', 'id': 2935180819, 'ino': 1099511631166, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-22.jpg'}, {'damage_type': 'backtrace', 'id': 2952562154, 'ino': 1099511630864, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-81.jpg'}, {'damage_type': 'backtrace', 'id': 2961757688, 'ino': 1099511630847, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-68.xml'}, {'damage_type': 'backtrace', 'id': 2987041157, 'ino': 1099511631292, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-4.jpg'}, {'damage_type': 'backtrace', 'id': 3022068579, 'ino': 1099511630942, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-56.jpg'}, {'damage_type': 'backtrace', 'id': 3030019791, 'ino': 1099511630824, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-12.xml'}, {'damage_type': 'backtrace', 'id': 3033361981, 'ino': 1099511630988, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-23.xml'}, {'damage_type': 'backtrace', 'id': 3039239612, 'ino': 1099511630929, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-42.xml'}, {'damage_type': 'backtrace', 'id': 3046002880, 'ino': 1099511631122, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-35.jpg'}, {'damage_type': 'backtrace', 'id': 3061088480, 'ino': 1099511631059, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-7.xml'}, {'damage_type': 'backtrace', 'id': 3068206700, 'ino': 1099511630971, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-28.xml'}, {'damage_type': 'backtrace', 'id': 3076846201, 'ino': 1099511631125, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-32.jpg'}, {'damage_type': 'backtrace', 'id': 3077964145, 'ino': 1099511630858, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-1.xml'}, {'damage_type': 'backtrace', 'id': 3088141567, 'ino': 1099511631261, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-8.xml'}, {'damage_type': 'backtrace', 'id': 3126763208, 'ino': 1099511630955, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-33.xml'}, {'damage_type': 'backtrace', 'id': 3133016251, 'ino': 1099511631206, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-11.xml'}, {'damage_type': 'backtrace', 'id': 3145833287, 'ino': 1099511631231, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-30.jpg'}, {'damage_type': 'backtrace', 'id': 3184982412, 'ino': 1099511630941, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-38.xml'}, {'damage_type': 'backtrace', 'id': 3210192486, 'ino': 1099511630932, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-41.xml'}, {'damage_type': 'backtrace', 'id': 3213081426, 'ino': 1099511631011, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-16.xml'}, {'damage_type': 'backtrace', 'id': 3217452053, 'ino': 1099511631073, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-4.xml'}, {'damage_type': 'backtrace', 'id': 3260192408, 'ino': 1099511631268, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-28.jpg'}, {'damage_type': 'backtrace', 'id': 3291101529, 'ino': 1099511630794, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-85.xml'}, {'damage_type': 'backtrace', 'id': 3357071724, 'ino': 1099511631249, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-29.jpg'}, {'damage_type': 'backtrace', 'id': 3371257531, 'ino': 1099511630939, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-57.jpg'}, {'damage_type': 'backtrace', 'id': 3389739121, 'ino': 1099511628293, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/blogbench.8'}, {'damage_type': 'backtrace', 'id': 3393273485, 'ino': 1099511630791, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-86.xml'}, {'damage_type': 'backtrace', 'id': 3408893139, 'ino': 1099511630790, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-23.xml'}, {'damage_type': 'backtrace', 'id': 3414767969, 'ino': 1099511631236, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-8.jpg'}, {'damage_type': 'backtrace', 'id': 3426821685, 'ino': 1099511630841, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-70.xml'}, {'damage_type': 'backtrace', 'id': 3429976806, 'ino': 1099511630969, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-47.jpg'}, {'damage_type': 'backtrace', 'id': 3453725544, 'ino': 1099511630956, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-51.jpg'}, {'damage_type': 'backtrace', 'id': 3478099366, 'ino': 1099511631002, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-18.xml'}, {'damage_type': 'backtrace', 'id': 3482737730, 'ino': 1099511628294, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/man/Makefile.am'}, {'damage_type': 'backtrace', 'id': 3495717671, 'ino': 1099511630780, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-25.xml'}, {'damage_type': 'backtrace', 'id': 3513547080, 'ino': 1099511631069, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-35.jpg'}, {'damage_type': 'backtrace', 'id': 3523400585, 'ino': 1099511631245, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-9.xml'}, {'damage_type': 'backtrace', 'id': 3596016628, 'ino': 1099511631208, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg'}, {'damage_type': 'backtrace', 'id': 3599479262, 'ino': 1099511631119, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/comment-15.xml'}, {'damage_type': 'backtrace', 'id': 3627462661, 'ino': 1099511630792, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-22.xml'}, {'damage_type': 'backtrace', 'id': 3655450662, 'ino': 1099511630802, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-82.xml'}, {'damage_type': 'backtrace', 'id': 3669233711, 'ino': 1099511630842, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-6.xml'}, {'damage_type': 'backtrace', 'id': 3683853032, 'ino': 1099511630852, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-3.xml'}, {'damage_type': 'backtrace', 'id': 3805714737, 'ino': 1099511630945, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-55.jpg'}, {'damage_type': 'backtrace', 'id': 3822465223, 'ino': 1099511631127, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-31.jpg'}, {'damage_type': 'backtrace', 'id': 3828750481, 'ino': 1099511631175, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-20.jpg'}, {'damage_type': 'backtrace', 'id': 3873886506, 'ino': 1099511631147, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-26.jpg'}, {'damage_type': 'backtrace', 'id': 3936304929, 'ino': 1099511630893, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-54.xml'}, {'damage_type': 'backtrace', 'id': 3943058417, 'ino': 1099511631086, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-1.xml'}, {'damage_type': 'backtrace', 'id': 3970664374, 'ino': 1099511631005, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-17.xml'}, {'damage_type': 'backtrace', 'id': 3972004974, 'ino': 1099511631308, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-1.jpg'}, {'damage_type': 'backtrace', 'id': 3973432103, 'ino': 1099511630896, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-53.xml'}, {'damage_type': 'backtrace', 'id': 4008393929, 'ino': 1099511630872, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-60.xml'}, {'damage_type': 'backtrace', 'id': 4009962745, 'ino': 1099511630915, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-65.jpg'}, {'damage_type': 'backtrace', 'id': 4055701442, 'ino': 1099511630848, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-4.xml'}, {'damage_type': 'backtrace', 'id': 4067492818, 'ino': 1099511630977, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-26.xml'}, {'damage_type': 'backtrace', 'id': 4068754583, 'ino': 1099511630903, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-69.jpg'}, {'damage_type': 'backtrace', 'id': 4087444034, 'ino': 1099511631307, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/picture-26.jpg'}, {'damage_type': 'backtrace', 'id': 4096161404, 'ino': 1099511631068, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-5.xml'}, {'damage_type': 'backtrace', 'id': 4101604561, 'ino': 1099511630912, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-66.jpg'}, {'damage_type': 'backtrace', 'id': 4110528567, 'ino': 1099511630965, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-48.jpg'}, {'damage_type': 'backtrace', 'id': 4118383068, 'ino': 1099511630867, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-80.jpg'}, {'damage_type': 'backtrace', 'id': 4122453697, 'ino': 1099511630876, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-77.jpg'}, {'damage_type': 'backtrace', 'id': 4134376484, 'ino': 1099511630938, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-39.xml'}, {'damage_type': 'backtrace', 'id': 4158186746, 'ino': 1099511630922, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-44.xml'}, {'damage_type': 'backtrace', 'id': 4171117759, 'ino': 1099511630783, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-88.xml'}, {'damage_type': 'backtrace', 'id': 4201151415, 'ino': 1099511630899, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-52.xml'}, {'damage_type': 'backtrace', 'id': 4243568538, 'ino': 1099511630829, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-74.xml'}, {'damage_type': 'backtrace', 'id': 4245261084, 'ino': 1099511631045, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-13/article-10.xml'}, {'damage_type': 'backtrace', 'id': 4287446791, 'ino': 1099511631089, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-97.xml'}, {'damage_type': 'backtrace', 'id': 4293505871, 'ino': 1099511631050, 'path': '/volumes/qa/sv_1/78711e5e-0ac4-4ad0-adea-054a648e2e00/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-63.xml'}] 2024-05-09T08:28:36.401 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-05-09T08:28:37.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:37 smithi028 ceph-mon[30119]: pgmap v72: 129 pgs: 129 active+clean; 209 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9.9 MiB/s wr, 1.24k op/s 2024-05-09T08:28:37.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:37 smithi165 ceph-mon[34540]: pgmap v72: 129 pgs: 129 active+clean; 209 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9.9 MiB/s wr, 1.24k op/s 2024-05-09T08:28:37.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:37 smithi184 ceph-mon[34509]: pgmap v72: 129 pgs: 129 active+clean; 209 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9.9 MiB/s wr, 1.24k op/s 2024-05-09T08:28:38.705 INFO:tasks.workunit.client.0.smithi028.stdout: 81 416533 533 301795 701 63446 229 2024-05-09T08:28:39.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:39 smithi028 ceph-mon[30119]: pgmap v73: 129 pgs: 129 active+clean; 209 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.6 MiB/s wr, 1.22k op/s 2024-05-09T08:28:39.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:39 smithi165 ceph-mon[34540]: pgmap v73: 129 pgs: 129 active+clean; 209 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.6 MiB/s wr, 1.22k op/s 2024-05-09T08:28:39.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:39 smithi184 ceph-mon[34509]: pgmap v73: 129 pgs: 129 active+clean; 209 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.6 MiB/s wr, 1.22k op/s 2024-05-09T08:28:41.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:41 smithi028 ceph-mon[30119]: pgmap v74: 129 pgs: 129 active+clean; 235 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-05-09T08:28:41.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:41 smithi165 ceph-mon[34540]: pgmap v74: 129 pgs: 129 active+clean; 235 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-05-09T08:28:41.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:41 smithi184 ceph-mon[34509]: pgmap v74: 129 pgs: 129 active+clean; 235 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-05-09T08:28:43.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:43 smithi028 ceph-mon[30119]: pgmap v75: 129 pgs: 129 active+clean; 235 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-05-09T08:28:43.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:43 smithi165 ceph-mon[34540]: pgmap v75: 129 pgs: 129 active+clean; 235 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-05-09T08:28:43.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:43 smithi184 ceph-mon[34509]: pgmap v75: 129 pgs: 129 active+clean; 235 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-05-09T08:28:45.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:45 smithi028 ceph-mon[30119]: pgmap v76: 129 pgs: 129 active+clean; 242 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-05-09T08:28:45.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:45 smithi165 ceph-mon[34540]: pgmap v76: 129 pgs: 129 active+clean; 242 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-05-09T08:28:45.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:45 smithi184 ceph-mon[34509]: pgmap v76: 129 pgs: 129 active+clean; 242 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-05-09T08:28:47.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.195+0000 7efd6eaba640 1 -- 172.21.15.28:0/533795973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd6810db00 msgr2=0x7efd6810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:47.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.195+0000 7efd6eaba640 1 --2- 172.21.15.28:0/533795973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd6810db00 0x7efd6810df80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd60009ca0 tx=0x7efd6002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.202+0000 7efd6eaba640 1 -- 172.21.15.28:0/533795973 shutdown_connections 2024-05-09T08:28:47.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.202+0000 7efd6eaba640 1 --2- 172.21.15.28:0/533795973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6810e5a0 0x7efd68108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.202+0000 7efd6eaba640 1 --2- 172.21.15.28:0/533795973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd6810db00 0x7efd6810df80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.202+0000 7efd6eaba640 1 --2- 172.21.15.28:0/533795973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6810fda0 0x7efd68110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.202+0000 7efd6eaba640 1 -- 172.21.15.28:0/533795973 >> 172.21.15.28:0/533795973 conn(0x7efd6806c7e0 msgr2=0x7efd6806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.204+0000 7efd6eaba640 1 -- 172.21.15.28:0/533795973 shutdown_connections 2024-05-09T08:28:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.204+0000 7efd6eaba640 1 -- 172.21.15.28:0/533795973 wait complete. 2024-05-09T08:28:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.205+0000 7efd6eaba640 1 Processor -- start 2024-05-09T08:28:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.205+0000 7efd6eaba640 1 -- start start 2024-05-09T08:28:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd6eaba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd6810db00 0x7efd6811e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd6eaba640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6810e5a0 0x7efd6811eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd6eaba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6810fda0 0x7efd68117810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd6eaba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd68120890 con 0x7efd6810fda0 2024-05-09T08:28:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd6eaba640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd68120710 con 0x7efd6810e5a0 2024-05-09T08:28:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd6eaba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd68120a10 con 0x7efd6810db00 2024-05-09T08:28:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd67fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6810e5a0 0x7efd6811eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd6c856640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd6810db00 0x7efd6811e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd6d057640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6810fda0 0x7efd68117810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd67fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6810e5a0 0x7efd6811eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35140/0 (socket says 172.21.15.28:35140) 2024-05-09T08:28:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd67fff640 1 -- 172.21.15.28:0/3889987093 learned_addr learned my addr 172.21.15.28:0/3889987093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:47.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd67fff640 1 -- 172.21.15.28:0/3889987093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd6810db00 msgr2=0x7efd6811e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:47.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd67fff640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd6810db00 0x7efd6811e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd67fff640 1 -- 172.21.15.28:0/3889987093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6810fda0 msgr2=0x7efd68117810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:47.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd67fff640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6810fda0 0x7efd68117810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.206+0000 7efd67fff640 1 -- 172.21.15.28:0/3889987093 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd681180d0 con 0x7efd6810e5a0 2024-05-09T08:28:47.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.207+0000 7efd67fff640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6810e5a0 0x7efd6811eaf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd60006010 tx=0x7efd60009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.228+0000 7efd65ffb640 1 -- 172.21.15.28:0/3889987093 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60003c70 con 0x7efd6810e5a0 2024-05-09T08:28:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.228+0000 7efd65ffb640 1 -- 172.21.15.28:0/3889987093 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd60003e10 con 0x7efd6810e5a0 2024-05-09T08:28:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.228+0000 7efd65ffb640 1 -- 172.21.15.28:0/3889987093 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60030d20 con 0x7efd6810e5a0 2024-05-09T08:28:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.228+0000 7efd6eaba640 1 -- 172.21.15.28:0/3889987093 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd68118360 con 0x7efd6810e5a0 2024-05-09T08:28:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.228+0000 7efd6eaba640 1 -- 172.21.15.28:0/3889987093 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd6811c310 con 0x7efd6810e5a0 2024-05-09T08:28:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.233+0000 7efd717c6740 1 Processor -- start 2024-05-09T08:28:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.233+0000 7efd717c6740 1 -- start start 2024-05-09T08:28:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.236+0000 7efd65ffb640 1 -- 172.21.15.28:0/3889987093 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7efd6003a400 con 0x7efd6810e5a0 2024-05-09T08:28:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.237+0000 7efd65ffb640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd500a2ec0 0x7efd500a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.237+0000 7efd65ffb640 1 -- 172.21.15.28:0/3889987093 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7efd600f0970 con 0x7efd6810e5a0 2024-05-09T08:28:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.238+0000 7efd717c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd884b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.238+0000 7efd6d057640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd500a2ec0 0x7efd500a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.238+0000 7efd717c6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884bef0 0x55cbd8856270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.238+0000 7efd717c6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd88567b0 0x55cbd8858ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.238+0000 7efd6c856640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd884b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.238+0000 7efd6c856640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd884b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57118/0 (socket says 172.21.15.28:57118) 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd6c856640 1 -- 172.21.15.28:0/3767841961 learned_addr learned my addr 172.21.15.28:0/3767841961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd717c6740 1 -- 172.21.15.28:0/3767841961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbd87b0340 con 0x55cbd884b5d0 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd717c6740 1 -- 172.21.15.28:0/3767841961 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55cbd8758290 con 0x55cbd884bef0 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd717c6740 1 -- 172.21.15.28:0/3767841961 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cbd87ac300 con 0x55cbd88567b0 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd67fff640 1 --2- 172.21.15.28:0/3767841961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884bef0 0x55cbd8856270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd6c856640 1 -- 172.21.15.28:0/3767841961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd88567b0 msgr2=0x55cbd8858ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd6c856640 1 --2- 172.21.15.28:0/3767841961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd88567b0 0x55cbd8858ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd6c856640 1 -- 172.21.15.28:0/3767841961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884bef0 msgr2=0x55cbd8856270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd6c856640 1 --2- 172.21.15.28:0/3767841961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884bef0 0x55cbd8856270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd6c856640 1 -- 172.21.15.28:0/3767841961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbd877c8e0 con 0x55cbd884b5d0 2024-05-09T08:28:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd6d057640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd500a2ec0 0x7efd500a5380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd6810ead0 tx=0x7efd58005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:47.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.239+0000 7efd6c856640 1 --2- 172.21.15.28:0/3767841961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd884b9b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00aba0 tx=0x7efd5c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852b729bba370797 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:47.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.248+0000 7efd4effd640 1 -- 172.21.15.28:0/3767841961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c01d070 con 0x55cbd884b5d0 2024-05-09T08:28:47.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.248+0000 7efd4effd640 1 -- 172.21.15.28:0/3767841961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd5c018b20 con 0x55cbd884b5d0 2024-05-09T08:28:47.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.249+0000 7efd717c6740 1 -- 172.21.15.28:0/3767841961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 msgr2=0x55cbd884b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:47.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.249+0000 7efd717c6740 1 --2- 172.21.15.28:0/3767841961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd884b9b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00aba0 tx=0x7efd5c00f790 comp rx=0 tx=0).stop 2024-05-09T08:28:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.265+0000 7efd717c6740 1 -- 172.21.15.28:0/3767841961 shutdown_connections 2024-05-09T08:28:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.265+0000 7efd717c6740 1 --2- 172.21.15.28:0/3767841961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd88567b0 0x55cbd8858ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.265+0000 7efd717c6740 1 --2- 172.21.15.28:0/3767841961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884bef0 0x55cbd8856270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.265+0000 7efd717c6740 1 --2- 172.21.15.28:0/3767841961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd884b9b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.265+0000 7efd717c6740 1 -- 172.21.15.28:0/3767841961 >> 172.21.15.28:0/3767841961 conn(0x55cbd8847ae0 msgr2=0x55cbd8847ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.265+0000 7efd717c6740 1 -- 172.21.15.28:0/3767841961 shutdown_connections 2024-05-09T08:28:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.265+0000 7efd717c6740 1 -- 172.21.15.28:0/3767841961 wait complete. 2024-05-09T08:28:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.266+0000 7efd717c6740 1 -- start start 2024-05-09T08:28:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.266+0000 7efd717c6740 1 Processor -- start 2024-05-09T08:28:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.266+0000 7efd717c6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd8853750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.266+0000 7efd717c6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd8853c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.266+0000 7efd717c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 0x55cbd88541d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.266+0000 7efd717c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbd88593e0 con 0x55cbd88567b0 2024-05-09T08:28:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.266+0000 7efd717c6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55cbd876b790 con 0x55cbd884b5d0 2024-05-09T08:28:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.266+0000 7efd717c6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cbd8859560 con 0x55cbd884bef0 2024-05-09T08:28:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.266+0000 7efd717c6740 10 client.0 _ll_register_callbacks cb 0x55cbd87a6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6c856640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd8853750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6c856640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd8853750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35156/0 (socket says 172.21.15.28:35156) 2024-05-09T08:28:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6c856640 1 -- 172.21.15.28:0/1357496506 learned_addr learned my addr 172.21.15.28:0/1357496506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd67fff640 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd8853c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6d057640 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 0x55cbd88541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6c856640 1 -- 172.21.15.28:0/1357496506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 msgr2=0x55cbd8853c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6c856640 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd8853c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6c856640 1 -- 172.21.15.28:0/1357496506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 msgr2=0x55cbd88541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6c856640 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 0x55cbd88541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6c856640 1 -- 172.21.15.28:0/1357496506 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbd84c4210 con 0x55cbd884b5d0 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.267+0000 7efd6d057640 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 0x55cbd88541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.268+0000 7efd6c856640 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd8853750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efd5c018650 tx=0x7efd5c018680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba6f94ecfd791fef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.268+0000 7efd4c7f8640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.268+0000 7efd4c7f8640 1 -- 172.21.15.28:0/1357496506 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c01d050 con 0x55cbd884b5d0 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.268+0000 7efd4c7f8640 1 -- 172.21.15.28:0/1357496506 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7efd5c005240 con 0x55cbd884b5d0 2024-05-09T08:28:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.268+0000 7efd717c6740 1 -- 172.21.15.28:0/1357496506 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd5c018b20 con 0x55cbd884b5d0 2024-05-09T08:28:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.268+0000 7efd4c7f8640 1 -- 172.21.15.28:0/1357496506 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7efd5c032050 con 0x55cbd884b5d0 2024-05-09T08:28:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.268+0000 7efd717c6740 10 client.25988 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.268+0000 7efd717c6740 1 -- 172.21.15.28:0/1357496506 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cbd8590620 con 0x55cbd884b5d0 2024-05-09T08:28:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.269+0000 7efd4c7f8640 1 -- 172.21.15.28:0/1357496506 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd5c018b20 con 0x55cbd884b5d0 2024-05-09T08:28:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.269+0000 7efd4c7f8640 1 -- 172.21.15.28:0/1357496506 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7efd5c004b40 con 0x55cbd884b5d0 2024-05-09T08:28:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.269+0000 7efd717c6740 10 client.25988 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.269+0000 7efd717c6740 10 client.25988 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.269+0000 7efd717c6740 20 client.25988 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.269+0000 7efd717c6740 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cbd88664d0 0x55cbd88688c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.269+0000 7efd717c6740 4 client.25988 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:28:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.269+0000 7efd717c6740 1 -- 172.21.15.28:0/1357496506 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbd87bda70 con 0x55cbd88664d0 2024-05-09T08:28:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.269+0000 7efd67fff640 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cbd88664d0 0x55cbd88688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.270+0000 7efd4c7f8640 10 client.25988 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.270+0000 7efd67fff640 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cbd88664d0 0x55cbd88688c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea9aa21171a59057 server_cookie=65ca6b5c0174cd40 in_seq=0 out_seq=0 2024-05-09T08:28:47.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.275+0000 7efd4c7f8640 1 -- 172.21.15.28:0/1357496506 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbd87bda70 con 0x55cbd88664d0 2024-05-09T08:28:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.275+0000 7efd4c7f8640 10 client.25988 handle_command_reply: tid=0 2024-05-09T08:28:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.275+0000 7efd717c6740 1 client.25988 shutdown 2024-05-09T08:28:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 -- 172.21.15.28:0/1357496506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 msgr2=0x55cbd8853750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:47.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd8853750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efd5c018650 tx=0x7efd5c018680 comp rx=0 tx=0).stop 2024-05-09T08:28:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 -- 172.21.15.28:0/1357496506 shutdown_connections 2024-05-09T08:28:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cbd88664d0 0x55cbd88688c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 0x55cbd88541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd8853c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 --2- 172.21.15.28:0/1357496506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd8853750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 -- 172.21.15.28:0/1357496506 >> 172.21.15.28:0/1357496506 conn(0x55cbd8847ae0 msgr2=0x55cbd884d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd4c7f8640 0 client.25988 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 -- 172.21.15.28:0/1357496506 shutdown_connections 2024-05-09T08:28:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 1 -- 172.21.15.28:0/1357496506 wait complete. 2024-05-09T08:28:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.276+0000 7efd717c6740 20 client.25988 trim_cache size 0 max 16384 2024-05-09T08:28:47.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.336+0000 7efd717c6740 1 Processor -- start 2024-05-09T08:28:47.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.336+0000 7efd717c6740 1 -- start start 2024-05-09T08:28:47.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd717c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd88622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd717c6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd886df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd717c6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd88567b0 0x55cbd886e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd6c856640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd88622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd717c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd681c62c0 con 0x55cbd884b5d0 2024-05-09T08:28:47.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd6c856640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd88622f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57142/0 (socket says 172.21.15.28:57142) 2024-05-09T08:28:47.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd6c856640 1 -- 172.21.15.28:0/3136790766 learned_addr learned my addr 172.21.15.28:0/3136790766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:47.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd717c6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd681c7ec0 con 0x55cbd88567b0 2024-05-09T08:28:47.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd6d057640 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd88567b0 0x55cbd886e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.338+0000 7efd67fff640 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd886df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.343+0000 7efd717c6740 1 -- 172.21.15.28:0/3136790766 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd681c6990 con 0x55cbd884bef0 2024-05-09T08:28:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.343+0000 7efd6d057640 1 -- 172.21.15.28:0/3136790766 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 msgr2=0x55cbd886df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.343+0000 7efd6d057640 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd886df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.343+0000 7efd6d057640 1 -- 172.21.15.28:0/3136790766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 msgr2=0x55cbd88622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:47.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.343+0000 7efd6d057640 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd88622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.343+0000 7efd6d057640 1 -- 172.21.15.28:0/3136790766 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbd8852700 con 0x55cbd88567b0 2024-05-09T08:28:47.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.343+0000 7efd67fff640 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd886df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:47.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.343+0000 7efd6c856640 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd88622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:28:47.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.344+0000 7efd6d057640 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd88567b0 0x55cbd886e490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55cbd8848f00 tx=0x7efd58014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ea8d4d9a1bb313f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:47.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.346+0000 7efd4e7fc640 1 -- 172.21.15.28:0/3136790766 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58003ac0 con 0x55cbd88567b0 2024-05-09T08:28:47.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.346+0000 7efd4e7fc640 1 -- 172.21.15.28:0/3136790766 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd58022070 con 0x55cbd88567b0 2024-05-09T08:28:47.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.346+0000 7efd4e7fc640 1 -- 172.21.15.28:0/3136790766 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5801b670 con 0x55cbd88567b0 2024-05-09T08:28:47.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.347+0000 7efd717c6740 1 -- 172.21.15.28:0/3136790766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd88567b0 msgr2=0x55cbd886e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:47.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.347+0000 7efd717c6740 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd88567b0 0x55cbd886e490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55cbd8848f00 tx=0x7efd58014000 comp rx=0 tx=0).stop 2024-05-09T08:28:47.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.347+0000 7efd717c6740 1 -- 172.21.15.28:0/3136790766 shutdown_connections 2024-05-09T08:28:47.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.347+0000 7efd717c6740 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd88567b0 0x55cbd886e490 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.347+0000 7efd717c6740 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd886df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.347+0000 7efd717c6740 1 --2- 172.21.15.28:0/3136790766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd884b5d0 0x55cbd88622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.347+0000 7efd717c6740 1 -- 172.21.15.28:0/3136790766 >> 172.21.15.28:0/3136790766 conn(0x55cbd8847ae0 msgr2=0x55cbd8847ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:47.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.349+0000 7efd717c6740 1 -- 172.21.15.28:0/3136790766 shutdown_connections 2024-05-09T08:28:47.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.349+0000 7efd717c6740 1 -- 172.21.15.28:0/3136790766 wait complete. 2024-05-09T08:28:47.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.349+0000 7efd717c6740 1 -- start start 2024-05-09T08:28:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.350+0000 7efd717c6740 1 Processor -- start 2024-05-09T08:28:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.350+0000 7efd717c6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd885ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.350+0000 7efd717c6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd8859e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd6c856640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd885ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd717c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 0x55cbd885a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd6c856640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd885ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35168/0 (socket says 172.21.15.28:35168) 2024-05-09T08:28:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd6c856640 1 -- 172.21.15.28:0/26739375 learned_addr learned my addr 172.21.15.28:0/26739375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:28:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd681c7730 con 0x55cbd88567b0 2024-05-09T08:28:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd681c7060 con 0x55cbd884b5d0 2024-05-09T08:28:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd681c42a0 con 0x55cbd884bef0 2024-05-09T08:28:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd6d057640 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 0x55cbd885a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd67fff640 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd8859e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd717c6740 10 client.0 _ll_register_callbacks cb 0x7efd5c01b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:28:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd6c856640 1 -- 172.21.15.28:0/26739375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 msgr2=0x55cbd8859e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd6c856640 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd8859e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd6c856640 1 -- 172.21.15.28:0/26739375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 msgr2=0x55cbd885a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:28:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd6c856640 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 0x55cbd885a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.351+0000 7efd6c856640 1 -- 172.21.15.28:0/26739375 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbd885aa70 con 0x55cbd884b5d0 2024-05-09T08:28:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.352+0000 7efd6c856640 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd885ead0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0246b0 tx=0x7efd5c00a450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7e527854b7a3b72 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.355+0000 7efd4c7f8640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:28:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.355+0000 7efd4c7f8640 1 -- 172.21.15.28:0/26739375 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c01d040 con 0x55cbd884b5d0 2024-05-09T08:28:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.355+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd58022070 con 0x55cbd884b5d0 2024-05-09T08:28:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.355+0000 7efd4c7f8640 1 -- 172.21.15.28:0/26739375 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7efd5c018c30 con 0x55cbd884b5d0 2024-05-09T08:28:47.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.356+0000 7efd4c7f8640 1 -- 172.21.15.28:0/26739375 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7efd5c047050 con 0x55cbd884b5d0 2024-05-09T08:28:47.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.356+0000 7efd717c6740 10 client.25994 fetch_fsmap learned FSMap version 34 2024-05-09T08:28:47.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.356+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cbd88cc6b0 con 0x55cbd884b5d0 2024-05-09T08:28:47.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.356+0000 7efd4c7f8640 1 -- 172.21.15.28:0/26739375 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd58022070 con 0x55cbd884b5d0 2024-05-09T08:28:47.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.356+0000 7efd4c7f8640 1 -- 172.21.15.28:0/26739375 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7efd5c024920 con 0x55cbd884b5d0 2024-05-09T08:28:47.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.357+0000 7efd717c6740 10 client.25994 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:28:47.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.357+0000 7efd717c6740 10 client.25994 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:28:47.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.357+0000 7efd717c6740 20 client.25994 populate_metadata read hostname 'smithi028' 2024-05-09T08:28:47.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.357+0000 7efd717c6740 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cbd88cce90 0x55cbd88cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:47.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.357+0000 7efd717c6740 4 client.25994 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:28:47.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.357+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cbd87b7dc0 con 0x55cbd88cce90 2024-05-09T08:28:47.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.357+0000 7efd67fff640 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cbd88cce90 0x55cbd88cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:47.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.358+0000 7efd67fff640 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cbd88cce90 0x55cbd88cf2b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f6d42193e69f2d3 server_cookie=862fe277cabcf23a in_seq=0 out_seq=0 2024-05-09T08:28:47.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:47.358+0000 7efd4c7f8640 10 client.25994 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:47.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:47 smithi028 ceph-mon[30119]: pgmap v77: 129 pgs: 129 active+clean; 243 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.6 MiB/s wr, 705 op/s 2024-05-09T08:28:47.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:47 smithi165 ceph-mon[34540]: pgmap v77: 129 pgs: 129 active+clean; 243 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.6 MiB/s wr, 705 op/s 2024-05-09T08:28:47.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:47 smithi184 ceph-mon[34509]: pgmap v77: 129 pgs: 129 active+clean; 243 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.6 MiB/s wr, 705 op/s 2024-05-09T08:28:48.708 INFO:tasks.workunit.client.0.smithi028.stdout: 108 1049978 1403 744318 1157 128308 151 2024-05-09T08:28:49.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:49 smithi028 ceph-mon[30119]: pgmap v78: 129 pgs: 129 active+clean; 243 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 145 op/s 2024-05-09T08:28:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.701+0000 7efd4c7f8640 1 -- 172.21.15.28:0/26739375 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cbd87b7dc0 con 0x55cbd88cce90 2024-05-09T08:28:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.701+0000 7efd4c7f8640 10 client.25994 handle_command_reply: tid=0 2024-05-09T08:28:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.702+0000 7efd717c6740 1 client.25994 shutdown 2024-05-09T08:28:49.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 msgr2=0x55cbd885ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:49.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd885ead0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0246b0 tx=0x7efd5c00a450 comp rx=0 tx=0).stop 2024-05-09T08:28:49.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 shutdown_connections 2024-05-09T08:28:49.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cbd88cce90 0x55cbd88cf2b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbd88567b0 0x55cbd885a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cbd884bef0 0x55cbd8859e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 --2- 172.21.15.28:0/26739375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cbd884b5d0 0x55cbd885ead0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 >> 172.21.15.28:0/26739375 conn(0x55cbd8847ae0 msgr2=0x55cbd884cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd4c7f8640 0 client.25994 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:28:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 shutdown_connections 2024-05-09T08:28:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 1 -- 172.21.15.28:0/26739375 wait complete. 2024-05-09T08:28:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.705+0000 7efd717c6740 20 client.25994 trim_cache size 0 max 16384 2024-05-09T08:28:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:28:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:28:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:28:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:28:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 -- 172.21.15.28:0/3889987093 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd500a2ec0 msgr2=0x7efd500a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd500a2ec0 0x7efd500a5380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd6810ead0 tx=0x7efd58005f90 comp rx=0 tx=0).stop 2024-05-09T08:28:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 -- 172.21.15.28:0/3889987093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6810e5a0 msgr2=0x7efd6811eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6810e5a0 0x7efd6811eaf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd60006010 tx=0x7efd60009360 comp rx=0 tx=0).stop 2024-05-09T08:28:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 -- 172.21.15.28:0/3889987093 shutdown_connections 2024-05-09T08:28:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6810fda0 0x7efd68117810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd6810e5a0 0x7efd6811eaf0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efd500a2ec0 0x7efd500a5380 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 --2- 172.21.15.28:0/3889987093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd6810db00 0x7efd6811e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.707+0000 7efd6eaba640 1 -- 172.21.15.28:0/3889987093 >> 172.21.15.28:0/3889987093 conn(0x7efd6806c7e0 msgr2=0x7efd68109e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.708+0000 7efd6eaba640 1 -- 172.21.15.28:0/3889987093 shutdown_connections 2024-05-09T08:28:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:28:49.708+0000 7efd6eaba640 1 -- 172.21.15.28:0/3889987093 wait complete. 2024-05-09T08:28:49.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:49 smithi165 ceph-mon[34540]: pgmap v78: 129 pgs: 129 active+clean; 243 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 145 op/s 2024-05-09T08:28:49.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:49 smithi184 ceph-mon[34509]: pgmap v78: 129 pgs: 129 active+clean; 243 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 145 op/s 2024-05-09T08:28:51.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:51 smithi028 ceph-mon[30119]: pgmap v79: 129 pgs: 129 active+clean; 249 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 863 op/s 2024-05-09T08:28:51.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:51 smithi165 ceph-mon[34540]: pgmap v79: 129 pgs: 129 active+clean; 249 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 863 op/s 2024-05-09T08:28:51.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:51 smithi184 ceph-mon[34509]: pgmap v79: 129 pgs: 129 active+clean; 249 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 863 op/s 2024-05-09T08:28:53.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:53 smithi028 ceph-mon[30119]: pgmap v80: 129 pgs: 129 active+clean; 249 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 759 op/s 2024-05-09T08:28:53.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:53 smithi165 ceph-mon[34540]: pgmap v80: 129 pgs: 129 active+clean; 249 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 759 op/s 2024-05-09T08:28:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:53 smithi184 ceph-mon[34509]: pgmap v80: 129 pgs: 129 active+clean; 249 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 759 op/s 2024-05-09T08:28:55.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:55 smithi028 ceph-mon[30119]: pgmap v81: 129 pgs: 129 active+clean; 240 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 782 op/s 2024-05-09T08:28:55.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:55 smithi165 ceph-mon[34540]: pgmap v81: 129 pgs: 129 active+clean; 240 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 782 op/s 2024-05-09T08:28:55.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:55 smithi184 ceph-mon[34509]: pgmap v81: 129 pgs: 129 active+clean; 240 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 782 op/s 2024-05-09T08:28:57.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:57 smithi028 ceph-mon[30119]: pgmap v82: 129 pgs: 129 active+clean; 228 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 1.3 MiB/s wr, 860 op/s 2024-05-09T08:28:57.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:57 smithi165 ceph-mon[34540]: pgmap v82: 129 pgs: 129 active+clean; 228 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 1.3 MiB/s wr, 860 op/s 2024-05-09T08:28:57.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:57 smithi184 ceph-mon[34509]: pgmap v82: 129 pgs: 129 active+clean; 228 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 1.3 MiB/s wr, 860 op/s 2024-05-09T08:28:58.708 INFO:tasks.workunit.client.0.smithi028.stdout: 120 977945 562 678839 593 113149 241 2024-05-09T08:28:59.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:28:59 smithi028 ceph-mon[30119]: pgmap v83: 129 pgs: 129 active+clean; 228 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.1 MiB/s wr, 844 op/s 2024-05-09T08:28:59.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:28:59 smithi165 ceph-mon[34540]: pgmap v83: 129 pgs: 129 active+clean; 228 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.1 MiB/s wr, 844 op/s 2024-05-09T08:28:59.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:28:59 smithi184 ceph-mon[34509]: pgmap v83: 129 pgs: 129 active+clean; 228 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.1 MiB/s wr, 844 op/s 2024-05-09T08:29:01.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:01 smithi165 ceph-mon[34540]: pgmap v84: 129 pgs: 129 active+clean; 298 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.3 MiB/s wr, 1.11k op/s 2024-05-09T08:29:01.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:01 smithi184 ceph-mon[34509]: pgmap v84: 129 pgs: 129 active+clean; 298 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.3 MiB/s wr, 1.11k op/s 2024-05-09T08:29:01.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:01 smithi028 ceph-mon[30119]: pgmap v84: 129 pgs: 129 active+clean; 298 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.3 MiB/s wr, 1.11k op/s 2024-05-09T08:29:02.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.907+0000 7f4ea595a640 1 -- 172.21.15.28:0/1847971581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0106720 msgr2=0x7f4ea0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:02.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.907+0000 7f4ea595a640 1 --2- 172.21.15.28:0/1847971581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0106720 0x7f4ea0106b00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009cd0 tx=0x7f4e9802a8f0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.907+0000 7f4ea595a640 1 -- 172.21.15.28:0/1847971581 shutdown_connections 2024-05-09T08:29:02.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.907+0000 7f4ea595a640 1 --2- 172.21.15.28:0/1847971581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea0101120 0x7f4ea010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.907+0000 7f4ea595a640 1 --2- 172.21.15.28:0/1847971581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ea0100760 0x7f4ea0100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.907+0000 7f4ea595a640 1 --2- 172.21.15.28:0/1847971581 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0106720 0x7f4ea0106b00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.907+0000 7f4ea595a640 1 -- 172.21.15.28:0/1847971581 >> 172.21.15.28:0/1847971581 conn(0x7f4ea00778b0 msgr2=0x7f4ea0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.907+0000 7f4ea595a640 1 -- 172.21.15.28:0/1847971581 shutdown_connections 2024-05-09T08:29:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.907+0000 7f4ea595a640 1 -- 172.21.15.28:0/1847971581 wait complete. 2024-05-09T08:29:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4ea595a640 1 Processor -- start 2024-05-09T08:29:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4ea595a640 1 -- start start 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4ea595a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0100760 0x7f4ea01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4ea595a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ea0101120 0x7f4ea01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4ea595a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea0106720 0x7f4ea019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4ea595a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea0110b40 con 0x7f4ea0106720 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4ea595a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01109c0 con 0x7f4ea0101120 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4ea595a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea0110cc0 con 0x7f4ea0100760 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ea0101120 0x7f4ea01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4e9effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0100760 0x7f4ea01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ea0101120 0x7f4ea01985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:53720/0 (socket says 172.21.15.28:53720) 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4e9effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0100760 0x7f4ea01980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51364/0 (socket says 172.21.15.28:51364) 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/2867187051 learned_addr learned my addr 172.21.15.28:0/2867187051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.908+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea0106720 0x7f4ea019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/2867187051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0100760 msgr2=0x7f4ea01980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0100760 0x7f4ea01980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea0106720 msgr2=0x7f4ea019c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea0106720 0x7f4ea019c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/2867187051 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea019d090 con 0x7f4ea0101120 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea0106720 0x7f4ea019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ea0101120 0x7f4ea01985e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400e9f0 tx=0x7f4e9400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4ea4958640 1 -- 172.21.15.28:0/2867187051 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400cd80 con 0x7f4ea0101120 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4ea4958640 1 -- 172.21.15.28:0/2867187051 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e94010910 con 0x7f4ea0101120 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4ea595a640 1 -- 172.21.15.28:0/2867187051 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea019d380 con 0x7f4ea0101120 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4ea4958640 1 -- 172.21.15.28:0/2867187051 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94018990 con 0x7f4ea0101120 2024-05-09T08:29:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.909+0000 7f4ea595a640 1 -- 172.21.15.28:0/2867187051 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea01a4c10 con 0x7f4ea0101120 2024-05-09T08:29:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.913+0000 7f4ea8664740 1 Processor -- start 2024-05-09T08:29:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.913+0000 7f4ea8664740 1 -- start start 2024-05-09T08:29:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.913+0000 7f4ea4958640 1 -- 172.21.15.28:0/2867187051 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4e94010ac0 con 0x7f4ea0101120 2024-05-09T08:29:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa6885d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa688680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6886aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4ea8664740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aa687c2340 con 0x55aa6885dd70 2024-05-09T08:29:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4ea8664740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55aa6876a290 con 0x55aa68868630 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4ea8664740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aa687be300 con 0x55aa6885d450 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6886aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4e9f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa6885d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4e9f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa6885d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51378/0 (socket says 172.21.15.28:51378) 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4e9f7fe640 1 -- 172.21.15.28:0/3171856692 learned_addr learned my addr 172.21.15.28:0/3171856692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6886aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:53722/0 (socket says 172.21.15.28:53722) 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.914+0000 7f4e9effd640 1 --2- 172.21.15.28:0/3171856692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa688680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.915+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/3171856692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 msgr2=0x55aa6885d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.915+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/3171856692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa6885d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.915+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/3171856692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 msgr2=0x55aa688680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.915+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/3171856692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa688680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.915+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/3171856692 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa6878e8e0 con 0x55aa68868630 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.915+0000 7f4e9effd640 1 --2- 172.21.15.28:0/3171856692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa688680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:29:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.915+0000 7f4ea4958640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f4e880a2da0 0x7f4e880a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.915+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/3171856692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6886aa20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4e940cce90 tx=0x7f4e940d8650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a71b094d2395cfdb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:02.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.915+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f4e880a2da0 0x7f4e880a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.916+0000 7f4e85ffb640 1 -- 172.21.15.28:0/3171856692 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e940df020 con 0x55aa68868630 2024-05-09T08:29:02.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.916+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f4e880a2da0 0x7f4e880a5260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e90005f60 tx=0x7f4e90005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:02.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.916+0000 7f4ea4958640 1 -- 172.21.15.28:0/2867187051 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f4e940cfb10 con 0x7f4ea0101120 2024-05-09T08:29:02.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.916+0000 7f4e85ffb640 1 -- 172.21.15.28:0/3171856692 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e9409bd10 con 0x55aa68868630 2024-05-09T08:29:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.918+0000 7f4ea8664740 1 -- 172.21.15.28:0/3171856692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 msgr2=0x55aa6886aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.918+0000 7f4ea8664740 1 --2- 172.21.15.28:0/3171856692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6886aa20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4e940cce90 tx=0x7f4e940d8650 comp rx=0 tx=0).stop 2024-05-09T08:29:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.918+0000 7f4ea8664740 1 -- 172.21.15.28:0/3171856692 shutdown_connections 2024-05-09T08:29:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.918+0000 7f4ea8664740 1 --2- 172.21.15.28:0/3171856692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6886aa20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.918+0000 7f4ea8664740 1 --2- 172.21.15.28:0/3171856692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa688680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.918+0000 7f4ea8664740 1 --2- 172.21.15.28:0/3171856692 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa6885d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.918+0000 7f4ea8664740 1 -- 172.21.15.28:0/3171856692 >> 172.21.15.28:0/3171856692 conn(0x55aa68859960 msgr2=0x55aa68859d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.918+0000 7f4ea8664740 1 -- 172.21.15.28:0/3171856692 shutdown_connections 2024-05-09T08:29:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.918+0000 7f4ea8664740 1 -- 172.21.15.28:0/3171856692 wait complete. 2024-05-09T08:29:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.919+0000 7f4ea8664740 1 -- start start 2024-05-09T08:29:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.919+0000 7f4ea8664740 1 Processor -- start 2024-05-09T08:29:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885d450 0x55aa68865540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885dd70 0x55aa68865a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68865fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885dd70 0x55aa68865a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885dd70 0x55aa68865a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51388/0 (socket says 172.21.15.28:51388) 2024-05-09T08:29:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4e9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885d450 0x55aa68865540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/1391286813 learned_addr learned my addr 172.21.15.28:0/1391286813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aa6886f670 con 0x55aa6885d450 2024-05-09T08:29:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55aa6877d790 con 0x55aa68868630 2024-05-09T08:29:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aa6886f7f0 con 0x55aa6885dd70 2024-05-09T08:29:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68865fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.920+0000 7f4ea8664740 10 client.0 _ll_register_callbacks cb 0x55aa687b8a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:29:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4e9f7fe640 1 -- 172.21.15.28:0/1391286813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885dd70 msgr2=0x55aa68865a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885dd70 0x55aa68865a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4e9f7fe640 1 -- 172.21.15.28:0/1391286813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885d450 msgr2=0x55aa68865540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885d450 0x55aa68865540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4e9f7fe640 1 -- 172.21.15.28:0/1391286813 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa684d6210 con 0x55aa68868630 2024-05-09T08:29:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68865fc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e90012f20 tx=0x7f4e90004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba272af1da86703 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4e7effd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:29:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4e7effd640 1 -- 172.21.15.28:0/1391286813 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90005020 con 0x55aa68868630 2024-05-09T08:29:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e9409bd10 con 0x55aa68868630 2024-05-09T08:29:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.921+0000 7f4e7effd640 1 -- 172.21.15.28:0/1391286813 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f4e900051c0 con 0x55aa68868630 2024-05-09T08:29:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.922+0000 7f4e7effd640 1 -- 172.21.15.28:0/1391286813 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4e900053c0 con 0x55aa68868630 2024-05-09T08:29:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.922+0000 7f4ea8664740 10 client.26006 fetch_fsmap learned FSMap version 34 2024-05-09T08:29:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.922+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55aa685a2620 con 0x55aa68868630 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.922+0000 7f4e7effd640 1 -- 172.21.15.28:0/1391286813 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e9409bd10 con 0x55aa68868630 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.922+0000 7f4e7effd640 1 -- 172.21.15.28:0/1391286813 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4e90027030 con 0x55aa68868630 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.922+0000 7f4ea8664740 10 client.26006 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.922+0000 7f4ea8664740 10 client.26006 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.922+0000 7f4ea8664740 20 client.26006 populate_metadata read hostname 'smithi028' 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.922+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aa6887cb20 0x55aa6887ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.923+0000 7f4ea8664740 4 client.26006 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.923+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa687cfa70 con 0x55aa6887cb20 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.923+0000 7f4e9effd640 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aa6887cb20 0x55aa6887ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.923+0000 7f4e7effd640 10 client.26006 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.923+0000 7f4e9effd640 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aa6887cb20 0x55aa6887ef80 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd9d88652c58d1a7 server_cookie=161315c925aa7f65 in_seq=0 out_seq=0 2024-05-09T08:29:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4e7effd640 1 -- 172.21.15.28:0/1391286813 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aa687cfa70 con 0x55aa6887cb20 2024-05-09T08:29:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4e7effd640 10 client.26006 handle_command_reply: tid=0 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4ea8664740 1 client.26006 shutdown 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 msgr2=0x55aa68865fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68865fc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e90012f20 tx=0x7f4e90004730 comp rx=0 tx=0).stop 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 shutdown_connections 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aa6887cb20 0x55aa6887ef80 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68865fc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885dd70 0x55aa68865a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1391286813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885d450 0x55aa68865540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.925+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 >> 172.21.15.28:0/1391286813 conn(0x55aa68859960 msgr2=0x55aa6885f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.926+0000 7f4e7effd640 0 client.26006 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.926+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 shutdown_connections 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.926+0000 7f4ea8664740 1 -- 172.21.15.28:0/1391286813 wait complete. 2024-05-09T08:29:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.926+0000 7f4ea8664740 20 client.26006 trim_cache size 0 max 16384 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.945+0000 7f4ea8664740 1 Processor -- start 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4ea8664740 1 -- start start 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa68867c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68864e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6885fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4ea8664740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01acbf0 con 0x55aa6885dd70 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4ea8664740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01ae7f0 con 0x55aa68868630 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4ea8664740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01ad2c0 con 0x55aa6885d450 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68864e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68864e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56398/0 (socket says 172.21.15.28:56398) 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/3870019035 learned_addr learned my addr 172.21.15.28:0/3870019035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/3870019035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6885fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9effd640 1 --2- 172.21.15.28:0/3870019035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa68867c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/3870019035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 msgr2=0x55aa68867c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/3870019035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa68867c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/3870019035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 msgr2=0x55aa6885fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/3870019035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6885fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/3870019035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa6886ad90 con 0x55aa6885dd70 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.946+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/3870019035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68864e20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400c6a0 tx=0x7f4e94000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a1797ecddc8548 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4e857fa640 1 -- 172.21.15.28:0/3870019035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e940df070 con 0x55aa6885dd70 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4e857fa640 1 -- 172.21.15.28:0/3870019035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e9409bd10 con 0x55aa6885dd70 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4e857fa640 1 -- 172.21.15.28:0/3870019035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e940a0010 con 0x55aa6885dd70 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4ea8664740 1 -- 172.21.15.28:0/3870019035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 msgr2=0x55aa68864e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4ea8664740 1 --2- 172.21.15.28:0/3870019035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68864e20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400c6a0 tx=0x7f4e94000bc0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4ea8664740 1 -- 172.21.15.28:0/3870019035 shutdown_connections 2024-05-09T08:29:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4ea8664740 1 --2- 172.21.15.28:0/3870019035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa6885fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4ea8664740 1 --2- 172.21.15.28:0/3870019035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68864e20 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4ea8664740 1 --2- 172.21.15.28:0/3870019035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa68867c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.947+0000 7f4ea8664740 1 -- 172.21.15.28:0/3870019035 >> 172.21.15.28:0/3870019035 conn(0x55aa68859960 msgr2=0x55aa68858a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 -- 172.21.15.28:0/3870019035 shutdown_connections 2024-05-09T08:29:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 -- 172.21.15.28:0/3870019035 wait complete. 2024-05-09T08:29:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 -- start start 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 Processor -- start 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa68881570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68881ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68881ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01ae060 con 0x55aa6885dd70 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01ad990 con 0x55aa68868630 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01aabd0 con 0x55aa6885d450 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.948+0000 7f4ea8664740 10 client.0 _ll_register_callbacks cb 0x7f4e90009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa68881570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68881ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa68881570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51398/0 (socket says 172.21.15.28:51398) 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9effd640 1 -- 172.21.15.28:0/1187234877 learned_addr learned my addr 172.21.15.28:0/1187234877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68881ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56410/0 (socket says 172.21.15.28:56410) 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68881ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/1187234877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 msgr2=0x55aa68881570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa68881570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/1187234877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 msgr2=0x55aa68881ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68881ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9e7fc640 1 -- 172.21.15.28:0/1187234877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa688e10a0 con 0x55aa6885dd70 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9f7fe640 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68881ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e9e7fc640 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68881ab0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x55aa68853d40 tx=0x7f4e9409a220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0ffefa0a1d0726 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e7effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:29:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e7effd640 1 -- 172.21.15.28:0/1187234877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e940df050 con 0x55aa6885dd70 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e7effd640 1 -- 172.21.15.28:0/1187234877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f4e940cd110 con 0x55aa6885dd70 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4ea8664740 1 -- 172.21.15.28:0/1187234877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e9409bd10 con 0x55aa6885dd70 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4e7effd640 1 -- 172.21.15.28:0/1187234877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4e940b4e60 con 0x55aa6885dd70 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4ea8664740 10 client.16539 fetch_fsmap learned FSMap version 34 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.949+0000 7f4ea8664740 1 -- 172.21.15.28:0/1187234877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55aa688e2390 con 0x55aa6885dd70 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4e7effd640 1 -- 172.21.15.28:0/1187234877 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e9409bd10 con 0x55aa6885dd70 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4e7effd640 1 -- 172.21.15.28:0/1187234877 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4e940da040 con 0x55aa6885dd70 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4ea8664740 10 client.16539 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4ea8664740 10 client.16539 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4ea8664740 20 client.16539 populate_metadata read hostname 'smithi028' 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aa688e25c0 0x55aa688e49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4ea8664740 4 client.16539 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4ea8664740 1 -- 172.21.15.28:0/1187234877 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aa687c9dc0 con 0x55aa688e25c0 2024-05-09T08:29:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4e9effd640 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aa688e25c0 0x55aa688e49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4e9effd640 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aa688e25c0 0x55aa688e49b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46e8a3ea8b21b09e server_cookie=fc89573547576b24 in_seq=0 out_seq=0 2024-05-09T08:29:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:02.950+0000 7f4e7effd640 10 client.16539 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.793+0000 7f4e7effd640 1 -- 172.21.15.28:0/1187234877 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aa687c9dc0 con 0x55aa688e25c0 2024-05-09T08:29:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.793+0000 7f4e7effd640 10 client.16539 handle_command_reply: tid=0 2024-05-09T08:29:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.793+0000 7f4ea8664740 1 client.16539 shutdown 2024-05-09T08:29:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 -- 172.21.15.28:0/1187234877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 msgr2=0x55aa68881ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68881ab0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x55aa68853d40 tx=0x7f4e9409a220 comp rx=0 tx=0).stop 2024-05-09T08:29:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 -- 172.21.15.28:0/1187234877 shutdown_connections 2024-05-09T08:29:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aa688e25c0 0x55aa688e49b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa68868630 0x55aa68881ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6885dd70 0x55aa68881ab0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 --2- 172.21.15.28:0/1187234877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aa6885d450 0x55aa68881570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 -- 172.21.15.28:0/1187234877 >> 172.21.15.28:0/1187234877 conn(0x55aa68859960 msgr2=0x55aa68865500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4e7effd640 0 client.16539 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 -- 172.21.15.28:0/1187234877 shutdown_connections 2024-05-09T08:29:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.794+0000 7f4ea8664740 1 -- 172.21.15.28:0/1187234877 wait complete. 2024-05-09T08:29:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea8664740 20 client.16539 trim_cache size 0 max 16384 2024-05-09T08:29:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:29:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:29:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:29:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:29:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 -- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f4e880a2da0 msgr2=0x7f4e880a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f4e880a2da0 0x7f4e880a5260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e90005f60 tx=0x7f4e90005f90 comp rx=0 tx=0).stop 2024-05-09T08:29:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 -- 172.21.15.28:0/2867187051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ea0101120 msgr2=0x7f4ea01985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ea0101120 0x7f4ea01985e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400e9f0 tx=0x7f4e9400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:29:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 -- 172.21.15.28:0/2867187051 shutdown_connections 2024-05-09T08:29:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea0106720 0x7f4ea019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ea0101120 0x7f4ea01985e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f4e880a2da0 0x7f4e880a5260 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 --2- 172.21.15.28:0/2867187051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ea0100760 0x7f4ea01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 -- 172.21.15.28:0/2867187051 >> 172.21.15.28:0/2867187051 conn(0x7f4ea00778b0 msgr2=0x7f4ea00ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 -- 172.21.15.28:0/2867187051 shutdown_connections 2024-05-09T08:29:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:03.795+0000 7f4ea595a640 1 -- 172.21.15.28:0/2867187051 wait complete. 2024-05-09T08:29:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:03 smithi165 ceph-mon[34540]: pgmap v85: 129 pgs: 129 active+clean; 298 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.5 MiB/s wr, 391 op/s 2024-05-09T08:29:03.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:03 smithi184 ceph-mon[34509]: pgmap v85: 129 pgs: 129 active+clean; 298 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.5 MiB/s wr, 391 op/s 2024-05-09T08:29:03.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:03 smithi028 ceph-mon[30119]: pgmap v85: 129 pgs: 129 active+clean; 298 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.5 MiB/s wr, 391 op/s 2024-05-09T08:29:05.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:05 smithi165 ceph-mon[34540]: pgmap v86: 129 pgs: 129 active+clean; 306 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-05-09T08:29:05.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:05 smithi184 ceph-mon[34509]: pgmap v86: 129 pgs: 129 active+clean; 306 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-05-09T08:29:05.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:05 smithi028 ceph-mon[30119]: pgmap v86: 129 pgs: 129 active+clean; 306 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-05-09T08:29:07.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:07 smithi165 ceph-mon[34540]: pgmap v87: 129 pgs: 129 active+clean; 348 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 924 op/s 2024-05-09T08:29:07.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:07 smithi184 ceph-mon[34509]: pgmap v87: 129 pgs: 129 active+clean; 348 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 924 op/s 2024-05-09T08:29:07.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:07 smithi028 ceph-mon[30119]: pgmap v87: 129 pgs: 129 active+clean; 348 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 924 op/s 2024-05-09T08:29:08.708 INFO:tasks.workunit.client.0.smithi028.stdout: 152 1009591 1993 696849 1373 142532 1384 2024-05-09T08:29:09.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:09 smithi165 ceph-mon[34540]: pgmap v88: 129 pgs: 129 active+clean; 348 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 822 op/s 2024-05-09T08:29:09.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:09 smithi184 ceph-mon[34509]: pgmap v88: 129 pgs: 129 active+clean; 348 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 822 op/s 2024-05-09T08:29:09.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:09 smithi028 ceph-mon[30119]: pgmap v88: 129 pgs: 129 active+clean; 348 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 822 op/s 2024-05-09T08:29:11.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:11 smithi165 ceph-mon[34540]: pgmap v89: 129 pgs: 129 active+clean; 364 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 13 MiB/s wr, 897 op/s 2024-05-09T08:29:11.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:11 smithi184 ceph-mon[34509]: pgmap v89: 129 pgs: 129 active+clean; 364 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 13 MiB/s wr, 897 op/s 2024-05-09T08:29:11.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:11 smithi028 ceph-mon[30119]: pgmap v89: 129 pgs: 129 active+clean; 364 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 13 MiB/s wr, 897 op/s 2024-05-09T08:29:13.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:13 smithi165 ceph-mon[34540]: pgmap v90: 129 pgs: 129 active+clean; 364 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 632 op/s 2024-05-09T08:29:13.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:13 smithi184 ceph-mon[34509]: pgmap v90: 129 pgs: 129 active+clean; 364 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 632 op/s 2024-05-09T08:29:13.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:13 smithi028 ceph-mon[30119]: pgmap v90: 129 pgs: 129 active+clean; 364 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 632 op/s 2024-05-09T08:29:15.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:15 smithi165 ceph-mon[34540]: pgmap v91: 129 pgs: 129 active+clean; 369 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.8 MiB/s wr, 665 op/s 2024-05-09T08:29:15.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:15 smithi184 ceph-mon[34509]: pgmap v91: 129 pgs: 129 active+clean; 369 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.8 MiB/s wr, 665 op/s 2024-05-09T08:29:15.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:15 smithi028 ceph-mon[30119]: pgmap v91: 129 pgs: 129 active+clean; 369 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.8 MiB/s wr, 665 op/s 2024-05-09T08:29:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.980+0000 7f829d82a640 1 -- 172.21.15.28:0/4104731284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8298110400 msgr2=0x7f8298108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.980+0000 7f829d82a640 1 --2- 172.21.15.28:0/4104731284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8298110400 0x7f8298108d70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8290009c10 tx=0x7f829002a790 comp rx=0 tx=0).stop 2024-05-09T08:29:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.983+0000 7f829d82a640 1 -- 172.21.15.28:0/4104731284 shutdown_connections 2024-05-09T08:29:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.983+0000 7f829d82a640 1 --2- 172.21.15.28:0/4104731284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f829811b010 0x7f829811d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.983+0000 7f829d82a640 1 --2- 172.21.15.28:0/4104731284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8298110400 0x7f8298108d70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.983+0000 7f829d82a640 1 --2- 172.21.15.28:0/4104731284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8298110020 0x7f8298108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.983+0000 7f829d82a640 1 -- 172.21.15.28:0/4104731284 >> 172.21.15.28:0/4104731284 conn(0x7f829806c900 msgr2=0x7f829806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.989+0000 7f829d82a640 1 -- 172.21.15.28:0/4104731284 shutdown_connections 2024-05-09T08:29:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.994+0000 7f829d82a640 1 -- 172.21.15.28:0/4104731284 wait complete. 2024-05-09T08:29:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.994+0000 7f829d82a640 1 Processor -- start 2024-05-09T08:29:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.996+0000 7f829d82a640 1 -- start start 2024-05-09T08:29:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.997+0000 7f829d82a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8298110020 0x7f82981a6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.997+0000 7f829d82a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f829811b010 0x7f82981a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.997+0000 7f829d82a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82981ab840 0x7f82981abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.997+0000 7f829d82a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8298120ae0 con 0x7f8298110020 2024-05-09T08:29:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.997+0000 7f829d82a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8298120960 con 0x7f829811b010 2024-05-09T08:29:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.997+0000 7f829d82a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8298120c60 con 0x7f82981ab840 2024-05-09T08:29:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.998+0000 7f8296ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8298110020 0x7f82981a6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.998+0000 7f8296ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8298110020 0x7f82981a6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39878/0 (socket says 172.21.15.28:39878) 2024-05-09T08:29:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.998+0000 7f8296ffd640 1 -- 172.21.15.28:0/2322750930 learned_addr learned my addr 172.21.15.28:0/2322750930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.999+0000 7f8296ffd640 1 -- 172.21.15.28:0/2322750930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82981ab840 msgr2=0x7f82981abcf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:29:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.999+0000 7f8296ffd640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82981ab840 0x7f82981abcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.999+0000 7f8296ffd640 1 -- 172.21.15.28:0/2322750930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f829811b010 msgr2=0x7f82981a74c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:29:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.999+0000 7f8296ffd640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f829811b010 0x7f82981a74c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.999+0000 7f8296ffd640 1 -- 172.21.15.28:0/2322750930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82981ac3a0 con 0x7f8298110020 2024-05-09T08:29:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:16.999+0000 7f8296ffd640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8298110020 0x7f82981a6f80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f828c00d950 tx=0x7f828c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.001+0000 7f829c828640 1 -- 172.21.15.28:0/2322750930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c017070 con 0x7f8298110020 2024-05-09T08:29:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.001+0000 7f829d82a640 1 -- 172.21.15.28:0/2322750930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82981107f0 con 0x7f8298110020 2024-05-09T08:29:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.001+0000 7f829d82a640 1 -- 172.21.15.28:0/2322750930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8298110d40 con 0x7f8298110020 2024-05-09T08:29:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.001+0000 7f829c828640 1 -- 172.21.15.28:0/2322750930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f828c009c30 con 0x7f8298110020 2024-05-09T08:29:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.001+0000 7f829c828640 1 -- 172.21.15.28:0/2322750930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c00f690 con 0x7f8298110020 2024-05-09T08:29:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.007+0000 7f829c828640 1 -- 172.21.15.28:0/2322750930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f828c0109d0 con 0x7f8298110020 2024-05-09T08:29:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.007+0000 7f82a0615740 1 Processor -- start 2024-05-09T08:29:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.007+0000 7f82a0615740 1 -- start start 2024-05-09T08:29:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.008+0000 7f82a0615740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93d450 0x55e1db93d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.008+0000 7f82a0615740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db93dd70 0x55e1db9480f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.008+0000 7f82a0615740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db948630 0x55e1db94aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.008+0000 7f82a0615740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e1db8a2340 con 0x55e1db948630 2024-05-09T08:29:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.008+0000 7f82a0615740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e1db84a290 con 0x55e1db93d450 2024-05-09T08:29:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.008+0000 7f82a0615740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e1db89e300 con 0x55e1db93dd70 2024-05-09T08:29:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.008+0000 7f829c828640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f82800a2f40 0x7f82800a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.009+0000 7f829c828640 1 -- 172.21.15.28:0/2322750930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f828c0cef70 con 0x7f8298110020 2024-05-09T08:29:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.010+0000 7f82977fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93d450 0x55e1db93d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.010+0000 7f82977fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93d450 0x55e1db93d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:54836/0 (socket says 172.21.15.28:54836) 2024-05-09T08:29:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.010+0000 7f82977fe640 1 -- 172.21.15.28:0/2970082342 learned_addr learned my addr 172.21.15.28:0/2970082342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.010+0000 7f82977fe640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f82800a2f40 0x7f82800a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.010+0000 7f82967fc640 1 --2- 172.21.15.28:0/2970082342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db948630 0x55e1db94aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.011+0000 7f82977fe640 1 -- 172.21.15.28:0/2970082342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db93dd70 msgr2=0x55e1db9480f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:29:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.011+0000 7f82977fe640 1 --2- 172.21.15.28:0/2970082342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db93dd70 0x55e1db9480f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.011+0000 7f82977fe640 1 -- 172.21.15.28:0/2970082342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db948630 msgr2=0x55e1db94aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.011+0000 7f82977fe640 1 --2- 172.21.15.28:0/2970082342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db948630 0x55e1db94aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.011+0000 7f82977fe640 1 -- 172.21.15.28:0/2970082342 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1db86e8e0 con 0x55e1db93d450 2024-05-09T08:29:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.011+0000 7f82977fe640 1 --2- 172.21.15.28:0/2970082342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93d450 0x55e1db93d830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f82880146c0 tx=0x7f8288014b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1af571e8643d4403 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.012+0000 7f82977fe640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f82800a2f40 0x7f82800a5400 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8288005000 tx=0x7f8288019000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.017+0000 7f827dffb640 1 -- 172.21.15.28:0/2970082342 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8288011040 con 0x55e1db93d450 2024-05-09T08:29:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.017+0000 7f827dffb640 1 -- 172.21.15.28:0/2970082342 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f828801b070 con 0x55e1db93d450 2024-05-09T08:29:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.018+0000 7f82a0615740 1 -- 172.21.15.28:0/2970082342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93d450 msgr2=0x55e1db93d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.018+0000 7f82a0615740 1 --2- 172.21.15.28:0/2970082342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93d450 0x55e1db93d830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f82880146c0 tx=0x7f8288014b80 comp rx=0 tx=0).stop 2024-05-09T08:29:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.019+0000 7f82a0615740 1 -- 172.21.15.28:0/2970082342 shutdown_connections 2024-05-09T08:29:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.019+0000 7f82a0615740 1 --2- 172.21.15.28:0/2970082342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db948630 0x55e1db94aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.019+0000 7f82a0615740 1 --2- 172.21.15.28:0/2970082342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db93dd70 0x55e1db9480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.019+0000 7f82a0615740 1 --2- 172.21.15.28:0/2970082342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93d450 0x55e1db93d830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.019+0000 7f82a0615740 1 -- 172.21.15.28:0/2970082342 >> 172.21.15.28:0/2970082342 conn(0x55e1db939960 msgr2=0x55e1db939d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.021+0000 7f82a0615740 1 -- 172.21.15.28:0/2970082342 shutdown_connections 2024-05-09T08:29:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.022+0000 7f82a0615740 1 -- 172.21.15.28:0/2970082342 wait complete. 2024-05-09T08:29:17.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.022+0000 7f82a0615740 1 -- start start 2024-05-09T08:29:17.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.023+0000 7f82a0615740 1 Processor -- start 2024-05-09T08:29:17.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.023+0000 7f82a0615740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93dd70 0x55e1db948060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.023+0000 7f82a0615740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db948630 0x55e1db945350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.023+0000 7f82a0615740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 0x55e1db9546f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.023+0000 7f82a0615740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e1db94b200 con 0x55e1db9459f0 2024-05-09T08:29:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.023+0000 7f82a0615740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e1db85d790 con 0x55e1db93dd70 2024-05-09T08:29:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.023+0000 7f82a0615740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e1db94b380 con 0x55e1db948630 2024-05-09T08:29:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.024+0000 7f82a0615740 10 client.0 _ll_register_callbacks cb 0x55e1db898a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:29:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.024+0000 7f82967fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db948630 0x55e1db945350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.024+0000 7f82967fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db948630 0x55e1db945350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:58586/0 (socket says 172.21.15.28:58586) 2024-05-09T08:29:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.024+0000 7f82967fc640 1 -- 172.21.15.28:0/3615184821 learned_addr learned my addr 172.21.15.28:0/3615184821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.024+0000 7f82967fc640 1 -- 172.21.15.28:0/3615184821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93dd70 msgr2=0x55e1db948060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:29:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.024+0000 7f82967fc640 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93dd70 0x55e1db948060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.024+0000 7f82967fc640 1 -- 172.21.15.28:0/3615184821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 msgr2=0x55e1db9546f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:29:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.024+0000 7f82967fc640 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 0x55e1db9546f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.024+0000 7f82967fc640 1 -- 172.21.15.28:0/3615184821 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1db5b6210 con 0x55e1db948630 2024-05-09T08:29:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.025+0000 7f82967fc640 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db948630 0x55e1db945350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8290000f80 tx=0x7f8290004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4e35272fcea6b29 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.025+0000 7f8276ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:29:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.025+0000 7f8276ffd640 1 -- 172.21.15.28:0/3615184821 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290040030 con 0x55e1db948630 2024-05-09T08:29:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.025+0000 7f8276ffd640 1 -- 172.21.15.28:0/3615184821 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f82900043e0 con 0x55e1db948630 2024-05-09T08:29:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.026+0000 7f82a0615740 1 -- 172.21.15.28:0/3615184821 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e1db8afa70 con 0x55e1db948630 2024-05-09T08:29:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.027+0000 7f8276ffd640 1 -- 172.21.15.28:0/3615184821 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8290036040 con 0x55e1db948630 2024-05-09T08:29:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.027+0000 7f82a0615740 10 client.25918 fetch_fsmap learned FSMap version 34 2024-05-09T08:29:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.027+0000 7f82a0615740 1 -- 172.21.15.28:0/3615184821 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e1db682620 con 0x55e1db948630 2024-05-09T08:29:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.028+0000 7f8276ffd640 1 -- 172.21.15.28:0/3615184821 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8290009800 con 0x55e1db948630 2024-05-09T08:29:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.028+0000 7f8276ffd640 1 -- 172.21.15.28:0/3615184821 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f829004f420 con 0x55e1db948630 2024-05-09T08:29:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.028+0000 7f82a0615740 10 client.25918 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:29:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.028+0000 7f82a0615740 10 client.25918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:29:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.028+0000 7f82a0615740 20 client.25918 populate_metadata read hostname 'smithi028' 2024-05-09T08:29:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.029+0000 7f82a0615740 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e1db9585a0 0x55e1db95a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.029+0000 7f82a0615740 4 client.25918 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:29:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.029+0000 7f82a0615740 1 -- 172.21.15.28:0/3615184821 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1db8afa70 con 0x55e1db9585a0 2024-05-09T08:29:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.030+0000 7f8296ffd640 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e1db9585a0 0x55e1db95a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.033+0000 7f8276ffd640 10 client.25918 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.033+0000 7f8296ffd640 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e1db9585a0 0x55e1db95a9c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51cccd80f25dc4cb server_cookie=5104f7cd763f239b in_seq=0 out_seq=0 2024-05-09T08:29:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.042+0000 7f8276ffd640 1 -- 172.21.15.28:0/3615184821 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e1db8afa70 con 0x55e1db9585a0 2024-05-09T08:29:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.042+0000 7f8276ffd640 10 client.25918 handle_command_reply: tid=0 2024-05-09T08:29:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.043+0000 7f82a0615740 1 client.25918 shutdown 2024-05-09T08:29:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.045+0000 7f82a0615740 1 -- 172.21.15.28:0/3615184821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db948630 msgr2=0x55e1db945350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.045+0000 7f82a0615740 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db948630 0x55e1db945350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8290000f80 tx=0x7f8290004270 comp rx=0 tx=0).stop 2024-05-09T08:29:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.052+0000 7f82a0615740 1 -- 172.21.15.28:0/3615184821 shutdown_connections 2024-05-09T08:29:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.052+0000 7f82a0615740 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e1db9585a0 0x55e1db95a9c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.052+0000 7f82a0615740 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 0x55e1db9546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.052+0000 7f82a0615740 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db948630 0x55e1db945350 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.052+0000 7f82a0615740 1 --2- 172.21.15.28:0/3615184821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db93dd70 0x55e1db948060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.052+0000 7f8276ffd640 0 client.25918 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.052+0000 7f82a0615740 1 -- 172.21.15.28:0/3615184821 >> 172.21.15.28:0/3615184821 conn(0x55e1db939960 msgr2=0x55e1db93ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.053+0000 7f82a0615740 1 -- 172.21.15.28:0/3615184821 shutdown_connections 2024-05-09T08:29:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.053+0000 7f82a0615740 1 -- 172.21.15.28:0/3615184821 wait complete. 2024-05-09T08:29:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.053+0000 7f82a0615740 20 client.25918 trim_cache size 0 max 16384 2024-05-09T08:29:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.089+0000 7f82a0615740 1 Processor -- start 2024-05-09T08:29:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.089+0000 7f82a0615740 1 -- start start 2024-05-09T08:29:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.090+0000 7f82a0615740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db93dd70 0x55e1db9442b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.090+0000 7f82a0615740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db9459f0 0x55e1db938880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.090+0000 7f82a0615740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db960290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.090+0000 7f82a0615740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8284010100 con 0x55e1db93dd70 2024-05-09T08:29:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.090+0000 7f82a0615740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8284011d00 con 0x55e1db948630 2024-05-09T08:29:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.090+0000 7f82a0615740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f82840107d0 con 0x55e1db9459f0 2024-05-09T08:29:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.090+0000 7f82977fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db960290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.090+0000 7f82977fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db960290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:54842/0 (socket says 172.21.15.28:54842) 2024-05-09T08:29:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.090+0000 7f82977fe640 1 -- 172.21.15.28:0/856920418 learned_addr learned my addr 172.21.15.28:0/856920418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.091+0000 7f82967fc640 1 --2- 172.21.15.28:0/856920418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db9459f0 0x55e1db938880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.091+0000 7f8296ffd640 1 --2- 172.21.15.28:0/856920418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db93dd70 0x55e1db9442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.092+0000 7f82967fc640 1 -- 172.21.15.28:0/856920418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 msgr2=0x55e1db960290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.092+0000 7f82967fc640 1 --2- 172.21.15.28:0/856920418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db960290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.092+0000 7f82967fc640 1 -- 172.21.15.28:0/856920418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db93dd70 msgr2=0x55e1db9442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.092+0000 7f82967fc640 1 --2- 172.21.15.28:0/856920418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db93dd70 0x55e1db9442b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.092+0000 7f82967fc640 1 -- 172.21.15.28:0/856920418 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1db934c20 con 0x55e1db9459f0 2024-05-09T08:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.092+0000 7f82967fc640 1 --2- 172.21.15.28:0/856920418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db9459f0 0x55e1db938880 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f829004f870 tx=0x7f829002a790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34c4070e1046ed78 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.102+0000 7f827d7fa640 1 -- 172.21.15.28:0/856920418 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290040070 con 0x55e1db9459f0 2024-05-09T08:29:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.102+0000 7f827d7fa640 1 -- 172.21.15.28:0/856920418 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8290053030 con 0x55e1db9459f0 2024-05-09T08:29:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.102+0000 7f827d7fa640 1 -- 172.21.15.28:0/856920418 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290031440 con 0x55e1db9459f0 2024-05-09T08:29:17.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.104+0000 7f82a0615740 1 -- 172.21.15.28:0/856920418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db9459f0 msgr2=0x55e1db938880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:17.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.104+0000 7f82a0615740 1 --2- 172.21.15.28:0/856920418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db9459f0 0x55e1db938880 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f829004f870 tx=0x7f829002a790 comp rx=0 tx=0).stop 2024-05-09T08:29:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.108+0000 7f82a0615740 1 -- 172.21.15.28:0/856920418 shutdown_connections 2024-05-09T08:29:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.108+0000 7f82a0615740 1 --2- 172.21.15.28:0/856920418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db960290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.108+0000 7f82a0615740 1 --2- 172.21.15.28:0/856920418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db9459f0 0x55e1db938880 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.108+0000 7f82a0615740 1 --2- 172.21.15.28:0/856920418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db93dd70 0x55e1db9442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.108+0000 7f82a0615740 1 -- 172.21.15.28:0/856920418 >> 172.21.15.28:0/856920418 conn(0x55e1db939960 msgr2=0x55e1db938170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:17.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.113+0000 7f82a0615740 1 -- 172.21.15.28:0/856920418 shutdown_connections 2024-05-09T08:29:17.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.115+0000 7f82a0615740 1 -- 172.21.15.28:0/856920418 wait complete. 2024-05-09T08:29:17.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.115+0000 7f82a0615740 1 -- start start 2024-05-09T08:29:17.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.116+0000 7f82a0615740 1 Processor -- start 2024-05-09T08:29:17.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.116+0000 7f82a0615740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db93dd70 0x55e1db950b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.116+0000 7f82a0615740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 0x55e1db942820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.116+0000 7f82a0615740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db942e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.116+0000 7f82a0615740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8284011570 con 0x55e1db9459f0 2024-05-09T08:29:17.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.116+0000 7f82a0615740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8284010ea0 con 0x55e1db948630 2024-05-09T08:29:17.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.116+0000 7f82a0615740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f828400e0e0 con 0x55e1db93dd70 2024-05-09T08:29:17.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.116+0000 7f82a0615740 10 client.0 _ll_register_callbacks cb 0x7f8290034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82977fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db942e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82977fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db942e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:54854/0 (socket says 172.21.15.28:54854) 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82977fe640 1 -- 172.21.15.28:0/2586241870 learned_addr learned my addr 172.21.15.28:0/2586241870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82967fc640 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 0x55e1db942820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82977fe640 1 -- 172.21.15.28:0/2586241870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db93dd70 msgr2=0x55e1db950b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82977fe640 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db93dd70 0x55e1db950b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82977fe640 1 -- 172.21.15.28:0/2586241870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 msgr2=0x55e1db942820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82977fe640 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 0x55e1db942820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82977fe640 1 -- 172.21.15.28:0/2586241870 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1db953c10 con 0x55e1db948630 2024-05-09T08:29:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.117+0000 7f82967fc640 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 0x55e1db942820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:29:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.119+0000 7f82977fe640 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db942e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f82880125c0 tx=0x7f8288004ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96a1b7694b9c670d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.123+0000 7f8276ffd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:29:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.123+0000 7f8276ffd640 1 -- 172.21.15.28:0/2586241870 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828802c060 con 0x55e1db948630 2024-05-09T08:29:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.123+0000 7f82a0615740 1 -- 172.21.15.28:0/2586241870 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8290053030 con 0x55e1db948630 2024-05-09T08:29:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.123+0000 7f8276ffd640 1 -- 172.21.15.28:0/2586241870 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f8288006760 con 0x55e1db948630 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.124+0000 7f8276ffd640 1 -- 172.21.15.28:0/2586241870 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8288033050 con 0x55e1db948630 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.125+0000 7f82a0615740 10 client.26024 fetch_fsmap learned FSMap version 34 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.125+0000 7f82a0615740 1 -- 172.21.15.28:0/2586241870 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e1db9bee40 con 0x55e1db948630 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.126+0000 7f8276ffd640 1 -- 172.21.15.28:0/2586241870 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8290053030 con 0x55e1db948630 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.126+0000 7f8276ffd640 1 -- 172.21.15.28:0/2586241870 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f8288012ca0 con 0x55e1db948630 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.126+0000 7f82a0615740 10 client.26024 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.126+0000 7f82a0615740 10 client.26024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.126+0000 7f82a0615740 20 client.26024 populate_metadata read hostname 'smithi028' 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.126+0000 7f82a0615740 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e1db9bf630 0x55e1db9c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.126+0000 7f82a0615740 4 client.26024 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:29:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.126+0000 7f82a0615740 1 -- 172.21.15.28:0/2586241870 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e1db8a9dc0 con 0x55e1db9bf630 2024-05-09T08:29:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.133+0000 7f8296ffd640 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e1db9bf630 0x55e1db9c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:17.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.142+0000 7f8296ffd640 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e1db9bf630 0x55e1db9c1a50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7223369f035ea46 server_cookie=7641f53e6f91a11d in_seq=0 out_seq=0 2024-05-09T08:29:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:17.142+0000 7f8276ffd640 10 client.26024 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:17.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:17 smithi165 ceph-mon[34540]: pgmap v92: 129 pgs: 129 active+clean; 378 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.6 MiB/s wr, 483 op/s 2024-05-09T08:29:17.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:17 smithi184 ceph-mon[34509]: pgmap v92: 129 pgs: 129 active+clean; 378 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.6 MiB/s wr, 483 op/s 2024-05-09T08:29:17.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:17 smithi028 ceph-mon[30119]: pgmap v92: 129 pgs: 129 active+clean; 378 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.6 MiB/s wr, 483 op/s 2024-05-09T08:29:18.708 INFO:tasks.workunit.client.0.smithi028.stdout: 181 553290 1417 376235 1675 97167 952 2024-05-09T08:29:19.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:19 smithi165 ceph-mon[34540]: pgmap v93: 129 pgs: 129 active+clean; 378 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 140 op/s 2024-05-09T08:29:19.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:19 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:19.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:19 smithi184 ceph-mon[34509]: pgmap v93: 129 pgs: 129 active+clean; 378 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 140 op/s 2024-05-09T08:29:19.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:19 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:19.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:19 smithi028 ceph-mon[30119]: pgmap v93: 129 pgs: 129 active+clean; 378 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 140 op/s 2024-05-09T08:29:19.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:19 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.043+0000 7f8276ffd640 1 -- 172.21.15.28:0/2586241870 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e1db8a9dc0 con 0x55e1db9bf630 2024-05-09T08:29:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.043+0000 7f8276ffd640 10 client.26024 handle_command_reply: tid=0 2024-05-09T08:29:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.043+0000 7f82a0615740 1 client.26024 shutdown 2024-05-09T08:29:20.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.100+0000 7f82a0615740 1 -- 172.21.15.28:0/2586241870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 msgr2=0x55e1db942e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:20.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.100+0000 7f82a0615740 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db942e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f82880125c0 tx=0x7f8288004ea0 comp rx=0 tx=0).stop 2024-05-09T08:29:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.102+0000 7f82a0615740 1 -- 172.21.15.28:0/2586241870 shutdown_connections 2024-05-09T08:29:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.102+0000 7f82a0615740 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e1db9bf630 0x55e1db9c1a50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.102+0000 7f82a0615740 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e1db948630 0x55e1db942e50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.102+0000 7f82a0615740 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1db9459f0 0x55e1db942820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.102+0000 7f82a0615740 1 --2- 172.21.15.28:0/2586241870 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e1db93dd70 0x55e1db950b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.102+0000 7f82a0615740 1 -- 172.21.15.28:0/2586241870 >> 172.21.15.28:0/2586241870 conn(0x55e1db939960 msgr2=0x55e1db938a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.102+0000 7f8276ffd640 0 client.26024 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.102+0000 7f82a0615740 1 -- 172.21.15.28:0/2586241870 shutdown_connections 2024-05-09T08:29:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.102+0000 7f82a0615740 1 -- 172.21.15.28:0/2586241870 wait complete. 2024-05-09T08:29:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.107+0000 7f82a0615740 20 client.26024 trim_cache size 0 max 16384 2024-05-09T08:29:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:29:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:29:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:29:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:29:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.148+0000 7f829d82a640 1 -- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f82800a2f40 msgr2=0x7f82800a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.148+0000 7f829d82a640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f82800a2f40 0x7f82800a5400 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8288005000 tx=0x7f8288019000 comp rx=0 tx=0).stop 2024-05-09T08:29:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.148+0000 7f829d82a640 1 -- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8298110020 msgr2=0x7f82981a6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.148+0000 7f829d82a640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8298110020 0x7f82981a6f80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f828c00d950 tx=0x7f828c00de10 comp rx=0 tx=0).stop 2024-05-09T08:29:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.148+0000 7f829d82a640 1 -- 172.21.15.28:0/2322750930 shutdown_connections 2024-05-09T08:29:20.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.149+0000 7f829d82a640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f82800a2f40 0x7f82800a5400 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:20.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.149+0000 7f829d82a640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82981ab840 0x7f82981abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:20.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.149+0000 7f829d82a640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f829811b010 0x7f82981a74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:20.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.149+0000 7f829d82a640 1 --2- 172.21.15.28:0/2322750930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8298110020 0x7f82981a6f80 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:20.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.149+0000 7f829d82a640 1 -- 172.21.15.28:0/2322750930 >> 172.21.15.28:0/2322750930 conn(0x7f829806c900 msgr2=0x7f829810dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:20.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.149+0000 7f829d82a640 1 -- 172.21.15.28:0/2322750930 shutdown_connections 2024-05-09T08:29:20.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:20.149+0000 7f829d82a640 1 -- 172.21.15.28:0/2322750930 wait complete. 2024-05-09T08:29:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:21 smithi165 ceph-mon[34540]: pgmap v94: 129 pgs: 129 active+clean; 526 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 16 MiB/s wr, 1.12k op/s 2024-05-09T08:29:21.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:21 smithi184 ceph-mon[34509]: pgmap v94: 129 pgs: 129 active+clean; 526 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 16 MiB/s wr, 1.12k op/s 2024-05-09T08:29:21.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:21 smithi028 ceph-mon[30119]: pgmap v94: 129 pgs: 129 active+clean; 526 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 16 MiB/s wr, 1.12k op/s 2024-05-09T08:29:22.258 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:21 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[63978]: *** Caught signal (Segmentation fault) ** 2024-05-09T08:29:22.259 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:21 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[63978]: in thread 7fb1d1a4c700 thread_name:MR_Finisher 2024-05-09T08:29:22.259 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:22 smithi165 podman[68014]: 2024-05-09 08:29:22.060307238 +0000 UTC m=+0.049159784 container died 5a02348fcbf90a5d78f3ecc7c42eec7864d8721fe8df9c06683b34486e040720 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-09T08:29:23.101 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:22 smithi165 podman[68014]: 2024-05-09 08:29:22.847576999 +0000 UTC m=+0.836429541 container remove 5a02348fcbf90a5d78f3ecc7c42eec7864d8721fe8df9c06683b34486e040720 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True) 2024-05-09T08:29:23.101 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:22 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Main process exited, code=exited, status=139/n/a 2024-05-09T08:29:23.508 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:23 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-09T08:29:23.508 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:23 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Consumed 12.498s CPU time. 2024-05-09T08:29:23.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:23 smithi165 ceph-mon[34540]: pgmap v95: 129 pgs: 129 active+clean; 526 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-05-09T08:29:23.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:23 smithi184 ceph-mon[34509]: pgmap v95: 129 pgs: 129 active+clean; 526 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-05-09T08:29:23.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:23 smithi028 ceph-mon[30119]: pgmap v95: 129 pgs: 129 active+clean; 526 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-05-09T08:29:24.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:24 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:24.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:24 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:24.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:24 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:25.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:25 smithi165 ceph-mon[34540]: pgmap v96: 129 pgs: 129 active+clean; 546 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 16 MiB/s wr, 1.15k op/s 2024-05-09T08:29:25.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:25 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:25.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:25 smithi184 ceph-mon[34509]: pgmap v96: 129 pgs: 129 active+clean; 546 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 16 MiB/s wr, 1.15k op/s 2024-05-09T08:29:25.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:25 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:25.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:25 smithi028 ceph-mon[30119]: pgmap v96: 129 pgs: 129 active+clean; 546 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 16 MiB/s wr, 1.15k op/s 2024-05-09T08:29:25.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:25 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:26.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:26 smithi165 ceph-mon[34540]: pgmap v97: 129 pgs: 129 active+clean; 594 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 20 MiB/s wr, 1.42k op/s 2024-05-09T08:29:26.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:26 smithi184 ceph-mon[34509]: pgmap v97: 129 pgs: 129 active+clean; 594 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 20 MiB/s wr, 1.42k op/s 2024-05-09T08:29:26.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:26 smithi028 ceph-mon[30119]: pgmap v97: 129 pgs: 129 active+clean; 594 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 20 MiB/s wr, 1.42k op/s 2024-05-09T08:29:28.708 INFO:tasks.workunit.client.0.smithi028.stdout: 201 1615328 1150 1093972 1071 303504 1241 2024-05-09T08:29:29.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:29 smithi184 ceph-mon[34509]: pgmap v98: 129 pgs: 129 active+clean; 594 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 20 MiB/s wr, 1.39k op/s 2024-05-09T08:29:29.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:29 smithi028 ceph-mon[30119]: pgmap v98: 129 pgs: 129 active+clean; 594 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 20 MiB/s wr, 1.39k op/s 2024-05-09T08:29:29.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:29 smithi165 ceph-mon[34540]: pgmap v98: 129 pgs: 129 active+clean; 594 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 20 MiB/s wr, 1.39k op/s 2024-05-09T08:29:31.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:31 smithi184 ceph-mon[34509]: pgmap v99: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 21 MiB/s wr, 1.49k op/s 2024-05-09T08:29:31.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:31 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:31.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:31 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:31.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:31 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:29:31.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:31 smithi028 ceph-mon[30119]: pgmap v99: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 21 MiB/s wr, 1.49k op/s 2024-05-09T08:29:31.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:31 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:31.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:31 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:31.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:31 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:29:31.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:31 smithi165 ceph-mon[34540]: pgmap v99: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 21 MiB/s wr, 1.49k op/s 2024-05-09T08:29:31.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:31 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:31.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:31 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:31.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:31 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:29:33.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:33 smithi165 ceph-mon[34540]: pgmap v100: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 8.1 MiB/s wr, 510 op/s 2024-05-09T08:29:33.366 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:33 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-05-09T08:29:33.366 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:33 smithi165 systemd[1]: Stopped Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:29:33.366 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:33 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Consumed 12.498s CPU time. 2024-05-09T08:29:33.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:33 smithi028 ceph-mon[30119]: pgmap v100: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 8.1 MiB/s wr, 510 op/s 2024-05-09T08:29:33.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:33 smithi184 ceph-mon[34509]: pgmap v100: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 8.1 MiB/s wr, 510 op/s 2024-05-09T08:29:33.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:33 smithi165 systemd[1]: Starting Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:29:33.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:33 smithi165 podman[68245]: 2024-05-09 08:29:33.501481276 +0000 UTC m=+0.018565750 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:29:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.890+0000 7f92e589a640 1 -- 172.21.15.28:0/3112611076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 msgr2=0x7f92e010df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.890+0000 7f92e589a640 1 --2- 172.21.15.28:0/3112611076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 0x7f92e010df80 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f92d401ab70 tx=0x7f92d403b6d0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.891+0000 7f92e589a640 1 -- 172.21.15.28:0/3112611076 shutdown_connections 2024-05-09T08:29:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.891+0000 7f92e589a640 1 --2- 172.21.15.28:0/3112611076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92e010e5a0 0x7f92e0108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.891+0000 7f92e589a640 1 --2- 172.21.15.28:0/3112611076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 0x7f92e010df80 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.891+0000 7f92e589a640 1 --2- 172.21.15.28:0/3112611076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92e010fda0 0x7f92e0110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.891+0000 7f92e589a640 1 -- 172.21.15.28:0/3112611076 >> 172.21.15.28:0/3112611076 conn(0x7f92e006c7e0 msgr2=0x7f92e006cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.892+0000 7f92e589a640 1 -- 172.21.15.28:0/3112611076 shutdown_connections 2024-05-09T08:29:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.892+0000 7f92e589a640 1 -- 172.21.15.28:0/3112611076 wait complete. 2024-05-09T08:29:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.893+0000 7f92e589a640 1 Processor -- start 2024-05-09T08:29:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.893+0000 7f92e589a640 1 -- start start 2024-05-09T08:29:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.893+0000 7f92e589a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 0x7f92e019e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.893+0000 7f92e589a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92e010e5a0 0x7f92e019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.893+0000 7f92e589a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92e010fda0 0x7f92e01a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.893+0000 7f92e589a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92e0112150 con 0x7f92e010db00 2024-05-09T08:29:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.893+0000 7f92e589a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f92e0111fd0 con 0x7f92e010e5a0 2024-05-09T08:29:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.893+0000 7f92e589a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f92e01122d0 con 0x7f92e010fda0 2024-05-09T08:29:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.894+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92e010fda0 0x7f92e01a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.894+0000 7f92deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 0x7f92e019e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.894+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92e010fda0 0x7f92e01a2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:35550/0 (socket says 172.21.15.28:35550) 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.894+0000 7f92df7fe640 1 -- 172.21.15.28:0/1844954528 learned_addr learned my addr 172.21.15.28:0/1844954528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.894+0000 7f92deffd640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 0x7f92e019e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38284/0 (socket says 172.21.15.28:38284) 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.894+0000 7f92de7fc640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92e010e5a0 0x7f92e019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.895+0000 7f92deffd640 1 -- 172.21.15.28:0/1844954528 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92e010fda0 msgr2=0x7f92e01a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.895+0000 7f92deffd640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92e010fda0 0x7f92e01a2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.895+0000 7f92deffd640 1 -- 172.21.15.28:0/1844954528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92e010e5a0 msgr2=0x7f92e019eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.895+0000 7f92deffd640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92e010e5a0 0x7f92e019eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.895+0000 7f92deffd640 1 -- 172.21.15.28:0/1844954528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e01a35e0 con 0x7f92e010db00 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.895+0000 7f92deffd640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 0x7f92e019e5f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f92d800bf90 tx=0x7f92d8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.895+0000 7f92df7fe640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92e010fda0 0x7f92e01a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:29:34.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.914+0000 7f92e4898640 1 -- 172.21.15.28:0/1844954528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8013600 con 0x7f92e010db00 2024-05-09T08:29:34.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.919+0000 7f92e589a640 1 -- 172.21.15.28:0/1844954528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e01a38d0 con 0x7f92e010db00 2024-05-09T08:29:34.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.919+0000 7f92e589a640 1 -- 172.21.15.28:0/1844954528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e01ab160 con 0x7f92e010db00 2024-05-09T08:29:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.932+0000 7f92e85a3740 1 Processor -- start 2024-05-09T08:29:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.932+0000 7f92e85a3740 1 -- start start 2024-05-09T08:29:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.932+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19600 0x564664b199e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.932+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19f20 0x564664b242a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.932+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b26bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.932+0000 7f92e85a3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564664a7e340 con 0x564664b19600 2024-05-09T08:29:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.933+0000 7f92e85a3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x564664a26290 con 0x564664b19f20 2024-05-09T08:29:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.933+0000 7f92e85a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564664a7a300 con 0x564664b247e0 2024-05-09T08:29:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.933+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19600 0x564664b199e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.933+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19600 0x564664b199e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38300/0 (socket says 172.21.15.28:38300) 2024-05-09T08:29:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.933+0000 7f92df7fe640 1 -- 172.21.15.28:0/1988802008 learned_addr learned my addr 172.21.15.28:0/1988802008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.933+0000 7f92deffd640 1 --2- 172.21.15.28:0/1988802008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19f20 0x564664b242a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.934+0000 7f92df7fe640 1 -- 172.21.15.28:0/1988802008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 msgr2=0x564664b26bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:29:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.934+0000 7f92df7fe640 1 --2- 172.21.15.28:0/1988802008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b26bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.934+0000 7f92df7fe640 1 -- 172.21.15.28:0/1988802008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19f20 msgr2=0x564664b242a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.934+0000 7f92df7fe640 1 --2- 172.21.15.28:0/1988802008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19f20 0x564664b242a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.934+0000 7f92df7fe640 1 -- 172.21.15.28:0/1988802008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564664a4a8e0 con 0x564664b19600 2024-05-09T08:29:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.934+0000 7f92df7fe640 1 --2- 172.21.15.28:0/1988802008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19600 0x564664b199e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f92d0007810 tx=0x7f92d0007cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1469b6bdff126383 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.943+0000 7f92e4898640 1 -- 172.21.15.28:0/1844954528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92d8013c80 con 0x7f92e010db00 2024-05-09T08:29:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.943+0000 7f92e4898640 1 -- 172.21.15.28:0/1844954528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8011300 con 0x7f92e010db00 2024-05-09T08:29:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.943+0000 7f92e4898640 1 -- 172.21.15.28:0/1844954528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f92d8011560 con 0x7f92e010db00 2024-05-09T08:29:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.944+0000 7f92e4898640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f92c80a2f30 0x7f92c80a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.945+0000 7f92deffd640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f92c80a2f30 0x7f92c80a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.949+0000 7f92deffd640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f92c80a2f30 0x7f92c80a53f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x564664b1aba0 tx=0x7f92d80bd040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.949+0000 7f92e4898640 1 -- 172.21.15.28:0/1844954528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f92d80d0de0 con 0x7f92e010db00 2024-05-09T08:29:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.953+0000 7f92c5ffb640 1 -- 172.21.15.28:0/1988802008 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d000d950 con 0x564664b19600 2024-05-09T08:29:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.953+0000 7f92c5ffb640 1 -- 172.21.15.28:0/1988802008 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92d0018070 con 0x564664b19600 2024-05-09T08:29:34.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.954+0000 7f92e85a3740 1 -- 172.21.15.28:0/1988802008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19600 msgr2=0x564664b199e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:34.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.954+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1988802008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19600 0x564664b199e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f92d0007810 tx=0x7f92d0007cd0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.961+0000 7f92e85a3740 1 -- 172.21.15.28:0/1988802008 shutdown_connections 2024-05-09T08:29:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.961+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1988802008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b26bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.961+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1988802008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19f20 0x564664b242a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.961+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1988802008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19600 0x564664b199e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.961+0000 7f92e85a3740 1 -- 172.21.15.28:0/1988802008 >> 172.21.15.28:0/1988802008 conn(0x564664b15b10 msgr2=0x564664b15ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.964+0000 7f92e85a3740 1 -- 172.21.15.28:0/1988802008 shutdown_connections 2024-05-09T08:29:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.964+0000 7f92e85a3740 1 -- 172.21.15.28:0/1988802008 wait complete. 2024-05-09T08:29:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.965+0000 7f92e85a3740 1 -- start start 2024-05-09T08:29:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.965+0000 7f92e85a3740 1 Processor -- start 2024-05-09T08:29:34.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.966+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b19600 0x564664b21650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.966+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.966+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b19600 0x564664b21650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.966+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b19600 0x564664b21650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:35580/0 (socket says 172.21.15.28:35580) 2024-05-09T08:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.966+0000 7f92df7fe640 1 -- 172.21.15.28:0/1129057589 learned_addr learned my addr 172.21.15.28:0/1129057589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.967+0000 7f92de7fc640 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.967+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b247e0 0x564664b220d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.967+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564664b27410 con 0x564664b19f20 2024-05-09T08:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.967+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x564664a39790 con 0x564664b247e0 2024-05-09T08:29:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.967+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564664b27590 con 0x564664b19600 2024-05-09T08:29:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.968+0000 7f92de7fc640 1 -- 172.21.15.28:0/1129057589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b19600 msgr2=0x564664b21650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.968+0000 7f92de7fc640 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b19600 0x564664b21650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.968+0000 7f92de7fc640 1 -- 172.21.15.28:0/1129057589 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b247e0 msgr2=0x564664b220d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:29:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.968+0000 7f92de7fc640 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b247e0 0x564664b220d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.968+0000 7f92de7fc640 1 -- 172.21.15.28:0/1129057589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564664792210 con 0x564664b19f20 2024-05-09T08:29:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.969+0000 7f92de7fc640 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21b90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f92d403bcc0 tx=0x7f92d4043dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e50ba87b2cafb11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.969+0000 7f92e85a3740 10 client.0 _ll_register_callbacks cb 0x564664a74a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:29:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.969+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92d0018070 con 0x564664b19f20 2024-05-09T08:29:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.980+0000 7f92c37f6640 10 client.16566 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:29:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.980+0000 7f92c37f6640 1 -- 172.21.15.28:0/1129057589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4046040 con 0x564664b19f20 2024-05-09T08:29:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.980+0000 7f92c37f6640 1 -- 172.21.15.28:0/1129057589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f92d4004250 con 0x564664b19f20 2024-05-09T08:29:34.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.980+0000 7f92c37f6640 1 -- 172.21.15.28:0/1129057589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f92d4004450 con 0x564664b19f20 2024-05-09T08:29:34.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.981+0000 7f92e85a3740 10 client.16566 fetch_fsmap learned FSMap version 36 2024-05-09T08:29:34.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.981+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=36}) -- 0x56466485e620 con 0x564664b19f20 2024-05-09T08:29:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.986+0000 7f92c37f6640 1 -- 172.21.15.28:0/1129057589 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7f92d4004fb0 con 0x564664b19f20 2024-05-09T08:29:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.986+0000 7f92e85a3740 10 client.16566 fetch_fsmap finished waiting for FSMap version 36 2024-05-09T08:29:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.986+0000 7f92e85a3740 10 client.16566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:29:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.986+0000 7f92e85a3740 20 client.16566 populate_metadata read hostname 'smithi028' 2024-05-09T08:29:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.986+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564664b347d0 0x564664b36c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.986+0000 7f92e85a3740 4 client.16566 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:29:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.986+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f92d0018070 con 0x564664b347d0 2024-05-09T08:29:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.987+0000 7f92df7fe640 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564664b347d0 0x564664b36c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.987+0000 7f92df7fe640 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564664b347d0 0x564664b36c90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b2435a35feb8fd9 server_cookie=d982ba0144fbf889 in_seq=0 out_seq=0 2024-05-09T08:29:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.988+0000 7f92c37f6640 10 client.16566 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.993+0000 7f92c37f6640 1 -- 172.21.15.28:0/1129057589 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f92d0018070 con 0x564664b347d0 2024-05-09T08:29:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.993+0000 7f92c37f6640 10 client.16566 handle_command_reply: tid=0 2024-05-09T08:29:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:34.993+0000 7f92e85a3740 1 client.16566 shutdown 2024-05-09T08:29:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.000+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 msgr2=0x564664b21b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.000+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21b90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f92d403bcc0 tx=0x7f92d4043dc0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.013+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 shutdown_connections 2024-05-09T08:29:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.013+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564664b347d0 0x564664b36c90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.013+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b247e0 0x564664b220d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.013+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21b90 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.013+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1129057589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b19600 0x564664b21650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.013+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 >> 172.21.15.28:0/1129057589 conn(0x564664b15b10 msgr2=0x564664b1b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.019+0000 7f92c37f6640 0 client.16566 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.019+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 shutdown_connections 2024-05-09T08:29:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.019+0000 7f92e85a3740 1 -- 172.21.15.28:0/1129057589 wait complete. 2024-05-09T08:29:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.019+0000 7f92e85a3740 20 client.16566 trim_cache size 0 max 16384 2024-05-09T08:29:35.114 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:34 smithi165 podman[68245]: 2024-05-09 08:29:34.857604252 +0000 UTC m=+1.374688720 container create eb338d2eadd20633527cb6a29d2986e9c9bdd543f8c1a2f310ca56315a3f5a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-09T08:29:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.260+0000 7f92e85a3740 1 Processor -- start 2024-05-09T08:29:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.261+0000 7f92e85a3740 1 -- start start 2024-05-09T08:29:35.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.266+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b23d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:35.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.266+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:35.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.266+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b23d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:35.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.267+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b23d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:38722/0 (socket says 172.21.15.28:38722) 2024-05-09T08:29:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.268+0000 7f92de7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.268+0000 7f92de7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38324/0 (socket says 172.21.15.28:38324) 2024-05-09T08:29:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.268+0000 7f92df7fe640 1 -- 172.21.15.28:0/1525432305 learned_addr learned my addr 172.21.15.28:0/1525432305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.268+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1525432305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b30350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.268+0000 7f92e85a3740 1 -- 172.21.15.28:0/1525432305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc0100b0 con 0x564664b19f20 2024-05-09T08:29:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.268+0000 7f92e85a3740 1 -- 172.21.15.28:0/1525432305 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc011d70 con 0x564664b19600 2024-05-09T08:29:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.270+0000 7f92e85a3740 1 -- 172.21.15.28:0/1525432305 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc010840 con 0x564664b247e0 2024-05-09T08:29:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.271+0000 7f92de7fc640 1 -- 172.21.15.28:0/1525432305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 msgr2=0x564664b30350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.271+0000 7f92de7fc640 1 --2- 172.21.15.28:0/1525432305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b30350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.271+0000 7f92de7fc640 1 -- 172.21.15.28:0/1525432305 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 msgr2=0x564664b23d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.271+0000 7f92de7fc640 1 --2- 172.21.15.28:0/1525432305 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b23d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.271+0000 7f92de7fc640 1 -- 172.21.15.28:0/1525432305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564664b24560 con 0x564664b19f20 2024-05-09T08:29:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.272+0000 7f92de7fc640 1 --2- 172.21.15.28:0/1525432305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21ef0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f92d401aac0 tx=0x7f92d404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4ed0f10db5c317 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.306+0000 7f92c57fa640 1 -- 172.21.15.28:0/1525432305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4046040 con 0x564664b19f20 2024-05-09T08:29:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.306+0000 7f92c57fa640 1 -- 172.21.15.28:0/1525432305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92d4050070 con 0x564664b19f20 2024-05-09T08:29:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.307+0000 7f92c57fa640 1 -- 172.21.15.28:0/1525432305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4004ba0 con 0x564664b19f20 2024-05-09T08:29:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.313+0000 7f92e85a3740 1 -- 172.21.15.28:0/1525432305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 msgr2=0x564664b21ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.313+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1525432305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21ef0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f92d401aac0 tx=0x7f92d404e000 comp rx=0 tx=0).stop 2024-05-09T08:29:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.316+0000 7f92e85a3740 1 -- 172.21.15.28:0/1525432305 shutdown_connections 2024-05-09T08:29:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.316+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1525432305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b30350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.316+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1525432305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b21ef0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.316+0000 7f92e85a3740 1 --2- 172.21.15.28:0/1525432305 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b23d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.316+0000 7f92e85a3740 1 -- 172.21.15.28:0/1525432305 >> 172.21.15.28:0/1525432305 conn(0x564664b15b10 msgr2=0x564664b1c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.320+0000 7f92e85a3740 1 -- 172.21.15.28:0/1525432305 shutdown_connections 2024-05-09T08:29:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.320+0000 7f92e85a3740 1 -- 172.21.15.28:0/1525432305 wait complete. 2024-05-09T08:29:35.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.340+0000 7f92e85a3740 1 -- start start 2024-05-09T08:29:35.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.340+0000 7f92e85a3740 1 Processor -- start 2024-05-09T08:29:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.355+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b27a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.355+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b27fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.355+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b27a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.355+0000 7f92df7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b27a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:38732/0 (socket says 172.21.15.28:38732) 2024-05-09T08:29:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92de7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b27fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92de7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b27fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38330/0 (socket says 172.21.15.28:38330) 2024-05-09T08:29:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92e85a3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b92dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92e85a3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc0115e0 con 0x564664b19f20 2024-05-09T08:29:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92e85a3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc010f10 con 0x564664b19600 2024-05-09T08:29:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92e85a3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc00e090 con 0x564664b247e0 2024-05-09T08:29:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92df7fe640 1 -- 172.21.15.28:0/36507099 learned_addr learned my addr 172.21.15.28:0/36507099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:29:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92e85a3740 10 client.0 _ll_register_callbacks cb 0x564664b24730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:29:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92df7fe640 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b92dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92de7fc640 1 -- 172.21.15.28:0/36507099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 msgr2=0x564664b92dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.356+0000 7f92de7fc640 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b92dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.357+0000 7f92de7fc640 1 -- 172.21.15.28:0/36507099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 msgr2=0x564664b27a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:29:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.357+0000 7f92de7fc640 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b27a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:35.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.357+0000 7f92de7fc640 1 -- 172.21.15.28:0/36507099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564664b93390 con 0x564664b19f20 2024-05-09T08:29:35.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.357+0000 7f92de7fc640 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b27fd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f92d401ab70 tx=0x7f92d4005990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34aafa33b775a85b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.368+0000 7f92c37f6640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:29:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.368+0000 7f92c37f6640 1 -- 172.21.15.28:0/36507099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4071020 con 0x564664b19f20 2024-05-09T08:29:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.369+0000 7f92c37f6640 1 -- 172.21.15.28:0/36507099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f92d4019040 con 0x564664b19f20 2024-05-09T08:29:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.369+0000 7f92e85a3740 1 -- 172.21.15.28:0/36507099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92d4050070 con 0x564664b19f20 2024-05-09T08:29:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.372+0000 7f92c37f6640 1 -- 172.21.15.28:0/36507099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f92d4046040 con 0x564664b19f20 2024-05-09T08:29:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.401+0000 7f92e85a3740 10 client.16572 fetch_fsmap learned FSMap version 36 2024-05-09T08:29:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.401+0000 7f92e85a3740 1 -- 172.21.15.28:0/36507099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564664b9a730 con 0x564664b19f20 2024-05-09T08:29:35.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.402+0000 7f92c37f6640 1 -- 172.21.15.28:0/36507099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92d401a530 con 0x564664b19f20 2024-05-09T08:29:35.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.402+0000 7f92c37f6640 1 -- 172.21.15.28:0/36507099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8394+0+0 (secure 0 0 0) 0x7f92d4003ce0 con 0x564664b19f20 2024-05-09T08:29:35.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.402+0000 7f92e85a3740 10 client.16572 fetch_fsmap finished waiting for FSMap version 36 2024-05-09T08:29:35.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.402+0000 7f92e85a3740 10 client.16572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:29:35.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.402+0000 7f92e85a3740 20 client.16572 populate_metadata read hostname 'smithi028' 2024-05-09T08:29:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.402+0000 7f92e85a3740 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564664b9ac60 0x564664b9d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.403+0000 7f92e85a3740 4 client.16572 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:29:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.403+0000 7f92e85a3740 1 -- 172.21.15.28:0/36507099 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f92d4050070 con 0x564664b9ac60 2024-05-09T08:29:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.403+0000 7f92df7fe640 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564664b9ac60 0x564664b9d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.403+0000 7f92c37f6640 10 client.16572 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:35.403+0000 7f92df7fe640 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564664b9ac60 0x564664b9d080 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea9c717a5f4e3fb3 server_cookie=c3cbb7a0d7dd8f38 in_seq=0 out_seq=0 2024-05-09T08:29:35.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:35 smithi028 ceph-mon[30119]: pgmap v101: 129 pgs: 129 active+clean; 616 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 9.0 MiB/s wr, 545 op/s 2024-05-09T08:29:35.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:35 smithi165 ceph-mon[34540]: pgmap v101: 129 pgs: 129 active+clean; 616 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 9.0 MiB/s wr, 545 op/s 2024-05-09T08:29:35.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:35 smithi184 ceph-mon[34509]: pgmap v101: 129 pgs: 129 active+clean; 616 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 9.0 MiB/s wr, 545 op/s 2024-05-09T08:29:37.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:37 smithi165 ceph-mon[34540]: pgmap v102: 129 pgs: 129 active+clean; 624 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.6 MiB/s wr, 483 op/s 2024-05-09T08:29:37.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:37 smithi165 ceph-mon[34540]: daemon mds.e restarted 2024-05-09T08:29:37.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:36 smithi165 podman[68245]: 2024-05-09 08:29:36.986504702 +0000 UTC m=+3.503589184 container init eb338d2eadd20633527cb6a29d2986e9c9bdd543f8c1a2f310ca56315a3f5a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-09T08:29:37.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:36 smithi165 podman[68245]: 2024-05-09 08:29:36.996091751 +0000 UTC m=+3.513176228 container start eb338d2eadd20633527cb6a29d2986e9c9bdd543f8c1a2f310ca56315a3f5a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.vendor=CentOS) 2024-05-09T08:29:37.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:37 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68262]: starting mds.e at 2024-05-09T08:29:37.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:37 smithi028 ceph-mon[30119]: pgmap v102: 129 pgs: 129 active+clean; 624 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.6 MiB/s wr, 483 op/s 2024-05-09T08:29:37.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:37 smithi028 ceph-mon[30119]: daemon mds.e restarted 2024-05-09T08:29:37.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:37 smithi184 ceph-mon[34509]: pgmap v102: 129 pgs: 129 active+clean; 624 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.6 MiB/s wr, 483 op/s 2024-05-09T08:29:37.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:37 smithi184 ceph-mon[34509]: daemon mds.e restarted 2024-05-09T08:29:38.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:38 smithi165 bash[68245]: eb338d2eadd20633527cb6a29d2986e9c9bdd543f8c1a2f310ca56315a3f5a91 2024-05-09T08:29:38.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:38 smithi165 systemd[1]: Started Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:29:38.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:38 smithi165 ceph-mon[34540]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T08:29:38.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:38 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6834/1976787395,v1:172.21.15.165:6835/1976787395] up:boot 2024-05-09T08:29:38.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:38 smithi165 ceph-mon[34540]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:29:38.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:38 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:29:38.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:38 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:38.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:38 smithi184 ceph-mon[34509]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T08:29:38.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:38 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6834/1976787395,v1:172.21.15.165:6835/1976787395] up:boot 2024-05-09T08:29:38.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:38 smithi184 ceph-mon[34509]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:29:38.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:38 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:29:38.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:38 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:38.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:38 smithi028 ceph-mon[30119]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T08:29:38.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:38 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6834/1976787395,v1:172.21.15.165:6835/1976787395] up:boot 2024-05-09T08:29:38.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:38 smithi028 ceph-mon[30119]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:29:38.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:38 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:29:38.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:38 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:38.708 INFO:tasks.workunit.client.0.smithi028.stdout: 237 2172253 1995 1448449 1662 450031 972 2024-05-09T08:29:39.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:39 smithi165 ceph-mon[34540]: pgmap v104: 129 pgs: 129 active+clean; 624 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 211 op/s 2024-05-09T08:29:39.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:39 smithi184 ceph-mon[34509]: pgmap v104: 129 pgs: 129 active+clean; 624 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 211 op/s 2024-05-09T08:29:39.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:39 smithi028 ceph-mon[30119]: pgmap v104: 129 pgs: 129 active+clean; 624 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 211 op/s 2024-05-09T08:29:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:41 smithi165 ceph-mon[34540]: pgmap v105: 129 pgs: 129 active+clean; 678 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.8 MiB/s wr, 888 op/s 2024-05-09T08:29:41.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:41 smithi184 ceph-mon[34509]: pgmap v105: 129 pgs: 129 active+clean; 678 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.8 MiB/s wr, 888 op/s 2024-05-09T08:29:41.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:41 smithi028 ceph-mon[30119]: pgmap v105: 129 pgs: 129 active+clean; 678 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.8 MiB/s wr, 888 op/s 2024-05-09T08:29:43.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:43 smithi165 ceph-mon[34540]: pgmap v106: 129 pgs: 129 active+clean; 685 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.5 MiB/s wr, 1.02k op/s 2024-05-09T08:29:43.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:43 smithi184 ceph-mon[34509]: pgmap v106: 129 pgs: 129 active+clean; 685 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.5 MiB/s wr, 1.02k op/s 2024-05-09T08:29:43.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:43 smithi028 ceph-mon[30119]: pgmap v106: 129 pgs: 129 active+clean; 685 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.5 MiB/s wr, 1.02k op/s 2024-05-09T08:29:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92c37f6640 1 -- 172.21.15.28:0/36507099 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f92d4050070 con 0x564664b9ac60 2024-05-09T08:29:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92c37f6640 10 client.16572 handle_command_reply: tid=0 2024-05-09T08:29:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 client.16572 shutdown 2024-05-09T08:29:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 -- 172.21.15.28:0/36507099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 msgr2=0x564664b27fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b27fd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f92d401ab70 tx=0x7f92d4005990 comp rx=0 tx=0).stop 2024-05-09T08:29:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 -- 172.21.15.28:0/36507099 shutdown_connections 2024-05-09T08:29:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564664b9ac60 0x564664b9d080 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564664b247e0 0x564664b92dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564664b19f20 0x564664b27fd0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 --2- 172.21.15.28:0/36507099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564664b19600 0x564664b27a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 -- 172.21.15.28:0/36507099 >> 172.21.15.28:0/36507099 conn(0x564664b15b10 msgr2=0x564664b1afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92c37f6640 0 client.16572 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:29:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 -- 172.21.15.28:0/36507099 shutdown_connections 2024-05-09T08:29:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.300+0000 7f92e85a3740 1 -- 172.21.15.28:0/36507099 wait complete. 2024-05-09T08:29:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.301+0000 7f92e85a3740 20 client.16572 trim_cache size 0 max 16384 2024-05-09T08:29:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:29:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:29:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:29:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:29:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.301+0000 7f92e589a640 1 -- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f92c80a2f30 msgr2=0x7f92c80a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.301+0000 7f92e589a640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f92c80a2f30 0x7f92c80a53f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x564664b1aba0 tx=0x7f92d80bd040 comp rx=0 tx=0).stop 2024-05-09T08:29:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.301+0000 7f92e589a640 1 -- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 msgr2=0x7f92e019e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.301+0000 7f92e589a640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 0x7f92e019e5f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f92d800bf90 tx=0x7f92d8009f10 comp rx=0 tx=0).stop 2024-05-09T08:29:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.302+0000 7f92e589a640 1 -- 172.21.15.28:0/1844954528 shutdown_connections 2024-05-09T08:29:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.302+0000 7f92e589a640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f92c80a2f30 0x7f92c80a53f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.302+0000 7f92e589a640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92e010fda0 0x7f92e01a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.302+0000 7f92e589a640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92e010e5a0 0x7f92e019eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.302+0000 7f92e589a640 1 --2- 172.21.15.28:0/1844954528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e010db00 0x7f92e019e5f0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.302+0000 7f92e589a640 1 -- 172.21.15.28:0/1844954528 >> 172.21.15.28:0/1844954528 conn(0x7f92e006c7e0 msgr2=0x7f92e0109e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.302+0000 7f92e589a640 1 -- 172.21.15.28:0/1844954528 shutdown_connections 2024-05-09T08:29:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:29:44.302+0000 7f92e589a640 1 -- 172.21.15.28:0/1844954528 wait complete. 2024-05-09T08:29:45.101 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:44 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68262]: ignoring --setuser ceph since I am not root 2024-05-09T08:29:45.101 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:44 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68262]: ignoring --setgroup ceph since I am not root 2024-05-09T08:29:45.101 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:44 smithi165 ceph-mds[68267]: -- 172.21.15.165:0/1582956069 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5601c61344e0 con 0x5601c610a800 2024-05-09T08:29:45.101 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:44 smithi165 ceph-mds[68267]: -- 172.21.15.165:0/1582956069 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5601c6137a40 con 0x5601c610a800 2024-05-09T08:29:45.101 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:29:44 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68262]: starting mds.e at 2024-05-09T08:29:45.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:45 smithi165 ceph-mon[34540]: pgmap v107: 129 pgs: 129 active+clean; 706 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9.7 MiB/s wr, 1.09k op/s 2024-05-09T08:29:45.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:45 smithi165 ceph-mon[34540]: daemon mds.e restarted 2024-05-09T08:29:45.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:45 smithi184 ceph-mon[34509]: pgmap v107: 129 pgs: 129 active+clean; 706 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9.7 MiB/s wr, 1.09k op/s 2024-05-09T08:29:45.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:45 smithi184 ceph-mon[34509]: daemon mds.e restarted 2024-05-09T08:29:45.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:45 smithi028 ceph-mon[30119]: pgmap v107: 129 pgs: 129 active+clean; 706 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9.7 MiB/s wr, 1.09k op/s 2024-05-09T08:29:45.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:45 smithi028 ceph-mon[30119]: daemon mds.e restarted 2024-05-09T08:29:46.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:46 smithi165 ceph-mon[34540]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T08:29:46.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:46 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6834/2059122732,v1:172.21.15.165:6835/2059122732] up:boot 2024-05-09T08:29:46.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:46 smithi165 ceph-mon[34540]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:29:46.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:46 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:29:46.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:46 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:46.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:46 smithi184 ceph-mon[34509]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T08:29:46.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:46 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6834/2059122732,v1:172.21.15.165:6835/2059122732] up:boot 2024-05-09T08:29:46.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:46 smithi184 ceph-mon[34509]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:29:46.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:46 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:29:46.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:46 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:46.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:46 smithi028 ceph-mon[30119]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T08:29:46.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:46 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6834/2059122732,v1:172.21.15.165:6835/2059122732] up:boot 2024-05-09T08:29:46.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:46 smithi028 ceph-mon[30119]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:29:46.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:46 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:29:46.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:46 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:29:47.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:47 smithi165 ceph-mon[34540]: pgmap v109: 129 pgs: 129 active+clean; 802 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 21 MiB/s wr, 1.74k op/s 2024-05-09T08:29:47.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:47 smithi184 ceph-mon[34509]: pgmap v109: 129 pgs: 129 active+clean; 802 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 21 MiB/s wr, 1.74k op/s 2024-05-09T08:29:47.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:47 smithi028 ceph-mon[30119]: pgmap v109: 129 pgs: 129 active+clean; 802 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 21 MiB/s wr, 1.74k op/s 2024-05-09T08:29:48.708 INFO:tasks.workunit.client.0.smithi028.stdout: 245 1133760 501 754680 386 228426 372 2024-05-09T08:29:49.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:49 smithi165 ceph-mon[34540]: pgmap v110: 129 pgs: 129 active+clean; 802 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 19 MiB/s wr, 1.55k op/s 2024-05-09T08:29:49.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:49 smithi184 ceph-mon[34509]: pgmap v110: 129 pgs: 129 active+clean; 802 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 19 MiB/s wr, 1.55k op/s 2024-05-09T08:29:49.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:49 smithi028 ceph-mon[30119]: pgmap v110: 129 pgs: 129 active+clean; 802 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 19 MiB/s wr, 1.55k op/s 2024-05-09T08:29:51.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:51 smithi165 ceph-mon[34540]: pgmap v111: 129 pgs: 129 active+clean; 797 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 14 MiB/s wr, 837 op/s 2024-05-09T08:29:51.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:51 smithi184 ceph-mon[34509]: pgmap v111: 129 pgs: 129 active+clean; 797 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 14 MiB/s wr, 837 op/s 2024-05-09T08:29:51.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:51 smithi028 ceph-mon[30119]: pgmap v111: 129 pgs: 129 active+clean; 797 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 14 MiB/s wr, 837 op/s 2024-05-09T08:29:53.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:53 smithi165 ceph-mon[34540]: pgmap v112: 129 pgs: 129 active+clean; 798 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 14 MiB/s wr, 717 op/s 2024-05-09T08:29:53.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:53 smithi184 ceph-mon[34509]: pgmap v112: 129 pgs: 129 active+clean; 798 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 14 MiB/s wr, 717 op/s 2024-05-09T08:29:53.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:53 smithi028 ceph-mon[30119]: pgmap v112: 129 pgs: 129 active+clean; 798 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 14 MiB/s wr, 717 op/s 2024-05-09T08:29:55.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:55 smithi165 ceph-mon[34540]: pgmap v113: 129 pgs: 129 active+clean; 803 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 12 MiB/s wr, 639 op/s 2024-05-09T08:29:55.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:55 smithi184 ceph-mon[34509]: pgmap v113: 129 pgs: 129 active+clean; 803 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 12 MiB/s wr, 639 op/s 2024-05-09T08:29:55.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:55 smithi028 ceph-mon[30119]: pgmap v113: 129 pgs: 129 active+clean; 803 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 12 MiB/s wr, 639 op/s 2024-05-09T08:29:57.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:57 smithi165 ceph-mon[34540]: pgmap v114: 129 pgs: 129 active+clean; 807 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.4 MiB/s wr, 147 op/s 2024-05-09T08:29:57.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:57 smithi184 ceph-mon[34509]: pgmap v114: 129 pgs: 129 active+clean; 807 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.4 MiB/s wr, 147 op/s 2024-05-09T08:29:57.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:57 smithi028 ceph-mon[30119]: pgmap v114: 129 pgs: 129 active+clean; 807 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.4 MiB/s wr, 147 op/s 2024-05-09T08:29:58.708 INFO:tasks.workunit.client.0.smithi028.stdout: 282 1760727 1962 1161289 1867 349828 238 2024-05-09T08:29:59.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:29:59 smithi165 ceph-mon[34540]: pgmap v115: 129 pgs: 129 active+clean; 807 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 133 op/s 2024-05-09T08:29:59.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:29:59 smithi184 ceph-mon[34509]: pgmap v115: 129 pgs: 129 active+clean; 807 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 133 op/s 2024-05-09T08:29:59.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:29:59 smithi028 ceph-mon[30119]: pgmap v115: 129 pgs: 129 active+clean; 807 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 133 op/s 2024-05-09T08:30:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.052+0000 7fab28763640 1 -- 172.21.15.28:0/3118993231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab2010fda0 msgr2=0x7fab20110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.052+0000 7fab28763640 1 --2- 172.21.15.28:0/3118993231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab2010fda0 0x7fab20110180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fab18009be0 tx=0x7fab1802a710 comp rx=0 tx=0).stop 2024-05-09T08:30:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.054+0000 7fab28763640 1 -- 172.21.15.28:0/3118993231 shutdown_connections 2024-05-09T08:30:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.054+0000 7fab28763640 1 --2- 172.21.15.28:0/3118993231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab2010e5a0 0x7fab20108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.054+0000 7fab28763640 1 --2- 172.21.15.28:0/3118993231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab2010db00 0x7fab2010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.054+0000 7fab28763640 1 --2- 172.21.15.28:0/3118993231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab2010fda0 0x7fab20110180 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.054+0000 7fab28763640 1 -- 172.21.15.28:0/3118993231 >> 172.21.15.28:0/3118993231 conn(0x7fab2006c7e0 msgr2=0x7fab2006cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.065+0000 7fab28763640 1 -- 172.21.15.28:0/3118993231 shutdown_connections 2024-05-09T08:30:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.065+0000 7fab28763640 1 -- 172.21.15.28:0/3118993231 wait complete. 2024-05-09T08:30:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.066+0000 7fab28763640 1 Processor -- start 2024-05-09T08:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.066+0000 7fab28763640 1 -- start start 2024-05-09T08:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.066+0000 7fab28763640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab2010db00 0x7fab2011e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.066+0000 7fab28763640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab2010e5a0 0x7fab20117640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.066+0000 7fab28763640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab2010fda0 0x7fab20117b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.066+0000 7fab28763640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab201208f0 con 0x7fab2010e5a0 2024-05-09T08:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.066+0000 7fab28763640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fab20120770 con 0x7fab2010db00 2024-05-09T08:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.066+0000 7fab28763640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab20120a70 con 0x7fab2010fda0 2024-05-09T08:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.069+0000 7fab25cfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab2010e5a0 0x7fab20117640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.069+0000 7fab25cfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab2010e5a0 0x7fab20117640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51130/0 (socket says 172.21.15.28:51130) 2024-05-09T08:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.069+0000 7fab25cfe640 1 -- 172.21.15.28:0/1551926422 learned_addr learned my addr 172.21.15.28:0/1551926422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.069+0000 7fab26d00640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab2010fda0 0x7fab20117b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.070+0000 7fab264ff640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab2010db00 0x7fab2011e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.070+0000 7fab26d00640 1 -- 172.21.15.28:0/1551926422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab2010db00 msgr2=0x7fab2011e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.070+0000 7fab26d00640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab2010db00 0x7fab2011e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.070+0000 7fab26d00640 1 -- 172.21.15.28:0/1551926422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab2010e5a0 msgr2=0x7fab20117640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.070+0000 7fab26d00640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab2010e5a0 0x7fab20117640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.070+0000 7fab26d00640 1 -- 172.21.15.28:0/1551926422 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab201180f0 con 0x7fab2010fda0 2024-05-09T08:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.073+0000 7fab26d00640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab2010fda0 0x7fab20117b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab1000b3e0 tx=0x7fab1000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.117+0000 7fab177fe640 1 -- 172.21.15.28:0/1551926422 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10015020 con 0x7fab2010fda0 2024-05-09T08:30:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.117+0000 7fab28763640 1 -- 172.21.15.28:0/1551926422 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab201183e0 con 0x7fab2010fda0 2024-05-09T08:30:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.117+0000 7fab28763640 1 -- 172.21.15.28:0/1551926422 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2011c3d0 con 0x7fab2010fda0 2024-05-09T08:30:00.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.121+0000 7fab2b478740 1 Processor -- start 2024-05-09T08:30:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.122+0000 7fab2b478740 1 -- start start 2024-05-09T08:30:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.122+0000 7fab2b478740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.122+0000 7fab2b478740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd1ef0 0x55cba3fdc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.122+0000 7fab2b478740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fdeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.122+0000 7fab2b478740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cba3f36340 con 0x55cba3fd1ef0 2024-05-09T08:30:00.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.122+0000 7fab2b478740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55cba3ede290 con 0x55cba3fd15d0 2024-05-09T08:30:00.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.122+0000 7fab2b478740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cba3f32300 con 0x55cba3fdc7b0 2024-05-09T08:30:00.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.123+0000 7fab25cfe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.123+0000 7fab25cfe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:37776/0 (socket says 172.21.15.28:37776) 2024-05-09T08:30:00.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.123+0000 7fab25cfe640 1 -- 172.21.15.28:0/866168215 learned_addr learned my addr 172.21.15.28:0/866168215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.123+0000 7fab25cfe640 1 --2- 172.21.15.28:0/866168215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fdeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.123+0000 7fab25cfe640 1 -- 172.21.15.28:0/866168215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 msgr2=0x55cba3fdeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.123+0000 7fab25cfe640 1 --2- 172.21.15.28:0/866168215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fdeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.123+0000 7fab25cfe640 1 -- 172.21.15.28:0/866168215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd1ef0 msgr2=0x55cba3fdc270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:30:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.123+0000 7fab25cfe640 1 --2- 172.21.15.28:0/866168215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd1ef0 0x55cba3fdc270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.123+0000 7fab25cfe640 1 -- 172.21.15.28:0/866168215 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cba3f028e0 con 0x55cba3fd15d0 2024-05-09T08:30:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.124+0000 7fab25cfe640 1 --2- 172.21.15.28:0/866168215 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd19b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fab1c005ab0 tx=0x7fab1c009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8428b43c08012073 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.126+0000 7fab177fe640 1 -- 172.21.15.28:0/1551926422 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab10011d10 con 0x7fab2010fda0 2024-05-09T08:30:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.126+0000 7fab177fe640 1 -- 172.21.15.28:0/1551926422 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1000f2e0 con 0x7fab2010fda0 2024-05-09T08:30:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.127+0000 7fab177fe640 1 -- 172.21.15.28:0/1551926422 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fab10018020 con 0x7fab2010fda0 2024-05-09T08:30:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.128+0000 7fab177fe640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fab0c0a2f30 0x7fab0c0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.128+0000 7fab177fe640 1 -- 172.21.15.28:0/1551926422 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9495+0+0 (secure 0 0 0) 0x7fab100cffb0 con 0x7fab2010fda0 2024-05-09T08:30:00.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.129+0000 7fab264ff640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fab0c0a2f30 0x7fab0c0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.133+0000 7fab264ff640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fab0c0a2f30 0x7fab0c0a53f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fab180099a0 tx=0x7fab18005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.160+0000 7fab14ff9640 1 -- 172.21.15.28:0/866168215 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c004050 con 0x55cba3fd15d0 2024-05-09T08:30:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.160+0000 7fab14ff9640 1 -- 172.21.15.28:0/866168215 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab1c0041f0 con 0x55cba3fd15d0 2024-05-09T08:30:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.161+0000 7fab2b478740 1 -- 172.21.15.28:0/866168215 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 msgr2=0x55cba3fd19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.161+0000 7fab2b478740 1 --2- 172.21.15.28:0/866168215 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd19b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fab1c005ab0 tx=0x7fab1c009700 comp rx=0 tx=0).stop 2024-05-09T08:30:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.243+0000 7fab2b478740 1 -- 172.21.15.28:0/866168215 shutdown_connections 2024-05-09T08:30:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.243+0000 7fab2b478740 1 --2- 172.21.15.28:0/866168215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fdeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.243+0000 7fab2b478740 1 --2- 172.21.15.28:0/866168215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd1ef0 0x55cba3fdc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.243+0000 7fab2b478740 1 --2- 172.21.15.28:0/866168215 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd19b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.243+0000 7fab2b478740 1 -- 172.21.15.28:0/866168215 >> 172.21.15.28:0/866168215 conn(0x55cba3fcdae0 msgr2=0x55cba3fcdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.247+0000 7fab2b478740 1 -- 172.21.15.28:0/866168215 shutdown_connections 2024-05-09T08:30:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.247+0000 7fab2b478740 1 -- 172.21.15.28:0/866168215 wait complete. 2024-05-09T08:30:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.248+0000 7fab2b478740 1 -- start start 2024-05-09T08:30:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.248+0000 7fab2b478740 1 Processor -- start 2024-05-09T08:30:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.248+0000 7fab2b478740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.248+0000 7fab2b478740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd1ef0 0x55cba3fd9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.248+0000 7fab2b478740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fda0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.248+0000 7fab2b478740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cba3fdf3e0 con 0x55cba3fd1ef0 2024-05-09T08:30:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.248+0000 7fab2b478740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55cba3ef1790 con 0x55cba3fd15d0 2024-05-09T08:30:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.248+0000 7fab2b478740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55cba3fdf560 con 0x55cba3fdc7b0 2024-05-09T08:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.248+0000 7fab2b478740 10 client.0 _ll_register_callbacks cb 0x55cba3f2ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab26d00640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fda0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab26d00640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fda0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:36006/0 (socket says 172.21.15.28:36006) 2024-05-09T08:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab26d00640 1 -- 172.21.15.28:0/4087425556 learned_addr learned my addr 172.21.15.28:0/4087425556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab264ff640 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab26d00640 1 -- 172.21.15.28:0/4087425556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 msgr2=0x55cba3fd9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab26d00640 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab26d00640 1 -- 172.21.15.28:0/4087425556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd1ef0 msgr2=0x55cba3fd9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab26d00640 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd1ef0 0x55cba3fd9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab26d00640 1 -- 172.21.15.28:0/4087425556 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cba3c4a210 con 0x55cba3fdc7b0 2024-05-09T08:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.249+0000 7fab264ff640 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.250+0000 7fab26d00640 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fda0f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fab1000bda0 tx=0x7fab1009ca70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6781f967ee34027b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:00.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:30:00.001+0000 7f8912d3a700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 4 MDSs report damaged metadata 2024-05-09T08:30:00.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:30:00.001+0000 7f8912d3a700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 4 MDSs report damaged metadata 2024-05-09T08:30:00.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:30:00.001+0000 7f8912d3a700 -1 log_channel(cluster) log [ERR] : mds.f(mds.4): Metadata damage detected 2024-05-09T08:30:00.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:30:00.001+0000 7f8912d3a700 -1 log_channel(cluster) log [ERR] : mds.h(mds.3): Metadata damage detected 2024-05-09T08:30:00.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:30:00.001+0000 7f8912d3a700 -1 log_channel(cluster) log [ERR] : mds.c(mds.2): Metadata damage detected 2024-05-09T08:30:00.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:30:00.001+0000 7f8912d3a700 -1 log_channel(cluster) log [ERR] : mds.i(mds.1): Metadata damage detected 2024-05-09T08:30:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.312+0000 7fab09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:30:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.312+0000 7fab09ffb640 1 -- 172.21.15.28:0/4087425556 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100b5060 con 0x55cba3fdc7b0 2024-05-09T08:30:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.312+0000 7fab2b478740 1 -- 172.21.15.28:0/4087425556 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab1c0041f0 con 0x55cba3fdc7b0 2024-05-09T08:30:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.312+0000 7fab09ffb640 1 -- 172.21.15.28:0/4087425556 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9495+0+0 (secure 0 0 0) 0x7fab100a0170 con 0x55cba3fdc7b0 2024-05-09T08:30:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.312+0000 7fab09ffb640 1 -- 172.21.15.28:0/4087425556 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fab100b5200 con 0x55cba3fdc7b0 2024-05-09T08:30:00.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.312+0000 7fab2b478740 10 client.25948 fetch_fsmap learned FSMap version 40 2024-05-09T08:30:00.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.312+0000 7fab2b478740 1 -- 172.21.15.28:0/4087425556 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55cba3d16620 con 0x55cba3fdc7b0 2024-05-09T08:30:00.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.314+0000 7fab09ffb640 1 -- 172.21.15.28:0/4087425556 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab1c0041f0 con 0x55cba3fdc7b0 2024-05-09T08:30:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.315+0000 7fab09ffb640 1 -- 172.21.15.28:0/4087425556 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 40) ==== 8394+0+0 (secure 0 0 0) 0x7fab1009c5a0 con 0x55cba3fdc7b0 2024-05-09T08:30:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.315+0000 7fab2b478740 10 client.25948 fetch_fsmap finished waiting for FSMap version 40 2024-05-09T08:30:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.315+0000 7fab2b478740 10 client.25948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:30:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.315+0000 7fab2b478740 20 client.25948 populate_metadata read hostname 'smithi028' 2024-05-09T08:30:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.315+0000 7fab2b478740 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cba400eb90 0x55cba4011050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.315+0000 7fab2b478740 4 client.25948 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:30:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.315+0000 7fab2b478740 1 -- 172.21.15.28:0/4087425556 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cba3f43a70 con 0x55cba400eb90 2024-05-09T08:30:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.315+0000 7fab264ff640 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cba400eb90 0x55cba4011050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.316+0000 7fab264ff640 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cba400eb90 0x55cba4011050 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=556df70577329651 server_cookie=c2521ca6b068d371 in_seq=0 out_seq=0 2024-05-09T08:30:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.317+0000 7fab09ffb640 10 client.25948 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.318+0000 7fab09ffb640 1 -- 172.21.15.28:0/4087425556 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cba3f43a70 con 0x55cba400eb90 2024-05-09T08:30:00.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.318+0000 7fab09ffb640 10 client.25948 handle_command_reply: tid=0 2024-05-09T08:30:00.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.318+0000 7fab2b478740 1 client.25948 shutdown 2024-05-09T08:30:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 -- 172.21.15.28:0/4087425556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 msgr2=0x55cba3fda0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fda0f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fab1000bda0 tx=0x7fab1009ca70 comp rx=0 tx=0).stop 2024-05-09T08:30:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 -- 172.21.15.28:0/4087425556 shutdown_connections 2024-05-09T08:30:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cba400eb90 0x55cba4011050 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fda0f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd1ef0 0x55cba3fd9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 --2- 172.21.15.28:0/4087425556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd15d0 0x55cba3fd9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 -- 172.21.15.28:0/4087425556 >> 172.21.15.28:0/4087425556 conn(0x55cba3fcdae0 msgr2=0x55cba3fd3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab09ffb640 0 client.25948 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 -- 172.21.15.28:0/4087425556 shutdown_connections 2024-05-09T08:30:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.322+0000 7fab2b478740 1 -- 172.21.15.28:0/4087425556 wait complete. 2024-05-09T08:30:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.323+0000 7fab2b478740 20 client.25948 trim_cache size 0 max 16384 2024-05-09T08:30:00.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:00 smithi165 ceph-mon[34540]: Health detail: HEALTH_ERR 4 MDSs report damaged metadata 2024-05-09T08:30:00.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:00 smithi165 ceph-mon[34540]: [ERR] MDS_DAMAGE: 4 MDSs report damaged metadata 2024-05-09T08:30:00.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:00 smithi165 ceph-mon[34540]: mds.f(mds.4): Metadata damage detected 2024-05-09T08:30:00.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:00 smithi165 ceph-mon[34540]: mds.h(mds.3): Metadata damage detected 2024-05-09T08:30:00.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:00 smithi165 ceph-mon[34540]: mds.c(mds.2): Metadata damage detected 2024-05-09T08:30:00.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:00 smithi165 ceph-mon[34540]: mds.i(mds.1): Metadata damage detected 2024-05-09T08:30:00.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:00 smithi184 ceph-mon[34509]: Health detail: HEALTH_ERR 4 MDSs report damaged metadata 2024-05-09T08:30:00.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:00 smithi184 ceph-mon[34509]: [ERR] MDS_DAMAGE: 4 MDSs report damaged metadata 2024-05-09T08:30:00.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:00 smithi184 ceph-mon[34509]: mds.f(mds.4): Metadata damage detected 2024-05-09T08:30:00.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:00 smithi184 ceph-mon[34509]: mds.h(mds.3): Metadata damage detected 2024-05-09T08:30:00.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:00 smithi184 ceph-mon[34509]: mds.c(mds.2): Metadata damage detected 2024-05-09T08:30:00.607 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:00 smithi184 ceph-mon[34509]: mds.i(mds.1): Metadata damage detected 2024-05-09T08:30:00.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-mon[30119]: Health detail: HEALTH_ERR 4 MDSs report damaged metadata 2024-05-09T08:30:00.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-mon[30119]: [ERR] MDS_DAMAGE: 4 MDSs report damaged metadata 2024-05-09T08:30:00.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-mon[30119]: mds.f(mds.4): Metadata damage detected 2024-05-09T08:30:00.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-mon[30119]: mds.h(mds.3): Metadata damage detected 2024-05-09T08:30:00.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-mon[30119]: mds.c(mds.2): Metadata damage detected 2024-05-09T08:30:00.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:00 smithi028 ceph-mon[30119]: mds.i(mds.1): Metadata damage detected 2024-05-09T08:30:00.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.700+0000 7fab2b478740 1 Processor -- start 2024-05-09T08:30:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.700+0000 7fab2b478740 1 -- start start 2024-05-09T08:30:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.700+0000 7fab2b478740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 0x55cba3fe8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.701+0000 7fab2b478740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba3fd3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.701+0000 7fab2b478740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba4016960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.701+0000 7fab25cfe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba3fd3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.701+0000 7fab25cfe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba3fd3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:37782/0 (socket says 172.21.15.28:37782) 2024-05-09T08:30:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.701+0000 7fab26d00640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba4016960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.701+0000 7fab26d00640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba4016960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:36022/0 (socket says 172.21.15.28:36022) 2024-05-09T08:30:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.701+0000 7fab25cfe640 1 -- 172.21.15.28:0/4129131885 learned_addr learned my addr 172.21.15.28:0/4129131885 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.702+0000 7fab2b478740 1 -- 172.21.15.28:0/4129131885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab0400ff60 con 0x55cba3fd15d0 2024-05-09T08:30:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.702+0000 7fab2b478740 1 -- 172.21.15.28:0/4129131885 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fab04011b60 con 0x55cba3fd1ef0 2024-05-09T08:30:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.702+0000 7fab2b478740 1 -- 172.21.15.28:0/4129131885 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab04010630 con 0x55cba3fdc7b0 2024-05-09T08:30:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.702+0000 7fab25cfe640 1 -- 172.21.15.28:0/4129131885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 msgr2=0x55cba4016960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.702+0000 7fab25cfe640 1 --2- 172.21.15.28:0/4129131885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba4016960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.702+0000 7fab25cfe640 1 -- 172.21.15.28:0/4129131885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 msgr2=0x55cba3fe8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.702+0000 7fab25cfe640 1 --2- 172.21.15.28:0/4129131885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 0x55cba3fe8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.702+0000 7fab25cfe640 1 -- 172.21.15.28:0/4129131885 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cba3fcd160 con 0x55cba3fd1ef0 2024-05-09T08:30:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.703+0000 7fab25cfe640 1 --2- 172.21.15.28:0/4129131885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba3fd3fe0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0098b0 tx=0x7fab1c005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1b7dfd0f6b168e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.751+0000 7fab0bfff640 1 -- 172.21.15.28:0/4129131885 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c028070 con 0x55cba3fd1ef0 2024-05-09T08:30:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.751+0000 7fab0bfff640 1 -- 172.21.15.28:0/4129131885 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab1c009e40 con 0x55cba3fd1ef0 2024-05-09T08:30:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.751+0000 7fab0bfff640 1 -- 172.21.15.28:0/4129131885 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c0035f0 con 0x55cba3fd1ef0 2024-05-09T08:30:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.752+0000 7fab2b478740 1 -- 172.21.15.28:0/4129131885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 msgr2=0x55cba3fd3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.752+0000 7fab2b478740 1 --2- 172.21.15.28:0/4129131885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba3fd3fe0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0098b0 tx=0x7fab1c005a20 comp rx=0 tx=0).stop 2024-05-09T08:30:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.755+0000 7fab2b478740 1 -- 172.21.15.28:0/4129131885 shutdown_connections 2024-05-09T08:30:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.755+0000 7fab2b478740 1 --2- 172.21.15.28:0/4129131885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba4016960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.755+0000 7fab2b478740 1 --2- 172.21.15.28:0/4129131885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba3fd3fe0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.755+0000 7fab2b478740 1 --2- 172.21.15.28:0/4129131885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 0x55cba3fe8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.755+0000 7fab2b478740 1 -- 172.21.15.28:0/4129131885 >> 172.21.15.28:0/4129131885 conn(0x55cba3fcdae0 msgr2=0x55cba3fd4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.757+0000 7fab2b478740 1 -- 172.21.15.28:0/4129131885 shutdown_connections 2024-05-09T08:30:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.757+0000 7fab2b478740 1 -- 172.21.15.28:0/4129131885 wait complete. 2024-05-09T08:30:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.758+0000 7fab2b478740 1 -- start start 2024-05-09T08:30:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.758+0000 7fab2b478740 1 Processor -- start 2024-05-09T08:30:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.758+0000 7fab2b478740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 0x55cba406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.759+0000 7fab2b478740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba406da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.759+0000 7fab2b478740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fdfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.759+0000 7fab264ff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 0x55cba406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.759+0000 7fab264ff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 0x55cba406d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51160/0 (socket says 172.21.15.28:51160) 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.759+0000 7fab25cfe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.759+0000 7fab25cfe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba406da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:37792/0 (socket says 172.21.15.28:37792) 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.759+0000 7fab264ff640 1 -- 172.21.15.28:0/561620091 learned_addr learned my addr 172.21.15.28:0/561620091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.759+0000 7fab26d00640 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fdfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.760+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab04011490 con 0x55cba3fd15d0 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.760+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fab04010d00 con 0x55cba3fd1ef0 2024-05-09T08:30:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.760+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab0400dff0 con 0x55cba3fdc7b0 2024-05-09T08:30:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.760+0000 7fab2b478740 10 client.0 _ll_register_callbacks cb 0x7fab10010f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:30:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.760+0000 7fab25cfe640 1 -- 172.21.15.28:0/561620091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 msgr2=0x55cba3fdfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.760+0000 7fab25cfe640 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fdfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.760+0000 7fab25cfe640 1 -- 172.21.15.28:0/561620091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 msgr2=0x55cba406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.760+0000 7fab25cfe640 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 0x55cba406d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.760+0000 7fab25cfe640 1 -- 172.21.15.28:0/561620091 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cba3fe0160 con 0x55cba3fd1ef0 2024-05-09T08:30:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.761+0000 7fab25cfe640 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba406da10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55cba3fe9290 tx=0x7fab1c00f6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b7d1d8d584b476f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.761+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cba3f3ddc0 con 0x55cba3fd1ef0 2024-05-09T08:30:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.802+0000 7fab09ffb640 10 client.26060 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:30:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.802+0000 7fab09ffb640 1 -- 172.21.15.28:0/561620091 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c028070 con 0x55cba3fd1ef0 2024-05-09T08:30:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.802+0000 7fab09ffb640 1 -- 172.21.15.28:0/561620091 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9495+0+0 (secure 0 0 0) 0x7fab1c005520 con 0x55cba3fd1ef0 2024-05-09T08:30:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.802+0000 7fab09ffb640 1 -- 172.21.15.28:0/561620091 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fab1c00f8c0 con 0x55cba3fd1ef0 2024-05-09T08:30:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.824+0000 7fab2b478740 10 client.26060 fetch_fsmap learned FSMap version 40 2024-05-09T08:30:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.824+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({fsmap=40}) -- 0x55cba4074b00 con 0x55cba3fd1ef0 2024-05-09T08:30:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.826+0000 7fab09ffb640 1 -- 172.21.15.28:0/561620091 <== mon.2 v2:172.21.15.165:3300/0 4 ==== fsmap(e 40) ==== 8394+0+0 (secure 0 0 0) 0x7fab1c01e070 con 0x55cba3fd1ef0 2024-05-09T08:30:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.826+0000 7fab2b478740 10 client.26060 fetch_fsmap finished waiting for FSMap version 40 2024-05-09T08:30:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.826+0000 7fab2b478740 10 client.26060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:30:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.826+0000 7fab2b478740 20 client.26060 populate_metadata read hostname 'smithi028' 2024-05-09T08:30:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.826+0000 7fab2b478740 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cba4074d20 0x55cba4077110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.826+0000 7fab2b478740 4 client.26060 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:30:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.826+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cba3f3ddc0 con 0x55cba4074d20 2024-05-09T08:30:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.826+0000 7fab264ff640 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cba4074d20 0x55cba4077110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.827+0000 7fab264ff640 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cba4074d20 0x55cba4077110 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d127020b6b55e0da server_cookie=444a74abd73ac8b8 in_seq=0 out_seq=0 2024-05-09T08:30:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:00.828+0000 7fab09ffb640 10 client.26060 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:01.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:01 smithi165 ceph-mon[34540]: pgmap v116: 129 pgs: 129 active+clean; 876 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.1 MiB/s wr, 393 op/s 2024-05-09T08:30:01.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:01 smithi184 ceph-mon[34509]: pgmap v116: 129 pgs: 129 active+clean; 876 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.1 MiB/s wr, 393 op/s 2024-05-09T08:30:01.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:01 smithi028 ceph-mon[30119]: pgmap v116: 129 pgs: 129 active+clean; 876 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.1 MiB/s wr, 393 op/s 2024-05-09T08:30:03.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:03 smithi165 ceph-mon[34540]: pgmap v117: 129 pgs: 129 active+clean; 886 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.9 MiB/s wr, 368 op/s 2024-05-09T08:30:03.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:03 smithi184 ceph-mon[34509]: pgmap v117: 129 pgs: 129 active+clean; 886 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.9 MiB/s wr, 368 op/s 2024-05-09T08:30:03.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:03 smithi028 ceph-mon[30119]: pgmap v117: 129 pgs: 129 active+clean; 886 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.9 MiB/s wr, 368 op/s 2024-05-09T08:30:05.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:05 smithi165 ceph-mon[34540]: pgmap v118: 129 pgs: 129 active+clean; 902 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.3 MiB/s wr, 515 op/s 2024-05-09T08:30:05.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:05 smithi184 ceph-mon[34509]: pgmap v118: 129 pgs: 129 active+clean; 902 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.3 MiB/s wr, 515 op/s 2024-05-09T08:30:05.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:05 smithi028 ceph-mon[30119]: pgmap v118: 129 pgs: 129 active+clean; 902 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.3 MiB/s wr, 515 op/s 2024-05-09T08:30:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.811+0000 7fab09ffb640 1 -- 172.21.15.28:0/561620091 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cba3f3ddc0 con 0x55cba4074d20 2024-05-09T08:30:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.811+0000 7fab09ffb640 10 client.26060 handle_command_reply: tid=0 2024-05-09T08:30:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.811+0000 7fab2b478740 1 client.26060 shutdown 2024-05-09T08:30:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 msgr2=0x55cba406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba406da10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55cba3fe9290 tx=0x7fab1c00f6d0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 shutdown_connections 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55cba4074d20 0x55cba4077110 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55cba3fdc7b0 0x55cba3fdfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cba3fd1ef0 0x55cba406da10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 --2- 172.21.15.28:0/561620091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cba3fd15d0 0x55cba406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 >> 172.21.15.28:0/561620091 conn(0x55cba3fcdae0 msgr2=0x55cba3fcc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab09ffb640 0 client.26060 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 shutdown_connections 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.812+0000 7fab2b478740 1 -- 172.21.15.28:0/561620091 wait complete. 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.813+0000 7fab2b478740 20 client.26060 trim_cache size 0 max 16384 2024-05-09T08:30:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:30:06.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:30:06.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:30:06.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:30:06.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 -- 172.21.15.28:0/1551926422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fab0c0a2f30 msgr2=0x7fab0c0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fab0c0a2f30 0x7fab0c0a53f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fab180099a0 tx=0x7fab18005fd0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 -- 172.21.15.28:0/1551926422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab2010fda0 msgr2=0x7fab20117b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab2010fda0 0x7fab20117b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab1000b3e0 tx=0x7fab1000b8a0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 -- 172.21.15.28:0/1551926422 shutdown_connections 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fab0c0a2f30 0x7fab0c0a53f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab2010fda0 0x7fab20117b80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab2010e5a0 0x7fab20117640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 --2- 172.21.15.28:0/1551926422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab2010db00 0x7fab2011e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.857+0000 7fab28763640 1 -- 172.21.15.28:0/1551926422 >> 172.21.15.28:0/1551926422 conn(0x7fab2006c7e0 msgr2=0x7fab20109e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.858+0000 7fab28763640 1 -- 172.21.15.28:0/1551926422 shutdown_connections 2024-05-09T08:30:06.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:06.858+0000 7fab28763640 1 -- 172.21.15.28:0/1551926422 wait complete. 2024-05-09T08:30:07.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:07 smithi165 ceph-mon[34540]: pgmap v119: 129 pgs: 129 active+clean; 950 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 886 op/s 2024-05-09T08:30:07.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:07 smithi184 ceph-mon[34509]: pgmap v119: 129 pgs: 129 active+clean; 950 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 886 op/s 2024-05-09T08:30:07.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:07 smithi028 ceph-mon[30119]: pgmap v119: 129 pgs: 129 active+clean; 950 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 886 op/s 2024-05-09T08:30:08.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:08 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:08.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:08 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68262]: *** Caught signal (Segmentation fault) ** 2024-05-09T08:30:08.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:08 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68262]: in thread 7fadd4f49700 thread_name:MR_Finisher 2024-05-09T08:30:08.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:08 smithi165 podman[68377]: 2024-05-09 08:30:08.312408213 +0000 UTC m=+0.021325467 container died eb338d2eadd20633527cb6a29d2986e9c9bdd543f8c1a2f310ca56315a3f5a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=) 2024-05-09T08:30:08.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:08 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:08.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:08 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:08.708 INFO:tasks.workunit.client.0.smithi028.stdout: 293 1863075 883 1231745 511 359714 716 2024-05-09T08:30:09.190 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:08 smithi165 podman[68377]: 2024-05-09 08:30:08.941330108 +0000 UTC m=+0.650247359 container remove eb338d2eadd20633527cb6a29d2986e9c9bdd543f8c1a2f310ca56315a3f5a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-09T08:30:09.190 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:08 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Main process exited, code=exited, status=139/n/a 2024-05-09T08:30:09.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:09 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-09T08:30:09.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:09 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Consumed 1.980s CPU time. 2024-05-09T08:30:09.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:09 smithi165 ceph-mon[34540]: pgmap v120: 129 pgs: 129 active+clean; 950 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 13 MiB/s wr, 860 op/s 2024-05-09T08:30:09.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:09 smithi184 ceph-mon[34509]: pgmap v120: 129 pgs: 129 active+clean; 950 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 13 MiB/s wr, 860 op/s 2024-05-09T08:30:09.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:09 smithi028 ceph-mon[30119]: pgmap v120: 129 pgs: 129 active+clean; 950 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 13 MiB/s wr, 860 op/s 2024-05-09T08:30:11.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:11 smithi165 ceph-mon[34540]: pgmap v121: 129 pgs: 129 active+clean; 951 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 14 MiB/s wr, 944 op/s 2024-05-09T08:30:11.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:11 smithi184 ceph-mon[34509]: pgmap v121: 129 pgs: 129 active+clean; 951 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 14 MiB/s wr, 944 op/s 2024-05-09T08:30:11.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:11 smithi028 ceph-mon[30119]: pgmap v121: 129 pgs: 129 active+clean; 951 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 14 MiB/s wr, 944 op/s 2024-05-09T08:30:13.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:13 smithi028 ceph-mon[30119]: pgmap v122: 129 pgs: 129 active+clean; 954 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 8.2 MiB/s wr, 730 op/s 2024-05-09T08:30:13.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:13 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:13.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:13 smithi165 ceph-mon[34540]: pgmap v122: 129 pgs: 129 active+clean; 954 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 8.2 MiB/s wr, 730 op/s 2024-05-09T08:30:13.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:13 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:13.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:13 smithi184 ceph-mon[34509]: pgmap v122: 129 pgs: 129 active+clean; 954 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 8.2 MiB/s wr, 730 op/s 2024-05-09T08:30:13.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:13 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:15.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:15 smithi028 ceph-mon[30119]: pgmap v123: 129 pgs: 129 active+clean; 965 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.4 MiB/s wr, 738 op/s 2024-05-09T08:30:15.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:15 smithi165 ceph-mon[34540]: pgmap v123: 129 pgs: 129 active+clean; 965 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.4 MiB/s wr, 738 op/s 2024-05-09T08:30:15.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:15 smithi184 ceph-mon[34509]: pgmap v123: 129 pgs: 129 active+clean; 965 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.4 MiB/s wr, 738 op/s 2024-05-09T08:30:17.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:17 smithi028 ceph-mon[30119]: pgmap v124: 129 pgs: 129 active+clean; 973 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.6 MiB/s wr, 621 op/s 2024-05-09T08:30:17.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:17 smithi165 ceph-mon[34540]: pgmap v124: 129 pgs: 129 active+clean; 973 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.6 MiB/s wr, 621 op/s 2024-05-09T08:30:17.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:17 smithi184 ceph-mon[34509]: pgmap v124: 129 pgs: 129 active+clean; 973 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.6 MiB/s wr, 621 op/s 2024-05-09T08:30:18.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:18 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:18.708 INFO:tasks.workunit.client.0.smithi028.stdout: 339 1912496 2235 1269647 2508 375327 741 2024-05-09T08:30:18.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:18 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:18.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:18 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:19.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:19 smithi028 ceph-mon[30119]: pgmap v125: 129 pgs: 129 active+clean; 973 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-05-09T08:30:19.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:19 smithi165 ceph-mon[34540]: pgmap v125: 129 pgs: 129 active+clean; 973 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-05-09T08:30:19.765 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:19 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-05-09T08:30:19.766 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:19 smithi165 systemd[1]: Stopped Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:30:19.766 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:19 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Consumed 1.980s CPU time. 2024-05-09T08:30:19.766 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:19 smithi165 systemd[1]: Starting Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:30:19.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:19 smithi184 ceph-mon[34509]: pgmap v125: 129 pgs: 129 active+clean; 973 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-05-09T08:30:20.069 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:19 smithi165 podman[68468]: 2024-05-09 08:30:19.669237189 +0000 UTC m=+0.033589788 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:20.069 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:19 smithi165 podman[68468]: 2024-05-09 08:30:19.801277745 +0000 UTC m=+0.165630330 container create 8f2f316a2496eafaaa14eb7eb128a6a7859df21b7479d8c32ca3151fb07787d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:30:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.266+0000 7fd67b2bc640 1 -- 172.21.15.28:0/3668406022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd674110020 msgr2=0x7fd674108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.266+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/3668406022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd674110020 0x7fd674108830 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009930 tx=0x7fd66c02a4f0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.266+0000 7fd67b2bc640 1 -- 172.21.15.28:0/3668406022 shutdown_connections 2024-05-09T08:30:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.266+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/3668406022 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd67411b010 0x7fd67411d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.266+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/3668406022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd674110400 0x7fd674108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.266+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/3668406022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd674110020 0x7fd674108830 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.266+0000 7fd67b2bc640 1 -- 172.21.15.28:0/3668406022 >> 172.21.15.28:0/3668406022 conn(0x7fd67406c900 msgr2=0x7fd67406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.276+0000 7fd67b2bc640 1 -- 172.21.15.28:0/3668406022 shutdown_connections 2024-05-09T08:30:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.276+0000 7fd67b2bc640 1 -- 172.21.15.28:0/3668406022 wait complete. 2024-05-09T08:30:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.277+0000 7fd67b2bc640 1 Processor -- start 2024-05-09T08:30:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.277+0000 7fd67b2bc640 1 -- start start 2024-05-09T08:30:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.278+0000 7fd67b2bc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd674110020 0x7fd6741a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.278+0000 7fd67b2bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67411b010 0x7fd6741a7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.278+0000 7fd67b2bc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6741ab8a0 0x7fd6741abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.278+0000 7fd67b2bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd674120b00 con 0x7fd67411b010 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.278+0000 7fd67b2bc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd674120980 con 0x7fd674110020 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.278+0000 7fd67b2bc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd674120c80 con 0x7fd6741ab8a0 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.278+0000 7fd679859640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6741ab8a0 0x7fd6741abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.278+0000 7fd679859640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6741ab8a0 0x7fd6741abd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37928/0 (socket says 172.21.15.28:37928) 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.278+0000 7fd679859640 1 -- 172.21.15.28:0/2471039800 learned_addr learned my addr 172.21.15.28:0/2471039800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.279+0000 7fd678857640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67411b010 0x7fd6741a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.279+0000 7fd679859640 1 -- 172.21.15.28:0/2471039800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd674110020 msgr2=0x7fd6741a6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.279+0000 7fd679859640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd674110020 0x7fd6741a6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.279+0000 7fd679859640 1 -- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67411b010 msgr2=0x7fd6741a7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.279+0000 7fd679859640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67411b010 0x7fd6741a7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.279+0000 7fd679859640 1 -- 172.21.15.28:0/2471039800 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6741ac390 con 0x7fd6741ab8a0 2024-05-09T08:30:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.280+0000 7fd679859640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6741ab8a0 0x7fd6741abd50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd66400b3c0 tx=0x7fd66400b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.281+0000 7fd66a7fc640 1 -- 172.21.15.28:0/2471039800 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664011830 con 0x7fd6741ab8a0 2024-05-09T08:30:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.281+0000 7fd67b2bc640 1 -- 172.21.15.28:0/2471039800 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6741ac680 con 0x7fd6741ab8a0 2024-05-09T08:30:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.281+0000 7fd67b2bc640 1 -- 172.21.15.28:0/2471039800 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd674110b10 con 0x7fd6741ab8a0 2024-05-09T08:30:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.281+0000 7fd66a7fc640 1 -- 172.21.15.28:0/2471039800 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd664016070 con 0x7fd6741ab8a0 2024-05-09T08:30:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.281+0000 7fd66a7fc640 1 -- 172.21.15.28:0/2471039800 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66400f2e0 con 0x7fd6741ab8a0 2024-05-09T08:30:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.286+0000 7fd67dfc5740 1 Processor -- start 2024-05-09T08:30:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.286+0000 7fd67dfc5740 1 -- start start 2024-05-09T08:30:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.286+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 0x55aca34e8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.286+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8d70 0x55aca34f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.286+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.286+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aca344d340 con 0x55aca34e8450 2024-05-09T08:30:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.286+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55aca33f5290 con 0x55aca34f3630 2024-05-09T08:30:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.286+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aca3449300 con 0x55aca34e8d70 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.288+0000 7fd679058640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8d70 0x55aca34f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.288+0000 7fd679058640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8d70 0x55aca34f30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37932/0 (socket says 172.21.15.28:37932) 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.288+0000 7fd679058640 1 -- 172.21.15.28:0/3454518153 learned_addr learned my addr 172.21.15.28:0/3454518153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.289+0000 7fd678857640 1 --2- 172.21.15.28:0/3454518153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.289+0000 7fd678857640 1 --2- 172.21.15.28:0/3454518153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 0x55aca34e8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.289+0000 7fd679058640 1 -- 172.21.15.28:0/3454518153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 msgr2=0x55aca34f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.289+0000 7fd679058640 1 --2- 172.21.15.28:0/3454518153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.289+0000 7fd679058640 1 -- 172.21.15.28:0/3454518153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 msgr2=0x55aca34e8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.289+0000 7fd679058640 1 --2- 172.21.15.28:0/3454518153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 0x55aca34e8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.289+0000 7fd679058640 1 -- 172.21.15.28:0/3454518153 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aca34198e0 con 0x55aca34e8d70 2024-05-09T08:30:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.293+0000 7fd679058640 1 --2- 172.21.15.28:0/3454518153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8d70 0x55aca34f30f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd66c005d10 tx=0x7fd66c005d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e21cb4dc8cdcd74f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.293+0000 7fd66a7fc640 1 -- 172.21.15.28:0/2471039800 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd664010380 con 0x7fd6741ab8a0 2024-05-09T08:30:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.294+0000 7fd66a7fc640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6600a2fc0 0x7fd6600a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.295+0000 7fd66a7fc640 1 -- 172.21.15.28:0/2471039800 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9495+0+0 (secure 0 0 0) 0x7fd6640ce880 con 0x7fd6741ab8a0 2024-05-09T08:30:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.296+0000 7fd65f7fe640 1 -- 172.21.15.28:0/3454518153 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c03e070 con 0x55aca34e8d70 2024-05-09T08:30:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.296+0000 7fd65f7fe640 1 -- 172.21.15.28:0/3454518153 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c004380 con 0x55aca34e8d70 2024-05-09T08:30:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.296+0000 7fd65f7fe640 1 -- 172.21.15.28:0/3454518153 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c035030 con 0x55aca34e8d70 2024-05-09T08:30:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.296+0000 7fd678857640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6600a2fc0 0x7fd6600a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.297+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3454518153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8d70 msgr2=0x55aca34f30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.297+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3454518153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8d70 0x55aca34f30f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd66c005d10 tx=0x7fd66c005d40 comp rx=0 tx=0).stop 2024-05-09T08:30:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.297+0000 7fd678857640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6600a2fc0 0x7fd6600a5480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x55aca34e5d80 tx=0x7fd670004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.298+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3454518153 shutdown_connections 2024-05-09T08:30:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.298+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3454518153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.298+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3454518153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8d70 0x55aca34f30f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.298+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3454518153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 0x55aca34e8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.298+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3454518153 >> 172.21.15.28:0/3454518153 conn(0x55aca34e4960 msgr2=0x55aca34e4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:20.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.303+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3454518153 shutdown_connections 2024-05-09T08:30:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.304+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3454518153 wait complete. 2024-05-09T08:30:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.304+0000 7fd67dfc5740 1 -- start start 2024-05-09T08:30:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.304+0000 7fd67dfc5740 1 Processor -- start 2024-05-09T08:30:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.305+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34e8450 0x55aca34f0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.305+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8d70 0x55aca34f0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.305+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34f3630 0x55aca34f0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.305+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aca34f6200 con 0x55aca34e8d70 2024-05-09T08:30:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.305+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55aca3408790 con 0x55aca34e8450 2024-05-09T08:30:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.305+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55aca34f6380 con 0x55aca34f3630 2024-05-09T08:30:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.305+0000 7fd67dfc5740 10 client.0 _ll_register_callbacks cb 0x55aca3443a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:30:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.306+0000 7fd679859640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34f3630 0x55aca34f0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.306+0000 7fd679859640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34f3630 0x55aca34f0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37946/0 (socket says 172.21.15.28:37946) 2024-05-09T08:30:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.306+0000 7fd679859640 1 -- 172.21.15.28:0/3778154148 learned_addr learned my addr 172.21.15.28:0/3778154148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.307+0000 7fd678857640 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8d70 0x55aca34f0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.307+0000 7fd679058640 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34e8450 0x55aca34f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.308+0000 7fd678857640 1 -- 172.21.15.28:0/3778154148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34f3630 msgr2=0x55aca34f0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.308+0000 7fd678857640 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34f3630 0x55aca34f0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.308+0000 7fd678857640 1 -- 172.21.15.28:0/3778154148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34e8450 msgr2=0x55aca34f0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.308+0000 7fd678857640 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34e8450 0x55aca34f0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.308+0000 7fd678857640 1 -- 172.21.15.28:0/3778154148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aca3161210 con 0x55aca34e8d70 2024-05-09T08:30:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.309+0000 7fd678857640 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8d70 0x55aca34f0a80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd670005c40 tx=0x7fd67000a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d1f63a53b383af server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.309+0000 7fd65cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:30:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.310+0000 7fd65cff9640 1 -- 172.21.15.28:0/3778154148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670004d20 con 0x55aca34e8d70 2024-05-09T08:30:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.310+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3778154148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd66c004380 con 0x55aca34e8d70 2024-05-09T08:30:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.310+0000 7fd65cff9640 1 -- 172.21.15.28:0/3778154148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9495+0+0 (secure 0 0 0) 0x7fd670021630 con 0x55aca34e8d70 2024-05-09T08:30:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.310+0000 7fd65cff9640 1 -- 172.21.15.28:0/3778154148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd670031050 con 0x55aca34e8d70 2024-05-09T08:30:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.310+0000 7fd67dfc5740 10 client.16599 fetch_fsmap learned FSMap version 43 2024-05-09T08:30:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.310+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3778154148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55aca322d620 con 0x55aca34e8d70 2024-05-09T08:30:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.311+0000 7fd65cff9640 1 -- 172.21.15.28:0/3778154148 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c004380 con 0x55aca34e8d70 2024-05-09T08:30:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.311+0000 7fd65cff9640 1 -- 172.21.15.28:0/3778154148 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8410+0+0 (secure 0 0 0) 0x7fd6700299a0 con 0x55aca34e8d70 2024-05-09T08:30:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.311+0000 7fd67dfc5740 10 client.16599 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T08:30:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.311+0000 7fd67dfc5740 10 client.16599 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:30:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.311+0000 7fd67dfc5740 20 client.16599 populate_metadata read hostname 'smithi028' 2024-05-09T08:30:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.311+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aca3503750 0x55aca3505bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.311+0000 7fd67dfc5740 4 client.16599 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:30:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.311+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3778154148 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aca345aa70 con 0x55aca3503750 2024-05-09T08:30:20.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.314+0000 7fd679058640 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aca3503750 0x55aca3505bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.318+0000 7fd679058640 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aca3503750 0x55aca3505bb0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c45299a30498bd1 server_cookie=5521cf3be8d2f944 in_seq=0 out_seq=0 2024-05-09T08:30:20.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.319+0000 7fd65cff9640 10 client.16599 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:20.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.323+0000 7fd65cff9640 1 -- 172.21.15.28:0/3778154148 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aca345aa70 con 0x55aca3503750 2024-05-09T08:30:20.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.323+0000 7fd65cff9640 10 client.16599 handle_command_reply: tid=0 2024-05-09T08:30:20.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.323+0000 7fd67dfc5740 1 client.16599 shutdown 2024-05-09T08:30:20.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.325+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3778154148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8d70 msgr2=0x55aca34f0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.325+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8d70 0x55aca34f0a80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd670005c40 tx=0x7fd67000a590 comp rx=0 tx=0).stop 2024-05-09T08:30:20.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.325+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3778154148 shutdown_connections 2024-05-09T08:30:20.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.325+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aca3503750 0x55aca3505bb0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.325+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34f3630 0x55aca34f0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.325+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8d70 0x55aca34f0a80 secure :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd670005c40 tx=0x7fd67000a590 comp rx=0 tx=0).stop 2024-05-09T08:30:20.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.325+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/3778154148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34e8450 0x55aca34f0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.325+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3778154148 >> 172.21.15.28:0/3778154148 conn(0x55aca34e4960 msgr2=0x55aca34ea6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:20.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.325+0000 7fd65cff9640 0 client.16599 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:20.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.330+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3778154148 shutdown_connections 2024-05-09T08:30:20.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.331+0000 7fd67dfc5740 1 -- 172.21.15.28:0/3778154148 wait complete. 2024-05-09T08:30:20.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.332+0000 7fd67dfc5740 20 client.16599 trim_cache size 0 max 16384 2024-05-09T08:30:20.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:20 smithi165 podman[68468]: 2024-05-09 08:30:20.070693336 +0000 UTC m=+0.435045939 container init 8f2f316a2496eafaaa14eb7eb128a6a7859df21b7479d8c32ca3151fb07787d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-96805dc) 2024-05-09T08:30:20.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:20 smithi165 podman[68468]: 2024-05-09 08:30:20.079287986 +0000 UTC m=+0.443640578 container start 8f2f316a2496eafaaa14eb7eb128a6a7859df21b7479d8c32ca3151fb07787d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-09T08:30:20.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:20 smithi165 ceph-mds[68483]: -- 172.21.15.165:0/3352138959 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a9b1495a40 con 0x55a9b1468800 2024-05-09T08:30:20.352 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:20 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68479]: starting mds.e at 2024-05-09T08:30:20.352 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:20 smithi165 bash[68468]: 8f2f316a2496eafaaa14eb7eb128a6a7859df21b7479d8c32ca3151fb07787d7 2024-05-09T08:30:20.352 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:20 smithi165 systemd[1]: Started Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:30:20.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.365+0000 7fd67dfc5740 1 Processor -- start 2024-05-09T08:30:20.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.365+0000 7fd67dfc5740 1 -- start start 2024-05-09T08:30:20.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.365+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 0x55aca34f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.366+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34ff120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.366+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34ef6a0 0x55aca34e3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.366+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd658010110 con 0x55aca34e8450 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.366+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd658011dd0 con 0x55aca34f3630 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.366+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6580107e0 con 0x55aca34ef6a0 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.366+0000 7fd679859640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34ef6a0 0x55aca34e3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.366+0000 7fd679859640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34ef6a0 0x55aca34e3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37962/0 (socket says 172.21.15.28:37962) 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.366+0000 7fd679859640 1 -- 172.21.15.28:0/600479313 learned_addr learned my addr 172.21.15.28:0/600479313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.366+0000 7fd679058640 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 0x55aca34f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.367+0000 7fd678857640 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34ff120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.367+0000 7fd679058640 1 -- 172.21.15.28:0/600479313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34ef6a0 msgr2=0x55aca34e3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.367+0000 7fd679058640 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34ef6a0 0x55aca34e3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.367+0000 7fd679058640 1 -- 172.21.15.28:0/600479313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 msgr2=0x55aca34ff120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.367+0000 7fd679058640 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34ff120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.367+0000 7fd679058640 1 -- 172.21.15.28:0/600479313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aca34e2f00 con 0x55aca34e8450 2024-05-09T08:30:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.367+0000 7fd678857640 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34ff120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.367+0000 7fd679058640 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 0x55aca34f2c70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009930 tx=0x7fd66c02ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b268fefdccd8739e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.368+0000 7fd65effd640 1 -- 172.21.15.28:0/600479313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c035050 con 0x55aca34e8450 2024-05-09T08:30:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.368+0000 7fd65effd640 1 -- 172.21.15.28:0/600479313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c031ad0 con 0x55aca34e8450 2024-05-09T08:30:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.368+0000 7fd65effd640 1 -- 172.21.15.28:0/600479313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c03e040 con 0x55aca34e8450 2024-05-09T08:30:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.369+0000 7fd67dfc5740 1 -- 172.21.15.28:0/600479313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 msgr2=0x55aca34f2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.369+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 0x55aca34f2c70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009930 tx=0x7fd66c02ac30 comp rx=0 tx=0).stop 2024-05-09T08:30:20.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.370+0000 7fd67dfc5740 1 -- 172.21.15.28:0/600479313 shutdown_connections 2024-05-09T08:30:20.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.370+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34ef6a0 0x55aca34e3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.370+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34ff120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.370+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/600479313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34e8450 0x55aca34f2c70 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.370+0000 7fd67dfc5740 1 -- 172.21.15.28:0/600479313 >> 172.21.15.28:0/600479313 conn(0x55aca34e4960 msgr2=0x55aca34e3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:20.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.375+0000 7fd67dfc5740 1 -- 172.21.15.28:0/600479313 shutdown_connections 2024-05-09T08:30:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.375+0000 7fd67dfc5740 1 -- 172.21.15.28:0/600479313 wait complete. 2024-05-09T08:30:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.376+0000 7fd67dfc5740 1 -- start start 2024-05-09T08:30:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.376+0000 7fd67dfc5740 1 Processor -- start 2024-05-09T08:30:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.377+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8450 0x55aca34ed460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.377+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34ef6a0 0x55aca34ed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.377+0000 7fd67dfc5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34edee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.377+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd658011700 con 0x55aca34ef6a0 2024-05-09T08:30:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.377+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd658010f70 con 0x55aca34f3630 2024-05-09T08:30:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.377+0000 7fd67dfc5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd65800e0f0 con 0x55aca34e8450 2024-05-09T08:30:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.377+0000 7fd67dfc5740 10 client.0 _ll_register_callbacks cb 0x7fd67000ae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:30:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.378+0000 7fd678857640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34ef6a0 0x55aca34ed9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.378+0000 7fd678857640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34ef6a0 0x55aca34ed9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35836/0 (socket says 172.21.15.28:35836) 2024-05-09T08:30:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.378+0000 7fd678857640 1 -- 172.21.15.28:0/2560536442 learned_addr learned my addr 172.21.15.28:0/2560536442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.379+0000 7fd679859640 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.379+0000 7fd678857640 1 -- 172.21.15.28:0/2560536442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8450 msgr2=0x55aca34ed460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.379+0000 7fd678857640 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8450 0x55aca34ed460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.379+0000 7fd678857640 1 -- 172.21.15.28:0/2560536442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 msgr2=0x55aca34edee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.379+0000 7fd678857640 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34edee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.379+0000 7fd678857640 1 -- 172.21.15.28:0/2560536442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aca3568f80 con 0x55aca34ef6a0 2024-05-09T08:30:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.380+0000 7fd678857640 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34ef6a0 0x55aca34ed9a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd670029d70 tx=0x7fd670004a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966b3b9f5a844071 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.381+0000 7fd65cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:30:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.381+0000 7fd65cff9640 1 -- 172.21.15.28:0/2560536442 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670003970 con 0x55aca34ef6a0 2024-05-09T08:30:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.381+0000 7fd65cff9640 1 -- 172.21.15.28:0/2560536442 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9495+0+0 (secure 0 0 0) 0x7fd670003b10 con 0x55aca34ef6a0 2024-05-09T08:30:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.382+0000 7fd67dfc5740 1 -- 172.21.15.28:0/2560536442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd66c031ad0 con 0x55aca34ef6a0 2024-05-09T08:30:20.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.388+0000 7fd65cff9640 1 -- 172.21.15.28:0/2560536442 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd6700205b0 con 0x55aca34ef6a0 2024-05-09T08:30:20.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.394+0000 7fd67dfc5740 10 client.16605 fetch_fsmap learned FSMap version 43 2024-05-09T08:30:20.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.394+0000 7fd67dfc5740 1 -- 172.21.15.28:0/2560536442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55aca358c1f0 con 0x55aca34ef6a0 2024-05-09T08:30:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.397+0000 7fd65cff9640 1 -- 172.21.15.28:0/2560536442 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c031ad0 con 0x55aca34ef6a0 2024-05-09T08:30:20.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.398+0000 7fd65cff9640 1 -- 172.21.15.28:0/2560536442 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8410+0+0 (secure 0 0 0) 0x7fd670029850 con 0x55aca34ef6a0 2024-05-09T08:30:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.398+0000 7fd67dfc5740 10 client.16605 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T08:30:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.398+0000 7fd67dfc5740 10 client.16605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:30:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.398+0000 7fd67dfc5740 20 client.16605 populate_metadata read hostname 'smithi028' 2024-05-09T08:30:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.398+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aca358c9d0 0x55aca358ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.398+0000 7fd67dfc5740 4 client.16605 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:30:20.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.398+0000 7fd67dfc5740 1 -- 172.21.15.28:0/2560536442 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aca3454dc0 con 0x55aca358c9d0 2024-05-09T08:30:20.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.399+0000 7fd679058640 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aca358c9d0 0x55aca358ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.407+0000 7fd679058640 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aca358c9d0 0x55aca358ee10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=979242f7a88eaab5 server_cookie=36899691abd2f772 in_seq=0 out_seq=0 2024-05-09T08:30:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:20.407+0000 7fd65cff9640 10 client.16605 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:20.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:20 smithi028 ceph-mon[30119]: daemon mds.e restarted 2024-05-09T08:30:20.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:20 smithi165 ceph-mon[34540]: daemon mds.e restarted 2024-05-09T08:30:20.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:20 smithi184 ceph-mon[34509]: daemon mds.e restarted 2024-05-09T08:30:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:21 smithi028 ceph-mon[30119]: pgmap v126: 129 pgs: 129 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.7 MiB/s wr, 510 op/s 2024-05-09T08:30:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:21 smithi028 ceph-mon[30119]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T08:30:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:21 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6834/290548075,v1:172.21.15.165:6835/290548075] up:boot 2024-05-09T08:30:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:21 smithi028 ceph-mon[30119]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:30:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:21 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:30:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:21 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:21 smithi165 ceph-mon[34540]: pgmap v126: 129 pgs: 129 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.7 MiB/s wr, 510 op/s 2024-05-09T08:30:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:21 smithi165 ceph-mon[34540]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T08:30:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:21 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6834/290548075,v1:172.21.15.165:6835/290548075] up:boot 2024-05-09T08:30:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:21 smithi165 ceph-mon[34540]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:30:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:21 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:30:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:21 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:21.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:21 smithi184 ceph-mon[34509]: pgmap v126: 129 pgs: 129 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.7 MiB/s wr, 510 op/s 2024-05-09T08:30:21.858 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:21 smithi184 ceph-mon[34509]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T08:30:21.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:21 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6834/290548075,v1:172.21.15.165:6835/290548075] up:boot 2024-05-09T08:30:21.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:21 smithi184 ceph-mon[34509]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:30:21.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:21 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:30:21.859 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:21 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:30:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.466+0000 7fd65cff9640 1 -- 172.21.15.28:0/2560536442 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aca3454dc0 con 0x55aca358c9d0 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.466+0000 7fd65cff9640 10 client.16605 handle_command_reply: tid=0 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.466+0000 7fd67dfc5740 1 client.16605 shutdown 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.466+0000 7fd67dfc5740 1 -- 172.21.15.28:0/2560536442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34ef6a0 msgr2=0x55aca34ed9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.466+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34ef6a0 0x55aca34ed9a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd670029d70 tx=0x7fd670004a00 comp rx=0 tx=0).stop 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd67dfc5740 1 -- 172.21.15.28:0/2560536442 shutdown_connections 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55aca358c9d0 0x55aca358ee10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aca34f3630 0x55aca34edee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aca34ef6a0 0x55aca34ed9a0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd67dfc5740 1 --2- 172.21.15.28:0/2560536442 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55aca34e8450 0x55aca34ed460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd67dfc5740 1 -- 172.21.15.28:0/2560536442 >> 172.21.15.28:0/2560536442 conn(0x55aca34e4960 msgr2=0x55aca34e3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd65cff9640 0 client.16605 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd67dfc5740 1 -- 172.21.15.28:0/2560536442 shutdown_connections 2024-05-09T08:30:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd67dfc5740 1 -- 172.21.15.28:0/2560536442 wait complete. 2024-05-09T08:30:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.467+0000 7fd67dfc5740 20 client.16605 trim_cache size 0 max 16384 2024-05-09T08:30:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:30:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:30:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:30:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:30:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 -- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6600a2fc0 msgr2=0x7fd6600a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6600a2fc0 0x7fd6600a5480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x55aca34e5d80 tx=0x7fd670004ab0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 -- 172.21.15.28:0/2471039800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6741ab8a0 msgr2=0x7fd6741abd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6741ab8a0 0x7fd6741abd50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd66400b3c0 tx=0x7fd66400b880 comp rx=0 tx=0).stop 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 -- 172.21.15.28:0/2471039800 shutdown_connections 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6600a2fc0 0x7fd6600a5480 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6741ab8a0 0x7fd6741abd50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67411b010 0x7fd6741a7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 --2- 172.21.15.28:0/2471039800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd674110020 0x7fd6741a6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.468+0000 7fd67b2bc640 1 -- 172.21.15.28:0/2471039800 >> 172.21.15.28:0/2471039800 conn(0x7fd67406c900 msgr2=0x7fd67410dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.469+0000 7fd67b2bc640 1 -- 172.21.15.28:0/2471039800 shutdown_connections 2024-05-09T08:30:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:23.469+0000 7fd67b2bc640 1 -- 172.21.15.28:0/2471039800 wait complete. 2024-05-09T08:30:23.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:23 smithi028 ceph-mon[30119]: pgmap v128: 129 pgs: 129 active+clean; 1.0 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 579 op/s 2024-05-09T08:30:23.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:23 smithi165 ceph-mon[34540]: pgmap v128: 129 pgs: 129 active+clean; 1.0 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 579 op/s 2024-05-09T08:30:23.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:23 smithi184 ceph-mon[34509]: pgmap v128: 129 pgs: 129 active+clean; 1.0 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 579 op/s 2024-05-09T08:30:25.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:25 smithi028 ceph-mon[30119]: pgmap v129: 129 pgs: 129 active+clean; 1.0 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 630 op/s 2024-05-09T08:30:25.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:25 smithi165 ceph-mon[34540]: pgmap v129: 129 pgs: 129 active+clean; 1.0 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 630 op/s 2024-05-09T08:30:25.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:25 smithi184 ceph-mon[34509]: pgmap v129: 129 pgs: 129 active+clean; 1.0 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 630 op/s 2024-05-09T08:30:27.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:27 smithi028 ceph-mon[30119]: pgmap v130: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 19 MiB/s wr, 1.36k op/s 2024-05-09T08:30:27.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:27 smithi165 ceph-mon[34540]: pgmap v130: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 19 MiB/s wr, 1.36k op/s 2024-05-09T08:30:27.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:27 smithi184 ceph-mon[34509]: pgmap v130: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 19 MiB/s wr, 1.36k op/s 2024-05-09T08:30:28.708 INFO:tasks.workunit.client.0.smithi028.stdout: 354 384202 1054 255477 850 73362 118 2024-05-09T08:30:29.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:29 smithi028 ceph-mon[30119]: pgmap v131: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 21 MiB/s wr, 1.47k op/s 2024-05-09T08:30:29.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:29 smithi165 ceph-mon[34540]: pgmap v131: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 21 MiB/s wr, 1.47k op/s 2024-05-09T08:30:29.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:29 smithi184 ceph-mon[34509]: pgmap v131: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 21 MiB/s wr, 1.47k op/s 2024-05-09T08:30:31.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:31 smithi028 ceph-mon[30119]: pgmap v132: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 1.18k op/s 2024-05-09T08:30:31.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:31 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:31.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:31 smithi165 ceph-mon[34540]: pgmap v132: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 1.18k op/s 2024-05-09T08:30:31.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:31 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:31 smithi184 ceph-mon[34509]: pgmap v132: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 1.18k op/s 2024-05-09T08:30:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:31 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:33 smithi028 ceph-mon[30119]: pgmap v133: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 11 MiB/s wr, 976 op/s 2024-05-09T08:30:33.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:33 smithi165 ceph-mon[34540]: pgmap v133: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 11 MiB/s wr, 976 op/s 2024-05-09T08:30:33.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:33 smithi184 ceph-mon[34509]: pgmap v133: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 11 MiB/s wr, 976 op/s 2024-05-09T08:30:35.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:35 smithi165 ceph-mon[34540]: pgmap v134: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 9.7 MiB/s wr, 903 op/s 2024-05-09T08:30:35.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:35 smithi184 ceph-mon[34509]: pgmap v134: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 9.7 MiB/s wr, 903 op/s 2024-05-09T08:30:35.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:35 smithi028 ceph-mon[30119]: pgmap v134: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 9.7 MiB/s wr, 903 op/s 2024-05-09T08:30:37.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:37 smithi165 ceph-mon[34540]: pgmap v135: 129 pgs: 129 active+clean; 1.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 13 MiB/s wr, 970 op/s 2024-05-09T08:30:37.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:37 smithi184 ceph-mon[34509]: pgmap v135: 129 pgs: 129 active+clean; 1.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 13 MiB/s wr, 970 op/s 2024-05-09T08:30:37.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:37 smithi028 ceph-mon[30119]: pgmap v135: 129 pgs: 129 active+clean; 1.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 13 MiB/s wr, 970 op/s 2024-05-09T08:30:38.708 INFO:tasks.workunit.client.0.smithi028.stdout: 376 1869744 1169 1245255 1211 320990 67 2024-05-09T08:30:38.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.931+0000 7fdb94dce640 1 -- 172.21.15.28:0/926950563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb90116830 msgr2=0x7fdb90118c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.935+0000 7fdb94dce640 1 --2- 172.21.15.28:0/926950563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb90116830 0x7fdb90118c20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009be0 tx=0x7fdb8802a6c0 comp rx=0 tx=0).stop 2024-05-09T08:30:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.937+0000 7fdb94dce640 1 -- 172.21.15.28:0/926950563 shutdown_connections 2024-05-09T08:30:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.938+0000 7fdb94dce640 1 --2- 172.21.15.28:0/926950563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb90116830 0x7fdb90118c20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.938+0000 7fdb94dce640 1 --2- 172.21.15.28:0/926950563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb90108d70 0x7fdb901091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.938+0000 7fdb94dce640 1 --2- 172.21.15.28:0/926950563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb9010bdd0 0x7fdb90108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.938+0000 7fdb94dce640 1 -- 172.21.15.28:0/926950563 >> 172.21.15.28:0/926950563 conn(0x7fdb9006c7e0 msgr2=0x7fdb9006cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:38.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.944+0000 7fdb94dce640 1 -- 172.21.15.28:0/926950563 shutdown_connections 2024-05-09T08:30:38.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.944+0000 7fdb94dce640 1 -- 172.21.15.28:0/926950563 wait complete. 2024-05-09T08:30:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.958+0000 7fdb94dce640 1 Processor -- start 2024-05-09T08:30:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.958+0000 7fdb94dce640 1 -- start start 2024-05-09T08:30:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.959+0000 7fdb94dce640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb90108d70 0x7fdb901a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.959+0000 7fdb94dce640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9010bdd0 0x7fdb901a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.959+0000 7fdb94dce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb90116830 0x7fdb901a6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.959+0000 7fdb8e59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb90108d70 0x7fdb901a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.959+0000 7fdb8e59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb90108d70 0x7fdb901a26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:45644/0 (socket says 172.21.15.28:45644) 2024-05-09T08:30:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.959+0000 7fdb8dd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9010bdd0 0x7fdb901a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.959+0000 7fdb8dd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9010bdd0 0x7fdb901a2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:59470/0 (socket says 172.21.15.28:59470) 2024-05-09T08:30:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.959+0000 7fdb8e59c640 1 -- 172.21.15.28:0/2129473265 learned_addr learned my addr 172.21.15.28:0/2129473265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.960+0000 7fdb94dce640 1 -- 172.21.15.28:0/2129473265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9011c400 con 0x7fdb90116830 2024-05-09T08:30:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.960+0000 7fdb94dce640 1 -- 172.21.15.28:0/2129473265 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9011c280 con 0x7fdb90108d70 2024-05-09T08:30:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.960+0000 7fdb94dce640 1 -- 172.21.15.28:0/2129473265 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9011c580 con 0x7fdb9010bdd0 2024-05-09T08:30:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.960+0000 7fdb8e59c640 1 -- 172.21.15.28:0/2129473265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9010bdd0 msgr2=0x7fdb901a2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.960+0000 7fdb8e59c640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9010bdd0 0x7fdb901a2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.960+0000 7fdb8e59c640 1 -- 172.21.15.28:0/2129473265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb90116830 msgr2=0x7fdb901a6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.960+0000 7fdb8e59c640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb90116830 0x7fdb901a6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.960+0000 7fdb8e59c640 1 -- 172.21.15.28:0/2129473265 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb901a7690 con 0x7fdb90108d70 2024-05-09T08:30:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.961+0000 7fdb8e59c640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb90108d70 0x7fdb901a26a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000e9c0 tx=0x7fdb8000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.979+0000 7fdb7f7fe640 1 -- 172.21.15.28:0/2129473265 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8000cde0 con 0x7fdb90108d70 2024-05-09T08:30:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.979+0000 7fdb7f7fe640 1 -- 172.21.15.28:0/2129473265 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb800035f0 con 0x7fdb90108d70 2024-05-09T08:30:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.980+0000 7fdb7f7fe640 1 -- 172.21.15.28:0/2129473265 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb80010690 con 0x7fdb90108d70 2024-05-09T08:30:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.981+0000 7fdb94dce640 1 -- 172.21.15.28:0/2129473265 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb901a7980 con 0x7fdb90108d70 2024-05-09T08:30:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.981+0000 7fdb94dce640 1 -- 172.21.15.28:0/2129473265 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb901af210 con 0x7fdb90108d70 2024-05-09T08:30:38.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.988+0000 7fdb97bbc740 1 Processor -- start 2024-05-09T08:30:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.989+0000 7fdb97bbc740 1 -- start start 2024-05-09T08:30:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.989+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c677e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.989+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677ea10 0x5606c6788d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.989+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c67892d0 0x5606c678b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.990+0000 7fdb8dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c67892d0 0x5606c678b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.990+0000 7fdb97bbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5606c66da340 con 0x5606c67892d0 2024-05-09T08:30:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.990+0000 7fdb97bbc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5606c6682290 con 0x5606c677e0f0 2024-05-09T08:30:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.990+0000 7fdb97bbc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5606c66d6300 con 0x5606c677ea10 2024-05-09T08:30:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.991+0000 7fdb8dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c67892d0 0x5606c678b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43716/0 (socket says 172.21.15.28:43716) 2024-05-09T08:30:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.991+0000 7fdb8dd9b640 1 -- 172.21.15.28:0/657946315 learned_addr learned my addr 172.21.15.28:0/657946315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.991+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/657946315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c677e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.991+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/657946315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677ea10 msgr2=0x5606c6788d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.991+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/657946315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677ea10 0x5606c6788d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.991+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/657946315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c67892d0 msgr2=0x5606c678b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.991+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/657946315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c67892d0 0x5606c678b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.991+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/657946315 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c66a68e0 con 0x5606c677e0f0 2024-05-09T08:30:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.992+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/657946315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c677e4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009b30 tx=0x7fdb88032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e39563733b61fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:38.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.996+0000 7fdb7f7fe640 1 -- 172.21.15.28:0/2129473265 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdb800037a0 con 0x7fdb90108d70 2024-05-09T08:30:38.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.998+0000 7fdb7f7fe640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb780a2ce0 0x7fdb780a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:38.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:38.999+0000 7fdb7f7fe640 1 -- 172.21.15.28:0/2129473265 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7fdb800cfce0 con 0x7fdb90108d70 2024-05-09T08:30:38.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.000+0000 7fdb8e59c640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb780a2ce0 0x7fdb780a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.013+0000 7fdb77fff640 1 -- 172.21.15.28:0/657946315 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88040020 con 0x5606c677e0f0 2024-05-09T08:30:39.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.014+0000 7fdb77fff640 1 -- 172.21.15.28:0/657946315 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb88032e50 con 0x5606c677e0f0 2024-05-09T08:30:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.015+0000 7fdb97bbc740 1 -- 172.21.15.28:0/657946315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 msgr2=0x5606c677e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.015+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/657946315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c677e4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009b30 tx=0x7fdb88032ce0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.017+0000 7fdb8e59c640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb780a2ce0 0x7fdb780a51a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x5606c677f690 tx=0x7fdb800980f0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.049+0000 7fdb97bbc740 1 -- 172.21.15.28:0/657946315 shutdown_connections 2024-05-09T08:30:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.049+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/657946315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c67892d0 0x5606c678b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.049+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/657946315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677ea10 0x5606c6788d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.049+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/657946315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c677e4d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.049+0000 7fdb97bbc740 1 -- 172.21.15.28:0/657946315 >> 172.21.15.28:0/657946315 conn(0x5606c677a600 msgr2=0x5606c677a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.049+0000 7fdb97bbc740 1 -- 172.21.15.28:0/657946315 shutdown_connections 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.049+0000 7fdb97bbc740 1 -- 172.21.15.28:0/657946315 wait complete. 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.050+0000 7fdb97bbc740 1 -- start start 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.050+0000 7fdb97bbc740 1 Processor -- start 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.050+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c6788ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.050+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677ea10 0x5606c6786030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.050+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 0x5606c6786640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.050+0000 7fdb97bbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5606c678bdb0 con 0x5606c677ea10 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.050+0000 7fdb97bbc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5606c6695790 con 0x5606c677e0f0 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.051+0000 7fdb97bbc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5606c678bf30 con 0x5606c67892d0 2024-05-09T08:30:39.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.051+0000 7fdb97bbc740 10 client.0 _ll_register_callbacks cb 0x5606c66d0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:30:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.052+0000 7fdb8ed9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c6788ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.052+0000 7fdb8ed9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c6788ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:45670/0 (socket says 172.21.15.28:45670) 2024-05-09T08:30:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.052+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/1239107731 learned_addr learned my addr 172.21.15.28:0/1239107731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:39.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.052+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 0x5606c6786640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.052+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/1239107731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 msgr2=0x5606c6786640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.052+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 0x5606c6786640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.052+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/1239107731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677ea10 msgr2=0x5606c6786030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.053+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677ea10 0x5606c6786030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.053+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/1239107731 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c63ee210 con 0x5606c677e0f0 2024-05-09T08:30:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.053+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c6788ca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdb88034040 tx=0x7fdb88003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40977421ac35ff09 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:39.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.067+0000 7fdb75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:30:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.068+0000 7fdb75ffb640 1 -- 172.21.15.28:0/1239107731 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88040070 con 0x5606c677e0f0 2024-05-09T08:30:39.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.069+0000 7fdb97bbc740 1 -- 172.21.15.28:0/1239107731 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb88032e50 con 0x5606c677e0f0 2024-05-09T08:30:39.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.069+0000 7fdb75ffb640 1 -- 172.21.15.28:0/1239107731 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7fdb88004040 con 0x5606c677e0f0 2024-05-09T08:30:39.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.069+0000 7fdb75ffb640 1 -- 172.21.15.28:0/1239107731 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fdb88004240 con 0x5606c677e0f0 2024-05-09T08:30:39.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.079+0000 7fdb97bbc740 10 client.26087 fetch_fsmap learned FSMap version 45 2024-05-09T08:30:39.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.079+0000 7fdb97bbc740 1 -- 172.21.15.28:0/1239107731 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5606c64ba620 con 0x5606c677e0f0 2024-05-09T08:30:39.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.080+0000 7fdb75ffb640 1 -- 172.21.15.28:0/1239107731 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb88032e50 con 0x5606c677e0f0 2024-05-09T08:30:39.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.080+0000 7fdb75ffb640 1 -- 172.21.15.28:0/1239107731 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 45) ==== 8410+0+0 (secure 0 0 0) 0x7fdb8803cb70 con 0x5606c677e0f0 2024-05-09T08:30:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.083+0000 7fdb97bbc740 10 client.26087 fetch_fsmap finished waiting for FSMap version 45 2024-05-09T08:30:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.083+0000 7fdb97bbc740 10 client.26087 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:30:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.083+0000 7fdb97bbc740 20 client.26087 populate_metadata read hostname 'smithi028' 2024-05-09T08:30:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.083+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5606c67755b0 0x5606c6777a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.083+0000 7fdb97bbc740 4 client.26087 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:30:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.083+0000 7fdb97bbc740 1 -- 172.21.15.28:0/1239107731 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606c66e7a70 con 0x5606c67755b0 2024-05-09T08:30:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.083+0000 7fdb8dd9b640 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5606c67755b0 0x5606c6777a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.084+0000 7fdb8dd9b640 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5606c67755b0 0x5606c6777a10 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cf55b044c80f257 server_cookie=9815d9df24d0c0a7 in_seq=0 out_seq=0 2024-05-09T08:30:39.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.085+0000 7fdb75ffb640 10 client.26087 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:39.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.090+0000 7fdb75ffb640 1 -- 172.21.15.28:0/1239107731 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606c66e7a70 con 0x5606c67755b0 2024-05-09T08:30:39.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.090+0000 7fdb75ffb640 10 client.26087 handle_command_reply: tid=0 2024-05-09T08:30:39.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.090+0000 7fdb97bbc740 1 client.26087 shutdown 2024-05-09T08:30:39.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.090+0000 7fdb97bbc740 1 -- 172.21.15.28:0/1239107731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 msgr2=0x5606c6788ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:39.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.090+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c6788ca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdb88034040 tx=0x7fdb88003a80 comp rx=0 tx=0).stop 2024-05-09T08:30:39.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb97bbc740 1 -- 172.21.15.28:0/1239107731 shutdown_connections 2024-05-09T08:30:39.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5606c67755b0 0x5606c6777a10 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 0x5606c6786640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677ea10 0x5606c6786030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/1239107731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677e0f0 0x5606c6788ca0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb97bbc740 1 -- 172.21.15.28:0/1239107731 >> 172.21.15.28:0/1239107731 conn(0x5606c677a600 msgr2=0x5606c6771050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb75ffb640 0 client.26087 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb97bbc740 1 -- 172.21.15.28:0/1239107731 shutdown_connections 2024-05-09T08:30:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb97bbc740 1 -- 172.21.15.28:0/1239107731 wait complete. 2024-05-09T08:30:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.105+0000 7fdb97bbc740 20 client.26087 trim_cache size 0 max 16384 2024-05-09T08:30:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.271+0000 7fdb97bbc740 1 Processor -- start 2024-05-09T08:30:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.271+0000 7fdb97bbc740 1 -- start start 2024-05-09T08:30:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.272+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677e0f0 0x5606c6794cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.272+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677ea10 0x5606c6788760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c67892d0 0x5606c6788ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb8ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677e0f0 0x5606c6794cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb97bbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901b4d90 con 0x5606c677ea10 2024-05-09T08:30:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb97bbc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901b6a50 con 0x5606c67892d0 2024-05-09T08:30:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb97bbc740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901b5460 con 0x5606c677e0f0 2024-05-09T08:30:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb8ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677e0f0 0x5606c6794cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:59498/0 (socket says 172.21.15.28:59498) 2024-05-09T08:30:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/3352235383 learned_addr learned my addr 172.21.15.28:0/3352235383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/3352235383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c67892d0 0x5606c6788ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/3352235383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c67892d0 msgr2=0x5606c6788ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/3352235383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c67892d0 0x5606c6788ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/3352235383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677ea10 msgr2=0x5606c6788760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/3352235383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677ea10 0x5606c6788760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.273+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/3352235383 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c678bf30 con 0x5606c677e0f0 2024-05-09T08:30:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.274+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/3352235383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677e0f0 0x5606c6794cc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdb88002230 tx=0x7fdb88005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2efc0ede107899aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.301+0000 7fdb777fe640 1 -- 172.21.15.28:0/3352235383 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88040040 con 0x5606c677e0f0 2024-05-09T08:30:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.301+0000 7fdb777fe640 1 -- 172.21.15.28:0/3352235383 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb8804a030 con 0x5606c677e0f0 2024-05-09T08:30:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.301+0000 7fdb777fe640 1 -- 172.21.15.28:0/3352235383 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88004b40 con 0x5606c677e0f0 2024-05-09T08:30:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.303+0000 7fdb97bbc740 1 -- 172.21.15.28:0/3352235383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677e0f0 msgr2=0x5606c6794cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.303+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/3352235383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677e0f0 0x5606c6794cc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdb88002230 tx=0x7fdb88005fd0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.303+0000 7fdb97bbc740 1 -- 172.21.15.28:0/3352235383 shutdown_connections 2024-05-09T08:30:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.303+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/3352235383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c67892d0 0x5606c6788ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.303+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/3352235383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677ea10 0x5606c6788760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.303+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/3352235383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c677e0f0 0x5606c6794cc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.303+0000 7fdb97bbc740 1 -- 172.21.15.28:0/3352235383 >> 172.21.15.28:0/3352235383 conn(0x5606c677a600 msgr2=0x5606c67707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.304+0000 7fdb97bbc740 1 -- 172.21.15.28:0/3352235383 shutdown_connections 2024-05-09T08:30:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.304+0000 7fdb97bbc740 1 -- 172.21.15.28:0/3352235383 wait complete. 2024-05-09T08:30:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.305+0000 7fdb97bbc740 1 -- start start 2024-05-09T08:30:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.305+0000 7fdb97bbc740 1 Processor -- start 2024-05-09T08:30:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.306+0000 7fdb97bbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677e0f0 0x5606c67915d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.307+0000 7fdb8ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677e0f0 0x5606c67915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.307+0000 7fdb8ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677e0f0 0x5606c67915d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43738/0 (socket says 172.21.15.28:43738) 2024-05-09T08:30:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.307+0000 7fdb8ed9d640 1 -- 172.21.15.28:0/635215797 learned_addr learned my addr 172.21.15.28:0/635215797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.307+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677ea10 0x5606c6778810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.307+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 0x5606c6778d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.307+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901b6380 con 0x5606c677e0f0 2024-05-09T08:30:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.307+0000 7fdb8dd9b640 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677ea10 0x5606c6778810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.307+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901b5bf0 con 0x5606c677ea10 2024-05-09T08:30:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.308+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901b2df0 con 0x5606c67892d0 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.308+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 0x5606c6778d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.308+0000 7fdb8dd9b640 1 -- 172.21.15.28:0/635215797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 msgr2=0x5606c6778d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.308+0000 7fdb8dd9b640 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 0x5606c6778d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.308+0000 7fdb8dd9b640 1 -- 172.21.15.28:0/635215797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677e0f0 msgr2=0x5606c67915d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.308+0000 7fdb8dd9b640 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677e0f0 0x5606c67915d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.308+0000 7fdb8dd9b640 1 -- 172.21.15.28:0/635215797 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c6779580 con 0x5606c677ea10 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.308+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 0x5606c6778d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.308+0000 7fdb97bbc740 10 client.0 _ll_register_callbacks cb 0x7fdb80005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.309+0000 7fdb8dd9b640 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677ea10 0x5606c6778810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x5606c6778250 tx=0x7fdb8401af90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3833710691b5bc53 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.309+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb8804a030 con 0x5606c677ea10 2024-05-09T08:30:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.324+0000 7fdb757fa640 10 client.26093 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:30:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.324+0000 7fdb757fa640 1 -- 172.21.15.28:0/635215797 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8401a540 con 0x5606c677ea10 2024-05-09T08:30:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.324+0000 7fdb757fa640 1 -- 172.21.15.28:0/635215797 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7fdb8401a6e0 con 0x5606c677ea10 2024-05-09T08:30:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.324+0000 7fdb757fa640 1 -- 172.21.15.28:0/635215797 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fdb8402a050 con 0x5606c677ea10 2024-05-09T08:30:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.328+0000 7fdb97bbc740 10 client.26093 fetch_fsmap learned FSMap version 45 2024-05-09T08:30:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.328+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({fsmap=45}) -- 0x5606c6807b10 con 0x5606c677ea10 2024-05-09T08:30:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.330+0000 7fdb757fa640 1 -- 172.21.15.28:0/635215797 <== mon.2 v2:172.21.15.165:3300/0 4 ==== fsmap(e 45) ==== 8410+0+0 (secure 0 0 0) 0x7fdb840035f0 con 0x5606c677ea10 2024-05-09T08:30:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.330+0000 7fdb97bbc740 10 client.26093 fetch_fsmap finished waiting for FSMap version 45 2024-05-09T08:30:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.330+0000 7fdb97bbc740 10 client.26093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:30:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.330+0000 7fdb97bbc740 20 client.26093 populate_metadata read hostname 'smithi028' 2024-05-09T08:30:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.330+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5606c6808110 0x5606c680a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.331+0000 7fdb97bbc740 4 client.26093 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:30:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.331+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fdb8804a030 con 0x5606c6808110 2024-05-09T08:30:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.331+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5606c6808110 0x5606c680a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.331+0000 7fdb8ed9d640 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5606c6808110 0x5606c680a530 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7798c6063f80544d server_cookie=2d19d571c1b178de in_seq=0 out_seq=0 2024-05-09T08:30:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:39.332+0000 7fdb757fa640 10 client.26093 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:39.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:39 smithi165 ceph-mon[34540]: pgmap v136: 129 pgs: 129 active+clean; 1.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.3 MiB/s wr, 328 op/s 2024-05-09T08:30:39.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:39 smithi184 ceph-mon[34509]: pgmap v136: 129 pgs: 129 active+clean; 1.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.3 MiB/s wr, 328 op/s 2024-05-09T08:30:39.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:39 smithi028 ceph-mon[30119]: pgmap v136: 129 pgs: 129 active+clean; 1.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.3 MiB/s wr, 328 op/s 2024-05-09T08:30:41.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb757fa640 1 -- 172.21.15.28:0/635215797 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fdb8804a030 con 0x5606c6808110 2024-05-09T08:30:41.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb757fa640 10 client.26093 handle_command_reply: tid=0 2024-05-09T08:30:41.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb97bbc740 1 client.26093 shutdown 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677ea10 msgr2=0x5606c6778810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677ea10 0x5606c6778810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x5606c6778250 tx=0x7fdb8401af90 comp rx=0 tx=0).stop 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 shutdown_connections 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5606c6808110 0x5606c680a530 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5606c67892d0 0x5606c6778d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5606c677ea10 0x5606c6778810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb97bbc740 1 --2- 172.21.15.28:0/635215797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606c677e0f0 0x5606c67915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 >> 172.21.15.28:0/635215797 conn(0x5606c677a600 msgr2=0x5606c6794540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.692+0000 7fdb757fa640 0 client.26093 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.693+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 shutdown_connections 2024-05-09T08:30:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.693+0000 7fdb97bbc740 1 -- 172.21.15.28:0/635215797 wait complete. 2024-05-09T08:30:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.693+0000 7fdb97bbc740 20 client.26093 trim_cache size 0 max 16384 2024-05-09T08:30:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:30:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:30:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:30:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:30:41.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.706+0000 7fdb7d7fa640 1 -- 172.21.15.28:0/2129473265 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb780a2ce0 msgr2=0x7fdb780a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.706+0000 7fdb7d7fa640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb780a2ce0 0x7fdb780a51a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x5606c677f690 tx=0x7fdb800980f0 comp rx=0 tx=0).stop 2024-05-09T08:30:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.707+0000 7fdb7d7fa640 1 -- 172.21.15.28:0/2129473265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb90108d70 msgr2=0x7fdb901a26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.707+0000 7fdb7d7fa640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb90108d70 0x7fdb901a26a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000e9c0 tx=0x7fdb8000ee80 comp rx=0 tx=0).stop 2024-05-09T08:30:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.710+0000 7fdb7d7fa640 1 -- 172.21.15.28:0/2129473265 shutdown_connections 2024-05-09T08:30:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.712+0000 7fdb7d7fa640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb90116830 0x7fdb901a6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.712+0000 7fdb7d7fa640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdb780a2ce0 0x7fdb780a51a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.712+0000 7fdb7d7fa640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9010bdd0 0x7fdb901a2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.712+0000 7fdb7d7fa640 1 --2- 172.21.15.28:0/2129473265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb90108d70 0x7fdb901a26a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.712+0000 7fdb7d7fa640 1 -- 172.21.15.28:0/2129473265 >> 172.21.15.28:0/2129473265 conn(0x7fdb9006c7e0 msgr2=0x7fdb90116c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.716+0000 7fdb7d7fa640 1 -- 172.21.15.28:0/2129473265 shutdown_connections 2024-05-09T08:30:41.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:41.716+0000 7fdb7d7fa640 1 -- 172.21.15.28:0/2129473265 wait complete. 2024-05-09T08:30:41.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:41 smithi165 ceph-mon[34540]: pgmap v137: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9.0 MiB/s wr, 428 op/s 2024-05-09T08:30:41.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:41 smithi184 ceph-mon[34509]: pgmap v137: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9.0 MiB/s wr, 428 op/s 2024-05-09T08:30:41.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:41 smithi028 ceph-mon[30119]: pgmap v137: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9.0 MiB/s wr, 428 op/s 2024-05-09T08:30:43.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:43 smithi165 ceph-mon[34540]: pgmap v138: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 10 MiB/s wr, 544 op/s 2024-05-09T08:30:43.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:43 smithi184 ceph-mon[34509]: pgmap v138: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 10 MiB/s wr, 544 op/s 2024-05-09T08:30:43.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:43 smithi028 ceph-mon[30119]: pgmap v138: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 10 MiB/s wr, 544 op/s 2024-05-09T08:30:45.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:45 smithi165 ceph-mon[34540]: pgmap v139: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 10 MiB/s wr, 589 op/s 2024-05-09T08:30:45.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:45 smithi184 ceph-mon[34509]: pgmap v139: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 10 MiB/s wr, 589 op/s 2024-05-09T08:30:45.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:45 smithi028 ceph-mon[30119]: pgmap v139: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 10 MiB/s wr, 589 op/s 2024-05-09T08:30:47.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:47 smithi165 ceph-mon[34540]: pgmap v140: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 805 op/s 2024-05-09T08:30:47.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:47 smithi184 ceph-mon[34509]: pgmap v140: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 805 op/s 2024-05-09T08:30:47.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:47 smithi028 ceph-mon[30119]: pgmap v140: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 805 op/s 2024-05-09T08:30:48.708 INFO:tasks.workunit.client.0.smithi028.stdout: 422 2209302 2781 1490369 2446 356249 77 2024-05-09T08:30:48.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:48 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:48.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:48 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:48.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:48 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:30:48.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:48 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:48.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:48 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:48.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:48 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:30:48.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:48 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:48.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:48 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:48.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:48 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:30:49.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:49 smithi165 ceph-mon[34540]: pgmap v141: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 7.5 MiB/s wr, 652 op/s 2024-05-09T08:30:49.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:49 smithi184 ceph-mon[34509]: pgmap v141: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 7.5 MiB/s wr, 652 op/s 2024-05-09T08:30:49.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:49 smithi028 ceph-mon[30119]: pgmap v141: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 7.5 MiB/s wr, 652 op/s 2024-05-09T08:30:50.956 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:50 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68479]: *** Caught signal (Segmentation fault) ** 2024-05-09T08:30:50.956 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:50 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68479]: in thread 7f5522d4c700 thread_name:msgr-worker-1 2024-05-09T08:30:50.956 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:50 smithi165 podman[68690]: 2024-05-09 08:30:50.662776916 +0000 UTC m=+0.026250481 container died 8f2f316a2496eafaaa14eb7eb128a6a7859df21b7479d8c32ca3151fb07787d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc) 2024-05-09T08:30:51.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:51 smithi165 ceph-mon[34540]: pgmap v142: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.1 MiB/s wr, 695 op/s 2024-05-09T08:30:51.560 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:51 smithi165 podman[68690]: 2024-05-09 08:30:51.283672936 +0000 UTC m=+0.647146494 container remove 8f2f316a2496eafaaa14eb7eb128a6a7859df21b7479d8c32ca3151fb07787d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-96805dc) 2024-05-09T08:30:51.560 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:51 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Main process exited, code=exited, status=139/n/a 2024-05-09T08:30:51.850 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:51 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-09T08:30:51.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:30:51 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Consumed 3.240s CPU time. 2024-05-09T08:30:51.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:51 smithi184 ceph-mon[34509]: pgmap v142: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.1 MiB/s wr, 695 op/s 2024-05-09T08:30:51.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:51 smithi028 ceph-mon[30119]: pgmap v142: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.1 MiB/s wr, 695 op/s 2024-05-09T08:30:53.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:53 smithi165 ceph-mon[34540]: pgmap v143: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.9 MiB/s wr, 528 op/s 2024-05-09T08:30:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:53 smithi184 ceph-mon[34509]: pgmap v143: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.9 MiB/s wr, 528 op/s 2024-05-09T08:30:53.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:53 smithi028 ceph-mon[30119]: pgmap v143: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.9 MiB/s wr, 528 op/s 2024-05-09T08:30:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.715+0000 7f84d1f63640 1 -- 172.21.15.28:0/4210239253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc11b010 msgr2=0x7f84cc11d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.715+0000 7f84d1f63640 1 --2- 172.21.15.28:0/4210239253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc11b010 0x7f84cc11d400 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f84c401ace0 tx=0x7f84c403b9e0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.719+0000 7f84d1f63640 1 -- 172.21.15.28:0/4210239253 shutdown_connections 2024-05-09T08:30:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.719+0000 7f84d1f63640 1 --2- 172.21.15.28:0/4210239253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc11b010 0x7f84cc11d400 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.719+0000 7f84d1f63640 1 --2- 172.21.15.28:0/4210239253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84cc110400 0x7f84cc108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.719+0000 7f84d1f63640 1 --2- 172.21.15.28:0/4210239253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84cc110020 0x7f84cc108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.719+0000 7f84d1f63640 1 -- 172.21.15.28:0/4210239253 >> 172.21.15.28:0/4210239253 conn(0x7f84cc06c900 msgr2=0x7f84cc06cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.722+0000 7f84d1f63640 1 -- 172.21.15.28:0/4210239253 shutdown_connections 2024-05-09T08:30:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.722+0000 7f84d1f63640 1 -- 172.21.15.28:0/4210239253 wait complete. 2024-05-09T08:30:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.723+0000 7f84d1f63640 1 Processor -- start 2024-05-09T08:30:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.723+0000 7f84d1f63640 1 -- start start 2024-05-09T08:30:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.723+0000 7f84d1f63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc110020 0x7f84cc11a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.723+0000 7f84d1f63640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84cc11b010 0x7f84cc1167c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.723+0000 7f84d1f63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84cc116d00 0x7f84cc1171b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.723+0000 7f84d1f63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc113180 con 0x7f84cc110020 2024-05-09T08:30:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.723+0000 7f84d1f63640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc113000 con 0x7f84cc11b010 2024-05-09T08:30:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.723+0000 7f84d1f63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc113300 con 0x7f84cc116d00 2024-05-09T08:30:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.724+0000 7f84cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc110020 0x7f84cc11a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.724+0000 7f84cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc110020 0x7f84cc11a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41486/0 (socket says 172.21.15.28:41486) 2024-05-09T08:30:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.724+0000 7f84cb7fe640 1 -- 172.21.15.28:0/2082162506 learned_addr learned my addr 172.21.15.28:0/2082162506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.725+0000 7f84cb7fe640 1 -- 172.21.15.28:0/2082162506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84cc116d00 msgr2=0x7f84cc1171b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.725+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84cc116d00 0x7f84cc1171b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.725+0000 7f84cb7fe640 1 -- 172.21.15.28:0/2082162506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84cc11b010 msgr2=0x7f84cc1167c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.725+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84cc11b010 0x7f84cc1167c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.725+0000 7f84cb7fe640 1 -- 172.21.15.28:0/2082162506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84cc1cce10 con 0x7f84cc110020 2024-05-09T08:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.726+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc110020 0x7f84cc11a730 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00d950 tx=0x7f84bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.729+0000 7f84c8ff9640 1 -- 172.21.15.28:0/2082162506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc017070 con 0x7f84cc110020 2024-05-09T08:30:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.729+0000 7f84c8ff9640 1 -- 172.21.15.28:0/2082162506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84bc009bb0 con 0x7f84cc110020 2024-05-09T08:30:55.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.729+0000 7f84c8ff9640 1 -- 172.21.15.28:0/2082162506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc00f600 con 0x7f84cc110020 2024-05-09T08:30:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.729+0000 7f84d1f63640 1 -- 172.21.15.28:0/2082162506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84cc1cd100 con 0x7f84cc110020 2024-05-09T08:30:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.733+0000 7f84d1f63640 1 -- 172.21.15.28:0/2082162506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84cc1cd5d0 con 0x7f84cc110020 2024-05-09T08:30:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.744+0000 7f84d4c77740 1 Processor -- start 2024-05-09T08:30:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.748+0000 7f84d4c77740 1 -- start start 2024-05-09T08:30:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.749+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619aa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.749+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.750+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.750+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619aa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:57804/0 (socket says 172.21.15.28:57804) 2024-05-09T08:30:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.750+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.750+0000 7f84d4c77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c46190f340 con 0x55c4619b5630 2024-05-09T08:30:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.750+0000 7f84d4c77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55c4618b7290 con 0x55c4619aad70 2024-05-09T08:30:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.750+0000 7f84d4c77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c46190b300 con 0x55c4619aa450 2024-05-09T08:30:55.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.750+0000 7f84cbfff640 1 -- 172.21.15.28:0/173008265 learned_addr learned my addr 172.21.15.28:0/173008265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.751+0000 7f84c8ff9640 1 -- 172.21.15.28:0/2082162506 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f84bc010940 con 0x7f84cc110020 2024-05-09T08:30:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.752+0000 7f84c8ff9640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84b40a2fd0 0x7f84b40a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.752+0000 7f84c8ff9640 1 -- 172.21.15.28:0/2082162506 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f84bc0cf220 con 0x7f84cc110020 2024-05-09T08:30:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.753+0000 7f84caffd640 1 --2- 172.21.15.28:0/173008265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.753+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/173008265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.754+0000 7f84cbfff640 1 -- 172.21.15.28:0/173008265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 msgr2=0x55c4619b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.754+0000 7f84cbfff640 1 --2- 172.21.15.28:0/173008265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619b50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.754+0000 7f84cbfff640 1 -- 172.21.15.28:0/173008265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 msgr2=0x55c4619b7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.754+0000 7f84cbfff640 1 --2- 172.21.15.28:0/173008265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.754+0000 7f84cbfff640 1 -- 172.21.15.28:0/173008265 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4618db8e0 con 0x55c4619aa450 2024-05-09T08:30:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.754+0000 7f84cbfff640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84b40a2fd0 0x7f84b40a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.754+0000 7f84cbfff640 1 --2- 172.21.15.28:0/173008265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619aa830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84c4005e30 tx=0x7f84c4047550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c741ce8e45f6b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.758+0000 7f84cbfff640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84b40a2fd0 0x7f84b40a5490 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f84c4017000 tx=0x7f84c4040b70 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.759+0000 7f84b1ffb640 1 -- 172.21.15.28:0/173008265 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4013030 con 0x55c4619aa450 2024-05-09T08:30:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.759+0000 7f84b1ffb640 1 -- 172.21.15.28:0/173008265 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c4003e50 con 0x55c4619aa450 2024-05-09T08:30:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.760+0000 7f84d4c77740 1 -- 172.21.15.28:0/173008265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 msgr2=0x55c4619aa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.760+0000 7f84d4c77740 1 --2- 172.21.15.28:0/173008265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619aa830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84c4005e30 tx=0x7f84c4047550 comp rx=0 tx=0).stop 2024-05-09T08:30:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.765+0000 7f84d4c77740 1 -- 172.21.15.28:0/173008265 shutdown_connections 2024-05-09T08:30:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.765+0000 7f84d4c77740 1 --2- 172.21.15.28:0/173008265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.765+0000 7f84d4c77740 1 --2- 172.21.15.28:0/173008265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.765+0000 7f84d4c77740 1 --2- 172.21.15.28:0/173008265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619aa830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.765+0000 7f84d4c77740 1 -- 172.21.15.28:0/173008265 >> 172.21.15.28:0/173008265 conn(0x55c4619a6960 msgr2=0x55c4619a6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.766+0000 7f84d4c77740 1 -- 172.21.15.28:0/173008265 shutdown_connections 2024-05-09T08:30:55.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.769+0000 7f84d4c77740 1 -- 172.21.15.28:0/173008265 wait complete. 2024-05-09T08:30:55.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.770+0000 7f84d4c77740 1 -- start start 2024-05-09T08:30:55.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.770+0000 7f84d4c77740 1 Processor -- start 2024-05-09T08:30:55.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.770+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 0x55c4619b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.770+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aad70 0x55c4619b2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.770+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.770+0000 7f84d4c77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4619b8200 con 0x55c4619b5630 2024-05-09T08:30:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.770+0000 7f84d4c77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55c4618ca790 con 0x55c4619aa450 2024-05-09T08:30:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.770+0000 7f84d4c77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c4619b8380 con 0x55c4619aad70 2024-05-09T08:30:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.770+0000 7f84d4c77740 10 client.0 _ll_register_callbacks cb 0x55c461905a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:30:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.772+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.772+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41500/0 (socket says 172.21.15.28:41500) 2024-05-09T08:30:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.772+0000 7f84cbfff640 1 -- 172.21.15.28:0/85259579 learned_addr learned my addr 172.21.15.28:0/85259579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.772+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 0x55c4619b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.772+0000 7f84caffd640 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aad70 0x55c4619b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.775+0000 7f84cbfff640 1 -- 172.21.15.28:0/85259579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aad70 msgr2=0x55c4619b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.775+0000 7f84cbfff640 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aad70 0x55c4619b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.775+0000 7f84cbfff640 1 -- 172.21.15.28:0/85259579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 msgr2=0x55c4619b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.776+0000 7f84cbfff640 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 0x55c4619b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.776+0000 7f84cbfff640 1 -- 172.21.15.28:0/85259579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c461623210 con 0x55c4619b5630 2024-05-09T08:30:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.776+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 0x55c4619b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:30:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.779+0000 7f84cbfff640 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b2bb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f84c4003e20 tx=0x7f84c4009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16379e2ee23cbe3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.780+0000 7f84aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:30:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.780+0000 7f84aaffd640 1 -- 172.21.15.28:0/85259579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4057070 con 0x55c4619b5630 2024-05-09T08:30:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.780+0000 7f84d4c77740 1 -- 172.21.15.28:0/85259579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84c4003e50 con 0x55c4619b5630 2024-05-09T08:30:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.780+0000 7f84aaffd640 1 -- 172.21.15.28:0/85259579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f84c4047ce0 con 0x55c4619b5630 2024-05-09T08:30:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.784+0000 7f84aaffd640 1 -- 172.21.15.28:0/85259579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f84c4061050 con 0x55c4619b5630 2024-05-09T08:30:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.784+0000 7f84d4c77740 10 client.16626 fetch_fsmap learned FSMap version 45 2024-05-09T08:30:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.784+0000 7f84d4c77740 1 -- 172.21.15.28:0/85259579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c4616ef620 con 0x55c4619b5630 2024-05-09T08:30:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.787+0000 7f84aaffd640 1 -- 172.21.15.28:0/85259579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c4003e50 con 0x55c4619b5630 2024-05-09T08:30:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.787+0000 7f84aaffd640 1 -- 172.21.15.28:0/85259579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 45) ==== 8410+0+0 (secure 0 0 0) 0x7f84c4049b10 con 0x55c4619b5630 2024-05-09T08:30:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.787+0000 7f84d4c77740 10 client.16626 fetch_fsmap finished waiting for FSMap version 45 2024-05-09T08:30:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.787+0000 7f84d4c77740 10 client.16626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:30:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.787+0000 7f84d4c77740 20 client.16626 populate_metadata read hostname 'smithi028' 2024-05-09T08:30:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.787+0000 7f84d4c77740 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c4619c54e0 0x55c4619c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.788+0000 7f84d4c77740 4 client.16626 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:30:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.788+0000 7f84d4c77740 1 -- 172.21.15.28:0/85259579 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c46191ca70 con 0x55c4619c54e0 2024-05-09T08:30:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.789+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c4619c54e0 0x55c4619c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.801+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c4619c54e0 0x55c4619c7980 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6c5962f50fa2ed7 server_cookie=fd3c73f322d2f4c8 in_seq=0 out_seq=0 2024-05-09T08:30:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.801+0000 7f84aaffd640 10 client.16626 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.805+0000 7f84aaffd640 1 -- 172.21.15.28:0/85259579 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c46191ca70 con 0x55c4619c54e0 2024-05-09T08:30:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.805+0000 7f84aaffd640 10 client.16626 handle_command_reply: tid=0 2024-05-09T08:30:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.807+0000 7f84d4c77740 1 client.16626 shutdown 2024-05-09T08:30:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.807+0000 7f84d4c77740 1 -- 172.21.15.28:0/85259579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 msgr2=0x55c4619b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.807+0000 7f84d4c77740 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b2bb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f84c4003e20 tx=0x7f84c4009000 comp rx=0 tx=0).stop 2024-05-09T08:30:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.813+0000 7f84d4c77740 1 -- 172.21.15.28:0/85259579 shutdown_connections 2024-05-09T08:30:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.813+0000 7f84d4c77740 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c4619c54e0 0x55c4619c7980 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.813+0000 7f84d4c77740 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619b2bb0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.813+0000 7f84d4c77740 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aad70 0x55c4619b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.813+0000 7f84d4c77740 1 --2- 172.21.15.28:0/85259579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 0x55c4619b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.813+0000 7f84d4c77740 1 -- 172.21.15.28:0/85259579 >> 172.21.15.28:0/85259579 conn(0x55c4619a6960 msgr2=0x55c4619ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.813+0000 7f84aaffd640 0 client.16626 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.815+0000 7f84d4c77740 1 -- 172.21.15.28:0/85259579 shutdown_connections 2024-05-09T08:30:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.815+0000 7f84d4c77740 1 -- 172.21.15.28:0/85259579 wait complete. 2024-05-09T08:30:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.819+0000 7f84d4c77740 20 client.16626 trim_cache size 0 max 16384 2024-05-09T08:30:55.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:55 smithi165 ceph-mon[34540]: pgmap v144: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 371 op/s 2024-05-09T08:30:55.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:55 smithi184 ceph-mon[34509]: pgmap v144: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 371 op/s 2024-05-09T08:30:55.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.857+0000 7f84d4c77740 1 Processor -- start 2024-05-09T08:30:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.857+0000 7f84d4c77740 1 -- start start 2024-05-09T08:30:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.858+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 0x55c4619b1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.858+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619aad70 0x55c4619b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.858+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619b5630 0x55c4619b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.858+0000 7f84d4c77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1d7510 con 0x55c4619aad70 2024-05-09T08:30:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.858+0000 7f84d4c77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1d91d0 con 0x55c4619aa450 2024-05-09T08:30:55.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.858+0000 7f84d4c77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1d7be0 con 0x55c4619b5630 2024-05-09T08:30:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.859+0000 7f84caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619aad70 0x55c4619b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.859+0000 7f84caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619aad70 0x55c4619b4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41516/0 (socket says 172.21.15.28:41516) 2024-05-09T08:30:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.859+0000 7f84caffd640 1 -- 172.21.15.28:0/3946299072 learned_addr learned my addr 172.21.15.28:0/3946299072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.859+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/3946299072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 0x55c4619b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.860+0000 7f84caffd640 1 -- 172.21.15.28:0/3946299072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619b5630 msgr2=0x55c4619b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.860+0000 7f84caffd640 1 --2- 172.21.15.28:0/3946299072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619b5630 0x55c4619b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.860+0000 7f84caffd640 1 -- 172.21.15.28:0/3946299072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 msgr2=0x55c4619b1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.860+0000 7f84caffd640 1 --2- 172.21.15.28:0/3946299072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 0x55c4619b1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.860+0000 7f84caffd640 1 -- 172.21.15.28:0/3946299072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4619b8380 con 0x55c4619aad70 2024-05-09T08:30:55.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.861+0000 7f84caffd640 1 --2- 172.21.15.28:0/3946299072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619aad70 0x55c4619b4ba0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f84cc06af00 tx=0x7f84c000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71092a5c5db226 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.863+0000 7f84b17fa640 1 -- 172.21.15.28:0/3946299072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c0010e80 con 0x55c4619aad70 2024-05-09T08:30:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.863+0000 7f84b17fa640 1 -- 172.21.15.28:0/3946299072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c0011920 con 0x55c4619aad70 2024-05-09T08:30:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.863+0000 7f84b17fa640 1 -- 172.21.15.28:0/3946299072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c001a9c0 con 0x55c4619aad70 2024-05-09T08:30:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.865+0000 7f84d4c77740 1 -- 172.21.15.28:0/3946299072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619aad70 msgr2=0x55c4619b4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.865+0000 7f84d4c77740 1 --2- 172.21.15.28:0/3946299072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619aad70 0x55c4619b4ba0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f84cc06af00 tx=0x7f84c000edf0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.870+0000 7f84d4c77740 1 -- 172.21.15.28:0/3946299072 shutdown_connections 2024-05-09T08:30:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.870+0000 7f84d4c77740 1 --2- 172.21.15.28:0/3946299072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619b5630 0x55c4619b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.870+0000 7f84d4c77740 1 --2- 172.21.15.28:0/3946299072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619aad70 0x55c4619b4ba0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.870+0000 7f84d4c77740 1 --2- 172.21.15.28:0/3946299072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aa450 0x55c4619b1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.870+0000 7f84d4c77740 1 -- 172.21.15.28:0/3946299072 >> 172.21.15.28:0/3946299072 conn(0x55c4619a6960 msgr2=0x55c4619b0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:55.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.874+0000 7f84d4c77740 1 -- 172.21.15.28:0/3946299072 shutdown_connections 2024-05-09T08:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.875+0000 7f84d4c77740 1 -- 172.21.15.28:0/3946299072 wait complete. 2024-05-09T08:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.876+0000 7f84d4c77740 1 -- start start 2024-05-09T08:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.877+0000 7f84d4c77740 1 Processor -- start 2024-05-09T08:30:55.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.877+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.877+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619aed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.878+0000 7f84d4c77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619af240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.878+0000 7f84d4c77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1d8b00 con 0x55c4619b5630 2024-05-09T08:30:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.878+0000 7f84d4c77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1d8370 con 0x55c4619aad70 2024-05-09T08:30:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.878+0000 7f84d4c77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1d54f0 con 0x55c4619aa450 2024-05-09T08:30:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.878+0000 7f84d4c77740 10 client.0 _ll_register_callbacks cb 0x7f84c4047430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:30:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.879+0000 7f84caffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.879+0000 7f84cb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.879+0000 7f84caffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619aed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:47188/0 (socket says 172.21.15.28:47188) 2024-05-09T08:30:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.879+0000 7f84caffd640 1 -- 172.21.15.28:0/2572784447 learned_addr learned my addr 172.21.15.28:0/2572784447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:30:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.880+0000 7f84caffd640 1 -- 172.21.15.28:0/2572784447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 msgr2=0x55c4619ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.880+0000 7f84caffd640 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619ae7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.880+0000 7f84caffd640 1 -- 172.21.15.28:0/2572784447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 msgr2=0x55c4619af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:30:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.880+0000 7f84caffd640 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.880+0000 7f84caffd640 1 -- 172.21.15.28:0/2572784447 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4619bc750 con 0x55c4619aad70 2024-05-09T08:30:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.880+0000 7f84caffd640 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619aed00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f84c00118f0 tx=0x7f84c000e4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=576910bc07f06a9a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.880+0000 7f84aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.884+0000 7f84aaffd640 1 -- 172.21.15.28:0/2572784447 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c001a640 con 0x55c4619aad70 2024-05-09T08:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.884+0000 7f84aaffd640 1 -- 172.21.15.28:0/2572784447 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f84c001a7e0 con 0x55c4619aad70 2024-05-09T08:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.885+0000 7f84d4c77740 1 -- 172.21.15.28:0/2572784447 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84c0011920 con 0x55c4619aad70 2024-05-09T08:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.886+0000 7f84aaffd640 1 -- 172.21.15.28:0/2572784447 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f84c001a9e0 con 0x55c4619aad70 2024-05-09T08:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.887+0000 7f84d4c77740 10 client.26108 fetch_fsmap learned FSMap version 45 2024-05-09T08:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.887+0000 7f84d4c77740 1 -- 172.21.15.28:0/2572784447 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c4619bda60 con 0x55c4619aad70 2024-05-09T08:30:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.890+0000 7f84aaffd640 1 -- 172.21.15.28:0/2572784447 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c0011920 con 0x55c4619aad70 2024-05-09T08:30:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.890+0000 7f84aaffd640 1 -- 172.21.15.28:0/2572784447 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 45) ==== 8410+0+0 (secure 0 0 0) 0x7f84c0028450 con 0x55c4619aad70 2024-05-09T08:30:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.890+0000 7f84d4c77740 10 client.26108 fetch_fsmap finished waiting for FSMap version 45 2024-05-09T08:30:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.890+0000 7f84d4c77740 10 client.26108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:30:55.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.890+0000 7f84d4c77740 20 client.26108 populate_metadata read hostname 'smithi028' 2024-05-09T08:30:55.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.890+0000 7f84d4c77740 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c461a2bac0 0x55c461a2deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:55.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.890+0000 7f84d4c77740 4 client.26108 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:30:55.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.890+0000 7f84d4c77740 1 -- 172.21.15.28:0/2572784447 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c461916dc0 con 0x55c461a2bac0 2024-05-09T08:30:55.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.893+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c461a2bac0 0x55c461a2deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:55.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.899+0000 7f84cb7fe640 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c461a2bac0 0x55c461a2deb0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49faed0149d157c5 server_cookie=c77dcd560a73b9fa in_seq=0 out_seq=0 2024-05-09T08:30:55.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:30:55.900+0000 7f84aaffd640 10 client.26108 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:30:55.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:55 smithi028 ceph-mon[30119]: pgmap v144: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 371 op/s 2024-05-09T08:30:57.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:57 smithi165 ceph-mon[34540]: pgmap v145: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.5 MiB/s wr, 470 op/s 2024-05-09T08:30:57.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:57 smithi184 ceph-mon[34509]: pgmap v145: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.5 MiB/s wr, 470 op/s 2024-05-09T08:30:57.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:57 smithi028 ceph-mon[30119]: pgmap v145: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.5 MiB/s wr, 470 op/s 2024-05-09T08:30:58.708 INFO:tasks.workunit.client.0.smithi028.stdout: 451 1567244 1130 1060084 1453 233543 817 2024-05-09T08:30:59.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:30:59 smithi165 ceph-mon[34540]: pgmap v146: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 265 op/s 2024-05-09T08:30:59.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:30:59 smithi184 ceph-mon[34509]: pgmap v146: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 265 op/s 2024-05-09T08:30:59.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:30:59 smithi028 ceph-mon[30119]: pgmap v146: 129 pgs: 129 active+clean; 1.3 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 265 op/s 2024-05-09T08:31:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84aaffd640 1 -- 172.21.15.28:0/2572784447 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c461916dc0 con 0x55c461a2bac0 2024-05-09T08:31:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84aaffd640 10 client.26108 handle_command_reply: tid=0 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 client.26108 shutdown 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 -- 172.21.15.28:0/2572784447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 msgr2=0x55c4619aed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619aed00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f84c00118f0 tx=0x7f84c000e4e0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 -- 172.21.15.28:0/2572784447 shutdown_connections 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c461a2bac0 0x55c461a2deb0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4619b5630 0x55c4619af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c4619aad70 0x55c4619aed00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84aaffd640 0 client.26108 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 --2- 172.21.15.28:0/2572784447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c4619aa450 0x55c4619ae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 -- 172.21.15.28:0/2572784447 >> 172.21.15.28:0/2572784447 conn(0x55c4619a6960 msgr2=0x55c4619b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 -- 172.21.15.28:0/2572784447 shutdown_connections 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.547+0000 7f84d4c77740 1 -- 172.21.15.28:0/2572784447 wait complete. 2024-05-09T08:31:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d4c77740 20 client.26108 trim_cache size 0 max 16384 2024-05-09T08:31:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:31:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:31:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:31:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 -- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84b40a2fd0 msgr2=0x7f84b40a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84b40a2fd0 0x7f84b40a5490 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f84c4017000 tx=0x7f84c4040b70 comp rx=0 tx=0).stop 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 -- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc110020 msgr2=0x7f84cc11a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc110020 0x7f84cc11a730 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00d950 tx=0x7f84bc00de10 comp rx=0 tx=0).stop 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 -- 172.21.15.28:0/2082162506 shutdown_connections 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84b40a2fd0 0x7f84b40a5490 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84cc116d00 0x7f84cc1171b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84cc11b010 0x7f84cc1167c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 --2- 172.21.15.28:0/2082162506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84cc110020 0x7f84cc11a730 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.548+0000 7f84d1f63640 1 -- 172.21.15.28:0/2082162506 >> 172.21.15.28:0/2082162506 conn(0x7f84cc06c900 msgr2=0x7f84cc11b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.549+0000 7f84d1f63640 1 -- 172.21.15.28:0/2082162506 shutdown_connections 2024-05-09T08:31:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:00.549+0000 7f84d1f63640 1 -- 172.21.15.28:0/2082162506 wait complete. 2024-05-09T08:31:01.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:01 smithi165 ceph-mon[34540]: pgmap v147: 129 pgs: 129 active+clean; 1.4 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 758 op/s 2024-05-09T08:31:01.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:01 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-05-09T08:31:01.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:01 smithi165 systemd[1]: Stopped Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:31:01.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:01 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Consumed 3.240s CPU time. 2024-05-09T08:31:01.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:01 smithi165 systemd[1]: Starting Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:31:01.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:01 smithi184 ceph-mon[34509]: pgmap v147: 129 pgs: 129 active+clean; 1.4 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 758 op/s 2024-05-09T08:31:01.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:01 smithi028 ceph-mon[30119]: pgmap v147: 129 pgs: 129 active+clean; 1.4 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 758 op/s 2024-05-09T08:31:02.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:01 smithi165 podman[68798]: 2024-05-09 08:31:01.778173186 +0000 UTC m=+0.024611304 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:31:03.548 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:03 smithi165 podman[68798]: 2024-05-09 08:31:03.159732845 +0000 UTC m=+1.406171051 container create 6e8c0a675b24fc008fd9f15a680878b0f6f6249cecfcd0b98e515421fcda754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-09T08:31:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:03 smithi165 ceph-mon[34540]: pgmap v148: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 18 MiB/s wr, 1.26k op/s 2024-05-09T08:31:03.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:03 smithi184 ceph-mon[34509]: pgmap v148: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 18 MiB/s wr, 1.26k op/s 2024-05-09T08:31:03.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:03 smithi028 ceph-mon[30119]: pgmap v148: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 18 MiB/s wr, 1.26k op/s 2024-05-09T08:31:05.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:05 smithi165 ceph-mon[34540]: pgmap v149: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 18 MiB/s wr, 1.25k op/s 2024-05-09T08:31:05.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:05 smithi165 ceph-mon[34540]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:31:05.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:05 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:31:05.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:05 smithi184 ceph-mon[34509]: pgmap v149: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 18 MiB/s wr, 1.25k op/s 2024-05-09T08:31:05.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:05 smithi184 ceph-mon[34509]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:31:05.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:05 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:31:05.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:05 smithi028 ceph-mon[30119]: pgmap v149: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 18 MiB/s wr, 1.25k op/s 2024-05-09T08:31:05.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:05 smithi028 ceph-mon[30119]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:31:05.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:05 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:31:06.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:06 smithi165 podman[68798]: 2024-05-09 08:31:06.022397648 +0000 UTC m=+4.268835761 container init 6e8c0a675b24fc008fd9f15a680878b0f6f6249cecfcd0b98e515421fcda754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:31:06.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:06 smithi165 podman[68798]: 2024-05-09 08:31:06.025963799 +0000 UTC m=+4.272401911 container start 6e8c0a675b24fc008fd9f15a680878b0f6f6249cecfcd0b98e515421fcda754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-09T08:31:06.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:06 smithi165 ceph-mds[68819]: -- 172.21.15.165:0/2407974545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d3e685ba40 con 0x55d3e682e800 2024-05-09T08:31:06.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:06 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: starting mds.e at 2024-05-09T08:31:07.557 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:07 smithi165 bash[68798]: 6e8c0a675b24fc008fd9f15a680878b0f6f6249cecfcd0b98e515421fcda754f 2024-05-09T08:31:07.557 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:31:07 smithi165 systemd[1]: Started Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:31:07.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:07 smithi165 ceph-mon[34540]: pgmap v150: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 19 MiB/s wr, 1.33k op/s 2024-05-09T08:31:07.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:07 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6834/764741091,v1:172.21.15.165:6835/764741091] up:boot 2024-05-09T08:31:07.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:07 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:07.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:07 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:31:07.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:07 smithi184 ceph-mon[34509]: pgmap v150: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 19 MiB/s wr, 1.33k op/s 2024-05-09T08:31:07.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:07 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6834/764741091,v1:172.21.15.165:6835/764741091] up:boot 2024-05-09T08:31:07.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:07 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:07.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:07 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:31:07.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:07 smithi028 ceph-mon[30119]: pgmap v150: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 19 MiB/s wr, 1.33k op/s 2024-05-09T08:31:07.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:07 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6834/764741091,v1:172.21.15.165:6835/764741091] up:boot 2024-05-09T08:31:07.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:07 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:07.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:07 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:31:08.708 INFO:tasks.workunit.client.0.smithi028.stdout: 460 99136 529 67262 363 18252 453 2024-05-09T08:31:09.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:09 smithi165 ceph-mon[34540]: pgmap v151: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 15 MiB/s wr, 1.18k op/s 2024-05-09T08:31:09.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:09 smithi184 ceph-mon[34509]: pgmap v151: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 15 MiB/s wr, 1.18k op/s 2024-05-09T08:31:09.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:09 smithi028 ceph-mon[30119]: pgmap v151: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 15 MiB/s wr, 1.18k op/s 2024-05-09T08:31:11.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:11 smithi165 ceph-mon[34540]: pgmap v152: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 15 MiB/s wr, 1.20k op/s 2024-05-09T08:31:11.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:11 smithi184 ceph-mon[34509]: pgmap v152: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 15 MiB/s wr, 1.20k op/s 2024-05-09T08:31:11.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:11 smithi028 ceph-mon[30119]: pgmap v152: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 15 MiB/s wr, 1.20k op/s 2024-05-09T08:31:13.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:13 smithi165 ceph-mon[34540]: pgmap v153: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 8.3 MiB/s wr, 738 op/s 2024-05-09T08:31:13.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:13 smithi184 ceph-mon[34509]: pgmap v153: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 8.3 MiB/s wr, 738 op/s 2024-05-09T08:31:13.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:13 smithi028 ceph-mon[30119]: pgmap v153: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 8.3 MiB/s wr, 738 op/s 2024-05-09T08:31:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.132+0000 7f230cb5c640 1 -- 172.21.15.28:0/2437932439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230810db00 msgr2=0x7f230810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.132+0000 7f230cb5c640 1 --2- 172.21.15.28:0/2437932439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230810db00 0x7f230810df80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2300009f00 tx=0x7f230002ab60 comp rx=0 tx=0).stop 2024-05-09T08:31:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.137+0000 7f230cb5c640 1 -- 172.21.15.28:0/2437932439 shutdown_connections 2024-05-09T08:31:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.137+0000 7f230cb5c640 1 --2- 172.21.15.28:0/2437932439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f230810e5a0 0x7f2308108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.137+0000 7f230cb5c640 1 --2- 172.21.15.28:0/2437932439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230810db00 0x7f230810df80 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.137+0000 7f230cb5c640 1 --2- 172.21.15.28:0/2437932439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230810fda0 0x7f2308110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.137+0000 7f230cb5c640 1 -- 172.21.15.28:0/2437932439 >> 172.21.15.28:0/2437932439 conn(0x7f230806c7e0 msgr2=0x7f230806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.137+0000 7f230cb5c640 1 -- 172.21.15.28:0/2437932439 shutdown_connections 2024-05-09T08:31:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.137+0000 7f230cb5c640 1 -- 172.21.15.28:0/2437932439 wait complete. 2024-05-09T08:31:15.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.140+0000 7f230cb5c640 1 Processor -- start 2024-05-09T08:31:15.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.140+0000 7f230cb5c640 1 -- start start 2024-05-09T08:31:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.141+0000 7f230cb5c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230810db00 0x7f23081a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.141+0000 7f230cb5c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f230810e5a0 0x7f23081a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.142+0000 7f230cb5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230810fda0 0x7f23081ab790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.142+0000 7f2305d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f230810e5a0 0x7f23081a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.142+0000 7f2305d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f230810e5a0 0x7f23081a7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38806/0 (socket says 172.21.15.28:38806) 2024-05-09T08:31:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.142+0000 7f2305d9b640 1 -- 172.21.15.28:0/3571484661 learned_addr learned my addr 172.21.15.28:0/3571484661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.143+0000 7f2306d9d640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230810fda0 0x7f23081ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.143+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2308120970 con 0x7f230810fda0 2024-05-09T08:31:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.143+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23081207f0 con 0x7f230810db00 2024-05-09T08:31:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.143+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2308120af0 con 0x7f230810e5a0 2024-05-09T08:31:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.144+0000 7f230659c640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230810db00 0x7f23081a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.144+0000 7f2305d9b640 1 -- 172.21.15.28:0/3571484661 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230810db00 msgr2=0x7f23081a6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.144+0000 7f2305d9b640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230810db00 0x7f23081a6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.144+0000 7f2305d9b640 1 -- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230810fda0 msgr2=0x7f23081ab790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.145+0000 7f2305d9b640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230810fda0 0x7f23081ab790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.145+0000 7f2305d9b640 1 -- 172.21.15.28:0/3571484661 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23081abec0 con 0x7f230810e5a0 2024-05-09T08:31:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.145+0000 7f2306d9d640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230810fda0 0x7f23081ab790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.145+0000 7f2305d9b640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f230810e5a0 0x7f23081a7410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2300000f80 tx=0x7f2300034770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.186+0000 7f22f77fe640 1 -- 172.21.15.28:0/3571484661 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300044020 con 0x7f230810e5a0 2024-05-09T08:31:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.186+0000 7f22f77fe640 1 -- 172.21.15.28:0/3571484661 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f230002c8c0 con 0x7f230810e5a0 2024-05-09T08:31:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.186+0000 7f22f77fe640 1 -- 172.21.15.28:0/3571484661 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230003e680 con 0x7f230810e5a0 2024-05-09T08:31:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.186+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23081ac150 con 0x7f230810e5a0 2024-05-09T08:31:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.187+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23081b3a40 con 0x7f230810e5a0 2024-05-09T08:31:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.192+0000 7f22f77fe640 1 -- 172.21.15.28:0/3571484661 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2300047020 con 0x7f230810e5a0 2024-05-09T08:31:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.196+0000 7f230f860740 1 Processor -- start 2024-05-09T08:31:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.197+0000 7f230f860740 1 -- start start 2024-05-09T08:31:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.197+0000 7f230f860740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cadc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.197+0000 7f2306d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cadc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.197+0000 7f2306d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cadc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38812/0 (socket says 172.21.15.28:38812) 2024-05-09T08:31:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.200+0000 7f230f860740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789cae72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.200+0000 7f230f860740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cae77e0 0x55789cae9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.200+0000 7f230f860740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55789ca41340 con 0x55789cadcf20 2024-05-09T08:31:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.200+0000 7f230f860740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55789c9e9290 con 0x55789cae77e0 2024-05-09T08:31:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.200+0000 7f230f860740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55789ca3d300 con 0x55789cadc600 2024-05-09T08:31:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.200+0000 7f2306d9d640 1 -- 172.21.15.28:0/867732391 learned_addr learned my addr 172.21.15.28:0/867732391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.200+0000 7f2305d9b640 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cae77e0 0x55789cae9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.201+0000 7f2306d9d640 1 -- 172.21.15.28:0/867732391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cae77e0 msgr2=0x55789cae9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.201+0000 7f2306d9d640 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cae77e0 0x55789cae9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.201+0000 7f2306d9d640 1 -- 172.21.15.28:0/867732391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 msgr2=0x55789cae72a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:31:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.201+0000 7f230659c640 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789cae72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.201+0000 7f2306d9d640 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789cae72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.201+0000 7f2306d9d640 1 -- 172.21.15.28:0/867732391 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55789ca0d8e0 con 0x55789cadc600 2024-05-09T08:31:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.201+0000 7f230659c640 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789cae72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.201+0000 7f2306d9d640 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cadc9e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f22f800b7f0 tx=0x7f22f800bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cd8894564d806e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.203+0000 7f22f77fe640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f22f00a2e80 0x7f22f00a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.203+0000 7f230659c640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f22f00a2e80 0x7f22f00a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.204+0000 7f22f77fe640 1 -- 172.21.15.28:0/3571484661 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f23000f13a0 con 0x7f230810e5a0 2024-05-09T08:31:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.204+0000 7f230659c640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f22f00a2e80 0x7f22f00a5340 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x55789caddba0 tx=0x7f22fc008040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.217+0000 7f22f4ff9640 1 -- 172.21.15.28:0/867732391 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f800d950 con 0x55789cadc600 2024-05-09T08:31:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.218+0000 7f22f4ff9640 1 -- 172.21.15.28:0/867732391 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22f8019070 con 0x55789cadc600 2024-05-09T08:31:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.219+0000 7f230f860740 1 -- 172.21.15.28:0/867732391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 msgr2=0x55789cadc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.219+0000 7f230f860740 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cadc9e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f22f800b7f0 tx=0x7f22f800bcb0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.222+0000 7f230f860740 1 -- 172.21.15.28:0/867732391 shutdown_connections 2024-05-09T08:31:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.222+0000 7f230f860740 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cae77e0 0x55789cae9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.222+0000 7f230f860740 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789cae72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.222+0000 7f230f860740 1 --2- 172.21.15.28:0/867732391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cadc9e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.222+0000 7f230f860740 1 -- 172.21.15.28:0/867732391 >> 172.21.15.28:0/867732391 conn(0x55789cad8b10 msgr2=0x55789cad8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.222+0000 7f230f860740 1 -- 172.21.15.28:0/867732391 shutdown_connections 2024-05-09T08:31:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.223+0000 7f230f860740 1 -- 172.21.15.28:0/867732391 wait complete. 2024-05-09T08:31:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.223+0000 7f230f860740 1 -- start start 2024-05-09T08:31:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.224+0000 7f230f860740 1 Processor -- start 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.224+0000 7f230f860740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadc600 0x55789cae46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.224+0000 7f230f860740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadcf20 0x55789cae4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.224+0000 7f230f860740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cae77e0 0x55789cae5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.224+0000 7f2305d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadcf20 0x55789cae4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f230f860740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55789caea410 con 0x55789cadc600 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f2305d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadcf20 0x55789cae4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38816/0 (socket says 172.21.15.28:38816) 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55789c9fc790 con 0x55789cae77e0 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f2305d9b640 1 -- 172.21.15.28:0/2892638182 learned_addr learned my addr 172.21.15.28:0/2892638182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55789caea590 con 0x55789cadcf20 2024-05-09T08:31:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f230f860740 10 client.0 _ll_register_callbacks cb 0x55789ca37a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:31:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f2305d9b640 1 -- 172.21.15.28:0/2892638182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cae77e0 msgr2=0x55789cae5170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:31:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f2305d9b640 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cae77e0 0x55789cae5170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f2305d9b640 1 -- 172.21.15.28:0/2892638182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadc600 msgr2=0x55789cae46f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:31:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.225+0000 7f230659c640 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadc600 0x55789cae46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.226+0000 7f2305d9b640 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadc600 0x55789cae46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.226+0000 7f2305d9b640 1 -- 172.21.15.28:0/2892638182 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55789c755210 con 0x55789cadcf20 2024-05-09T08:31:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.226+0000 7f230659c640 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadc600 0x55789cae46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.226+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22f8019070 con 0x55789cadcf20 2024-05-09T08:31:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.226+0000 7f2305d9b640 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadcf20 0x55789cae4c30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f230002cd00 tx=0x7f23000bd380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ebe7bc75c694c8b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.239+0000 7f22edffb640 10 client.26005 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:31:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.240+0000 7f22edffb640 1 -- 172.21.15.28:0/2892638182 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23000bd4f0 con 0x55789cadcf20 2024-05-09T08:31:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.240+0000 7f22edffb640 1 -- 172.21.15.28:0/2892638182 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f23000c5cd0 con 0x55789cadcf20 2024-05-09T08:31:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.241+0000 7f22edffb640 1 -- 172.21.15.28:0/2892638182 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f23000c5f10 con 0x55789cadcf20 2024-05-09T08:31:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.241+0000 7f230f860740 10 client.26005 fetch_fsmap learned FSMap version 47 2024-05-09T08:31:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.242+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=47}) -- 0x55789c821620 con 0x55789cadcf20 2024-05-09T08:31:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.242+0000 7f22edffb640 1 -- 172.21.15.28:0/2892638182 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 47) ==== 8410+0+0 (secure 0 0 0) 0x7f230000b040 con 0x55789cadcf20 2024-05-09T08:31:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.242+0000 7f230f860740 10 client.26005 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T08:31:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.242+0000 7f230f860740 10 client.26005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:31:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.242+0000 7f230f860740 20 client.26005 populate_metadata read hostname 'smithi028' 2024-05-09T08:31:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.243+0000 7f230f860740 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55789caf7640 0x55789caf9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.243+0000 7f230f860740 4 client.26005 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:31:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.243+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55789ca4ea70 con 0x55789caf7640 2024-05-09T08:31:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.243+0000 7f230659c640 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55789caf7640 0x55789caf9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.243+0000 7f230659c640 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55789caf7640 0x55789caf9ac0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a1b7e0c7d752e52 server_cookie=2265d3660c9cf411 in_seq=0 out_seq=0 2024-05-09T08:31:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.243+0000 7f22edffb640 10 client.26005 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:15.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.246+0000 7f22edffb640 1 -- 172.21.15.28:0/2892638182 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55789ca4ea70 con 0x55789caf7640 2024-05-09T08:31:15.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.246+0000 7f22edffb640 10 client.26005 handle_command_reply: tid=0 2024-05-09T08:31:15.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.246+0000 7f230f860740 1 client.26005 shutdown 2024-05-09T08:31:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.256+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadcf20 msgr2=0x55789cae4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.256+0000 7f230f860740 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadcf20 0x55789cae4c30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f230002cd00 tx=0x7f23000bd380 comp rx=0 tx=0).stop 2024-05-09T08:31:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.256+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 shutdown_connections 2024-05-09T08:31:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.256+0000 7f230f860740 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55789caf7640 0x55789caf9ac0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.256+0000 7f230f860740 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cae77e0 0x55789cae5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.256+0000 7f230f860740 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadcf20 0x55789cae4c30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.256+0000 7f230f860740 1 --2- 172.21.15.28:0/2892638182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadc600 0x55789cae46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.256+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 >> 172.21.15.28:0/2892638182 conn(0x55789cad8b10 msgr2=0x55789cade890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.256+0000 7f22edffb640 0 client.26005 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.257+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 shutdown_connections 2024-05-09T08:31:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.257+0000 7f230f860740 1 -- 172.21.15.28:0/2892638182 wait complete. 2024-05-09T08:31:15.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.257+0000 7f230f860740 20 client.26005 trim_cache size 0 max 16384 2024-05-09T08:31:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.415+0000 7f230f860740 1 Processor -- start 2024-05-09T08:31:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.417+0000 7f230f860740 1 -- start start 2024-05-09T08:31:15.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.417+0000 7f230f860740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cae6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.417+0000 7f230f860740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadcf20 0x55789caf3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.418+0000 7f230659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cae6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.418+0000 7f230659c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cae6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38828/0 (socket says 172.21.15.28:38828) 2024-05-09T08:31:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.418+0000 7f230659c640 1 -- 172.21.15.28:0/1847901388 learned_addr learned my addr 172.21.15.28:0/1847901388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.418+0000 7f2305d9b640 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadcf20 0x55789caf3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.418+0000 7f230f860740 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cae77e0 0x55789cae3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.418+0000 7f230f860740 1 -- 172.21.15.28:0/1847901388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23081bdb30 con 0x55789cae77e0 2024-05-09T08:31:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.419+0000 7f230f860740 1 -- 172.21.15.28:0/1847901388 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23081bf8b0 con 0x55789cadcf20 2024-05-09T08:31:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.419+0000 7f230f860740 1 -- 172.21.15.28:0/1847901388 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23081be2c0 con 0x55789cadc600 2024-05-09T08:31:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.419+0000 7f2306d9d640 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cae77e0 0x55789cae3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.419+0000 7f230659c640 1 -- 172.21.15.28:0/1847901388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadcf20 msgr2=0x55789caf3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.419+0000 7f230659c640 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadcf20 0x55789caf3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.419+0000 7f230659c640 1 -- 172.21.15.28:0/1847901388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cae77e0 msgr2=0x55789cae3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.419+0000 7f230659c640 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cae77e0 0x55789cae3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.419+0000 7f230659c640 1 -- 172.21.15.28:0/1847901388 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55789cae2c60 con 0x55789cadc600 2024-05-09T08:31:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.419+0000 7f2306d9d640 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cae77e0 0x55789cae3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:31:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.420+0000 7f230659c640 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cae6e20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00ecf0 tx=0x7f22fc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80c73c86ad92a204 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.518+0000 7f22effff640 1 -- 172.21.15.28:0/1847901388 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc00f040 con 0x55789cadc600 2024-05-09T08:31:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.519+0000 7f22effff640 1 -- 172.21.15.28:0/1847901388 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22fc02b070 con 0x55789cadc600 2024-05-09T08:31:15.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.523+0000 7f230f860740 1 -- 172.21.15.28:0/1847901388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 msgr2=0x55789cae6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.523+0000 7f230f860740 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cae6e20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00ecf0 tx=0x7f22fc014000 comp rx=0 tx=0).stop 2024-05-09T08:31:15.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.523+0000 7f230f860740 1 -- 172.21.15.28:0/1847901388 shutdown_connections 2024-05-09T08:31:15.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.523+0000 7f230f860740 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cae77e0 0x55789cae3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.523+0000 7f230f860740 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadcf20 0x55789caf3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.524+0000 7f230f860740 1 --2- 172.21.15.28:0/1847901388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cadc600 0x55789cae6e20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.524+0000 7f230f860740 1 -- 172.21.15.28:0/1847901388 >> 172.21.15.28:0/1847901388 conn(0x55789cad8b10 msgr2=0x55789cad7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.526+0000 7f230f860740 1 -- 172.21.15.28:0/1847901388 shutdown_connections 2024-05-09T08:31:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.527+0000 7f230f860740 1 -- 172.21.15.28:0/1847901388 wait complete. 2024-05-09T08:31:15.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.528+0000 7f230f860740 1 -- start start 2024-05-09T08:31:15.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.529+0000 7f230f860740 1 Processor -- start 2024-05-09T08:31:15.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.529+0000 7f230f860740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadc600 0x55789caeb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f230f860740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789caeb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f230f860740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cae77e0 0x55789caebd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f230659c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadc600 0x55789caeb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f230659c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadc600 0x55789caeb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:37912/0 (socket says 172.21.15.28:37912) 2024-05-09T08:31:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f2306d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cae77e0 0x55789caebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f2306d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cae77e0 0x55789caebd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38838/0 (socket says 172.21.15.28:38838) 2024-05-09T08:31:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f2305d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789caeb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f230659c640 1 -- 172.21.15.28:0/2904966774 learned_addr learned my addr 172.21.15.28:0/2904966774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f2305d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789caeb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40460/0 (socket says 172.21.15.28:40460) 2024-05-09T08:31:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23081bf120 con 0x55789cadcf20 2024-05-09T08:31:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23081be990 con 0x55789cadc600 2024-05-09T08:31:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23081bbb10 con 0x55789cae77e0 2024-05-09T08:31:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.530+0000 7f230f860740 10 client.0 _ll_register_callbacks cb 0x7f23000b9860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:31:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.531+0000 7f2305d9b640 1 -- 172.21.15.28:0/2904966774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cae77e0 msgr2=0x55789caebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.531+0000 7f2305d9b640 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cae77e0 0x55789caebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.531+0000 7f2305d9b640 1 -- 172.21.15.28:0/2904966774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadc600 msgr2=0x55789caeb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.531+0000 7f2305d9b640 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadc600 0x55789caeb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.531+0000 7f2305d9b640 1 -- 172.21.15.28:0/2904966774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55789cb5cc90 con 0x55789cadcf20 2024-05-09T08:31:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.531+0000 7f2306d9d640 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cae77e0 0x55789caebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:31:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.531+0000 7f2305d9b640 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789caeb7d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f23000b9ec0 tx=0x7f2300009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fae447a156b3f88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.554+0000 7f22edffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:31:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.554+0000 7f22edffb640 1 -- 172.21.15.28:0/2904966774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23000f6030 con 0x55789cadcf20 2024-05-09T08:31:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.554+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22fc02b070 con 0x55789cadcf20 2024-05-09T08:31:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.558+0000 7f22edffb640 1 -- 172.21.15.28:0/2904966774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f2300004d40 con 0x55789cadcf20 2024-05-09T08:31:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.558+0000 7f22edffb640 1 -- 172.21.15.28:0/2904966774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f23000bd4f0 con 0x55789cadcf20 2024-05-09T08:31:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.558+0000 7f230f860740 10 client.16647 fetch_fsmap learned FSMap version 47 2024-05-09T08:31:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.562+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55789cb5df80 con 0x55789cadcf20 2024-05-09T08:31:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.563+0000 7f22edffb640 1 -- 172.21.15.28:0/2904966774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22f8019070 con 0x55789cadcf20 2024-05-09T08:31:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.563+0000 7f22edffb640 1 -- 172.21.15.28:0/2904966774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 47) ==== 8410+0+0 (secure 0 0 0) 0x7f23000bc3b0 con 0x55789cadcf20 2024-05-09T08:31:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.563+0000 7f230f860740 10 client.16647 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T08:31:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.563+0000 7f230f860740 10 client.16647 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:31:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.563+0000 7f230f860740 20 client.16647 populate_metadata read hostname 'smithi028' 2024-05-09T08:31:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.563+0000 7f230f860740 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55789cb5e590 0x55789cb609b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.563+0000 7f230f860740 4 client.16647 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:31:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.563+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f22fc02b070 con 0x55789cb5e590 2024-05-09T08:31:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.564+0000 7f230659c640 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55789cb5e590 0x55789cb609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.564+0000 7f230659c640 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55789cb5e590 0x55789cb609b0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a34a7ea653f38a server_cookie=aaf820de648e1861 in_seq=0 out_seq=0 2024-05-09T08:31:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:15.566+0000 7f22edffb640 10 client.16647 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:15.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:15 smithi165 ceph-mon[34540]: pgmap v154: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-05-09T08:31:15.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:15 smithi184 ceph-mon[34509]: pgmap v154: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-05-09T08:31:15.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:15 smithi028 ceph-mon[30119]: pgmap v154: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-05-09T08:31:17.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:17 smithi165 ceph-mon[34540]: pgmap v155: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.3 MiB/s wr, 198 op/s 2024-05-09T08:31:17.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:17 smithi184 ceph-mon[34509]: pgmap v155: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.3 MiB/s wr, 198 op/s 2024-05-09T08:31:17.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:17 smithi028 ceph-mon[30119]: pgmap v155: 129 pgs: 129 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.3 MiB/s wr, 198 op/s 2024-05-09T08:31:18.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.005+0000 7f22edffb640 1 -- 172.21.15.28:0/2904966774 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f22fc02b070 con 0x55789cb5e590 2024-05-09T08:31:18.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.005+0000 7f22edffb640 10 client.16647 handle_command_reply: tid=0 2024-05-09T08:31:18.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.005+0000 7f230f860740 1 client.16647 shutdown 2024-05-09T08:31:18.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.006+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 msgr2=0x55789caeb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:18.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.006+0000 7f230f860740 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789caeb7d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f23000b9ec0 tx=0x7f2300009990 comp rx=0 tx=0).stop 2024-05-09T08:31:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.008+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 shutdown_connections 2024-05-09T08:31:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.008+0000 7f230f860740 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55789cb5e590 0x55789cb609b0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.008+0000 7f230f860740 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55789cae77e0 0x55789caebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.008+0000 7f230f860740 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55789cadcf20 0x55789caeb7d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.008+0000 7f230f860740 1 --2- 172.21.15.28:0/2904966774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55789cadc600 0x55789caeb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.008+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 >> 172.21.15.28:0/2904966774 conn(0x55789cad8b10 msgr2=0x55789cae4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.008+0000 7f22edffb640 0 client.16647 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.008+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 shutdown_connections 2024-05-09T08:31:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.009+0000 7f230f860740 1 -- 172.21.15.28:0/2904966774 wait complete. 2024-05-09T08:31:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.009+0000 7f230f860740 20 client.16647 trim_cache size 0 max 16384 2024-05-09T08:31:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:31:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:31:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:31:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:31:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f22f00a2e80 msgr2=0x7f22f00a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f22f00a2e80 0x7f22f00a5340 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x55789caddba0 tx=0x7f22fc008040 comp rx=0 tx=0).stop 2024-05-09T08:31:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f230810e5a0 msgr2=0x7f23081a7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f230810e5a0 0x7f23081a7410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2300000f80 tx=0x7f2300034770 comp rx=0 tx=0).stop 2024-05-09T08:31:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 shutdown_connections 2024-05-09T08:31:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230810fda0 0x7f23081ab790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f22f00a2e80 0x7f22f00a5340 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f230810e5a0 0x7f23081a7410 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2300000f80 tx=0x7f2300034770 comp rx=0 tx=0).stop 2024-05-09T08:31:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 --2- 172.21.15.28:0/3571484661 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230810db00 0x7f23081a6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.014+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 >> 172.21.15.28:0/3571484661 conn(0x7f230806c7e0 msgr2=0x7f2308109e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.038+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 shutdown_connections 2024-05-09T08:31:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:18.038+0000 7f230cb5c640 1 -- 172.21.15.28:0/3571484661 wait complete. 2024-05-09T08:31:18.708 INFO:tasks.workunit.client.0.smithi028.stdout: 490 2025242 2026 1363406 1741 327176 92 2024-05-09T08:31:19.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:19 smithi165 ceph-mon[34540]: pgmap v156: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 168 op/s 2024-05-09T08:31:19.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:19 smithi184 ceph-mon[34509]: pgmap v156: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 168 op/s 2024-05-09T08:31:19.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:19 smithi028 ceph-mon[30119]: pgmap v156: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 168 op/s 2024-05-09T08:31:21.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:21 smithi028 ceph-mon[30119]: pgmap v157: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 341 op/s 2024-05-09T08:31:22.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:21 smithi165 ceph-mon[34540]: pgmap v157: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 341 op/s 2024-05-09T08:31:22.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:21 smithi184 ceph-mon[34509]: pgmap v157: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 341 op/s 2024-05-09T08:31:23.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:23 smithi028 ceph-mon[30119]: pgmap v158: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 475 op/s 2024-05-09T08:31:24.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:23 smithi165 ceph-mon[34540]: pgmap v158: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 475 op/s 2024-05-09T08:31:24.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:23 smithi184 ceph-mon[34509]: pgmap v158: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 475 op/s 2024-05-09T08:31:25.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:25 smithi028 ceph-mon[30119]: pgmap v159: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 508 op/s 2024-05-09T08:31:25.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:25 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:26.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:25 smithi165 ceph-mon[34540]: pgmap v159: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 508 op/s 2024-05-09T08:31:26.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:25 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:25 smithi184 ceph-mon[34509]: pgmap v159: 129 pgs: 129 active+clean; 1.5 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 508 op/s 2024-05-09T08:31:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:25 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:27.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:27 smithi028 ceph-mon[30119]: pgmap v160: 129 pgs: 129 active+clean; 1.5 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.5 MiB/s wr, 648 op/s 2024-05-09T08:31:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:27 smithi165 ceph-mon[34540]: pgmap v160: 129 pgs: 129 active+clean; 1.5 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.5 MiB/s wr, 648 op/s 2024-05-09T08:31:28.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:27 smithi184 ceph-mon[34509]: pgmap v160: 129 pgs: 129 active+clean; 1.5 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.5 MiB/s wr, 648 op/s 2024-05-09T08:31:28.709 INFO:tasks.workunit.client.0.smithi028.stdout: 504 1644298 883 1107624 706 259654 1158 2024-05-09T08:31:29.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:28 smithi165 ceph-mon[34540]: pgmap v161: 129 pgs: 129 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 8.4 MiB/s wr, 733 op/s 2024-05-09T08:31:29.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:28 smithi184 ceph-mon[34509]: pgmap v161: 129 pgs: 129 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 8.4 MiB/s wr, 733 op/s 2024-05-09T08:31:29.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:28 smithi028 ceph-mon[30119]: pgmap v161: 129 pgs: 129 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 8.4 MiB/s wr, 733 op/s 2024-05-09T08:31:31.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:30 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:31.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:30 smithi165 ceph-mon[34540]: pgmap v162: 129 pgs: 129 active+clean; 1.6 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 15 MiB/s wr, 914 op/s 2024-05-09T08:31:31.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:30 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:31.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:30 smithi184 ceph-mon[34509]: pgmap v162: 129 pgs: 129 active+clean; 1.6 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 15 MiB/s wr, 914 op/s 2024-05-09T08:31:31.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:30 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:31.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:30 smithi028 ceph-mon[30119]: pgmap v162: 129 pgs: 129 active+clean; 1.6 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 15 MiB/s wr, 914 op/s 2024-05-09T08:31:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.286+0000 7f04759b0640 1 -- 172.21.15.28:0/2766627842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0470110400 msgr2=0x7f0470108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.286+0000 7f04759b0640 1 --2- 172.21.15.28:0/2766627842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0470110400 0x7f0470108d70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0464009900 tx=0x7f046402a4d0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.288+0000 7f04759b0640 1 -- 172.21.15.28:0/2766627842 shutdown_connections 2024-05-09T08:31:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.288+0000 7f04759b0640 1 --2- 172.21.15.28:0/2766627842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f047011b010 0x7f047011d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.288+0000 7f04759b0640 1 --2- 172.21.15.28:0/2766627842 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0470110400 0x7f0470108d70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.288+0000 7f04759b0640 1 --2- 172.21.15.28:0/2766627842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0470110020 0x7f0470108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.288+0000 7f04759b0640 1 -- 172.21.15.28:0/2766627842 >> 172.21.15.28:0/2766627842 conn(0x7f047006c900 msgr2=0x7f047006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:31.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.289+0000 7f04759b0640 1 -- 172.21.15.28:0/2766627842 shutdown_connections 2024-05-09T08:31:31.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.290+0000 7f04759b0640 1 -- 172.21.15.28:0/2766627842 wait complete. 2024-05-09T08:31:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.291+0000 7f04759b0640 1 Processor -- start 2024-05-09T08:31:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.291+0000 7f04759b0640 1 -- start start 2024-05-09T08:31:31.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.291+0000 7f04759b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0470110020 0x7f047011a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.292+0000 7f04759b0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047011b010 0x7f047011aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.292+0000 7f04759b0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0470113710 0x7f0470113bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.292+0000 7f04759b0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0470120b00 con 0x7f0470110020 2024-05-09T08:31:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.292+0000 7f04759b0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0470120980 con 0x7f0470113710 2024-05-09T08:31:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.292+0000 7f04759b0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0470120c80 con 0x7f047011b010 2024-05-09T08:31:31.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.292+0000 7f046effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0470110020 0x7f047011a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.292+0000 7f046effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0470110020 0x7f047011a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54356/0 (socket says 172.21.15.28:54356) 2024-05-09T08:31:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.292+0000 7f046effd640 1 -- 172.21.15.28:0/938340495 learned_addr learned my addr 172.21.15.28:0/938340495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.293+0000 7f046f7fe640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0470113710 0x7f0470113bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.293+0000 7f046e7fc640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047011b010 0x7f047011aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.295+0000 7f046e7fc640 1 -- 172.21.15.28:0/938340495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0470113710 msgr2=0x7f0470113bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.295+0000 7f046e7fc640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0470113710 0x7f0470113bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.295+0000 7f046e7fc640 1 -- 172.21.15.28:0/938340495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0470110020 msgr2=0x7f047011a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.295+0000 7f046e7fc640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0470110020 0x7f047011a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.295+0000 7f046e7fc640 1 -- 172.21.15.28:0/938340495 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0470118050 con 0x7f047011b010 2024-05-09T08:31:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.297+0000 7f046e7fc640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047011b010 0x7f047011aa70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0464009850 tx=0x7f0464003bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.297+0000 7f046f7fe640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0470113710 0x7f0470113bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:31.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.299+0000 7f04749ae640 1 -- 172.21.15.28:0/938340495 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046403f070 con 0x7f047011b010 2024-05-09T08:31:31.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.299+0000 7f04749ae640 1 -- 172.21.15.28:0/938340495 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0464031c20 con 0x7f047011b010 2024-05-09T08:31:31.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.299+0000 7f04749ae640 1 -- 172.21.15.28:0/938340495 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464035030 con 0x7f047011b010 2024-05-09T08:31:31.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.299+0000 7f04759b0640 1 -- 172.21.15.28:0/938340495 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0470118280 con 0x7f047011b010 2024-05-09T08:31:31.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.299+0000 7f04759b0640 1 -- 172.21.15.28:0/938340495 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04701186f0 con 0x7f047011b010 2024-05-09T08:31:31.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.304+0000 7f04787a0740 1 Processor -- start 2024-05-09T08:31:31.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.313+0000 7f04749ae640 1 -- 172.21.15.28:0/938340495 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0464042070 con 0x7f047011b010 2024-05-09T08:31:31.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.314+0000 7f04787a0740 1 -- start start 2024-05-09T08:31:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.317+0000 7f04787a0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa57830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.320+0000 7f04787a0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57d70 0x55668aa620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.321+0000 7f04787a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aa64a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.321+0000 7f04787a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55668a9bc340 con 0x55668aa62630 2024-05-09T08:31:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.321+0000 7f04787a0740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55668a964290 con 0x55668aa57450 2024-05-09T08:31:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.321+0000 7f04787a0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55668a9b8300 con 0x55668aa57d70 2024-05-09T08:31:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.321+0000 7f046f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa57830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.321+0000 7f046f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa57830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:45676/0 (socket says 172.21.15.28:45676) 2024-05-09T08:31:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.321+0000 7f046f7fe640 1 -- 172.21.15.28:0/3636447443 learned_addr learned my addr 172.21.15.28:0/3636447443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.323+0000 7f046effd640 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57d70 0x55668aa620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.324+0000 7f046e7fc640 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aa64a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.325+0000 7f04749ae640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04580a2fd0 0x7f04580a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.325+0000 7f046f7fe640 1 -- 172.21.15.28:0/3636447443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57d70 msgr2=0x55668aa620f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.325+0000 7f046f7fe640 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57d70 0x55668aa620f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.325+0000 7f046f7fe640 1 -- 172.21.15.28:0/3636447443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 msgr2=0x55668aa64a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.325+0000 7f04749ae640 1 -- 172.21.15.28:0/938340495 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f04640efde0 con 0x7f047011b010 2024-05-09T08:31:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.325+0000 7f046f7fe640 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aa64a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.325+0000 7f046f7fe640 1 -- 172.21.15.28:0/3636447443 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55668a9888e0 con 0x55668aa57450 2024-05-09T08:31:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.326+0000 7f046e7fc640 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aa64a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.326+0000 7f046f7fe640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04580a2fd0 0x7f04580a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.326+0000 7f046f7fe640 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa57830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0460009d00 tx=0x7f046000e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f41a5ca20d09795a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:31.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.329+0000 7f046f7fe640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04580a2fd0 0x7f04580a5490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f046001d000 tx=0x7f046001e000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:31.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.331+0000 7f0455ffb640 1 -- 172.21.15.28:0/3636447443 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460020070 con 0x55668aa57450 2024-05-09T08:31:31.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.332+0000 7f0455ffb640 1 -- 172.21.15.28:0/3636447443 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0460012a90 con 0x55668aa57450 2024-05-09T08:31:31.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.332+0000 7f04787a0740 1 -- 172.21.15.28:0/3636447443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 msgr2=0x55668aa57830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:31.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.332+0000 7f04787a0740 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa57830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0460009d00 tx=0x7f046000e6c0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.334+0000 7f04787a0740 1 -- 172.21.15.28:0/3636447443 shutdown_connections 2024-05-09T08:31:31.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.334+0000 7f04787a0740 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aa64a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.334+0000 7f04787a0740 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57d70 0x55668aa620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.334+0000 7f04787a0740 1 --2- 172.21.15.28:0/3636447443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa57830 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.334+0000 7f04787a0740 1 -- 172.21.15.28:0/3636447443 >> 172.21.15.28:0/3636447443 conn(0x55668aa53960 msgr2=0x55668aa53d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.338+0000 7f04787a0740 1 -- 172.21.15.28:0/3636447443 shutdown_connections 2024-05-09T08:31:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.341+0000 7f04787a0740 1 -- 172.21.15.28:0/3636447443 wait complete. 2024-05-09T08:31:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.342+0000 7f04787a0740 1 -- start start 2024-05-09T08:31:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.342+0000 7f04787a0740 1 Processor -- start 2024-05-09T08:31:31.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f04787a0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 0x55668aa5f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f04787a0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57d70 0x55668aa5f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f04787a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aa5ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f04787a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55668aa652b0 con 0x55668aa62630 2024-05-09T08:31:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f04787a0740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55668a977790 con 0x55668aa57d70 2024-05-09T08:31:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f04787a0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55668aa65430 con 0x55668aa57450 2024-05-09T08:31:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f04787a0740 10 client.0 _ll_register_callbacks cb 0x55668a9b2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:31:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f046e7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57d70 0x55668aa5f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f046e7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57d70 0x55668aa5f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:45686/0 (socket says 172.21.15.28:45686) 2024-05-09T08:31:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.345+0000 7f046e7fc640 1 -- 172.21.15.28:0/3321091537 learned_addr learned my addr 172.21.15.28:0/3321091537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.347+0000 7f046f7fe640 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aa5ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.348+0000 7f046e7fc640 1 -- 172.21.15.28:0/3321091537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 msgr2=0x55668aa5f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.348+0000 7f046e7fc640 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 0x55668aa5f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.348+0000 7f046e7fc640 1 -- 172.21.15.28:0/3321091537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 msgr2=0x55668aa5ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.348+0000 7f046e7fc640 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aa5ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.348+0000 7f046e7fc640 1 -- 172.21.15.28:0/3321091537 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55668a6d0210 con 0x55668aa57d70 2024-05-09T08:31:31.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.356+0000 7f046e7fc640 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57d70 0x55668aa5f9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f04640bbe60 tx=0x7f04640bbdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa7eb9212160229d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.358+0000 7f044effd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:31:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.358+0000 7f044effd640 1 -- 172.21.15.28:0/3321091537 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04640bbed0 con 0x55668aa57d70 2024-05-09T08:31:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.358+0000 7f04787a0740 1 -- 172.21.15.28:0/3321091537 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0460012a90 con 0x55668aa57d70 2024-05-09T08:31:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.358+0000 7f044effd640 1 -- 172.21.15.28:0/3321091537 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f04640b8540 con 0x55668aa57d70 2024-05-09T08:31:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.360+0000 7f044effd640 1 -- 172.21.15.28:0/3321091537 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f04640c0280 con 0x55668aa57d70 2024-05-09T08:31:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.366+0000 7f04787a0740 10 client.26132 fetch_fsmap learned FSMap version 49 2024-05-09T08:31:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.366+0000 7f04787a0740 1 -- 172.21.15.28:0/3321091537 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55668a79c620 con 0x55668aa57d70 2024-05-09T08:31:31.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.368+0000 7f044effd640 1 -- 172.21.15.28:0/3321091537 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0460012a90 con 0x55668aa57d70 2024-05-09T08:31:31.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.368+0000 7f044effd640 1 -- 172.21.15.28:0/3321091537 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 49) ==== 8426+0+0 (secure 0 0 0) 0x7f04640bb320 con 0x55668aa57d70 2024-05-09T08:31:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.371+0000 7f04787a0740 10 client.26132 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T08:31:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.372+0000 7f04787a0740 10 client.26132 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:31:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.372+0000 7f04787a0740 20 client.26132 populate_metadata read hostname 'smithi028' 2024-05-09T08:31:31.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.372+0000 7f04787a0740 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55668aa72350 0x55668aa74810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.372+0000 7f04787a0740 4 client.26132 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:31:31.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.372+0000 7f04787a0740 1 -- 172.21.15.28:0/3321091537 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55668a9c9a70 con 0x55668aa72350 2024-05-09T08:31:31.374 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.374+0000 7f046effd640 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55668aa72350 0x55668aa74810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.382+0000 7f044effd640 10 client.26132 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.382+0000 7f046effd640 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55668aa72350 0x55668aa74810 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5799932ce8e7b25 server_cookie=d584fc40c589994c in_seq=0 out_seq=0 2024-05-09T08:31:31.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.385+0000 7f044effd640 1 -- 172.21.15.28:0/3321091537 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55668a9c9a70 con 0x55668aa72350 2024-05-09T08:31:31.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.385+0000 7f044effd640 10 client.26132 handle_command_reply: tid=0 2024-05-09T08:31:31.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.385+0000 7f04787a0740 1 client.26132 shutdown 2024-05-09T08:31:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.387+0000 7f04787a0740 1 -- 172.21.15.28:0/3321091537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57d70 msgr2=0x55668aa5f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.387+0000 7f04787a0740 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57d70 0x55668aa5f9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f04640bbe60 tx=0x7f04640bbdd0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.389+0000 7f04787a0740 1 -- 172.21.15.28:0/3321091537 shutdown_connections 2024-05-09T08:31:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.389+0000 7f04787a0740 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55668aa72350 0x55668aa74810 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.389+0000 7f04787a0740 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aa5ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.389+0000 7f04787a0740 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57d70 0x55668aa5f9f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.389+0000 7f04787a0740 1 --2- 172.21.15.28:0/3321091537 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 0x55668aa5f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.389+0000 7f04787a0740 1 -- 172.21.15.28:0/3321091537 >> 172.21.15.28:0/3321091537 conn(0x55668aa53960 msgr2=0x55668aa59690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:31.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.391+0000 7f044effd640 0 client.26132 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:31.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.392+0000 7f04787a0740 1 -- 172.21.15.28:0/3321091537 shutdown_connections 2024-05-09T08:31:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.396+0000 7f04787a0740 1 -- 172.21.15.28:0/3321091537 wait complete. 2024-05-09T08:31:31.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.398+0000 7f04787a0740 20 client.26132 trim_cache size 0 max 16384 2024-05-09T08:31:31.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.429+0000 7f04787a0740 1 Processor -- start 2024-05-09T08:31:31.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.431+0000 7f04787a0740 1 -- start start 2024-05-09T08:31:31.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.431+0000 7f04787a0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 0x55668aa61be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.431+0000 7f04787a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa57d70 0x55668aa7a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.431+0000 7f04787a0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa62630 0x55668aa7a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.431+0000 7f04787a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04701c62c0 con 0x55668aa57d70 2024-05-09T08:31:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.431+0000 7f04787a0740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f04701c8040 con 0x55668aa62630 2024-05-09T08:31:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.431+0000 7f04787a0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04701c6a50 con 0x55668aa57450 2024-05-09T08:31:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.433+0000 7f046f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa62630 0x55668aa7a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.433+0000 7f046effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 0x55668aa61be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.433+0000 7f046effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 0x55668aa61be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:59006/0 (socket says 172.21.15.28:59006) 2024-05-09T08:31:31.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.433+0000 7f046effd640 1 -- 172.21.15.28:0/3408630254 learned_addr learned my addr 172.21.15.28:0/3408630254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:31.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.434+0000 7f046e7fc640 1 --2- 172.21.15.28:0/3408630254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa57d70 0x55668aa7a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.434+0000 7f046effd640 1 -- 172.21.15.28:0/3408630254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa62630 msgr2=0x55668aa7a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:31.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.434+0000 7f046effd640 1 --2- 172.21.15.28:0/3408630254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa62630 0x55668aa7a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.434+0000 7f046effd640 1 -- 172.21.15.28:0/3408630254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa57d70 msgr2=0x55668aa7a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:31.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.434+0000 7f046effd640 1 --2- 172.21.15.28:0/3408630254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa57d70 0x55668aa7a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.434+0000 7f046effd640 1 -- 172.21.15.28:0/3408630254 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55668aa74e90 con 0x55668aa57450 2024-05-09T08:31:31.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.438+0000 7f046effd640 1 --2- 172.21.15.28:0/3408630254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 0x55668aa61be0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55668aa75060 tx=0x7f0468007c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e80323142bdf1fe7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.440+0000 7f04557fa640 1 -- 172.21.15.28:0/3408630254 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468007e40 con 0x55668aa57450 2024-05-09T08:31:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.440+0000 7f04557fa640 1 -- 172.21.15.28:0/3408630254 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f046801f030 con 0x55668aa57450 2024-05-09T08:31:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.440+0000 7f04557fa640 1 -- 172.21.15.28:0/3408630254 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468023630 con 0x55668aa57450 2024-05-09T08:31:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.442+0000 7f04787a0740 1 -- 172.21.15.28:0/3408630254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 msgr2=0x55668aa61be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.442+0000 7f04787a0740 1 --2- 172.21.15.28:0/3408630254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 0x55668aa61be0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55668aa75060 tx=0x7f0468007c50 comp rx=0 tx=0).stop 2024-05-09T08:31:31.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.445+0000 7f04787a0740 1 -- 172.21.15.28:0/3408630254 shutdown_connections 2024-05-09T08:31:31.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.445+0000 7f04787a0740 1 --2- 172.21.15.28:0/3408630254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa62630 0x55668aa7a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.445+0000 7f04787a0740 1 --2- 172.21.15.28:0/3408630254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa57d70 0x55668aa7a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.445+0000 7f04787a0740 1 --2- 172.21.15.28:0/3408630254 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57450 0x55668aa61be0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.445+0000 7f04787a0740 1 -- 172.21.15.28:0/3408630254 >> 172.21.15.28:0/3408630254 conn(0x55668aa53960 msgr2=0x55668aa52a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.449+0000 7f04787a0740 1 -- 172.21.15.28:0/3408630254 shutdown_connections 2024-05-09T08:31:31.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.452+0000 7f04787a0740 1 -- 172.21.15.28:0/3408630254 wait complete. 2024-05-09T08:31:31.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.453+0000 7f04787a0740 1 -- start start 2024-05-09T08:31:31.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.453+0000 7f04787a0740 1 Processor -- start 2024-05-09T08:31:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.454+0000 7f04787a0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa66720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.454+0000 7f04787a0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57d70 0x55668aa66c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.454+0000 7f04787a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aad7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.454+0000 7f046effd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa66720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.454+0000 7f046effd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa66720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:45706/0 (socket says 172.21.15.28:45706) 2024-05-09T08:31:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.454+0000 7f046effd640 1 -- 172.21.15.28:0/12346761 learned_addr learned my addr 172.21.15.28:0/12346761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.454+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04701c78b0 con 0x55668aa62630 2024-05-09T08:31:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.454+0000 7f046f7fe640 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aad7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.455+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f04701c7120 con 0x55668aa57450 2024-05-09T08:31:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.455+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04701c42a0 con 0x55668aa57d70 2024-05-09T08:31:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.455+0000 7f04787a0740 10 client.0 _ll_register_callbacks cb 0x7f0464003f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:31:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.455+0000 7f046effd640 1 -- 172.21.15.28:0/12346761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57d70 msgr2=0x55668aa66c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:31:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.455+0000 7f046effd640 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57d70 0x55668aa66c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.455+0000 7f046effd640 1 -- 172.21.15.28:0/12346761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 msgr2=0x55668aad7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.455+0000 7f046effd640 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aad7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.455+0000 7f046effd640 1 -- 172.21.15.28:0/12346761 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55668aad7ed0 con 0x55668aa57450 2024-05-09T08:31:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.455+0000 7f046effd640 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa66720 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0468004aa0 tx=0x7f04680074f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dbd3f4db019f92d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.458+0000 7f044effd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:31:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.458+0000 7f044effd640 1 -- 172.21.15.28:0/12346761 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468017940 con 0x55668aa57450 2024-05-09T08:31:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.458+0000 7f044effd640 1 -- 172.21.15.28:0/12346761 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f0468017ae0 con 0x55668aa57450 2024-05-09T08:31:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.458+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f046801f030 con 0x55668aa57450 2024-05-09T08:31:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.458+0000 7f044effd640 1 -- 172.21.15.28:0/12346761 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f046802c420 con 0x55668aa57450 2024-05-09T08:31:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.458+0000 7f04787a0740 10 client.26135 fetch_fsmap learned FSMap version 49 2024-05-09T08:31:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.458+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55668aad90c0 con 0x55668aa57450 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.459+0000 7f044effd640 1 -- 172.21.15.28:0/12346761 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f046801f030 con 0x55668aa57450 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.459+0000 7f044effd640 1 -- 172.21.15.28:0/12346761 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 49) ==== 8426+0+0 (secure 0 0 0) 0x7f0468017ce0 con 0x55668aa57450 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.459+0000 7f04787a0740 10 client.26135 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.459+0000 7f04787a0740 10 client.26135 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.459+0000 7f04787a0740 20 client.26135 populate_metadata read hostname 'smithi028' 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.459+0000 7f04787a0740 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55668aad9740 0x55668aadbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.459+0000 7f04787a0740 4 client.26135 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.459+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55668a9c3dc0 con 0x55668aad9740 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.459+0000 7f046e7fc640 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55668aad9740 0x55668aadbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.460+0000 7f044effd640 10 client.26135 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:31.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:31.460+0000 7f046e7fc640 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55668aad9740 0x55668aadbba0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4167e9c714d643e server_cookie=e99010083bddc45 in_seq=0 out_seq=0 2024-05-09T08:31:33.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:33 smithi165 ceph-mon[34540]: pgmap v163: 129 pgs: 129 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 17 MiB/s wr, 839 op/s 2024-05-09T08:31:33.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:33 smithi184 ceph-mon[34509]: pgmap v163: 129 pgs: 129 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 17 MiB/s wr, 839 op/s 2024-05-09T08:31:33.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:33 smithi028 ceph-mon[30119]: pgmap v163: 129 pgs: 129 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 17 MiB/s wr, 839 op/s 2024-05-09T08:31:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f044effd640 1 -- 172.21.15.28:0/12346761 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55668a9c3dc0 con 0x55668aad9740 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f044effd640 10 client.26135 handle_command_reply: tid=0 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f04787a0740 1 client.26135 shutdown 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 msgr2=0x55668aa66720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f04787a0740 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa66720 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0468004aa0 tx=0x7f04680074f0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 shutdown_connections 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f04787a0740 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55668aad9740 0x55668aadbba0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f04787a0740 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668aa62630 0x55668aad7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f04787a0740 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55668aa57d70 0x55668aa66c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f04787a0740 1 --2- 172.21.15.28:0/12346761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55668aa57450 0x55668aa66720 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f044effd640 0 client.26135 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.293+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 >> 172.21.15.28:0/12346761 conn(0x55668aa53960 msgr2=0x55668aa5f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.294+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 shutdown_connections 2024-05-09T08:31:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.294+0000 7f04787a0740 1 -- 172.21.15.28:0/12346761 wait complete. 2024-05-09T08:31:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.294+0000 7f04787a0740 20 client.26135 trim_cache size 0 max 16384 2024-05-09T08:31:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:31:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:31:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:31:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:31:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.295+0000 7f04759b0640 1 -- 172.21.15.28:0/938340495 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04580a2fd0 msgr2=0x7f04580a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.295+0000 7f04759b0640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04580a2fd0 0x7f04580a5490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f046001d000 tx=0x7f046001e000 comp rx=0 tx=0).stop 2024-05-09T08:31:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.295+0000 7f04759b0640 1 -- 172.21.15.28:0/938340495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047011b010 msgr2=0x7f047011aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.295+0000 7f04759b0640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047011b010 0x7f047011aa70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0464009850 tx=0x7f0464003bb0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.296+0000 7f04759b0640 1 -- 172.21.15.28:0/938340495 shutdown_connections 2024-05-09T08:31:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.296+0000 7f04759b0640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0470113710 0x7f0470113bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.296+0000 7f04759b0640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04580a2fd0 0x7f04580a5490 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.296+0000 7f04759b0640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047011b010 0x7f047011aa70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.296+0000 7f04759b0640 1 --2- 172.21.15.28:0/938340495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0470110020 0x7f047011a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.296+0000 7f04759b0640 1 -- 172.21.15.28:0/938340495 >> 172.21.15.28:0/938340495 conn(0x7f047006c900 msgr2=0x7f047011b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.296+0000 7f04759b0640 1 -- 172.21.15.28:0/938340495 shutdown_connections 2024-05-09T08:31:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:35.296+0000 7f04759b0640 1 -- 172.21.15.28:0/938340495 wait complete. 2024-05-09T08:31:35.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:35 smithi165 ceph-mon[34540]: pgmap v164: 129 pgs: 129 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 17 MiB/s wr, 771 op/s 2024-05-09T08:31:35.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:35 smithi184 ceph-mon[34509]: pgmap v164: 129 pgs: 129 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 17 MiB/s wr, 771 op/s 2024-05-09T08:31:35.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:35 smithi028 ceph-mon[30119]: pgmap v164: 129 pgs: 129 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 17 MiB/s wr, 771 op/s 2024-05-09T08:31:37.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:37 smithi165 ceph-mon[34540]: pgmap v165: 129 pgs: 129 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 18 MiB/s wr, 1.02k op/s 2024-05-09T08:31:37.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:37 smithi184 ceph-mon[34509]: pgmap v165: 129 pgs: 129 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 18 MiB/s wr, 1.02k op/s 2024-05-09T08:31:37.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:37 smithi028 ceph-mon[30119]: pgmap v165: 129 pgs: 129 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 18 MiB/s wr, 1.02k op/s 2024-05-09T08:31:38.682 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:38 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[68094]: *** Caught signal (Segmentation fault) ** 2024-05-09T08:31:38.682 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:38 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[68094]: in thread 7fab4a83e700 thread_name:MR_Finisher 2024-05-09T08:31:38.682 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:38 smithi028 podman[91438]: 2024-05-09 08:31:38.528948511 +0000 UTC m=+0.080657367 container died de5d67530b5945c7057680633d64799de036c1ce31af84fd23aeea69969966b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-09T08:31:38.709 INFO:tasks.workunit.client.0.smithi028.stdout: 521 1217248 813 819789 758 217844 2168 2024-05-09T08:31:39.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:39 smithi165 ceph-mon[34540]: pgmap v166: 129 pgs: 129 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-05-09T08:31:39.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:39 smithi184 ceph-mon[34509]: pgmap v166: 129 pgs: 129 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-05-09T08:31:39.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:39 smithi028 ceph-mon[30119]: pgmap v166: 129 pgs: 129 active+clean; 1.7 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-05-09T08:31:39.683 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:39 smithi028 podman[91438]: 2024-05-09 08:31:39.290741305 +0000 UTC m=+0.842450150 container remove de5d67530b5945c7057680633d64799de036c1ce31af84fd23aeea69969966b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, RELEASE=squid-96805dc) 2024-05-09T08:31:39.683 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:39 smithi028 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.d.service: Main process exited, code=exited, status=139/n/a 2024-05-09T08:31:40.433 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:40 smithi028 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-09T08:31:40.433 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:40 smithi028 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.d.service: Consumed 5.617s CPU time. 2024-05-09T08:31:41.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:41 smithi028 ceph-mon[30119]: pgmap v167: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 15 MiB/s wr, 1.05k op/s 2024-05-09T08:31:41.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:41 smithi165 ceph-mon[34540]: pgmap v167: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 15 MiB/s wr, 1.05k op/s 2024-05-09T08:31:41.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:41 smithi184 ceph-mon[34509]: pgmap v167: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 15 MiB/s wr, 1.05k op/s 2024-05-09T08:31:43.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:43 smithi165 ceph-mon[34540]: pgmap v168: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.1 MiB/s wr, 837 op/s 2024-05-09T08:31:43.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:43 smithi184 ceph-mon[34509]: pgmap v168: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.1 MiB/s wr, 837 op/s 2024-05-09T08:31:43.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:43 smithi028 ceph-mon[30119]: pgmap v168: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.1 MiB/s wr, 837 op/s 2024-05-09T08:31:45.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:45 smithi165 ceph-mon[34540]: pgmap v169: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.8 MiB/s wr, 746 op/s 2024-05-09T08:31:45.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:45 smithi184 ceph-mon[34509]: pgmap v169: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.8 MiB/s wr, 746 op/s 2024-05-09T08:31:45.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:45 smithi028 ceph-mon[30119]: pgmap v169: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.8 MiB/s wr, 746 op/s 2024-05-09T08:31:47.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:47 smithi165 ceph-mon[34540]: pgmap v170: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.0 MiB/s wr, 684 op/s 2024-05-09T08:31:47.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:47 smithi184 ceph-mon[34509]: pgmap v170: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.0 MiB/s wr, 684 op/s 2024-05-09T08:31:47.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:47 smithi028 ceph-mon[30119]: pgmap v170: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.0 MiB/s wr, 684 op/s 2024-05-09T08:31:48.709 INFO:tasks.workunit.client.0.smithi028.stdout: 555 1949556 1574 1314561 1683 385347 1893 2024-05-09T08:31:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.809+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/142410600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ac11b010 msgr2=0x7fa1ac11d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.809+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/142410600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ac11b010 0x7fa1ac11d400 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa1a401acb0 tx=0x7fa1a403b9b0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.810+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/142410600 shutdown_connections 2024-05-09T08:31:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.810+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/142410600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ac11b010 0x7fa1ac11d400 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.810+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/142410600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ac110400 0x7fa1ac108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.810+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/142410600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac110020 0x7fa1ac108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.810+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/142410600 >> 172.21.15.28:0/142410600 conn(0x7fa1ac06c900 msgr2=0x7fa1ac06cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:48.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.813+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/142410600 shutdown_connections 2024-05-09T08:31:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.815+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/142410600 wait complete. 2024-05-09T08:31:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.815+0000 7fa1b2fc1640 1 Processor -- start 2024-05-09T08:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.816+0000 7fa1b2fc1640 1 -- start start 2024-05-09T08:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.817+0000 7fa1b2fc1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac110020 0x7fa1ac11a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.817+0000 7fa1b2fc1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ac11b010 0x7fa1ac116750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.817+0000 7fa1b2fc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ac116c90 0x7fa1ac117140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.817+0000 7fa1b2fc1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1131e0 con 0x7fa1ac116c90 2024-05-09T08:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.817+0000 7fa1b2fc1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac113060 con 0x7fa1ac110020 2024-05-09T08:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.817+0000 7fa1b2fc1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac113360 con 0x7fa1ac11b010 2024-05-09T08:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.818+0000 7fa1b0d5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac110020 0x7fa1ac11a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.818+0000 7fa1b155e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ac116c90 0x7fa1ac117140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.818+0000 7fa1b155e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ac116c90 0x7fa1ac117140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42116/0 (socket says 172.21.15.28:42116) 2024-05-09T08:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.818+0000 7fa1b155e640 1 -- 172.21.15.28:0/1162610447 learned_addr learned my addr 172.21.15.28:0/1162610447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.818+0000 7fa1b155e640 1 -- 172.21.15.28:0/1162610447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ac11b010 msgr2=0x7fa1ac116750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.818+0000 7fa1b155e640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ac11b010 0x7fa1ac116750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.818+0000 7fa1b155e640 1 -- 172.21.15.28:0/1162610447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac110020 msgr2=0x7fa1ac11a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.818+0000 7fa1b155e640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac110020 0x7fa1ac11a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.818+0000 7fa1b155e640 1 -- 172.21.15.28:0/1162610447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ac1aae00 con 0x7fa1ac116c90 2024-05-09T08:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.819+0000 7fa1b155e640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ac116c90 0x7fa1ac117140 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4005c40 tx=0x7fa1a4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.819+0000 7fa1a9ffb640 1 -- 172.21.15.28:0/1162610447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4044050 con 0x7fa1ac116c90 2024-05-09T08:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.819+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/1162610447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ac1ab090 con 0x7fa1ac116c90 2024-05-09T08:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.819+0000 7fa1a9ffb640 1 -- 172.21.15.28:0/1162610447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1a4047d70 con 0x7fa1ac116c90 2024-05-09T08:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.819+0000 7fa1a9ffb640 1 -- 172.21.15.28:0/1162610447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4004680 con 0x7fa1ac116c90 2024-05-09T08:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.819+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/1162610447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ac1ab550 con 0x7fa1ac116c90 2024-05-09T08:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.824+0000 7fa1b5dae740 1 Processor -- start 2024-05-09T08:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.824+0000 7fa1a9ffb640 1 -- 172.21.15.28:0/1162610447 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa1a4047070 con 0x7fa1ac116c90 2024-05-09T08:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.824+0000 7fa1b5dae740 1 -- start start 2024-05-09T08:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.824+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148dee4450 0x56148dee4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.824+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148deef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.824+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148deef630 0x56148def1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.824+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56148de49340 con 0x56148dee4d70 2024-05-09T08:31:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.824+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56148ddf1290 con 0x56148deef630 2024-05-09T08:31:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.824+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56148de45300 con 0x56148dee4450 2024-05-09T08:31:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.825+0000 7fa1b0d5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148deef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.825+0000 7fa1b0d5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148deef0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42126/0 (socket says 172.21.15.28:42126) 2024-05-09T08:31:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.825+0000 7fa1b0d5d640 1 -- 172.21.15.28:0/1755979802 learned_addr learned my addr 172.21.15.28:0/1755979802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.826+0000 7fa1a9ffb640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1940a2ef0 0x7fa1940a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.826+0000 7fa1b0d5d640 1 -- 172.21.15.28:0/1755979802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148dee4450 msgr2=0x56148dee4830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:31:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.826+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/1755979802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148dee4450 0x56148dee4830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.826+0000 7fa1b0d5d640 1 -- 172.21.15.28:0/1755979802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148deef630 msgr2=0x56148def1a20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:31:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.826+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/1755979802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148deef630 0x56148def1a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.826+0000 7fa1b0d5d640 1 -- 172.21.15.28:0/1755979802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56148de158e0 con 0x56148dee4d70 2024-05-09T08:31:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.826+0000 7fa1a9ffb640 1 -- 172.21.15.28:0/1162610447 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7fa1a41010a0 con 0x7fa1ac116c90 2024-05-09T08:31:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.827+0000 7fa1b155e640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1940a2ef0 0x7fa1940a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.827+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/1755979802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148deef0f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009cf0 tx=0x7fa1a0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b333e20159af473 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.828+0000 7fa192ffd640 1 -- 172.21.15.28:0/1755979802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0004230 con 0x56148dee4d70 2024-05-09T08:31:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.828+0000 7fa192ffd640 1 -- 172.21.15.28:0/1755979802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1a00043d0 con 0x56148dee4d70 2024-05-09T08:31:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.828+0000 7fa192ffd640 1 -- 172.21.15.28:0/1755979802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0019960 con 0x56148dee4d70 2024-05-09T08:31:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.829+0000 7fa1b5dae740 1 -- 172.21.15.28:0/1755979802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 msgr2=0x56148deef0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.829+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/1755979802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148deef0f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009cf0 tx=0x7fa1a0011810 comp rx=0 tx=0).stop 2024-05-09T08:31:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.829+0000 7fa1b155e640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1940a2ef0 0x7fa1940a53b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4056040 tx=0x7fa1a403b9b0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.829+0000 7fa1b5dae740 1 -- 172.21.15.28:0/1755979802 shutdown_connections 2024-05-09T08:31:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.829+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/1755979802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148deef630 0x56148def1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.829+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/1755979802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148deef0f0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.829+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/1755979802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148dee4450 0x56148dee4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.829+0000 7fa1b5dae740 1 -- 172.21.15.28:0/1755979802 >> 172.21.15.28:0/1755979802 conn(0x56148dee0960 msgr2=0x56148dee0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:48.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.832+0000 7fa1b5dae740 1 -- 172.21.15.28:0/1755979802 shutdown_connections 2024-05-09T08:31:48.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.832+0000 7fa1b5dae740 1 -- 172.21.15.28:0/1755979802 wait complete. 2024-05-09T08:31:48.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.832+0000 7fa1b5dae740 1 -- start start 2024-05-09T08:31:48.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.832+0000 7fa1b5dae740 1 Processor -- start 2024-05-09T08:31:48.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.833+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deec570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.833+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148dee4d70 0x56148deecab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.833+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148deef630 0x56148deecff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.833+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56148def2260 con 0x56148deef630 2024-05-09T08:31:48.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.833+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56148de04790 con 0x56148dee4450 2024-05-09T08:31:48.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.833+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56148def23e0 con 0x56148dee4d70 2024-05-09T08:31:48.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.833+0000 7fa1b5dae740 10 client.0 _ll_register_callbacks cb 0x56148de3fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:31:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.835+0000 7fa1abfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.835+0000 7fa1abfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deec570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:34132/0 (socket says 172.21.15.28:34132) 2024-05-09T08:31:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.835+0000 7fa1abfff640 1 -- 172.21.15.28:0/3322092800 learned_addr learned my addr 172.21.15.28:0/3322092800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.835+0000 7fa1abfff640 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148deef630 0x56148deecff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.835+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148dee4d70 0x56148deecab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.835+0000 7fa1abfff640 1 -- 172.21.15.28:0/3322092800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148dee4d70 msgr2=0x56148deecab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.835+0000 7fa1abfff640 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148dee4d70 0x56148deecab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.835+0000 7fa1abfff640 1 -- 172.21.15.28:0/3322092800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148deef630 msgr2=0x56148deecff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.836+0000 7fa1abfff640 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148deef630 0x56148deecff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.836+0000 7fa1abfff640 1 -- 172.21.15.28:0/3322092800 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56148db5d210 con 0x56148dee4450 2024-05-09T08:31:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.837+0000 7fa1abfff640 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deec570 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac06af00 tx=0x7fa19c009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee3949152cf8a590 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.837+0000 7fa18bfff640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:31:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.838+0000 7fa18bfff640 1 -- 172.21.15.28:0/3322092800 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c012040 con 0x56148dee4450 2024-05-09T08:31:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.838+0000 7fa18bfff640 1 -- 172.21.15.28:0/3322092800 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7fa19c004be0 con 0x56148dee4450 2024-05-09T08:31:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.838+0000 7fa1b5dae740 1 -- 172.21.15.28:0/3322092800 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1a00043d0 con 0x56148dee4450 2024-05-09T08:31:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.842+0000 7fa18bfff640 1 -- 172.21.15.28:0/3322092800 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fa19c004de0 con 0x56148dee4450 2024-05-09T08:31:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.843+0000 7fa1b5dae740 10 client.26144 fetch_fsmap learned FSMap version 49 2024-05-09T08:31:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.843+0000 7fa1b5dae740 1 -- 172.21.15.28:0/3322092800 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56148dc29620 con 0x56148dee4450 2024-05-09T08:31:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.843+0000 7fa18bfff640 1 -- 172.21.15.28:0/3322092800 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1a00043d0 con 0x56148dee4450 2024-05-09T08:31:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.843+0000 7fa18bfff640 1 -- 172.21.15.28:0/3322092800 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 49) ==== 8426+0+0 (secure 0 0 0) 0x7fa19c013b60 con 0x56148dee4450 2024-05-09T08:31:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.844+0000 7fa1b5dae740 10 client.26144 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T08:31:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.844+0000 7fa1b5dae740 10 client.26144 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:31:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.844+0000 7fa1b5dae740 20 client.26144 populate_metadata read hostname 'smithi028' 2024-05-09T08:31:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.844+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56148deff870 0x56148df01cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.844+0000 7fa1b5dae740 4 client.26144 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:31:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.844+0000 7fa1b5dae740 1 -- 172.21.15.28:0/3322092800 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56148de56a70 con 0x56148deff870 2024-05-09T08:31:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.845+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56148deff870 0x56148df01cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.858+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56148deff870 0x56148df01cf0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=683ded5d525ee70d server_cookie=4f1d390cd2a57c82 in_seq=0 out_seq=0 2024-05-09T08:31:48.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.859+0000 7fa18bfff640 10 client.26144 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.869+0000 7fa18bfff640 1 -- 172.21.15.28:0/3322092800 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56148de56a70 con 0x56148deff870 2024-05-09T08:31:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.869+0000 7fa18bfff640 10 client.26144 handle_command_reply: tid=0 2024-05-09T08:31:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.869+0000 7fa1b5dae740 1 client.26144 shutdown 2024-05-09T08:31:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.870+0000 7fa1b5dae740 1 -- 172.21.15.28:0/3322092800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 msgr2=0x56148deec570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.870+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deec570 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac06af00 tx=0x7fa19c009da0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.880+0000 7fa1b5dae740 1 -- 172.21.15.28:0/3322092800 shutdown_connections 2024-05-09T08:31:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.880+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56148deff870 0x56148df01cf0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.881+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148deef630 0x56148deecff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.881+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148dee4d70 0x56148deecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.881+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/3322092800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deec570 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.881+0000 7fa1b5dae740 1 -- 172.21.15.28:0/3322092800 >> 172.21.15.28:0/3322092800 conn(0x56148dee0960 msgr2=0x56148dee66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.882+0000 7fa18bfff640 0 client.26144 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:48.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.887+0000 7fa1b5dae740 1 -- 172.21.15.28:0/3322092800 shutdown_connections 2024-05-09T08:31:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.893+0000 7fa1b5dae740 1 -- 172.21.15.28:0/3322092800 wait complete. 2024-05-09T08:31:48.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.893+0000 7fa1b5dae740 20 client.26144 trim_cache size 0 max 16384 2024-05-09T08:31:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.937+0000 7fa1b5dae740 1 Processor -- start 2024-05-09T08:31:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.938+0000 7fa1b5dae740 1 -- start start 2024-05-09T08:31:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.938+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deebe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.938+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148dee6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.938+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148deef630 0x56148deee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.938+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b54e0 con 0x56148dee4d70 2024-05-09T08:31:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.938+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b7260 con 0x56148dee4450 2024-05-09T08:31:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.938+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b5c70 con 0x56148deef630 2024-05-09T08:31:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.939+0000 7fa1abfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deebe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.939+0000 7fa1abfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148deef630 0x56148deee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.939+0000 7fa1abfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deebe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:34138/0 (socket says 172.21.15.28:34138) 2024-05-09T08:31:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.939+0000 7fa1abfff640 1 -- 172.21.15.28:0/2612548778 learned_addr learned my addr 172.21.15.28:0/2612548778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:48.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.941+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/2612548778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148dee6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.941+0000 7fa1abfff640 1 -- 172.21.15.28:0/2612548778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148deef630 msgr2=0x56148deee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.941+0000 7fa1abfff640 1 --2- 172.21.15.28:0/2612548778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148deef630 0x56148deee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.941+0000 7fa1abfff640 1 -- 172.21.15.28:0/2612548778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 msgr2=0x56148dee6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.941+0000 7fa1abfff640 1 --2- 172.21.15.28:0/2612548778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148dee6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.941+0000 7fa1abfff640 1 -- 172.21.15.28:0/2612548778 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56148dee7350 con 0x56148dee4450 2024-05-09T08:31:48.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.942+0000 7fa1abfff640 1 --2- 172.21.15.28:0/2612548778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deebe50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa19c013fb0 tx=0x7fa19c000e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b8e1f8d513bfc53 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.944+0000 7fa1927fc640 1 -- 172.21.15.28:0/2612548778 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c0197f0 con 0x56148dee4450 2024-05-09T08:31:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.944+0000 7fa1927fc640 1 -- 172.21.15.28:0/2612548778 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19c0049e0 con 0x56148dee4450 2024-05-09T08:31:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.945+0000 7fa1b5dae740 1 -- 172.21.15.28:0/2612548778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 msgr2=0x56148deebe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.945+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/2612548778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deebe50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa19c013fb0 tx=0x7fa19c000e60 comp rx=0 tx=0).stop 2024-05-09T08:31:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.947+0000 7fa1b5dae740 1 -- 172.21.15.28:0/2612548778 shutdown_connections 2024-05-09T08:31:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.947+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/2612548778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148deef630 0x56148deee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.947+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/2612548778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148dee6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.947+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/2612548778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148deebe50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.947+0000 7fa1b5dae740 1 -- 172.21.15.28:0/2612548778 >> 172.21.15.28:0/2612548778 conn(0x56148dee0960 msgr2=0x56148deedc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:48.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.949+0000 7fa1b5dae740 1 -- 172.21.15.28:0/2612548778 shutdown_connections 2024-05-09T08:31:48.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.950+0000 7fa1b5dae740 1 -- 172.21.15.28:0/2612548778 wait complete. 2024-05-09T08:31:48.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.950+0000 7fa1b5dae740 1 -- start start 2024-05-09T08:31:48.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.950+0000 7fa1b5dae740 1 Processor -- start 2024-05-09T08:31:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.951+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148def7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.951+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148def7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.951+0000 7fa1b5dae740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148deef630 0x56148dee8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.951+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b6ad0 con 0x56148dee4d70 2024-05-09T08:31:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.951+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b6340 con 0x56148dee4450 2024-05-09T08:31:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.951+0000 7fa1b5dae740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b34c0 con 0x56148deef630 2024-05-09T08:31:48.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.951+0000 7fa1b5dae740 10 client.0 _ll_register_callbacks cb 0x7fa19c0012f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:31:48.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.952+0000 7fa1b0d5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148def7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.952+0000 7fa1b0d5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148def7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42170/0 (socket says 172.21.15.28:42170) 2024-05-09T08:31:48.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.952+0000 7fa1b0d5d640 1 -- 172.21.15.28:0/586955400 learned_addr learned my addr 172.21.15.28:0/586955400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:31:48.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.953+0000 7fa1b0d5d640 1 -- 172.21.15.28:0/586955400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148deef630 msgr2=0x56148dee8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:48.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.953+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148deef630 0x56148dee8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.953+0000 7fa1b0d5d640 1 -- 172.21.15.28:0/586955400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 msgr2=0x56148def7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:48.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.953+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148def7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.953+0000 7fa1b0d5d640 1 -- 172.21.15.28:0/586955400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56148dee9010 con 0x56148dee4d70 2024-05-09T08:31:48.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.955+0000 7fa1b0d5d640 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148def7bc0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009250 tx=0x7fa1a0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfb82a4f4919572 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.955+0000 7fa18bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:31:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.955+0000 7fa18bfff640 1 -- 172.21.15.28:0/586955400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0009910 con 0x56148dee4d70 2024-05-09T08:31:48.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.956+0000 7fa1b5dae740 1 -- 172.21.15.28:0/586955400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa19c0049e0 con 0x56148dee4d70 2024-05-09T08:31:48.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.956+0000 7fa18bfff640 1 -- 172.21.15.28:0/586955400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7fa1a002bd60 con 0x56148dee4d70 2024-05-09T08:31:48.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.956+0000 7fa18bfff640 1 -- 172.21.15.28:0/586955400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fa1a003b050 con 0x56148dee4d70 2024-05-09T08:31:48.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.957+0000 7fa1b5dae740 10 client.16677 fetch_fsmap learned FSMap version 49 2024-05-09T08:31:48.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.957+0000 7fa1b5dae740 1 -- 172.21.15.28:0/586955400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56148df658f0 con 0x56148dee4d70 2024-05-09T08:31:48.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.957+0000 7fa18bfff640 1 -- 172.21.15.28:0/586955400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19c0049e0 con 0x56148dee4d70 2024-05-09T08:31:48.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.957+0000 7fa18bfff640 1 -- 172.21.15.28:0/586955400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) ==== 8426+0+0 (secure 0 0 0) 0x7fa1a002b2d0 con 0x56148dee4d70 2024-05-09T08:31:48.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.958+0000 7fa1b5dae740 10 client.16677 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T08:31:48.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.958+0000 7fa1b5dae740 10 client.16677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:31:48.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.958+0000 7fa1b5dae740 20 client.16677 populate_metadata read hostname 'smithi028' 2024-05-09T08:31:48.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.958+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56148df66110 0x56148df68530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.958+0000 7fa1b5dae740 4 client.16677 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:31:48.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.958+0000 7fa1b5dae740 1 -- 172.21.15.28:0/586955400 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56148de50dc0 con 0x56148df66110 2024-05-09T08:31:48.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.959+0000 7fa1abfff640 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56148df66110 0x56148df68530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.962+0000 7fa1abfff640 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56148df66110 0x56148df68530 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=998698b941117b25 server_cookie=179be31f1848ddd4 in_seq=0 out_seq=0 2024-05-09T08:31:48.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:31:48.963+0000 7fa18bfff640 10 client.16677 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:31:49.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:49 smithi165 ceph-mon[34540]: pgmap v171: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.0 MiB/s wr, 413 op/s 2024-05-09T08:31:49.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:49 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:31:49.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:49 smithi184 ceph-mon[34509]: pgmap v171: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.0 MiB/s wr, 413 op/s 2024-05-09T08:31:49.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:49 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:31:49.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:49 smithi028 ceph-mon[30119]: pgmap v171: 129 pgs: 129 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.0 MiB/s wr, 413 op/s 2024-05-09T08:31:49.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:49 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:31:50.682 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:50 smithi028 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-05-09T08:31:50.682 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:50 smithi028 systemd[1]: Stopped Ceph mds.d for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:31:50.683 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:50 smithi028 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.d.service: Consumed 5.617s CPU time. 2024-05-09T08:31:50.683 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:50 smithi028 systemd[1]: Starting Ceph mds.d for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:31:51.365 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:51 smithi028 podman[91668]: 2024-05-09 08:31:50.935922978 +0000 UTC m=+0.117742211 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:31:51.365 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:51 smithi028 podman[91668]: 2024-05-09 08:31:51.210561521 +0000 UTC m=+0.392380786 container create 6f72f5305c15b165b06aac9c8b8f84d0a7c0f90b7dfc8c7f91c6b5863384d4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc) 2024-05-09T08:31:51.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:51 smithi165 ceph-mon[34540]: pgmap v172: 129 pgs: 129 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 8.9 MiB/s wr, 513 op/s 2024-05-09T08:31:51.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:51 smithi184 ceph-mon[34509]: pgmap v172: 129 pgs: 129 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 8.9 MiB/s wr, 513 op/s 2024-05-09T08:31:51.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:51 smithi028 ceph-mon[30119]: pgmap v172: 129 pgs: 129 active+clean; 1.8 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 8.9 MiB/s wr, 513 op/s 2024-05-09T08:31:51.933 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:51 smithi028 podman[91668]: 2024-05-09 08:31:51.512518409 +0000 UTC m=+0.694337646 container init 6f72f5305c15b165b06aac9c8b8f84d0a7c0f90b7dfc8c7f91c6b5863384d4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:31:51.933 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:51 smithi028 podman[91668]: 2024-05-09 08:31:51.521098259 +0000 UTC m=+0.702917484 container start 6f72f5305c15b165b06aac9c8b8f84d0a7c0f90b7dfc8c7f91c6b5863384d4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-09T08:31:51.933 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:51 smithi028 bash[91668]: 6f72f5305c15b165b06aac9c8b8f84d0a7c0f90b7dfc8c7f91c6b5863384d4ab 2024-05-09T08:31:51.933 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:51 smithi028 systemd[1]: Started Ceph mds.d for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:31:51.933 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:31:51 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[91682]: starting mds.d at 2024-05-09T08:31:52.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:52 smithi165 ceph-mon[34540]: daemon mds.d restarted 2024-05-09T08:31:52.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:52 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:31:52.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:52 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:31:52.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:52 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:31:52.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:52 smithi165 ceph-mon[34540]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T08:31:52.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:52 smithi184 ceph-mon[34509]: daemon mds.d restarted 2024-05-09T08:31:52.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:52 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:31:52.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:52 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:31:52.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:52 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:31:52.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:52 smithi184 ceph-mon[34509]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T08:31:52.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:52 smithi028 ceph-mon[30119]: daemon mds.d restarted 2024-05-09T08:31:52.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:52 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:31:52.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:52 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:31:52.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:52 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:31:52.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:52 smithi028 ceph-mon[30119]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T08:31:53.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:53 smithi165 ceph-mon[34540]: pgmap v173: 129 pgs: 129 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.7 MiB/s wr, 544 op/s 2024-05-09T08:31:53.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:53 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.28:6836/2982138678,v1:172.21.15.28:6837/2982138678] up:boot 2024-05-09T08:31:53.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:53 smithi165 ceph-mon[34540]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:31:53.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:53 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:31:53.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:53 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:53 smithi184 ceph-mon[34509]: pgmap v173: 129 pgs: 129 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.7 MiB/s wr, 544 op/s 2024-05-09T08:31:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:53 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.28:6836/2982138678,v1:172.21.15.28:6837/2982138678] up:boot 2024-05-09T08:31:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:53 smithi184 ceph-mon[34509]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:31:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:53 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:31:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:53 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:53.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:53 smithi028 ceph-mon[30119]: pgmap v173: 129 pgs: 129 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.7 MiB/s wr, 544 op/s 2024-05-09T08:31:53.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:53 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.28:6836/2982138678,v1:172.21.15.28:6837/2982138678] up:boot 2024-05-09T08:31:53.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:53 smithi028 ceph-mon[30119]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:31:53.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:53 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:31:53.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:53 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:55.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:55 smithi165 ceph-mon[34540]: pgmap v175: 129 pgs: 129 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 13 MiB/s wr, 759 op/s 2024-05-09T08:31:55.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:55 smithi184 ceph-mon[34509]: pgmap v175: 129 pgs: 129 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 13 MiB/s wr, 759 op/s 2024-05-09T08:31:55.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:55 smithi028 ceph-mon[30119]: pgmap v175: 129 pgs: 129 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 13 MiB/s wr, 759 op/s 2024-05-09T08:31:56.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:56 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:56.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:56 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:56.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:56 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:31:57.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:57 smithi165 ceph-mon[34540]: pgmap v176: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 14 MiB/s wr, 1.10k op/s 2024-05-09T08:31:57.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:57 smithi184 ceph-mon[34509]: pgmap v176: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 14 MiB/s wr, 1.10k op/s 2024-05-09T08:31:57.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:57 smithi028 ceph-mon[30119]: pgmap v176: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 14 MiB/s wr, 1.10k op/s 2024-05-09T08:31:58.709 INFO:tasks.workunit.client.0.smithi028.stdout: 557 1110770 162 750309 170 231438 403 2024-05-09T08:31:59.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:31:59 smithi028 ceph-mon[30119]: pgmap v177: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 1.32k op/s 2024-05-09T08:32:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:31:59 smithi165 ceph-mon[34540]: pgmap v177: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 1.32k op/s 2024-05-09T08:32:00.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:31:59 smithi184 ceph-mon[34509]: pgmap v177: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 1.32k op/s 2024-05-09T08:32:01.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:01 smithi028 ceph-mon[30119]: pgmap v178: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-05-09T08:32:01.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:01 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:01 smithi165 ceph-mon[34540]: pgmap v178: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-05-09T08:32:02.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:01 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:02.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:01 smithi184 ceph-mon[34509]: pgmap v178: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-05-09T08:32:02.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:01 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:03.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:03 smithi165 ceph-mon[34540]: pgmap v179: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-05-09T08:32:03.907 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:03 smithi184 ceph-mon[34509]: pgmap v179: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-05-09T08:32:03.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:03 smithi028 ceph-mon[30119]: pgmap v179: 129 pgs: 129 active+clean; 1.9 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-05-09T08:32:05.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:05 smithi028 ceph-mon[30119]: pgmap v180: 129 pgs: 129 active+clean; 1.9 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 9.9 MiB/s wr, 929 op/s 2024-05-09T08:32:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:05 smithi165 ceph-mon[34540]: pgmap v180: 129 pgs: 129 active+clean; 1.9 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 9.9 MiB/s wr, 929 op/s 2024-05-09T08:32:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:05 smithi184 ceph-mon[34509]: pgmap v180: 129 pgs: 129 active+clean; 1.9 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 9.9 MiB/s wr, 929 op/s 2024-05-09T08:32:07.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:06 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:07.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:06 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:07.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:06 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:07.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.667+0000 7fa18bfff640 1 -- 172.21.15.28:0/586955400 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56148de50dc0 con 0x56148df66110 2024-05-09T08:32:07.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.667+0000 7fa18bfff640 10 client.16677 handle_command_reply: tid=0 2024-05-09T08:32:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 client.16677 shutdown 2024-05-09T08:32:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 -- 172.21.15.28:0/586955400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 msgr2=0x56148def7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148def7bc0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009250 tx=0x7fa1a0005850 comp rx=0 tx=0).stop 2024-05-09T08:32:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 -- 172.21.15.28:0/586955400 shutdown_connections 2024-05-09T08:32:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56148df66110 0x56148df68530 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56148deef630 0x56148dee8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56148dee4d70 0x56148def7bc0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 --2- 172.21.15.28:0/586955400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56148dee4450 0x56148def7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 -- 172.21.15.28:0/586955400 >> 172.21.15.28:0/586955400 conn(0x56148dee0960 msgr2=0x56148deee3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa18bfff640 0 client.16677 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.672+0000 7fa1b5dae740 1 -- 172.21.15.28:0/586955400 shutdown_connections 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.673+0000 7fa1b5dae740 1 -- 172.21.15.28:0/586955400 wait complete. 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.673+0000 7fa1b5dae740 20 client.16677 trim_cache size 0 max 16384 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:32:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:32:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.733+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1940a2ef0 msgr2=0x7fa1940a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.733+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1940a2ef0 0x7fa1940a53b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4056040 tx=0x7fa1a403b9b0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.733+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ac116c90 msgr2=0x7fa1ac117140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.733+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ac116c90 0x7fa1ac117140 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4005c40 tx=0x7fa1a4003ba0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.743+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/1162610447 shutdown_connections 2024-05-09T08:32:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.743+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ac116c90 0x7fa1ac117140 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.743+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1940a2ef0 0x7fa1940a53b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.743+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ac11b010 0x7fa1ac116750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.743+0000 7fa1b2fc1640 1 --2- 172.21.15.28:0/1162610447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac110020 0x7fa1ac11a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:07.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.743+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/1162610447 >> 172.21.15.28:0/1162610447 conn(0x7fa1ac06c900 msgr2=0x7fa1ac11b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:07.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.746+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/1162610447 shutdown_connections 2024-05-09T08:32:07.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:07.746+0000 7fa1b2fc1640 1 -- 172.21.15.28:0/1162610447 wait complete. 2024-05-09T08:32:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:07 smithi165 ceph-mon[34540]: pgmap v181: 129 pgs: 129 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 11 MiB/s wr, 915 op/s 2024-05-09T08:32:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:07 smithi184 ceph-mon[34509]: pgmap v181: 129 pgs: 129 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 11 MiB/s wr, 915 op/s 2024-05-09T08:32:08.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:07 smithi028 ceph-mon[30119]: pgmap v181: 129 pgs: 129 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 11 MiB/s wr, 915 op/s 2024-05-09T08:32:08.718 INFO:tasks.workunit.client.0.smithi028.stdout: 564 1684747 214 1136904 454 380227 2164 2024-05-09T08:32:09.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:08 smithi165 ceph-mon[34540]: pgmap v182: 129 pgs: 129 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 8.2 MiB/s wr, 630 op/s 2024-05-09T08:32:09.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:08 smithi184 ceph-mon[34509]: pgmap v182: 129 pgs: 129 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 8.2 MiB/s wr, 630 op/s 2024-05-09T08:32:09.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:08 smithi028 ceph-mon[30119]: pgmap v182: 129 pgs: 129 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 8.2 MiB/s wr, 630 op/s 2024-05-09T08:32:11.102 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: *** Caught signal (Segmentation fault) ** 2024-05-09T08:32:11.102 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: in thread 7f12a8c79700 thread_name:md_log_replay 2024-05-09T08:32:11.102 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:32:11.103 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f12ba03bd20] 2024-05-09T08:32:11.103 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0xe3) [0x7f12bbd01573] 2024-05-09T08:32:11.103 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 3: (tcmalloc::ThreadCache::Cleanup()+0x44) [0x7f12bbd01624] 2024-05-09T08:32:11.103 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 4: (tcmalloc::ThreadCache::DeleteCache(tcmalloc::ThreadCache*)+0x12) [0x7f12bbd01b02] 2024-05-09T08:32:11.103 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 5: /lib64/libpthread.so.0(+0x7451) [0x7f12ba030451] 2024-05-09T08:32:11.103 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 6: /lib64/libpthread.so.0(+0x81eb) [0x7f12ba0311eb] 2024-05-09T08:32:11.104 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 7: clone() 2024-05-09T08:32:11.104 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2024-05-09T08:32:10.830+0000 7f12a8c79700 -1 *** Caught signal (Segmentation fault) ** 2024-05-09T08:32:11.104 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: in thread 7f12a8c79700 thread_name:md_log_replay 2024-05-09T08:32:11.104 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2024-05-09T08:32:11.104 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:32:11.104 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f12ba03bd20] 2024-05-09T08:32:11.105 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0xe3) [0x7f12bbd01573] 2024-05-09T08:32:11.105 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 3: (tcmalloc::ThreadCache::Cleanup()+0x44) [0x7f12bbd01624] 2024-05-09T08:32:11.105 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 4: (tcmalloc::ThreadCache::DeleteCache(tcmalloc::ThreadCache*)+0x12) [0x7f12bbd01b02] 2024-05-09T08:32:11.105 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 5: /lib64/libpthread.so.0(+0x7451) [0x7f12ba030451] 2024-05-09T08:32:11.105 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 6: /lib64/libpthread.so.0(+0x81eb) [0x7f12ba0311eb] 2024-05-09T08:32:11.105 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 7: clone() 2024-05-09T08:32:11.106 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:32:11.106 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2024-05-09T08:32:11.106 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 0> 2024-05-09T08:32:10.830+0000 7f12a8c79700 -1 *** Caught signal (Segmentation fault) ** 2024-05-09T08:32:11.106 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: in thread 7f12a8c79700 thread_name:md_log_replay 2024-05-09T08:32:11.106 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2024-05-09T08:32:11.106 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:32:11.106 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f12ba03bd20] 2024-05-09T08:32:11.108 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0xe3) [0x7f12bbd01573] 2024-05-09T08:32:11.108 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 3: (tcmalloc::ThreadCache::Cleanup()+0x44) [0x7f12bbd01624] 2024-05-09T08:32:11.108 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 4: (tcmalloc::ThreadCache::DeleteCache(tcmalloc::ThreadCache*)+0x12) [0x7f12bbd01b02] 2024-05-09T08:32:11.108 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 5: /lib64/libpthread.so.0(+0x7451) [0x7f12ba030451] 2024-05-09T08:32:11.108 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 6: /lib64/libpthread.so.0(+0x81eb) [0x7f12ba0311eb] 2024-05-09T08:32:11.108 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 7: clone() 2024-05-09T08:32:11.108 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:32:11.108 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2024-05-09T08:32:11.108 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 0> 2024-05-09T08:32:10.830+0000 7f12a8c79700 -1 *** Caught signal (Segmentation fault) ** 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: in thread 7f12a8c79700 thread_name:md_log_replay 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f12ba03bd20] 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0xe3) [0x7f12bbd01573] 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 3: (tcmalloc::ThreadCache::Cleanup()+0x44) [0x7f12bbd01624] 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 4: (tcmalloc::ThreadCache::DeleteCache(tcmalloc::ThreadCache*)+0x12) [0x7f12bbd01b02] 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 5: /lib64/libpthread.so.0(+0x7451) [0x7f12ba030451] 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 6: /lib64/libpthread.so.0(+0x81eb) [0x7f12ba0311eb] 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 7: clone() 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[68814]: 2024-05-09T08:32:11.109 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:10 smithi165 podman[69008]: 2024-05-09 08:32:10.91439556 +0000 UTC m=+0.025677938 container died 6e8c0a675b24fc008fd9f15a680878b0f6f6249cecfcd0b98e515421fcda754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc) 2024-05-09T08:32:11.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:11 smithi165 ceph-mon[34540]: pgmap v183: 129 pgs: 129 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.1 MiB/s wr, 514 op/s 2024-05-09T08:32:11.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:11 smithi184 ceph-mon[34509]: pgmap v183: 129 pgs: 129 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.1 MiB/s wr, 514 op/s 2024-05-09T08:32:11.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:11 smithi028 ceph-mon[30119]: pgmap v183: 129 pgs: 129 active+clean; 2.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.1 MiB/s wr, 514 op/s 2024-05-09T08:32:13.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:13 smithi165 ceph-mon[34540]: pgmap v184: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.0 MiB/s wr, 447 op/s 2024-05-09T08:32:13.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:13 smithi184 ceph-mon[34509]: pgmap v184: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.0 MiB/s wr, 447 op/s 2024-05-09T08:32:13.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:13 smithi028 ceph-mon[30119]: pgmap v184: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.0 MiB/s wr, 447 op/s 2024-05-09T08:32:14.056 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:13 smithi165 podman[69008]: 2024-05-09 08:32:13.754305729 +0000 UTC m=+2.865588103 container remove 6e8c0a675b24fc008fd9f15a680878b0f6f6249cecfcd0b98e515421fcda754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=) 2024-05-09T08:32:14.057 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:13 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Main process exited, code=exited, status=139/n/a 2024-05-09T08:32:14.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:14 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-05-09T08:32:14.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:14 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Consumed 3.083s CPU time. 2024-05-09T08:32:15.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:15 smithi165 ceph-mon[34540]: pgmap v185: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 422 op/s 2024-05-09T08:32:15.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:15 smithi184 ceph-mon[34509]: pgmap v185: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 422 op/s 2024-05-09T08:32:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:15 smithi028 ceph-mon[30119]: pgmap v185: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 422 op/s 2024-05-09T08:32:17.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:17 smithi165 ceph-mon[34540]: pgmap v186: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.7 MiB/s wr, 474 op/s 2024-05-09T08:32:17.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:17 smithi184 ceph-mon[34509]: pgmap v186: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.7 MiB/s wr, 474 op/s 2024-05-09T08:32:17.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:17 smithi028 ceph-mon[30119]: pgmap v186: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.7 MiB/s wr, 474 op/s 2024-05-09T08:32:18.715 INFO:tasks.workunit.client.0.smithi028.stdout: 587 1943261 1264 1310782 1025 479157 2530 2024-05-09T08:32:19.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:19 smithi165 ceph-mon[34540]: pgmap v187: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.5 MiB/s wr, 410 op/s 2024-05-09T08:32:19.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:19 smithi184 ceph-mon[34509]: pgmap v187: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.5 MiB/s wr, 410 op/s 2024-05-09T08:32:19.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:19 smithi028 ceph-mon[30119]: pgmap v187: 129 pgs: 129 active+clean; 2.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.5 MiB/s wr, 410 op/s 2024-05-09T08:32:21.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:21 smithi165 ceph-mon[34540]: pgmap v188: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.3 MiB/s wr, 560 op/s 2024-05-09T08:32:21.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:21 smithi184 ceph-mon[34509]: pgmap v188: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.3 MiB/s wr, 560 op/s 2024-05-09T08:32:21.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:21 smithi028 ceph-mon[30119]: pgmap v188: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.3 MiB/s wr, 560 op/s 2024-05-09T08:32:23.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:23 smithi165 ceph-mon[34540]: pgmap v189: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.6 MiB/s wr, 549 op/s 2024-05-09T08:32:23.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:23 smithi184 ceph-mon[34509]: pgmap v189: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.6 MiB/s wr, 549 op/s 2024-05-09T08:32:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.610+0000 7f238b729640 1 -- 172.21.15.28:0/49901663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2384108aa0 msgr2=0x7f2384108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.610+0000 7f238b729640 1 --2- 172.21.15.28:0/49901663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2384108aa0 0x7f2384108e80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2374009900 tx=0x7f237402a410 comp rx=0 tx=0).stop 2024-05-09T08:32:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.611+0000 7f238b729640 1 -- 172.21.15.28:0/49901663 shutdown_connections 2024-05-09T08:32:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.611+0000 7f238b729640 1 --2- 172.21.15.28:0/49901663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f238410cff0 0x7f2384079ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.611+0000 7f238b729640 1 --2- 172.21.15.28:0/49901663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2384109450 0x7f238410c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.611+0000 7f238b729640 1 --2- 172.21.15.28:0/49901663 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2384108aa0 0x7f2384108e80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.611+0000 7f238b729640 1 -- 172.21.15.28:0/49901663 >> 172.21.15.28:0/49901663 conn(0x7f238406c7e0 msgr2=0x7f238406cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.613+0000 7f238b729640 1 -- 172.21.15.28:0/49901663 shutdown_connections 2024-05-09T08:32:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.613+0000 7f238b729640 1 -- 172.21.15.28:0/49901663 wait complete. 2024-05-09T08:32:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.619+0000 7f238b729640 1 Processor -- start 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.619+0000 7f238b729640 1 -- start start 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f238b729640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2384108aa0 0x7f238407e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f238b729640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f238410cff0 0x7f238407eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f238b729640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f238407f0b0 0x7f23841bb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f238b729640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f238407d0e0 con 0x7f238410cff0 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f238b729640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f238407cf60 con 0x7f2384108aa0 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f238b729640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f238407d260 con 0x7f238407f0b0 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f23894c5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2384108aa0 0x7f238407e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f23894c5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2384108aa0 0x7f238407e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:49062/0 (socket says 172.21.15.28:49062) 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f23894c5640 1 -- 172.21.15.28:0/1741846938 learned_addr learned my addr 172.21.15.28:0/1741846938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f2389cc6640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f238407f0b0 0x7f23841bb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f23894c5640 1 -- 172.21.15.28:0/1741846938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f238407f0b0 msgr2=0x7f23841bb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f23894c5640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f238407f0b0 0x7f23841bb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.620+0000 7f23894c5640 1 -- 172.21.15.28:0/1741846938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f238410cff0 msgr2=0x7f238407eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.621+0000 7f23894c5640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f238410cff0 0x7f238407eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.621+0000 7f23894c5640 1 -- 172.21.15.28:0/1741846938 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23841bbdb0 con 0x7f2384108aa0 2024-05-09T08:32:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.621+0000 7f23894c5640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2384108aa0 0x7f238407e630 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2374006010 tx=0x7f23740055e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:23.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.636+0000 7f237a7fc640 1 -- 172.21.15.28:0/1741846938 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237403f030 con 0x7f2384108aa0 2024-05-09T08:32:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.637+0000 7f238b729640 1 -- 172.21.15.28:0/1741846938 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23841bbfe0 con 0x7f2384108aa0 2024-05-09T08:32:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.638+0000 7f238b729640 1 -- 172.21.15.28:0/1741846938 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23841bc440 con 0x7f2384108aa0 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.645+0000 7f238e438740 1 Processor -- start 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.645+0000 7f238e438740 1 -- start start 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.646+0000 7f238e438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c5a400 0x560df4c5a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.646+0000 7f238e438740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4c762b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.646+0000 7f238e438740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c767f0 0x560df4c78be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.646+0000 7f238e438740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560df4bbf340 con 0x560df4c5a400 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.646+0000 7f238e438740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560df4b67290 con 0x560df4c5ad20 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.646+0000 7f238e438740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560df4bbb300 con 0x560df4c767f0 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.646+0000 7f2388cc4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c767f0 0x560df4c78be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.646+0000 7f2388cc4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c767f0 0x560df4c78be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:36600/0 (socket says 172.21.15.28:36600) 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.647+0000 7f2388cc4640 1 -- 172.21.15.28:0/662826707 learned_addr learned my addr 172.21.15.28:0/662826707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.647+0000 7f2388cc4640 1 -- 172.21.15.28:0/662826707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 msgr2=0x560df4c762b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.647+0000 7f2388cc4640 1 --2- 172.21.15.28:0/662826707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4c762b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.647+0000 7f2388cc4640 1 -- 172.21.15.28:0/662826707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c5a400 msgr2=0x560df4c5a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.647+0000 7f2388cc4640 1 --2- 172.21.15.28:0/662826707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c5a400 0x560df4c5a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.647+0000 7f2388cc4640 1 -- 172.21.15.28:0/662826707 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df4b8b8e0 con 0x560df4c767f0 2024-05-09T08:32:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.647+0000 7f2388cc4640 1 --2- 172.21.15.28:0/662826707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c767f0 0x560df4c78be0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2380009b40 tx=0x7f238000e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6d9927f30de601b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.658+0000 7f237a7fc640 1 -- 172.21.15.28:0/1741846938 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2374004db0 con 0x7f2384108aa0 2024-05-09T08:32:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.658+0000 7f237a7fc640 1 -- 172.21.15.28:0/1741846938 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374035070 con 0x7f2384108aa0 2024-05-09T08:32:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.658+0000 7f237a7fc640 1 -- 172.21.15.28:0/1741846938 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2374042030 con 0x7f2384108aa0 2024-05-09T08:32:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.663+0000 7f237a7fc640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f23700a2f30 0x7f23700a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.663+0000 7f237a7fc640 1 -- 172.21.15.28:0/1741846938 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9495+0+0 (secure 0 0 0) 0x7f23740f0bb0 con 0x7f2384108aa0 2024-05-09T08:32:23.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.667+0000 7f2389cc6640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f23700a2f30 0x7f23700a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.668+0000 7f2389cc6640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f23700a2f30 0x7f23700a53f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f237c00d850 tx=0x7f237c00f040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.669+0000 7f236f7fe640 1 -- 172.21.15.28:0/662826707 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238000ed80 con 0x560df4c767f0 2024-05-09T08:32:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.670+0000 7f236f7fe640 1 -- 172.21.15.28:0/662826707 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2380005020 con 0x560df4c767f0 2024-05-09T08:32:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.671+0000 7f238e438740 1 -- 172.21.15.28:0/662826707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c767f0 msgr2=0x560df4c78be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.671+0000 7f238e438740 1 --2- 172.21.15.28:0/662826707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c767f0 0x560df4c78be0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2380009b40 tx=0x7f238000e350 comp rx=0 tx=0).stop 2024-05-09T08:32:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.671+0000 7f238e438740 1 -- 172.21.15.28:0/662826707 shutdown_connections 2024-05-09T08:32:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.671+0000 7f238e438740 1 --2- 172.21.15.28:0/662826707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c767f0 0x560df4c78be0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.671+0000 7f238e438740 1 --2- 172.21.15.28:0/662826707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4c762b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.671+0000 7f238e438740 1 --2- 172.21.15.28:0/662826707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c5a400 0x560df4c5a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.672+0000 7f238e438740 1 -- 172.21.15.28:0/662826707 >> 172.21.15.28:0/662826707 conn(0x560df4c56b10 msgr2=0x560df4c56ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:23.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:23 smithi028 ceph-mon[30119]: pgmap v189: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.6 MiB/s wr, 549 op/s 2024-05-09T08:32:23.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.687+0000 7f238e438740 1 -- 172.21.15.28:0/662826707 shutdown_connections 2024-05-09T08:32:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.687+0000 7f238e438740 1 -- 172.21.15.28:0/662826707 wait complete. 2024-05-09T08:32:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.687+0000 7f238e438740 1 -- start start 2024-05-09T08:32:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.688+0000 7f238e438740 1 Processor -- start 2024-05-09T08:32:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.688+0000 7f238e438740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5a400 0x560df4c67a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.688+0000 7f238e438740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5ad20 0x560df4c67fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.688+0000 7f238e438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 0x560df4c62ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.688+0000 7f238e438740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560df4c5d530 con 0x560df4c767f0 2024-05-09T08:32:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.688+0000 7f238e438740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560df4b7a790 con 0x560df4c5a400 2024-05-09T08:32:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.688+0000 7f238e438740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560df4c5d6b0 con 0x560df4c5ad20 2024-05-09T08:32:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.688+0000 7f238e438740 10 client.0 _ll_register_callbacks cb 0x560df4bb5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.689+0000 7f23894c5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5a400 0x560df4c67a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.689+0000 7f23894c5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5a400 0x560df4c67a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:49064/0 (socket says 172.21.15.28:49064) 2024-05-09T08:32:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.689+0000 7f23894c5640 1 -- 172.21.15.28:0/4179335319 learned_addr learned my addr 172.21.15.28:0/4179335319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.689+0000 7f2388cc4640 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5ad20 0x560df4c67fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.689+0000 7f23894c5640 1 -- 172.21.15.28:0/4179335319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5ad20 msgr2=0x560df4c67fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.689+0000 7f23894c5640 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5ad20 0x560df4c67fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.689+0000 7f23894c5640 1 -- 172.21.15.28:0/4179335319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 msgr2=0x560df4c62ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.689+0000 7f23894c5640 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 0x560df4c62ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.689+0000 7f23894c5640 1 -- 172.21.15.28:0/4179335319 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df48d3210 con 0x560df4c5a400 2024-05-09T08:32:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.690+0000 7f23894c5640 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5a400 0x560df4c67a90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f237402aa00 tx=0x7f23740bcaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bff7c78fc8c131ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.710+0000 7f236cff9640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:32:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.710+0000 7f236cff9640 1 -- 172.21.15.28:0/4179335319 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374049040 con 0x560df4c5a400 2024-05-09T08:32:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.710+0000 7f238e438740 1 -- 172.21.15.28:0/4179335319 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2380005020 con 0x560df4c5a400 2024-05-09T08:32:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.710+0000 7f236cff9640 1 -- 172.21.15.28:0/4179335319 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9495+0+0 (secure 0 0 0) 0x7f23740ee1d0 con 0x560df4c5a400 2024-05-09T08:32:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.710+0000 7f236cff9640 1 -- 172.21.15.28:0/4179335319 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f23740319b0 con 0x560df4c5a400 2024-05-09T08:32:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.710+0000 7f238e438740 10 client.26162 fetch_fsmap learned FSMap version 54 2024-05-09T08:32:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.710+0000 7f238e438740 1 -- 172.21.15.28:0/4179335319 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x560df499f620 con 0x560df4c5a400 2024-05-09T08:32:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.710+0000 7f236cff9640 1 -- 172.21.15.28:0/4179335319 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2380005020 con 0x560df4c5a400 2024-05-09T08:32:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.711+0000 7f236cff9640 1 -- 172.21.15.28:0/4179335319 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 54) ==== 8414+0+0 (secure 0 0 0) 0x7f23740c5be0 con 0x560df4c5a400 2024-05-09T08:32:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.711+0000 7f238e438740 10 client.26162 fetch_fsmap finished waiting for FSMap version 54 2024-05-09T08:32:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.711+0000 7f238e438740 10 client.26162 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:32:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.711+0000 7f238e438740 20 client.26162 populate_metadata read hostname 'smithi028' 2024-05-09T08:32:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.711+0000 7f238e438740 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560df4c72fa0 0x560df4c75420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.711+0000 7f238e438740 4 client.26162 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:32:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.711+0000 7f238e438740 1 -- 172.21.15.28:0/4179335319 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560df4bcca70 con 0x560df4c72fa0 2024-05-09T08:32:23.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.725+0000 7f2388cc4640 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560df4c72fa0 0x560df4c75420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.726+0000 7f2388cc4640 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560df4c72fa0 0x560df4c75420 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cac8dbe9e69bb023 server_cookie=bd17fac5ee449e63 in_seq=0 out_seq=0 2024-05-09T08:32:23.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.726+0000 7f236cff9640 10 client.26162 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:32:23.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.727+0000 7f236cff9640 1 -- 172.21.15.28:0/4179335319 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560df4bcca70 con 0x560df4c72fa0 2024-05-09T08:32:23.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.727+0000 7f236cff9640 10 client.26162 handle_command_reply: tid=0 2024-05-09T08:32:23.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 client.26162 shutdown 2024-05-09T08:32:23.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 -- 172.21.15.28:0/4179335319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5a400 msgr2=0x560df4c67a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:23.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5a400 0x560df4c67a90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f237402aa00 tx=0x7f23740bcaf0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 -- 172.21.15.28:0/4179335319 shutdown_connections 2024-05-09T08:32:23.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560df4c72fa0 0x560df4c75420 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 0x560df4c62ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5ad20 0x560df4c67fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 --2- 172.21.15.28:0/4179335319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5a400 0x560df4c67a90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 -- 172.21.15.28:0/4179335319 >> 172.21.15.28:0/4179335319 conn(0x560df4c56b10 msgr2=0x560df4c5c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f236cff9640 0 client.26162 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:32:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 -- 172.21.15.28:0/4179335319 shutdown_connections 2024-05-09T08:32:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 1 -- 172.21.15.28:0/4179335319 wait complete. 2024-05-09T08:32:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.728+0000 7f238e438740 20 client.26162 trim_cache size 0 max 16384 2024-05-09T08:32:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.940+0000 7f238e438740 1 Processor -- start 2024-05-09T08:32:23.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.951+0000 7f238e438740 1 -- start start 2024-05-09T08:32:23.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.951+0000 7f238e438740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4ca0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.952+0000 7f238e438740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4ca06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.952+0000 7f238e438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 0x560df4c617b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.952+0000 7f23894c5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4ca0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.952+0000 7f23894c5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4ca0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:36608/0 (socket says 172.21.15.28:36608) 2024-05-09T08:32:23.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.952+0000 7f23894c5640 1 -- 172.21.15.28:0/2687910124 learned_addr learned my addr 172.21.15.28:0/2687910124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:23.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.952+0000 7f238e438740 1 -- 172.21.15.28:0/2687910124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23680100a0 con 0x560df4c767f0 2024-05-09T08:32:23.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.952+0000 7f238e438740 1 -- 172.21.15.28:0/2687910124 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2368011d90 con 0x560df4c5ad20 2024-05-09T08:32:23.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.952+0000 7f238e438740 1 -- 172.21.15.28:0/2687910124 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2368010830 con 0x560df4c5a400 2024-05-09T08:32:23.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.959+0000 7f23894c5640 1 -- 172.21.15.28:0/2687910124 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 msgr2=0x560df4ca06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.959+0000 7f23894c5640 1 --2- 172.21.15.28:0/2687910124 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4ca06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.959+0000 7f23894c5640 1 -- 172.21.15.28:0/2687910124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 msgr2=0x560df4c617b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.959+0000 7f23894c5640 1 --2- 172.21.15.28:0/2687910124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 0x560df4c617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.959+0000 7f23894c5640 1 -- 172.21.15.28:0/2687910124 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df4c61430 con 0x560df4c5a400 2024-05-09T08:32:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.960+0000 7f23894c5640 1 --2- 172.21.15.28:0/2687910124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4ca0160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2374005dc0 tx=0x7f23740bcc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1086f039d3f2439 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.968+0000 7f236effd640 1 -- 172.21.15.28:0/2687910124 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374049070 con 0x560df4c5a400 2024-05-09T08:32:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.968+0000 7f236effd640 1 -- 172.21.15.28:0/2687910124 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23740bcd20 con 0x560df4c5a400 2024-05-09T08:32:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.968+0000 7f236effd640 1 -- 172.21.15.28:0/2687910124 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23740ee1d0 con 0x560df4c5a400 2024-05-09T08:32:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.971+0000 7f238e438740 1 -- 172.21.15.28:0/2687910124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 msgr2=0x560df4ca0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.971+0000 7f238e438740 1 --2- 172.21.15.28:0/2687910124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4ca0160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2374005dc0 tx=0x7f23740bcc20 comp rx=0 tx=0).stop 2024-05-09T08:32:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.975+0000 7f238e438740 1 -- 172.21.15.28:0/2687910124 shutdown_connections 2024-05-09T08:32:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.975+0000 7f238e438740 1 --2- 172.21.15.28:0/2687910124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 0x560df4c617b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.975+0000 7f238e438740 1 --2- 172.21.15.28:0/2687910124 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4ca06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.975+0000 7f238e438740 1 --2- 172.21.15.28:0/2687910124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4ca0160 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.975+0000 7f238e438740 1 -- 172.21.15.28:0/2687910124 >> 172.21.15.28:0/2687910124 conn(0x560df4c56b10 msgr2=0x560df4c75cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.986+0000 7f238e438740 1 -- 172.21.15.28:0/2687910124 shutdown_connections 2024-05-09T08:32:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.986+0000 7f238e438740 1 -- 172.21.15.28:0/2687910124 wait complete. 2024-05-09T08:32:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.986+0000 7f238e438740 1 -- start start 2024-05-09T08:32:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.987+0000 7f238e438740 1 Processor -- start 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.987+0000 7f238e438740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4c6f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.987+0000 7f238e438740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4c693c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.987+0000 7f238e438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 0x560df4c69900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.987+0000 7f238e438740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23680116c0 con 0x560df4c767f0 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.987+0000 7f238e438740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2368010fc0 con 0x560df4c5ad20 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.987+0000 7f238e438740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f236800e080 con 0x560df4c5a400 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.987+0000 7f238e438740 10 client.0 _ll_register_callbacks cb 0x7f2374009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.988+0000 7f2388cc4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4c693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.988+0000 7f2388cc4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4c693c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:49092/0 (socket says 172.21.15.28:49092) 2024-05-09T08:32:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.988+0000 7f2388cc4640 1 -- 172.21.15.28:0/2824459824 learned_addr learned my addr 172.21.15.28:0/2824459824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.988+0000 7f23894c5640 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4c6f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.989+0000 7f2388cc4640 1 -- 172.21.15.28:0/2824459824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 msgr2=0x560df4c6f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.989+0000 7f2388cc4640 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4c6f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.989+0000 7f2388cc4640 1 -- 172.21.15.28:0/2824459824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 msgr2=0x560df4c69900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.989+0000 7f2388cc4640 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 0x560df4c69900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.989+0000 7f2388cc4640 1 -- 172.21.15.28:0/2824459824 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df4c69fc0 con 0x560df4c5ad20 2024-05-09T08:32:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:23.989+0000 7f2388cc4640 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4c693c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f238000e850 tx=0x7f23800097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=703624d6a48224d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:24.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.016+0000 7f236cff9640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:32:24.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.016+0000 7f236cff9640 1 -- 172.21.15.28:0/2824459824 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238002c060 con 0x560df4c5ad20 2024-05-09T08:32:24.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.016+0000 7f236cff9640 1 -- 172.21.15.28:0/2824459824 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9495+0+0 (secure 0 0 0) 0x7f2380027430 con 0x560df4c5ad20 2024-05-09T08:32:24.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.018+0000 7f238e438740 1 -- 172.21.15.28:0/2824459824 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23740bcd20 con 0x560df4c5ad20 2024-05-09T08:32:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.023+0000 7f236cff9640 1 -- 172.21.15.28:0/2824459824 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f2380004810 con 0x560df4c5ad20 2024-05-09T08:32:24.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.027+0000 7f238e438740 10 client.26165 fetch_fsmap learned FSMap version 54 2024-05-09T08:32:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.027+0000 7f238e438740 1 -- 172.21.15.28:0/2824459824 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x560df4cfd8c0 con 0x560df4c5ad20 2024-05-09T08:32:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.028+0000 7f236cff9640 1 -- 172.21.15.28:0/2824459824 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23740bcd20 con 0x560df4c5ad20 2024-05-09T08:32:24.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.030+0000 7f236cff9640 1 -- 172.21.15.28:0/2824459824 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 54) ==== 8414+0+0 (secure 0 0 0) 0x7f23800051b0 con 0x560df4c5ad20 2024-05-09T08:32:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.030+0000 7f238e438740 10 client.26165 fetch_fsmap finished waiting for FSMap version 54 2024-05-09T08:32:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.030+0000 7f238e438740 10 client.26165 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:32:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.030+0000 7f238e438740 20 client.26165 populate_metadata read hostname 'smithi028' 2024-05-09T08:32:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.030+0000 7f238e438740 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560df4cfe020 0x560df4d00440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.030+0000 7f238e438740 4 client.26165 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:32:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.030+0000 7f238e438740 1 -- 172.21.15.28:0/2824459824 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560df4bc6dc0 con 0x560df4cfe020 2024-05-09T08:32:24.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.031+0000 7f23894c5640 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560df4cfe020 0x560df4d00440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:24.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.031+0000 7f23894c5640 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560df4cfe020 0x560df4d00440 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8772098d81dc81e1 server_cookie=7b1aa0f0360f5d8a in_seq=0 out_seq=0 2024-05-09T08:32:24.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:24.031+0000 7f236cff9640 10 client.26165 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:32:24.600 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:24 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-05-09T08:32:24.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:24 smithi165 systemd[1]: Stopped Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:32:24.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:24 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service: Consumed 3.083s CPU time. 2024-05-09T08:32:24.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:24 smithi165 systemd[1]: Starting Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:32:24.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:24 smithi165 podman[69098]: 2024-05-09 08:32:24.2508362 +0000 UTC m=+0.019842567 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:32:25.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:25 smithi165 ceph-mon[34540]: pgmap v190: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.8 MiB/s wr, 496 op/s 2024-05-09T08:32:25.601 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:25 smithi165 podman[69098]: 2024-05-09 08:32:25.457283898 +0000 UTC m=+1.226290261 container create b378486507542c3c301b47ecc43ce3b094681f92c2cacf02d2d6cf03009ebcec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:32:25.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:25 smithi184 ceph-mon[34509]: pgmap v190: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.8 MiB/s wr, 496 op/s 2024-05-09T08:32:25.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:25 smithi028 ceph-mon[30119]: pgmap v190: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.8 MiB/s wr, 496 op/s 2024-05-09T08:32:26.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:26 smithi165 ceph-mon[34540]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:32:26.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:26 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:32:26.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:26 smithi184 ceph-mon[34509]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:32:26.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:26 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:32:26.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:26 smithi028 ceph-mon[30119]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-05-09T08:32:26.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:26 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-09T08:32:27.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:27 smithi165 ceph-mon[34540]: pgmap v191: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 10 MiB/s wr, 611 op/s 2024-05-09T08:32:27.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:27 smithi165 podman[69098]: 2024-05-09 08:32:27.518241478 +0000 UTC m=+3.287247834 container init b378486507542c3c301b47ecc43ce3b094681f92c2cacf02d2d6cf03009ebcec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T08:32:27.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:27 smithi165 podman[69098]: 2024-05-09 08:32:27.522479212 +0000 UTC m=+3.291485575 container start b378486507542c3c301b47ecc43ce3b094681f92c2cacf02d2d6cf03009ebcec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2) 2024-05-09T08:32:27.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:27 smithi165 ceph-mds[69116]: -- 172.21.15.165:0/1044330501 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5636c8359a40 con 0x5636c832c800 2024-05-09T08:32:27.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:27 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[69112]: starting mds.e at 2024-05-09T08:32:27.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:27 smithi165 bash[69098]: b378486507542c3c301b47ecc43ce3b094681f92c2cacf02d2d6cf03009ebcec 2024-05-09T08:32:27.851 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:32:27 smithi165 systemd[1]: Started Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:32:27.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:27 smithi184 ceph-mon[34509]: pgmap v191: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 10 MiB/s wr, 611 op/s 2024-05-09T08:32:27.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:27 smithi028 ceph-mon[30119]: pgmap v191: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 10 MiB/s wr, 611 op/s 2024-05-09T08:32:28.715 INFO:tasks.workunit.client.0.smithi028.stdout: 590 1188541 80 807068 137 302801 491 2024-05-09T08:32:29.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:29 smithi165 ceph-mon[34540]: pgmap v192: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 12 MiB/s wr, 745 op/s 2024-05-09T08:32:29.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:29 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6834/2426983483,v1:172.21.15.165:6835/2426983483] up:boot 2024-05-09T08:32:29.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:29 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:29.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:29 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:32:29.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:29 smithi184 ceph-mon[34509]: pgmap v192: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 12 MiB/s wr, 745 op/s 2024-05-09T08:32:29.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:29 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6834/2426983483,v1:172.21.15.165:6835/2426983483] up:boot 2024-05-09T08:32:29.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:29 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:29.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:29 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:32:29.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:29 smithi028 ceph-mon[30119]: pgmap v192: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 12 MiB/s wr, 745 op/s 2024-05-09T08:32:29.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:29 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6834/2426983483,v1:172.21.15.165:6835/2426983483] up:boot 2024-05-09T08:32:29.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:29 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:29.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:29 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:32:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.773+0000 7f236cff9640 1 -- 172.21.15.28:0/2824459824 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560df4bc6dc0 con 0x560df4cfe020 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.773+0000 7f236cff9640 10 client.26165 handle_command_reply: tid=0 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.773+0000 7f238e438740 1 client.26165 shutdown 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.773+0000 7f238e438740 1 -- 172.21.15.28:0/2824459824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 msgr2=0x560df4c693c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.773+0000 7f238e438740 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4c693c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f238000e850 tx=0x7f23800097b0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f238e438740 1 -- 172.21.15.28:0/2824459824 shutdown_connections 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f238e438740 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560df4cfe020 0x560df4d00440 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f238e438740 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4c767f0 0x560df4c69900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f238e438740 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560df4c5ad20 0x560df4c693c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f238e438740 1 --2- 172.21.15.28:0/2824459824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560df4c5a400 0x560df4c6f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f238e438740 1 -- 172.21.15.28:0/2824459824 >> 172.21.15.28:0/2824459824 conn(0x560df4c56b10 msgr2=0x560df4c61640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f236cff9640 0 client.26165 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f238e438740 1 -- 172.21.15.28:0/2824459824 shutdown_connections 2024-05-09T08:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f238e438740 1 -- 172.21.15.28:0/2824459824 wait complete. 2024-05-09T08:32:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.774+0000 7f238e438740 20 client.26165 trim_cache size 0 max 16384 2024-05-09T08:32:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:32:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:32:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:32:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:32:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 -- 172.21.15.28:0/1741846938 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f23700a2f30 msgr2=0x7f23700a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f23700a2f30 0x7f23700a53f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f237c00d850 tx=0x7f237c00f040 comp rx=0 tx=0).stop 2024-05-09T08:32:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 -- 172.21.15.28:0/1741846938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2384108aa0 msgr2=0x7f238407e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2384108aa0 0x7f238407e630 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2374006010 tx=0x7f23740055e0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 -- 172.21.15.28:0/1741846938 shutdown_connections 2024-05-09T08:32:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f23700a2f30 0x7f23700a53f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f238407f0b0 0x7f23841bb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f238410cff0 0x7f238407eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 --2- 172.21.15.28:0/1741846938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2384108aa0 0x7f238407e630 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:30.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 -- 172.21.15.28:0/1741846938 >> 172.21.15.28:0/1741846938 conn(0x7f238406c7e0 msgr2=0x7f23840711b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:30.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 -- 172.21.15.28:0/1741846938 shutdown_connections 2024-05-09T08:32:30.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:30.775+0000 7f238b729640 1 -- 172.21.15.28:0/1741846938 wait complete. 2024-05-09T08:32:31.182 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:32:30 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[91682]: ignoring --setuser ceph since I am not root 2024-05-09T08:32:31.182 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:32:30 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[91682]: ignoring --setgroup ceph since I am not root 2024-05-09T08:32:31.182 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:32:30 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[91682]: starting mds.d at 2024-05-09T08:32:31.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:31 smithi165 ceph-mon[34540]: pgmap v193: 129 pgs: 129 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 13 MiB/s wr, 999 op/s 2024-05-09T08:32:31.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:31 smithi165 ceph-mon[34540]: daemon mds.d restarted 2024-05-09T08:32:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:31 smithi184 ceph-mon[34509]: pgmap v193: 129 pgs: 129 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 13 MiB/s wr, 999 op/s 2024-05-09T08:32:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:31 smithi184 ceph-mon[34509]: daemon mds.d restarted 2024-05-09T08:32:31.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:31 smithi028 ceph-mon[30119]: pgmap v193: 129 pgs: 129 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 13 MiB/s wr, 999 op/s 2024-05-09T08:32:31.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:31 smithi028 ceph-mon[30119]: daemon mds.d restarted 2024-05-09T08:32:32.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:32 smithi165 ceph-mon[34540]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T08:32:32.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:32 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.28:6836/2330663776,v1:172.21.15.28:6837/2330663776] up:boot 2024-05-09T08:32:32.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:32 smithi165 ceph-mon[34540]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:32:32.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:32 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:32:32.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:32 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:32.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:32 smithi184 ceph-mon[34509]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T08:32:32.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:32 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.28:6836/2330663776,v1:172.21.15.28:6837/2330663776] up:boot 2024-05-09T08:32:32.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:32 smithi184 ceph-mon[34509]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:32:32.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:32 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:32:32.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:32 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:32.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:32 smithi028 ceph-mon[30119]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T08:32:32.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:32 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.28:6836/2330663776,v1:172.21.15.28:6837/2330663776] up:boot 2024-05-09T08:32:32.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:32 smithi028 ceph-mon[30119]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:32:32.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:32 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:32:32.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:32 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:33.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:33 smithi165 ceph-mon[34540]: pgmap v195: 129 pgs: 129 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 1.15k op/s 2024-05-09T08:32:33.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:33 smithi184 ceph-mon[34509]: pgmap v195: 129 pgs: 129 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 1.15k op/s 2024-05-09T08:32:33.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:33 smithi028 ceph-mon[30119]: pgmap v195: 129 pgs: 129 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 12 MiB/s wr, 1.15k op/s 2024-05-09T08:32:35.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:35 smithi165 ceph-mon[34540]: pgmap v196: 129 pgs: 129 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 11 MiB/s wr, 1.17k op/s 2024-05-09T08:32:35.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:35 smithi184 ceph-mon[34509]: pgmap v196: 129 pgs: 129 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 11 MiB/s wr, 1.17k op/s 2024-05-09T08:32:35.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:35 smithi028 ceph-mon[30119]: pgmap v196: 129 pgs: 129 active+clean; 2.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 11 MiB/s wr, 1.17k op/s 2024-05-09T08:32:37.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:37 smithi165 ceph-mon[34540]: pgmap v197: 129 pgs: 129 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.9 MiB/s wr, 1.08k op/s 2024-05-09T08:32:37.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:37 smithi184 ceph-mon[34509]: pgmap v197: 129 pgs: 129 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.9 MiB/s wr, 1.08k op/s 2024-05-09T08:32:37.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:37 smithi028 ceph-mon[30119]: pgmap v197: 129 pgs: 129 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.9 MiB/s wr, 1.08k op/s 2024-05-09T08:32:38.746 INFO:tasks.workunit.client.0.smithi028.stdout: 610 1445573 1294 974083 985 398176 2246 2024-05-09T08:32:39.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:39 smithi165 ceph-mon[34540]: pgmap v198: 129 pgs: 129 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 7.1 MiB/s wr, 857 op/s 2024-05-09T08:32:39.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:39 smithi028 ceph-mon[30119]: pgmap v198: 129 pgs: 129 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 7.1 MiB/s wr, 857 op/s 2024-05-09T08:32:40.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:39 smithi184 ceph-mon[34509]: pgmap v198: 129 pgs: 129 active+clean; 2.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 7.1 MiB/s wr, 857 op/s 2024-05-09T08:32:41.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:41 smithi028 ceph-mon[30119]: pgmap v199: 129 pgs: 129 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.0 MiB/s wr, 681 op/s 2024-05-09T08:32:42.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:41 smithi165 ceph-mon[34540]: pgmap v199: 129 pgs: 129 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.0 MiB/s wr, 681 op/s 2024-05-09T08:32:42.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:41 smithi184 ceph-mon[34509]: pgmap v199: 129 pgs: 129 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.0 MiB/s wr, 681 op/s 2024-05-09T08:32:43.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:43 smithi028 ceph-mon[30119]: pgmap v200: 129 pgs: 129 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 9.2 MiB/s wr, 663 op/s 2024-05-09T08:32:44.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:43 smithi165 ceph-mon[34540]: pgmap v200: 129 pgs: 129 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 9.2 MiB/s wr, 663 op/s 2024-05-09T08:32:44.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:43 smithi184 ceph-mon[34509]: pgmap v200: 129 pgs: 129 active+clean; 2.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 9.2 MiB/s wr, 663 op/s 2024-05-09T08:32:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.619+0000 7f0f916c0640 1 -- 172.21.15.28:0/2278066501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c10d960 msgr2=0x7f0f8c10ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.619+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2278066501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c10d960 0x7f0f8c10ddc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009ca0 tx=0x7f0f8402a900 comp rx=0 tx=0).stop 2024-05-09T08:32:45.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.619+0000 7f0f916c0640 1 -- 172.21.15.28:0/2278066501 shutdown_connections 2024-05-09T08:32:45.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.619+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2278066501 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f8c10e300 0x7f0f8c079a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.619+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2278066501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c10d960 0x7f0f8c10ddc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.619+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2278066501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8c10ccd0 0x7f0f8c10d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.619+0000 7f0f916c0640 1 -- 172.21.15.28:0/2278066501 >> 172.21.15.28:0/2278066501 conn(0x7f0f8c06c770 msgr2=0x7f0f8c06cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.626+0000 7f0f916c0640 1 -- 172.21.15.28:0/2278066501 shutdown_connections 2024-05-09T08:32:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.626+0000 7f0f916c0640 1 -- 172.21.15.28:0/2278066501 wait complete. 2024-05-09T08:32:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.627+0000 7f0f916c0640 1 Processor -- start 2024-05-09T08:32:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.627+0000 7f0f916c0640 1 -- start start 2024-05-09T08:32:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f916c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8c10ccd0 0x7f0f8c079910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f916c0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f8c10e300 0x7f0f8c073980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f916c0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c073ec0 0x7f0f8c074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f916c0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8c07c920 con 0x7f0f8c10ccd0 2024-05-09T08:32:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f916c0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8c07c7a0 con 0x7f0f8c073ec0 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f916c0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8c07caa0 con 0x7f0f8c10e300 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f90ebf640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c073ec0 0x7f0f8c074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f90ebf640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c073ec0 0x7f0f8c074350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33220/0 (socket says 172.21.15.28:33220) 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f90ebf640 1 -- 172.21.15.28:0/2487126316 learned_addr learned my addr 172.21.15.28:0/2487126316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f8c10e300 0x7f0f8c073980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f90ebf640 1 -- 172.21.15.28:0/2487126316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f8c10e300 msgr2=0x7f0f8c073980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f90ebf640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f8c10e300 0x7f0f8c073980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f90ebf640 1 -- 172.21.15.28:0/2487126316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8c10ccd0 msgr2=0x7f0f8c079910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f90ebf640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8c10ccd0 0x7f0f8c079910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.628+0000 7f0f90ebf640 1 -- 172.21.15.28:0/2487126316 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8c136d30 con 0x7f0f8c073ec0 2024-05-09T08:32:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.629+0000 7f0f90ebf640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c073ec0 0x7f0f8c074350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000a850 tx=0x7f0f8000ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.640+0000 7f0f897fa640 1 -- 172.21.15.28:0/2487126316 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f800079b0 con 0x7f0f8c073ec0 2024-05-09T08:32:45.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.640+0000 7f0f916c0640 1 -- 172.21.15.28:0/2487126316 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f8c137020 con 0x7f0f8c073ec0 2024-05-09T08:32:45.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.640+0000 7f0f916c0640 1 -- 172.21.15.28:0/2487126316 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f8c137540 con 0x7f0f8c073ec0 2024-05-09T08:32:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.647+0000 7f0f95632740 1 Processor -- start 2024-05-09T08:32:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.648+0000 7f0f95632740 1 -- start start 2024-05-09T08:32:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.648+0000 7f0f95632740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcb99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.648+0000 7f0f95632740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.648+0000 7f0f8b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcb99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.648+0000 7f0f8b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcb99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:47720/0 (socket says 172.21.15.28:47720) 2024-05-09T08:32:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.648+0000 7f0f8b7fe640 1 -- 172.21.15.28:0/3164004627 learned_addr learned my addr 172.21.15.28:0/3164004627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.648+0000 7f0f95632740 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f95632740 1 -- 172.21.15.28:0/3164004627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd8bc1e340 con 0x55dd8bcb9f20 2024-05-09T08:32:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f95632740 1 -- 172.21.15.28:0/3164004627 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55dd8bbc6290 con 0x55dd8bcc47e0 2024-05-09T08:32:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f95632740 1 -- 172.21.15.28:0/3164004627 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dd8bc1a300 con 0x55dd8bcb9600 2024-05-09T08:32:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f8b7fe640 1 -- 172.21.15.28:0/3164004627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc47e0 msgr2=0x55dd8bcc6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f8b7fe640 1 -- 172.21.15.28:0/3164004627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 msgr2=0x55dd8bcc42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f8b7fe640 1 -- 172.21.15.28:0/3164004627 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd8bbea8e0 con 0x55dd8bcb9600 2024-05-09T08:32:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.649+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.650+0000 7f0f897fa640 1 -- 172.21.15.28:0/2487126316 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f80005500 con 0x7f0f8c073ec0 2024-05-09T08:32:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.650+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcb99e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0f84005fe0 tx=0x7f0f84005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eaf9d6dbc532ced server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.650+0000 7f0f897fa640 1 -- 172.21.15.28:0/2487126316 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80004e00 con 0x7f0f8c073ec0 2024-05-09T08:32:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.651+0000 7f0f897fa640 1 -- 172.21.15.28:0/2487126316 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0f80014410 con 0x7f0f8c073ec0 2024-05-09T08:32:45.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.655+0000 7f0f897fa640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0f740a2f80 0x7f0f740a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.653+0000 7f0f727fc640 1 -- 172.21.15.28:0/3164004627 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f84044020 con 0x55dd8bcb9600 2024-05-09T08:32:45.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.658+0000 7f0f897fa640 1 -- 172.21.15.28:0/2487126316 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9495+0+0 (secure 0 0 0) 0x7f0f800cfbe0 con 0x7f0f8c073ec0 2024-05-09T08:32:45.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.658+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0f740a2f80 0x7f0f740a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.659+0000 7f0f727fc640 1 -- 172.21.15.28:0/3164004627 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f84036070 con 0x55dd8bcb9600 2024-05-09T08:32:45.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.661+0000 7f0f95632740 1 -- 172.21.15.28:0/3164004627 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 msgr2=0x55dd8bcb99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.661+0000 7f0f95632740 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcb99e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0f84005fe0 tx=0x7f0f84005f30 comp rx=0 tx=0).stop 2024-05-09T08:32:45.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.667+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0f740a2f80 0x7f0f740a5440 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55dd8bcbaba0 tx=0x7f0f7c009530 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.672+0000 7f0f95632740 1 -- 172.21.15.28:0/3164004627 shutdown_connections 2024-05-09T08:32:45.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.672+0000 7f0f95632740 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.672+0000 7f0f95632740 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.672+0000 7f0f95632740 1 --2- 172.21.15.28:0/3164004627 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcb99e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.672+0000 7f0f95632740 1 -- 172.21.15.28:0/3164004627 >> 172.21.15.28:0/3164004627 conn(0x55dd8bcb5b10 msgr2=0x55dd8bcb5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.675+0000 7f0f95632740 1 -- 172.21.15.28:0/3164004627 shutdown_connections 2024-05-09T08:32:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.675+0000 7f0f95632740 1 -- 172.21.15.28:0/3164004627 wait complete. 2024-05-09T08:32:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.675+0000 7f0f95632740 1 -- start start 2024-05-09T08:32:45.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.675+0000 7f0f95632740 1 Processor -- start 2024-05-09T08:32:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.676+0000 7f0f95632740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.676+0000 7f0f95632740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.676+0000 7f0f95632740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.676+0000 7f0f95632740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd8bcc73b0 con 0x55dd8bcb9f20 2024-05-09T08:32:45.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.676+0000 7f0f95632740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55dd8bbd9790 con 0x55dd8bcb9600 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.676+0000 7f0f95632740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dd8bcc7530 con 0x55dd8bcc47e0 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.676+0000 7f0f95632740 10 client.0 _ll_register_callbacks cb 0x55dd8bc14a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.676+0000 7f0f90ebf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.676+0000 7f0f8bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.677+0000 7f0f8bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33232/0 (socket says 172.21.15.28:33232) 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.677+0000 7f0f8bfff640 1 -- 172.21.15.28:0/117806035 learned_addr learned my addr 172.21.15.28:0/117806035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.677+0000 7f0f8bfff640 1 -- 172.21.15.28:0/117806035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 msgr2=0x55dd8bcc2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.677+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.677+0000 7f0f8bfff640 1 -- 172.21.15.28:0/117806035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 msgr2=0x55dd8bcc1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.677+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.677+0000 7f0f8bfff640 1 -- 172.21.15.28:0/117806035 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd8b932210 con 0x55dd8bcb9600 2024-05-09T08:32:45.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.677+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc16e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c009830 tx=0x7f0f7c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a943951302d48b4e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.696+0000 7f0f6fff7640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:32:45.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.696+0000 7f0f6fff7640 1 -- 172.21.15.28:0/117806035 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c023070 con 0x55dd8bcb9600 2024-05-09T08:32:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.697+0000 7f0f95632740 1 -- 172.21.15.28:0/117806035 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f84036070 con 0x55dd8bcb9600 2024-05-09T08:32:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.697+0000 7f0f6fff7640 1 -- 172.21.15.28:0/117806035 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9495+0+0 (secure 0 0 0) 0x7f0f7c004460 con 0x55dd8bcb9600 2024-05-09T08:32:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.700+0000 7f0f6fff7640 1 -- 172.21.15.28:0/117806035 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f0f7c02f050 con 0x55dd8bcb9600 2024-05-09T08:32:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.700+0000 7f0f95632740 10 client.26189 fetch_fsmap learned FSMap version 59 2024-05-09T08:32:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.700+0000 7f0f95632740 1 -- 172.21.15.28:0/117806035 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55dd8b9fe620 con 0x55dd8bcb9600 2024-05-09T08:32:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.701+0000 7f0f6fff7640 1 -- 172.21.15.28:0/117806035 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f84036070 con 0x55dd8bcb9600 2024-05-09T08:32:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.701+0000 7f0f6fff7640 1 -- 172.21.15.28:0/117806035 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 59) ==== 8418+0+0 (secure 0 0 0) 0x7f0f7c01f420 con 0x55dd8bcb9600 2024-05-09T08:32:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.704+0000 7f0f95632740 10 client.26189 fetch_fsmap finished waiting for FSMap version 59 2024-05-09T08:32:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.704+0000 7f0f95632740 10 client.26189 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:32:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.704+0000 7f0f95632740 20 client.26189 populate_metadata read hostname 'smithi028' 2024-05-09T08:32:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.704+0000 7f0f95632740 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dd8bcd4af0 0x55dd8bcd6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.704+0000 7f0f95632740 4 client.26189 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:32:45.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.704+0000 7f0f95632740 1 -- 172.21.15.28:0/117806035 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd8bc2ba70 con 0x55dd8bcd4af0 2024-05-09T08:32:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.710+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dd8bcd4af0 0x55dd8bcd6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.712+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dd8bcd4af0 0x55dd8bcd6f90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb37a670f845b313 server_cookie=28ac6694fa549317 in_seq=0 out_seq=0 2024-05-09T08:32:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.721+0000 7f0f6fff7640 10 client.26189 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:32:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.721+0000 7f0f6fff7640 1 -- 172.21.15.28:0/117806035 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd8bc2ba70 con 0x55dd8bcd4af0 2024-05-09T08:32:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.721+0000 7f0f6fff7640 10 client.26189 handle_command_reply: tid=0 2024-05-09T08:32:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.721+0000 7f0f95632740 1 client.26189 shutdown 2024-05-09T08:32:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.722+0000 7f0f95632740 1 -- 172.21.15.28:0/117806035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9600 msgr2=0x55dd8bcc16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.722+0000 7f0f95632740 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc16e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c009830 tx=0x7f0f7c0046c0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.726+0000 7f0f95632740 1 -- 172.21.15.28:0/117806035 shutdown_connections 2024-05-09T08:32:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.726+0000 7f0f95632740 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dd8bcd4af0 0x55dd8bcd6f90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.726+0000 7f0f95632740 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.726+0000 7f0f95632740 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.726+0000 7f0f95632740 1 --2- 172.21.15.28:0/117806035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc16e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.726+0000 7f0f95632740 1 -- 172.21.15.28:0/117806035 >> 172.21.15.28:0/117806035 conn(0x55dd8bcb5b10 msgr2=0x55dd8bcbb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.726+0000 7f0f6fff7640 0 client.26189 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:32:45.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.726+0000 7f0f95632740 1 -- 172.21.15.28:0/117806035 shutdown_connections 2024-05-09T08:32:45.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.730+0000 7f0f95632740 1 -- 172.21.15.28:0/117806035 wait complete. 2024-05-09T08:32:45.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.734+0000 7f0f95632740 20 client.26189 trim_cache size 0 max 16384 2024-05-09T08:32:45.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:45 smithi028 ceph-mon[30119]: pgmap v201: 129 pgs: 129 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.0 MiB/s wr, 471 op/s 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.936+0000 7f0f95632740 1 Processor -- start 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.936+0000 7f0f95632740 1 -- start start 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f95632740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f95632740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcd0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f95632740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcbbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f95632740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f78010140 con 0x55dd8bcc47e0 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f95632740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0f78011e60 con 0x55dd8bcb9f20 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f95632740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0f780108d0 con 0x55dd8bcb9600 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f90ebf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcbbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f8bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f8bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:47736/0 (socket says 172.21.15.28:47736) 2024-05-09T08:32:45.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f8bfff640 1 -- 172.21.15.28:0/3171062709 learned_addr learned my addr 172.21.15.28:0/3171062709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:45.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/3171062709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcd0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.937+0000 7f0f8bfff640 1 -- 172.21.15.28:0/3171062709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9f20 msgr2=0x55dd8bcd0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:45.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.938+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/3171062709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcd0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.938+0000 7f0f8bfff640 1 -- 172.21.15.28:0/3171062709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcc47e0 msgr2=0x55dd8bcbbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:45.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.938+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/3171062709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcbbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.938+0000 7f0f8bfff640 1 -- 172.21.15.28:0/3171062709 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd8bcb9d50 con 0x55dd8bcb9600 2024-05-09T08:32:45.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.938+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/3171062709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc3e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c01f7f0 tx=0x7f0f7c011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=315be90e4cd47f9f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.946+0000 7f0f71ffb640 1 -- 172.21.15.28:0/3171062709 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c037070 con 0x55dd8bcb9600 2024-05-09T08:32:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.946+0000 7f0f71ffb640 1 -- 172.21.15.28:0/3171062709 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7c005c50 con 0x55dd8bcb9600 2024-05-09T08:32:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.946+0000 7f0f71ffb640 1 -- 172.21.15.28:0/3171062709 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c023050 con 0x55dd8bcb9600 2024-05-09T08:32:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.948+0000 7f0f95632740 1 -- 172.21.15.28:0/3171062709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 msgr2=0x55dd8bcc3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.948+0000 7f0f95632740 1 --2- 172.21.15.28:0/3171062709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc3e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c01f7f0 tx=0x7f0f7c011a50 comp rx=0 tx=0).stop 2024-05-09T08:32:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.949+0000 7f0f95632740 1 -- 172.21.15.28:0/3171062709 shutdown_connections 2024-05-09T08:32:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.949+0000 7f0f95632740 1 --2- 172.21.15.28:0/3171062709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcbbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.949+0000 7f0f95632740 1 --2- 172.21.15.28:0/3171062709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcd0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.949+0000 7f0f95632740 1 --2- 172.21.15.28:0/3171062709 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcb9600 0x55dd8bcc3e10 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c01f7f0 tx=0x7f0f7c011a50 comp rx=0 tx=0).stop 2024-05-09T08:32:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.949+0000 7f0f95632740 1 -- 172.21.15.28:0/3171062709 >> 172.21.15.28:0/3171062709 conn(0x55dd8bcb5b10 msgr2=0x55dd8bcbffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.954+0000 7f0f95632740 1 -- 172.21.15.28:0/3171062709 shutdown_connections 2024-05-09T08:32:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.954+0000 7f0f95632740 1 -- 172.21.15.28:0/3171062709 wait complete. 2024-05-09T08:32:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.954+0000 7f0f95632740 1 -- start start 2024-05-09T08:32:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.955+0000 7f0f95632740 1 Processor -- start 2024-05-09T08:32:45.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.955+0000 7f0f95632740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f95632740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f95632740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc5280 0x55dd8bcc5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f8b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f8b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:47752/0 (socket says 172.21.15.28:47752) 2024-05-09T08:32:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f8b7fe640 1 -- 172.21.15.28:0/812922788 learned_addr learned my addr 172.21.15.28:0/812922788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:32:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f90ebf640 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc5280 0x55dd8bcc5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f78011790 con 0x55dd8bcb9f20 2024-05-09T08:32:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0f78011060 con 0x55dd8bcc5280 2024-05-09T08:32:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7800e120 con 0x55dd8bcc47e0 2024-05-09T08:32:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.956+0000 7f0f95632740 10 client.0 _ll_register_callbacks cb 0x7f0f7c009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.957+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.957+0000 7f0f8bfff640 1 -- 172.21.15.28:0/812922788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 msgr2=0x55dd8bcc4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.957+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.958+0000 7f0f8bfff640 1 -- 172.21.15.28:0/812922788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc5280 msgr2=0x55dd8bcc5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.958+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc5280 0x55dd8bcc5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.958+0000 7f0f8bfff640 1 -- 172.21.15.28:0/812922788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd8bccb930 con 0x55dd8bcb9f20 2024-05-09T08:32:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.958+0000 7f0f8bfff640 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc6a40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x55dd8bcc2760 tx=0x7f0f7c01f650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477106c17d224850 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.958+0000 7f0f6fff7640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:32:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.959+0000 7f0f6fff7640 1 -- 172.21.15.28:0/812922788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c041070 con 0x55dd8bcb9f20 2024-05-09T08:32:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.959+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.959+0000 7f0f90ebf640 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc5280 0x55dd8bcc5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.959+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f7c005c50 con 0x55dd8bcb9f20 2024-05-09T08:32:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.959+0000 7f0f6fff7640 1 -- 172.21.15.28:0/812922788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9495+0+0 (secure 0 0 0) 0x7f0f7c01fbe0 con 0x55dd8bcb9f20 2024-05-09T08:32:45.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.959+0000 7f0f6fff7640 1 -- 172.21.15.28:0/812922788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f0f7c049050 con 0x55dd8bcb9f20 2024-05-09T08:32:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.961+0000 7f0f95632740 10 client.16704 fetch_fsmap learned FSMap version 59 2024-05-09T08:32:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.961+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55dd8bcccc40 con 0x55dd8bcb9f20 2024-05-09T08:32:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.971+0000 7f0f6fff7640 1 -- 172.21.15.28:0/812922788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7c005c50 con 0x55dd8bcb9f20 2024-05-09T08:32:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.971+0000 7f0f6fff7640 1 -- 172.21.15.28:0/812922788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 59) ==== 8418+0+0 (secure 0 0 0) 0x7f0f7c044030 con 0x55dd8bcb9f20 2024-05-09T08:32:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.974+0000 7f0f95632740 10 client.16704 fetch_fsmap finished waiting for FSMap version 59 2024-05-09T08:32:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.974+0000 7f0f95632740 10 client.16704 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:32:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.974+0000 7f0f95632740 20 client.16704 populate_metadata read hostname 'smithi028' 2024-05-09T08:32:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.974+0000 7f0f95632740 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dd8bd3b110 0x55dd8bd3d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.974+0000 7f0f95632740 4 client.16704 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:32:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.974+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd8bc25dc0 con 0x55dd8bd3b110 2024-05-09T08:32:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.974+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dd8bd3b110 0x55dd8bd3d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.976+0000 7f0f8b7fe640 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dd8bd3b110 0x55dd8bd3d500 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecc01bd13dc7c265 server_cookie=305677da3856787 in_seq=0 out_seq=0 2024-05-09T08:32:45.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:45.976+0000 7f0f6fff7640 10 client.16704 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:32:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:45 smithi165 ceph-mon[34540]: pgmap v201: 129 pgs: 129 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.0 MiB/s wr, 471 op/s 2024-05-09T08:32:46.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:45 smithi184 ceph-mon[34509]: pgmap v201: 129 pgs: 129 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.0 MiB/s wr, 471 op/s 2024-05-09T08:32:47.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:46 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:47.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:46 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:47.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:46 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:47 smithi165 ceph-mon[34540]: pgmap v202: 129 pgs: 129 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.6 MiB/s wr, 530 op/s 2024-05-09T08:32:48.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:47 smithi184 ceph-mon[34509]: pgmap v202: 129 pgs: 129 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.6 MiB/s wr, 530 op/s 2024-05-09T08:32:48.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:47 smithi028 ceph-mon[30119]: pgmap v202: 129 pgs: 129 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.6 MiB/s wr, 530 op/s 2024-05-09T08:32:48.718 INFO:tasks.workunit.client.0.smithi028.stdout: 616 1449547 410 973064 485 406510 1593 2024-05-09T08:32:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:49 smithi165 ceph-mon[34540]: pgmap v203: 129 pgs: 129 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 9.9 MiB/s wr, 594 op/s 2024-05-09T08:32:50.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:49 smithi184 ceph-mon[34509]: pgmap v203: 129 pgs: 129 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 9.9 MiB/s wr, 594 op/s 2024-05-09T08:32:50.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:49 smithi028 ceph-mon[30119]: pgmap v203: 129 pgs: 129 active+clean; 2.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 9.9 MiB/s wr, 594 op/s 2024-05-09T08:32:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.809+0000 7f0f6fff7640 1 -- 172.21.15.28:0/812922788 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd8bc25dc0 con 0x55dd8bd3b110 2024-05-09T08:32:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.809+0000 7f0f6fff7640 10 client.16704 handle_command_reply: tid=0 2024-05-09T08:32:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.809+0000 7f0f95632740 1 client.16704 shutdown 2024-05-09T08:32:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.809+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 msgr2=0x55dd8bcc6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.809+0000 7f0f95632740 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc6a40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x55dd8bcc2760 tx=0x7f0f7c01f650 comp rx=0 tx=0).stop 2024-05-09T08:32:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 shutdown_connections 2024-05-09T08:32:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f95632740 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dd8bd3b110 0x55dd8bd3d500 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f95632740 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dd8bcc5280 0x55dd8bcc5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f95632740 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dd8bcc47e0 0x55dd8bcc4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f95632740 1 --2- 172.21.15.28:0/812922788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd8bcb9f20 0x55dd8bcc6a40 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 >> 172.21.15.28:0/812922788 conn(0x55dd8bcb5b10 msgr2=0x55dd8bcb4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f6fff7640 0 client.16704 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:32:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 shutdown_connections 2024-05-09T08:32:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f95632740 1 -- 172.21.15.28:0/812922788 wait complete. 2024-05-09T08:32:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.810+0000 7f0f95632740 20 client.16704 trim_cache size 0 max 16384 2024-05-09T08:32:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:32:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:32:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:32:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:32:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 -- 172.21.15.28:0/2487126316 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0f740a2f80 msgr2=0x7f0f740a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0f740a2f80 0x7f0f740a5440 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55dd8bcbaba0 tx=0x7f0f7c009530 comp rx=0 tx=0).stop 2024-05-09T08:32:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 -- 172.21.15.28:0/2487126316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c073ec0 msgr2=0x7f0f8c074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c073ec0 0x7f0f8c074350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000a850 tx=0x7f0f8000ad10 comp rx=0 tx=0).stop 2024-05-09T08:32:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 -- 172.21.15.28:0/2487126316 shutdown_connections 2024-05-09T08:32:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f8c073ec0 0x7f0f8c074350 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0f740a2f80 0x7f0f740a5440 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f8c10e300 0x7f0f8c073980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 --2- 172.21.15.28:0/2487126316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8c10ccd0 0x7f0f8c079910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:50.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 -- 172.21.15.28:0/2487126316 >> 172.21.15.28:0/2487126316 conn(0x7f0f8c06c770 msgr2=0x7f0f8c072910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:50.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 -- 172.21.15.28:0/2487126316 shutdown_connections 2024-05-09T08:32:50.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:32:50.811+0000 7f0f916c0640 1 -- 172.21.15.28:0/2487126316 wait complete. 2024-05-09T08:32:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:51 smithi165 ceph-mon[34540]: pgmap v204: 129 pgs: 129 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.3 MiB/s wr, 811 op/s 2024-05-09T08:32:52.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:51 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:52.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:51 smithi184 ceph-mon[34509]: pgmap v204: 129 pgs: 129 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.3 MiB/s wr, 811 op/s 2024-05-09T08:32:52.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:51 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:52.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:51 smithi028 ceph-mon[30119]: pgmap v204: 129 pgs: 129 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.3 MiB/s wr, 811 op/s 2024-05-09T08:32:52.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:51 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:32:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:53 smithi165 ceph-mon[34540]: pgmap v205: 129 pgs: 129 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.5 MiB/s wr, 826 op/s 2024-05-09T08:32:54.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:53 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:32:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:53 smithi184 ceph-mon[34509]: pgmap v205: 129 pgs: 129 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.5 MiB/s wr, 826 op/s 2024-05-09T08:32:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:53 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:32:54.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:53 smithi028 ceph-mon[30119]: pgmap v205: 129 pgs: 129 active+clean; 2.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.5 MiB/s wr, 826 op/s 2024-05-09T08:32:54.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:53 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:32:55.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:54 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:32:55.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:54 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:32:55.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:54 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:32:55.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:54 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:32:55.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:54 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:32:55.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:54 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:32:55.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:54 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:32:55.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:54 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:32:55.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:54 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:32:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:55 smithi165 ceph-mon[34540]: pgmap v206: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.4 MiB/s wr, 1.01k op/s 2024-05-09T08:32:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:55 smithi184 ceph-mon[34509]: pgmap v206: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.4 MiB/s wr, 1.01k op/s 2024-05-09T08:32:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:55 smithi028 ceph-mon[30119]: pgmap v206: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.4 MiB/s wr, 1.01k op/s 2024-05-09T08:32:57.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:56 smithi165 ceph-mon[34540]: pgmap v207: 129 pgs: 129 active+clean; 2.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.6 MiB/s wr, 1.05k op/s 2024-05-09T08:32:57.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:56 smithi184 ceph-mon[34509]: pgmap v207: 129 pgs: 129 active+clean; 2.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.6 MiB/s wr, 1.05k op/s 2024-05-09T08:32:57.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:56 smithi028 ceph-mon[30119]: pgmap v207: 129 pgs: 129 active+clean; 2.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.6 MiB/s wr, 1.05k op/s 2024-05-09T08:32:59.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:32:59 smithi165 ceph-mon[34540]: pgmap v208: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.8 MiB/s wr, 950 op/s 2024-05-09T08:32:59.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:32:59 smithi184 ceph-mon[34509]: pgmap v208: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.8 MiB/s wr, 950 op/s 2024-05-09T08:32:59.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:32:59 smithi028 ceph-mon[30119]: pgmap v208: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.8 MiB/s wr, 950 op/s 2024-05-09T08:33:01.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:01 smithi165 ceph-mon[34540]: pgmap v209: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 877 op/s 2024-05-09T08:33:01.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:01 smithi184 ceph-mon[34509]: pgmap v209: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 877 op/s 2024-05-09T08:33:01.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:01 smithi028 ceph-mon[30119]: pgmap v209: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 877 op/s 2024-05-09T08:33:03.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:03 smithi165 ceph-mon[34540]: pgmap v210: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 653 op/s 2024-05-09T08:33:03.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:03 smithi184 ceph-mon[34509]: pgmap v210: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 653 op/s 2024-05-09T08:33:03.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:03 smithi028 ceph-mon[30119]: pgmap v210: 129 pgs: 129 active+clean; 2.3 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 653 op/s 2024-05-09T08:33:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.919+0000 7f2bff4e1640 1 -- 172.21.15.28:0/3032507168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 msgr2=0x7f2bf810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.919+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/3032507168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 0x7f2bf810d5e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0009cd0 tx=0x7f2bf002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.919+0000 7f2bff4e1640 1 -- 172.21.15.28:0/3032507168 shutdown_connections 2024-05-09T08:33:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.919+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/3032507168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 0x7f2bf810d5e0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.919+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/3032507168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf8100760 0x7f2bf8100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.919+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/3032507168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf8106720 0x7f2bf8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.919+0000 7f2bff4e1640 1 -- 172.21.15.28:0/3032507168 >> 172.21.15.28:0/3032507168 conn(0x7f2bf80778b0 msgr2=0x7f2bf8077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.919+0000 7f2bff4e1640 1 -- 172.21.15.28:0/3032507168 shutdown_connections 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.919+0000 7f2bff4e1640 1 -- 172.21.15.28:0/3032507168 wait complete. 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bff4e1640 1 Processor -- start 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bff4e1640 1 -- start start 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bff4e1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf8100760 0x7f2bf8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bff4e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 0x7f2bf8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bff4e1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf8106720 0x7f2bf819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bff4e1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf8110b30 con 0x7f2bf8101120 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bff4e1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf81109b0 con 0x7f2bf8106720 2024-05-09T08:33:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bff4e1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf8110cb0 con 0x7f2bf8100760 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bfca7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 0x7f2bf8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bfca7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 0x7f2bf8198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57278/0 (socket says 172.21.15.28:57278) 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.920+0000 7f2bfca7c640 1 -- 172.21.15.28:0/958100906 learned_addr learned my addr 172.21.15.28:0/958100906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bfd27d640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf8100760 0x7f2bf8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf8106720 0x7f2bf819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bfca7c640 1 -- 172.21.15.28:0/958100906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf8100760 msgr2=0x7f2bf8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf8100760 0x7f2bf8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bfca7c640 1 -- 172.21.15.28:0/958100906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf8106720 msgr2=0x7f2bf819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf8106720 0x7f2bf819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bfca7c640 1 -- 172.21.15.28:0/958100906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf819d010 con 0x7f2bf8101120 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 0x7f2bf8198560 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2be800c920 tx=0x7f2be800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bee7fc640 1 -- 172.21.15.28:0/958100906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be8007c70 con 0x7f2bf8101120 2024-05-09T08:33:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bee7fc640 1 -- 172.21.15.28:0/958100906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be8007e10 con 0x7f2bf8101120 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bff4e1640 1 -- 172.21.15.28:0/958100906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf819d300 con 0x7f2bf8101120 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bee7fc640 1 -- 172.21.15.28:0/958100906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be8018890 con 0x7f2bf8101120 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.921+0000 7f2bff4e1640 1 -- 172.21.15.28:0/958100906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf81a4b90 con 0x7f2bf8101120 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.923+0000 7f2bee7fc640 1 -- 172.21.15.28:0/958100906 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2be8018a70 con 0x7f2bf8101120 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.924+0000 7f2c021f5740 1 Processor -- start 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.924+0000 7f2c021f5740 1 -- start start 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.924+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 0x56399fda9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.924+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdb8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.924+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdbae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.924+0000 7f2c021f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56399fd0e340 con 0x56399fdb8a40 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.924+0000 7f2c021f5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56399fcb6290 con 0x56399fda9c50 2024-05-09T08:33:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.924+0000 7f2c021f5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56399fd0a300 con 0x56399fda9330 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfca7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdbae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bee7fc640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2be40a2da0 0x7f2be40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfca7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdbae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57282/0 (socket says 172.21.15.28:57282) 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfca7c640 1 -- 172.21.15.28:0/4132411396 learned_addr learned my addr 172.21.15.28:0/4132411396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/4132411396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 0x56399fda9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfd27d640 1 --2- 172.21.15.28:0/4132411396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdb8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2be40a2da0 0x7f2be40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bee7fc640 1 -- 172.21.15.28:0/958100906 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9495+0+0 (secure 0 0 0) 0x7f2be80cfa80 con 0x7f2bf8101120 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfca7c640 1 -- 172.21.15.28:0/4132411396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 msgr2=0x56399fda9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/4132411396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 0x56399fda9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfca7c640 1 -- 172.21.15.28:0/4132411396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 msgr2=0x56399fdb8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/4132411396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdb8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfca7c640 1 -- 172.21.15.28:0/4132411396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56399fcda8e0 con 0x56399fdb8a40 2024-05-09T08:33:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/4132411396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdbae30 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2be8097fd0 tx=0x7f2be8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b012497ac0e5f05a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2be37fe640 1 -- 172.21.15.28:0/4132411396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be809bb70 con 0x56399fdb8a40 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2be40a2da0 0x7f2be40a5260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0009430 tx=0x7f2bf003f040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2be37fe640 1 -- 172.21.15.28:0/4132411396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be809ff20 con 0x56399fdb8a40 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.925+0000 7f2be37fe640 1 -- 172.21.15.28:0/4132411396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be8098200 con 0x56399fdb8a40 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 -- 172.21.15.28:0/4132411396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 msgr2=0x56399fdbae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 --2- 172.21.15.28:0/4132411396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdbae30 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2be8097fd0 tx=0x7f2be8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 -- 172.21.15.28:0/4132411396 shutdown_connections 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 --2- 172.21.15.28:0/4132411396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdbae30 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 --2- 172.21.15.28:0/4132411396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdb8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 --2- 172.21.15.28:0/4132411396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 0x56399fda9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 -- 172.21.15.28:0/4132411396 >> 172.21.15.28:0/4132411396 conn(0x56399fda5960 msgr2=0x56399fda5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 -- 172.21.15.28:0/4132411396 shutdown_connections 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 -- 172.21.15.28:0/4132411396 wait complete. 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.926+0000 7f2c021f5740 1 -- start start 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2c021f5740 1 Processor -- start 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 0x56399fdb5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdb5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2c021f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56399fdbb610 con 0x56399fdb8a40 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2c021f5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x56399fcc9790 con 0x56399fda9c50 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2c021f5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x56399fdbb790 con 0x56399fda9330 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2c021f5740 10 client.0 _ll_register_callbacks cb 0x56399fd04a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2bfda7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2bfda7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57296/0 (socket says 172.21.15.28:57296) 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2bfda7e640 1 -- 172.21.15.28:0/2158969475 learned_addr learned my addr 172.21.15.28:0/2158969475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdb5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.927+0000 7f2bfd27d640 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 0x56399fdb5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2bfda7e640 1 -- 172.21.15.28:0/2158969475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 msgr2=0x56399fdb5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 0x56399fdb5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2bfda7e640 1 -- 172.21.15.28:0/2158969475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 msgr2=0x56399fdb5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdb5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2bfda7e640 1 -- 172.21.15.28:0/2158969475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56399fa22210 con 0x56399fdb8a40 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb6410 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0031860 tx=0x7f2bf00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c9f032224c703c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2be0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2158969475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf004f070 con 0x56399fdb8a40 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2158969475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9495+0+0 (secure 0 0 0) 0x7f2bf0004240 con 0x56399fdb8a40 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2c021f5740 1 -- 172.21.15.28:0/2158969475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2be809ff20 con 0x56399fdb8a40 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2158969475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f2bf005b050 con 0x56399fdb8a40 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2c021f5740 10 client.16716 fetch_fsmap learned FSMap version 60 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2c021f5740 1 -- 172.21.15.28:0/2158969475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x56399faee620 con 0x56399fdb8a40 2024-05-09T08:33:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.928+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2158969475 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be809ff20 con 0x56399fdb8a40 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2158969475 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 60) ==== 8422+0+0 (secure 0 0 0) 0x7f2bf0030070 con 0x56399fdb8a40 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2c021f5740 10 client.16716 fetch_fsmap finished waiting for FSMap version 60 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2c021f5740 10 client.16716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2c021f5740 20 client.16716 populate_metadata read hostname 'smithi028' 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56399fdc8810 0x56399fdcac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2c021f5740 4 client.16716 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2c021f5740 1 -- 172.21.15.28:0/2158969475 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56399fd1ba70 con 0x56399fdc8810 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2bfd27d640 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56399fdc8810 0x56399fdcac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2bfd27d640 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56399fdc8810 0x56399fdcac20 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee203d14d4972d77 server_cookie=c59198b89d3fe9fa in_seq=0 out_seq=0 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.929+0000 7f2be0ff9640 10 client.16716 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.930+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2158969475 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56399fd1ba70 con 0x56399fdc8810 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.930+0000 7f2be0ff9640 10 client.16716 handle_command_reply: tid=0 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.930+0000 7f2c021f5740 1 client.16716 shutdown 2024-05-09T08:33:03.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.930+0000 7f2c021f5740 1 -- 172.21.15.28:0/2158969475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 msgr2=0x56399fdb6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.930+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb6410 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0031860 tx=0x7f2bf00095c0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2c021f5740 1 -- 172.21.15.28:0/2158969475 shutdown_connections 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56399fdc8810 0x56399fdcac20 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb6410 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdb5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2158969475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9330 0x56399fdb5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2be0ff9640 0 client.16716 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2c021f5740 1 -- 172.21.15.28:0/2158969475 >> 172.21.15.28:0/2158969475 conn(0x56399fda5960 msgr2=0x56399fdab7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2c021f5740 1 -- 172.21.15.28:0/2158969475 shutdown_connections 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2c021f5740 1 -- 172.21.15.28:0/2158969475 wait complete. 2024-05-09T08:33:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.931+0000 7f2c021f5740 20 client.16716 trim_cache size 0 max 16384 2024-05-09T08:33:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.946+0000 7f2c021f5740 1 Processor -- start 2024-05-09T08:33:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.946+0000 7f2c021f5740 1 -- start start 2024-05-09T08:33:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.946+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9330 0x56399fdb80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.946+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9c50 0x56399fdc4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.946+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.946+0000 7f2c021f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf81acb00 con 0x56399fdb8a40 2024-05-09T08:33:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.946+0000 7f2c021f5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf81ae850 con 0x56399fda9330 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.946+0000 7f2c021f5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf81ad290 con 0x56399fda9c50 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.946+0000 7f2bfda7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfda7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57302/0 (socket says 172.21.15.28:57302) 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfda7e640 1 -- 172.21.15.28:0/928865929 learned_addr learned my addr 172.21.15.28:0/928865929 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfd27d640 1 --2- 172.21.15.28:0/928865929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9330 0x56399fdb80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/928865929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9c50 0x56399fdc4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfda7e640 1 -- 172.21.15.28:0/928865929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9c50 msgr2=0x56399fdc4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/928865929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9c50 0x56399fdc4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfda7e640 1 -- 172.21.15.28:0/928865929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9330 msgr2=0x56399fdb80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/928865929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9330 0x56399fdb80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfda7e640 1 -- 172.21.15.28:0/928865929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56399fda3f00 con 0x56399fdb8a40 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/928865929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb4970 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0009980 tx=0x7f2bf0005f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4237ee91c37a680 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2be2ffd640 1 -- 172.21.15.28:0/928865929 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0064070 con 0x56399fdb8a40 2024-05-09T08:33:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2be2ffd640 1 -- 172.21.15.28:0/928865929 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bf0036040 con 0x56399fdb8a40 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.947+0000 7f2be2ffd640 1 -- 172.21.15.28:0/928865929 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf004f040 con 0x56399fdb8a40 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 -- 172.21.15.28:0/928865929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 msgr2=0x56399fdb4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 --2- 172.21.15.28:0/928865929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb4970 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0009980 tx=0x7f2bf0005f50 comp rx=0 tx=0).stop 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 -- 172.21.15.28:0/928865929 shutdown_connections 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 --2- 172.21.15.28:0/928865929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fdb8a40 0x56399fdb4970 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 --2- 172.21.15.28:0/928865929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fda9c50 0x56399fdc4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 --2- 172.21.15.28:0/928865929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9330 0x56399fdb80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 -- 172.21.15.28:0/928865929 >> 172.21.15.28:0/928865929 conn(0x56399fda5960 msgr2=0x56399fdb4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 -- 172.21.15.28:0/928865929 shutdown_connections 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 -- 172.21.15.28:0/928865929 wait complete. 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 -- start start 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.948+0000 7f2c021f5740 1 Processor -- start 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fda9330 0x56399fdc0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdc0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2c021f5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fdb8a40 0x56399fdb1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2c021f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf81ae180 con 0x56399fda9330 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2c021f5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf81ada20 con 0x56399fda9c50 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2c021f5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf81aaae0 con 0x56399fdb8a40 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2c021f5740 10 client.0 _ll_register_callbacks cb 0x7f2bf00317b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfd27d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fda9330 0x56399fdc0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfd27d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fda9330 0x56399fdc0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57304/0 (socket says 172.21.15.28:57304) 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfd27d640 1 -- 172.21.15.28:0/2724719962 learned_addr learned my addr 172.21.15.28:0/2724719962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfda7e640 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fdb8a40 0x56399fdb1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdc0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfd27d640 1 -- 172.21.15.28:0/2724719962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fdb8a40 msgr2=0x56399fdb1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfd27d640 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fdb8a40 0x56399fdb1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfd27d640 1 -- 172.21.15.28:0/2724719962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 msgr2=0x56399fdc0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfd27d640 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdc0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfd27d640 1 -- 172.21.15.28:0/2724719962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56399fdb22e0 con 0x56399fda9330 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdc0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2bfd27d640 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fda9330 0x56399fdc0a50 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400bae0 tx=0x7f2bf4013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5fb1b91b9b4c820 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2be0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2724719962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4004230 con 0x56399fda9330 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2724719962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9495+0+0 (secure 0 0 0) 0x7f2bf40043d0 con 0x56399fda9330 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2c021f5740 1 -- 172.21.15.28:0/2724719962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bf0036040 con 0x56399fda9330 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.949+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2724719962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f2bf402b050 con 0x56399fda9330 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2c021f5740 10 client.16722 fetch_fsmap learned FSMap version 60 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2c021f5740 1 -- 172.21.15.28:0/2724719962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x56399fe2e750 con 0x56399fda9330 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2724719962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bf0036040 con 0x56399fda9330 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2724719962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 60) ==== 8422+0+0 (secure 0 0 0) 0x7f2bf4003ca0 con 0x56399fda9330 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2c021f5740 10 client.16722 fetch_fsmap finished waiting for FSMap version 60 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2c021f5740 10 client.16722 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:33:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2c021f5740 20 client.16722 populate_metadata read hostname 'smithi028' 2024-05-09T08:33:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56399fe2ef90 0x56399fe313b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2c021f5740 4 client.16722 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:33:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2c021f5740 1 -- 172.21.15.28:0/2724719962 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56399fd15dc0 con 0x56399fe2ef90 2024-05-09T08:33:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.950+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56399fe2ef90 0x56399fe313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.951+0000 7f2be0ff9640 10 client.16722 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:03.951+0000 7f2bfca7c640 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56399fe2ef90 0x56399fe313b0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e466ff12ba4fef44 server_cookie=9fc8fe1e8a25c168 in_seq=0 out_seq=0 2024-05-09T08:33:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.096+0000 7f2be0ff9640 1 -- 172.21.15.28:0/2724719962 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56399fd15dc0 con 0x56399fe2ef90 2024-05-09T08:33:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.096+0000 7f2be0ff9640 10 client.16722 handle_command_reply: tid=0 2024-05-09T08:33:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.096+0000 7f2c021f5740 1 client.16722 shutdown 2024-05-09T08:33:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 -- 172.21.15.28:0/2724719962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fda9330 msgr2=0x56399fdc0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fda9330 0x56399fdc0a50 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400bae0 tx=0x7f2bf4013830 comp rx=0 tx=0).stop 2024-05-09T08:33:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 -- 172.21.15.28:0/2724719962 shutdown_connections 2024-05-09T08:33:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x56399fe2ef90 0x56399fe313b0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x56399fdb8a40 0x56399fdb1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2be0ff9640 0 client.16722 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56399fda9c50 0x56399fdc0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 --2- 172.21.15.28:0/2724719962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56399fda9330 0x56399fdc0a50 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 -- 172.21.15.28:0/2724719962 >> 172.21.15.28:0/2724719962 conn(0x56399fda5960 msgr2=0x56399fdb4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 -- 172.21.15.28:0/2724719962 shutdown_connections 2024-05-09T08:33:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 1 -- 172.21.15.28:0/2724719962 wait complete. 2024-05-09T08:33:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.097+0000 7f2c021f5740 20 client.16722 trim_cache size 0 max 16384 2024-05-09T08:33:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:33:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:33:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:33:04.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:33:04.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 -- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2be40a2da0 msgr2=0x7f2be40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:04.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2be40a2da0 0x7f2be40a5260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0009430 tx=0x7f2bf003f040 comp rx=0 tx=0).stop 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 -- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 msgr2=0x7f2bf8198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 0x7f2bf8198560 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2be800c920 tx=0x7f2be800cde0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 -- 172.21.15.28:0/958100906 shutdown_connections 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf8106720 0x7f2bf819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf8101120 0x7f2bf8198560 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2be40a2da0 0x7f2be40a5260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 --2- 172.21.15.28:0/958100906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bf8100760 0x7f2bf8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 -- 172.21.15.28:0/958100906 >> 172.21.15.28:0/958100906 conn(0x7f2bf80778b0 msgr2=0x7f2bf8069160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 -- 172.21.15.28:0/958100906 shutdown_connections 2024-05-09T08:33:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:04.098+0000 7f2bff4e1640 1 -- 172.21.15.28:0/958100906 wait complete. 2024-05-09T08:33:05.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:05 smithi165 ceph-mon[34540]: pgmap v211: 129 pgs: 129 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.5 MiB/s wr, 617 op/s 2024-05-09T08:33:05.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:05 smithi184 ceph-mon[34509]: pgmap v211: 129 pgs: 129 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.5 MiB/s wr, 617 op/s 2024-05-09T08:33:05.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:05 smithi028 ceph-mon[30119]: pgmap v211: 129 pgs: 129 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.5 MiB/s wr, 617 op/s 2024-05-09T08:33:07.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:07 smithi165 ceph-mon[34540]: pgmap v212: 129 pgs: 129 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 398 op/s 2024-05-09T08:33:07.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:07 smithi165 ceph-mon[34540]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T08:33:07.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:07 smithi184 ceph-mon[34509]: pgmap v212: 129 pgs: 129 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 398 op/s 2024-05-09T08:33:07.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:07 smithi184 ceph-mon[34509]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T08:33:07.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:07 smithi028 ceph-mon[30119]: pgmap v212: 129 pgs: 129 active+clean; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 398 op/s 2024-05-09T08:33:07.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:07 smithi028 ceph-mon[30119]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T08:33:09.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:09 smithi165 ceph-mon[34540]: pgmap v214: 129 pgs: 10 active+clean+snaptrim, 88 active+clean, 31 active+clean+snaptrim_wait; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.0 MiB/s wr, 378 op/s 2024-05-09T08:33:09.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:09 smithi184 ceph-mon[34509]: pgmap v214: 129 pgs: 10 active+clean+snaptrim, 88 active+clean, 31 active+clean+snaptrim_wait; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.0 MiB/s wr, 378 op/s 2024-05-09T08:33:09.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:09 smithi028 ceph-mon[30119]: pgmap v214: 129 pgs: 10 active+clean+snaptrim, 88 active+clean, 31 active+clean+snaptrim_wait; 2.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.0 MiB/s wr, 378 op/s 2024-05-09T08:33:11.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:11 smithi165 ceph-mon[34540]: pgmap v215: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 321 op/s 2024-05-09T08:33:11.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:11 smithi184 ceph-mon[34509]: pgmap v215: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 321 op/s 2024-05-09T08:33:11.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:11 smithi028 ceph-mon[30119]: pgmap v215: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 321 op/s 2024-05-09T08:33:13.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:13 smithi165 ceph-mon[34540]: pgmap v216: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 271 op/s 2024-05-09T08:33:13.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:13 smithi184 ceph-mon[34509]: pgmap v216: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 271 op/s 2024-05-09T08:33:13.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:13 smithi028 ceph-mon[30119]: pgmap v216: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 271 op/s 2024-05-09T08:33:15.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:15 smithi165 ceph-mon[34540]: pgmap v217: 129 pgs: 18 active+clean+snaptrim, 69 active+clean, 42 active+clean+snaptrim_wait; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-05-09T08:33:15.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:15 smithi184 ceph-mon[34509]: pgmap v217: 129 pgs: 18 active+clean+snaptrim, 69 active+clean, 42 active+clean+snaptrim_wait; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-05-09T08:33:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:15 smithi028 ceph-mon[30119]: pgmap v217: 129 pgs: 18 active+clean+snaptrim, 69 active+clean, 42 active+clean+snaptrim_wait; 2.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-05-09T08:33:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.244+0000 7fa52159a640 1 -- 172.21.15.28:0/427257817 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51402c040 con 0x7fa524073bd0 2024-05-09T08:33:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.244+0000 7fa528d16640 1 -- 172.21.15.28:0/427257817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa524073bd0 msgr2=0x7fa524073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.244+0000 7fa528d16640 1 --2- 172.21.15.28:0/427257817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa524073bd0 0x7fa524073fb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa514009ca0 tx=0x7fa51402a950 comp rx=0 tx=0).stop 2024-05-09T08:33:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.246+0000 7fa528d16640 1 -- 172.21.15.28:0/427257817 shutdown_connections 2024-05-09T08:33:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.246+0000 7fa528d16640 1 --2- 172.21.15.28:0/427257817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa524112820 0x7fa524114c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.246+0000 7fa528d16640 1 --2- 172.21.15.28:0/427257817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5240744f0 0x7fa5241122e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.246+0000 7fa528d16640 1 --2- 172.21.15.28:0/427257817 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa524073bd0 0x7fa524073fb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.246+0000 7fa528d16640 1 -- 172.21.15.28:0/427257817 >> 172.21.15.28:0/427257817 conn(0x7fa5240fd2e0 msgr2=0x7fa5240ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.247+0000 7fa528d16640 1 -- 172.21.15.28:0/427257817 shutdown_connections 2024-05-09T08:33:17.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.247+0000 7fa528d16640 1 -- 172.21.15.28:0/427257817 wait complete. 2024-05-09T08:33:17.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.248+0000 7fa528d16640 1 Processor -- start 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.248+0000 7fa528d16640 1 -- start start 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa528d16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa524073bd0 0x7fa5240734a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa528d16640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5240744f0 0x7fa524070520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa528d16640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa524112820 0x7fa524070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa528d16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241060c0 con 0x7fa524073bd0 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa528d16640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa524105f40 con 0x7fa524112820 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa528d16640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa524106240 con 0x7fa5240744f0 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa521d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5240744f0 0x7fa524070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa521d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5240744f0 0x7fa524070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:55904/0 (socket says 172.21.15.28:55904) 2024-05-09T08:33:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa521d9b640 1 -- 172.21.15.28:0/3780236258 learned_addr learned my addr 172.21.15.28:0/3780236258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa521d9b640 1 -- 172.21.15.28:0/3780236258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa524112820 msgr2=0x7fa524070c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:33:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa521d9b640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa524112820 0x7fa524070c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa521d9b640 1 -- 172.21.15.28:0/3780236258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa524073bd0 msgr2=0x7fa5240734a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa521d9b640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa524073bd0 0x7fa5240734a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.249+0000 7fa521d9b640 1 -- 172.21.15.28:0/3780236258 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52410f8e0 con 0x7fa5240744f0 2024-05-09T08:33:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.250+0000 7fa521d9b640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5240744f0 0x7fa524070520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa51800b4a0 tx=0x7fa51800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.251+0000 7fa5137fe640 1 -- 172.21.15.28:0/3780236258 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518014070 con 0x7fa5240744f0 2024-05-09T08:33:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.251+0000 7fa5137fe640 1 -- 172.21.15.28:0/3780236258 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa518010af0 con 0x7fa5240744f0 2024-05-09T08:33:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.251+0000 7fa5137fe640 1 -- 172.21.15.28:0/3780236258 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518017cc0 con 0x7fa5240744f0 2024-05-09T08:33:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.252+0000 7fa528d16640 1 -- 172.21.15.28:0/3780236258 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa52410fbd0 con 0x7fa5240744f0 2024-05-09T08:33:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.252+0000 7fa528d16640 1 -- 172.21.15.28:0/3780236258 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5241100d0 con 0x7fa5240744f0 2024-05-09T08:33:17.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.255+0000 7fa5137fe640 1 -- 172.21.15.28:0/3780236258 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa518017070 con 0x7fa5240744f0 2024-05-09T08:33:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.256+0000 7fa5137fe640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa50c0a2ef0 0x7fa50c0a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.257+0000 7fa5137fe640 1 -- 172.21.15.28:0/3780236258 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9671+0+0 (secure 0 0 0) 0x7fa5180cec50 con 0x7fa5240744f0 2024-05-09T08:33:17.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.258+0000 7fa52ba1c740 1 Processor -- start 2024-05-09T08:33:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa52ba1c740 1 -- start start 2024-05-09T08:33:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150f5450 0x558e150f5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e151000f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e15102a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa52ba1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e1505a340 con 0x558e150f5450 2024-05-09T08:33:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa52ba1c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558e15002290 con 0x558e150f5d70 2024-05-09T08:33:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa52ba1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558e15056300 con 0x558e15100630 2024-05-09T08:33:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa52259c640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa50c0a2ef0 0x7fa50c0a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa521d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e151000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa521d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e151000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:39620/0 (socket says 172.21.15.28:39620) 2024-05-09T08:33:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa521d9b640 1 -- 172.21.15.28:0/84949392 learned_addr learned my addr 172.21.15.28:0/84949392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.260+0000 7fa522d9d640 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e15102a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.261+0000 7fa52259c640 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150f5450 0x558e150f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.261+0000 7fa52259c640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa50c0a2ef0 0x7fa50c0a53b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa514009bf0 tx=0x7fa514005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.261+0000 7fa52259c640 1 -- 172.21.15.28:0/84949392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 msgr2=0x558e15102a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.261+0000 7fa52259c640 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e15102a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.261+0000 7fa52259c640 1 -- 172.21.15.28:0/84949392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 msgr2=0x558e151000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.261+0000 7fa52259c640 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e151000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.261+0000 7fa52259c640 1 -- 172.21.15.28:0/84949392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e150268e0 con 0x558e150f5450 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.261+0000 7fa521d9b640 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e151000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.262+0000 7fa52259c640 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150f5450 0x558e150f5830 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa514009ca0 tx=0x7fa514047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68a43ef84bdfa29 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.262+0000 7fa510ff9640 1 -- 172.21.15.28:0/84949392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514034040 con 0x558e150f5450 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.262+0000 7fa510ff9640 1 -- 172.21.15.28:0/84949392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa51402fc00 con 0x558e150f5450 2024-05-09T08:33:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.262+0000 7fa510ff9640 1 -- 172.21.15.28:0/84949392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51403d7a0 con 0x558e150f5450 2024-05-09T08:33:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.263+0000 7fa52ba1c740 1 -- 172.21.15.28:0/84949392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150f5450 msgr2=0x558e150f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.263+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150f5450 0x558e150f5830 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa514009ca0 tx=0x7fa514047000 comp rx=0 tx=0).stop 2024-05-09T08:33:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.263+0000 7fa52ba1c740 1 -- 172.21.15.28:0/84949392 shutdown_connections 2024-05-09T08:33:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.263+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e15102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.263+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e151000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.263+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/84949392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150f5450 0x558e150f5830 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.263+0000 7fa52ba1c740 1 -- 172.21.15.28:0/84949392 >> 172.21.15.28:0/84949392 conn(0x558e150f1960 msgr2=0x558e150f1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.263+0000 7fa52ba1c740 1 -- 172.21.15.28:0/84949392 shutdown_connections 2024-05-09T08:33:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.264+0000 7fa52ba1c740 1 -- 172.21.15.28:0/84949392 wait complete. 2024-05-09T08:33:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.264+0000 7fa52ba1c740 1 -- start start 2024-05-09T08:33:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.264+0000 7fa52ba1c740 1 Processor -- start 2024-05-09T08:33:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.264+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150fdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e1510c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa52ba1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e15103200 con 0x558e150fe250 2024-05-09T08:33:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa52ba1c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558e15015790 con 0x558e150f5d70 2024-05-09T08:33:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa52ba1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558e15103380 con 0x558e15100630 2024-05-09T08:33:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa52ba1c740 10 client.0 _ll_register_callbacks cb 0x558e15050a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa52259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa52259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:39622/0 (socket says 172.21.15.28:39622) 2024-05-09T08:33:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa52259c640 1 -- 172.21.15.28:0/287671901 learned_addr learned my addr 172.21.15.28:0/287671901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.265+0000 7fa521d9b640 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150fdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa521d9b640 1 -- 172.21.15.28:0/287671901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 msgr2=0x558e150fd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa521d9b640 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa521d9b640 1 -- 172.21.15.28:0/287671901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 msgr2=0x558e1510c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa521d9b640 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e1510c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa521d9b640 1 -- 172.21.15.28:0/287671901 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e14d6e210 con 0x558e15100630 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa521d9b640 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150fdd10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa518097200 tx=0x7fa5180174b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f5a6ac6f4c86cfd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa509ffb640 1 -- 172.21.15.28:0/287671901 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51801e040 con 0x558e15100630 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa52ba1c740 1 -- 172.21.15.28:0/287671901 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa51402fc00 con 0x558e15100630 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.266+0000 7fa509ffb640 1 -- 172.21.15.28:0/287671901 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9671+0+0 (secure 0 0 0) 0x7fa5180cc140 con 0x558e15100630 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.267+0000 7fa509ffb640 1 -- 172.21.15.28:0/287671901 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7fa5180cc340 con 0x558e15100630 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.267+0000 7fa52ba1c740 10 client.26104 fetch_fsmap learned FSMap version 60 2024-05-09T08:33:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.267+0000 7fa52ba1c740 1 -- 172.21.15.28:0/287671901 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x558e14e3a620 con 0x558e15100630 2024-05-09T08:33:17.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.267+0000 7fa509ffb640 1 -- 172.21.15.28:0/287671901 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa51402fc00 con 0x558e15100630 2024-05-09T08:33:17.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.267+0000 7fa509ffb640 1 -- 172.21.15.28:0/287671901 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 60) ==== 8422+0+0 (secure 0 0 0) 0x7fa518096af0 con 0x558e15100630 2024-05-09T08:33:17.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.268+0000 7fa52ba1c740 10 client.26104 fetch_fsmap finished waiting for FSMap version 60 2024-05-09T08:33:17.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.268+0000 7fa52ba1c740 10 client.26104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:33:17.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.268+0000 7fa52ba1c740 20 client.26104 populate_metadata read hostname 'smithi028' 2024-05-09T08:33:17.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.268+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558e15110a20 0x558e15112ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.268+0000 7fa52ba1c740 4 client.26104 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.268+0000 7fa52ba1c740 1 -- 172.21.15.28:0/287671901 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e15067a70 con 0x558e15110a20 2024-05-09T08:33:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.269+0000 7fa52259c640 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558e15110a20 0x558e15112ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.269+0000 7fa52259c640 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558e15110a20 0x558e15112ec0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb8f768fa126c335 server_cookie=8f8420f519114493 in_seq=0 out_seq=0 2024-05-09T08:33:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.270+0000 7fa509ffb640 10 client.26104 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.270+0000 7fa509ffb640 1 -- 172.21.15.28:0/287671901 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558e15067a70 con 0x558e15110a20 2024-05-09T08:33:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.270+0000 7fa509ffb640 10 client.26104 handle_command_reply: tid=0 2024-05-09T08:33:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.270+0000 7fa52ba1c740 1 client.26104 shutdown 2024-05-09T08:33:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.271+0000 7fa52ba1c740 1 -- 172.21.15.28:0/287671901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 msgr2=0x558e150fdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.271+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150fdd10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa518097200 tx=0x7fa5180174b0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa52ba1c740 1 -- 172.21.15.28:0/287671901 shutdown_connections 2024-05-09T08:33:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558e15110a20 0x558e15112ec0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e1510c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150fdd10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/287671901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa52ba1c740 1 -- 172.21.15.28:0/287671901 >> 172.21.15.28:0/287671901 conn(0x558e150f1960 msgr2=0x558e150f76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa509ffb640 0 client.26104 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa52ba1c740 1 -- 172.21.15.28:0/287671901 shutdown_connections 2024-05-09T08:33:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa52ba1c740 1 -- 172.21.15.28:0/287671901 wait complete. 2024-05-09T08:33:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.272+0000 7fa52ba1c740 20 client.26104 trim_cache size 0 max 16384 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.300+0000 7fa52ba1c740 1 Processor -- start 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.300+0000 7fa52ba1c740 1 -- start start 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.300+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.300+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e150fd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150f0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:39624/0 (socket says 172.21.15.28:39624) 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52259c640 1 -- 172.21.15.28:0/1473082257 learned_addr learned my addr 172.21.15.28:0/1473082257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52ba1c740 1 -- 172.21.15.28:0/1473082257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241b57a0 con 0x558e150fe250 2024-05-09T08:33:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52ba1c740 1 -- 172.21.15.28:0/1473082257 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241b74f0 con 0x558e150f5d70 2024-05-09T08:33:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52ba1c740 1 -- 172.21.15.28:0/1473082257 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241b5f30 con 0x558e15100630 2024-05-09T08:33:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa522d9d640 1 --2- 172.21.15.28:0/1473082257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa521d9b640 1 --2- 172.21.15.28:0/1473082257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e150fd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52259c640 1 -- 172.21.15.28:0/1473082257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 msgr2=0x558e150f0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:17.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52259c640 1 --2- 172.21.15.28:0/1473082257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150f0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52259c640 1 -- 172.21.15.28:0/1473082257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 msgr2=0x558e150fd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:17.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52259c640 1 --2- 172.21.15.28:0/1473082257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e150fd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.301+0000 7fa52259c640 1 -- 172.21.15.28:0/1473082257 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e150f1790 con 0x558e150f5d70 2024-05-09T08:33:17.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.302+0000 7fa52259c640 1 --2- 172.21.15.28:0/1473082257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fc990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa5140049a0 tx=0x7fa51403d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d39818d8feb1a342 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:17.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.304+0000 7fa50bfff640 1 -- 172.21.15.28:0/1473082257 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51405e070 con 0x558e150f5d70 2024-05-09T08:33:17.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.304+0000 7fa50bfff640 1 -- 172.21.15.28:0/1473082257 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5140044e0 con 0x558e150f5d70 2024-05-09T08:33:17.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.304+0000 7fa50bfff640 1 -- 172.21.15.28:0/1473082257 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514034040 con 0x558e150f5d70 2024-05-09T08:33:17.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.305+0000 7fa52ba1c740 1 -- 172.21.15.28:0/1473082257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 msgr2=0x558e150fc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:17.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.305+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/1473082257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fc990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa5140049a0 tx=0x7fa51403d460 comp rx=0 tx=0).stop 2024-05-09T08:33:17.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.305+0000 7fa52ba1c740 1 -- 172.21.15.28:0/1473082257 shutdown_connections 2024-05-09T08:33:17.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.305+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/1473082257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150f0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.305+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/1473082257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e150fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.305+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/1473082257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e150fc990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.305+0000 7fa52ba1c740 1 -- 172.21.15.28:0/1473082257 >> 172.21.15.28:0/1473082257 conn(0x558e150f1960 msgr2=0x558e150f0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:17.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.305+0000 7fa52ba1c740 1 -- 172.21.15.28:0/1473082257 shutdown_connections 2024-05-09T08:33:17.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.305+0000 7fa52ba1c740 1 -- 172.21.15.28:0/1473082257 wait complete. 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.306+0000 7fa52ba1c740 1 -- start start 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.306+0000 7fa52ba1c740 1 Processor -- start 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.306+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e15108830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.306+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e150fa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.306+0000 7fa52ba1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150fa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.306+0000 7fa52ba1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241b6e20 con 0x558e150fe250 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.306+0000 7fa52ba1c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241b66c0 con 0x558e150f5d70 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.306+0000 7fa52ba1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241b37a0 con 0x558e15100630 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.306+0000 7fa52ba1c740 10 client.0 _ll_register_callbacks cb 0x7fa51809a750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa52259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e15108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa52259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e15108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:39626/0 (socket says 172.21.15.28:39626) 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa52259c640 1 -- 172.21.15.28:0/2477027418 learned_addr learned my addr 172.21.15.28:0/2477027418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa52259c640 1 -- 172.21.15.28:0/2477027418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 msgr2=0x558e150fa5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa52259c640 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150fa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa52259c640 1 -- 172.21.15.28:0/2477027418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 msgr2=0x558e150fa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa52259c640 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e150fa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa52259c640 1 -- 172.21.15.28:0/2477027418 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e150fac50 con 0x558e150f5d70 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa52259c640 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e15108830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa514000f80 tx=0x7fa514009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f8913a8c2b09272 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:33:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.307+0000 7fa509ffb640 1 -- 172.21.15.28:0/2477027418 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51405e070 con 0x558e150f5d70 2024-05-09T08:33:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.308+0000 7fa52ba1c740 1 -- 172.21.15.28:0/2477027418 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5140044e0 con 0x558e150f5d70 2024-05-09T08:33:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.308+0000 7fa509ffb640 1 -- 172.21.15.28:0/2477027418 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9671+0+0 (secure 0 0 0) 0x7fa514004000 con 0x558e150f5d70 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.308+0000 7fa509ffb640 1 -- 172.21.15.28:0/2477027418 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7fa514034040 con 0x558e150f5d70 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.308+0000 7fa52ba1c740 10 client.26225 fetch_fsmap learned FSMap version 60 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.308+0000 7fa52ba1c740 1 -- 172.21.15.28:0/2477027418 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x558e15176e80 con 0x558e150f5d70 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.308+0000 7fa509ffb640 1 -- 172.21.15.28:0/2477027418 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5140044e0 con 0x558e150f5d70 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.309+0000 7fa509ffb640 1 -- 172.21.15.28:0/2477027418 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 60) ==== 8422+0+0 (secure 0 0 0) 0x7fa514059650 con 0x558e150f5d70 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.309+0000 7fa52ba1c740 10 client.26225 fetch_fsmap finished waiting for FSMap version 60 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.309+0000 7fa52ba1c740 10 client.26225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.309+0000 7fa52ba1c740 20 client.26225 populate_metadata read hostname 'smithi028' 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.309+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558e15177100 0x558e15179550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.309+0000 7fa52ba1c740 4 client.26225 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.309+0000 7fa52ba1c740 1 -- 172.21.15.28:0/2477027418 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558e150fe0a0 con 0x558e15177100 2024-05-09T08:33:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.309+0000 7fa521d9b640 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558e15177100 0x558e15179550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.310+0000 7fa521d9b640 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558e15177100 0x558e15179550 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0250dcac0c74d7e server_cookie=b2b0d185a256a592 in_seq=0 out_seq=0 2024-05-09T08:33:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.311+0000 7fa509ffb640 10 client.26225 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa509ffb640 1 -- 172.21.15.28:0/2477027418 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558e150fe0a0 con 0x558e15177100 2024-05-09T08:33:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa509ffb640 10 client.26225 handle_command_reply: tid=0 2024-05-09T08:33:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 client.26225 shutdown 2024-05-09T08:33:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 -- 172.21.15.28:0/2477027418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 msgr2=0x558e15108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e15108830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa514000f80 tx=0x7fa514009b20 comp rx=0 tx=0).stop 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 -- 172.21.15.28:0/2477027418 shutdown_connections 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558e15177100 0x558e15179550 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558e15100630 0x558e150fa5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e150fe250 0x558e150fa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 --2- 172.21.15.28:0/2477027418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558e150f5d70 0x558e15108830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 -- 172.21.15.28:0/2477027418 >> 172.21.15.28:0/2477027418 conn(0x558e150f1960 msgr2=0x558e150f0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa509ffb640 0 client.26225 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 -- 172.21.15.28:0/2477027418 shutdown_connections 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.461+0000 7fa52ba1c740 1 -- 172.21.15.28:0/2477027418 wait complete. 2024-05-09T08:33:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.462+0000 7fa52ba1c740 20 client.26225 trim_cache size 0 max 16384 2024-05-09T08:33:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:33:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:33:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:33:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.462+0000 7fa528d16640 1 -- 172.21.15.28:0/3780236258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa50c0a2ef0 msgr2=0x7fa50c0a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.462+0000 7fa528d16640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa50c0a2ef0 0x7fa50c0a53b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa514009bf0 tx=0x7fa514005fd0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.462+0000 7fa528d16640 1 -- 172.21.15.28:0/3780236258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5240744f0 msgr2=0x7fa524070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.462+0000 7fa528d16640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5240744f0 0x7fa524070520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa51800b4a0 tx=0x7fa51800b960 comp rx=0 tx=0).stop 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.463+0000 7fa528d16640 1 -- 172.21.15.28:0/3780236258 shutdown_connections 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.463+0000 7fa528d16640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa524112820 0x7fa524070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.463+0000 7fa528d16640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa50c0a2ef0 0x7fa50c0a53b0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa514009bf0 tx=0x7fa514005fd0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.463+0000 7fa528d16640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5240744f0 0x7fa524070520 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.463+0000 7fa528d16640 1 --2- 172.21.15.28:0/3780236258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa524073bd0 0x7fa5240734a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.463+0000 7fa528d16640 1 -- 172.21.15.28:0/3780236258 >> 172.21.15.28:0/3780236258 conn(0x7fa5240fd2e0 msgr2=0x7fa5240fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.463+0000 7fa528d16640 1 -- 172.21.15.28:0/3780236258 shutdown_connections 2024-05-09T08:33:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:17.463+0000 7fa528d16640 1 -- 172.21.15.28:0/3780236258 wait complete. 2024-05-09T08:33:17.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:17 smithi165 ceph-mon[34540]: pgmap v218: 129 pgs: 14 active+clean+snaptrim, 78 active+clean, 37 active+clean+snaptrim_wait; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 345 KiB/s wr, 57 op/s 2024-05-09T08:33:17.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:17 smithi184 ceph-mon[34509]: pgmap v218: 129 pgs: 14 active+clean+snaptrim, 78 active+clean, 37 active+clean+snaptrim_wait; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 345 KiB/s wr, 57 op/s 2024-05-09T08:33:17.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:17 smithi028 ceph-mon[30119]: pgmap v218: 129 pgs: 14 active+clean+snaptrim, 78 active+clean, 37 active+clean+snaptrim_wait; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 345 KiB/s wr, 57 op/s 2024-05-09T08:33:19.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:19 smithi165 ceph-mon[34540]: pgmap v219: 129 pgs: 8 active+clean+snaptrim, 104 active+clean, 17 active+clean+snaptrim_wait; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 351 KiB/s wr, 55 op/s 2024-05-09T08:33:19.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:19 smithi184 ceph-mon[34509]: pgmap v219: 129 pgs: 8 active+clean+snaptrim, 104 active+clean, 17 active+clean+snaptrim_wait; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 351 KiB/s wr, 55 op/s 2024-05-09T08:33:19.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:19 smithi028 ceph-mon[30119]: pgmap v219: 129 pgs: 8 active+clean+snaptrim, 104 active+clean, 17 active+clean+snaptrim_wait; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 351 KiB/s wr, 55 op/s 2024-05-09T08:33:21.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:21 smithi165 ceph-mon[34540]: pgmap v220: 129 pgs: 4 active+clean+snaptrim, 120 active+clean, 5 active+clean+snaptrim_wait; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 363 KiB/s wr, 51 op/s 2024-05-09T08:33:21.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:21 smithi184 ceph-mon[34509]: pgmap v220: 129 pgs: 4 active+clean+snaptrim, 120 active+clean, 5 active+clean+snaptrim_wait; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 363 KiB/s wr, 51 op/s 2024-05-09T08:33:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:21 smithi028 ceph-mon[30119]: pgmap v220: 129 pgs: 4 active+clean+snaptrim, 120 active+clean, 5 active+clean+snaptrim_wait; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 363 KiB/s wr, 51 op/s 2024-05-09T08:33:23.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:23 smithi165 ceph-mon[34540]: pgmap v221: 129 pgs: 2 active+clean+snaptrim, 122 active+clean, 5 active+clean+snaptrim_wait; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 383 KiB/s wr, 42 op/s 2024-05-09T08:33:23.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:23 smithi184 ceph-mon[34509]: pgmap v221: 129 pgs: 2 active+clean+snaptrim, 122 active+clean, 5 active+clean+snaptrim_wait; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 383 KiB/s wr, 42 op/s 2024-05-09T08:33:23.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:23 smithi028 ceph-mon[30119]: pgmap v221: 129 pgs: 2 active+clean+snaptrim, 122 active+clean, 5 active+clean+snaptrim_wait; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 383 KiB/s wr, 42 op/s 2024-05-09T08:33:25.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:25 smithi165 ceph-mon[34540]: pgmap v222: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 55 op/s 2024-05-09T08:33:25.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:25 smithi184 ceph-mon[34509]: pgmap v222: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 55 op/s 2024-05-09T08:33:25.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:25 smithi028 ceph-mon[30119]: pgmap v222: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 55 op/s 2024-05-09T08:33:26.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:26 smithi165 ceph-mon[34540]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T08:33:26.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:26 smithi184 ceph-mon[34509]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T08:33:26.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:26 smithi028 ceph-mon[30119]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T08:33:27.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:27 smithi165 ceph-mon[34540]: pgmap v224: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 61 op/s 2024-05-09T08:33:27.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:27 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:33:27.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:27 smithi184 ceph-mon[34509]: pgmap v224: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 61 op/s 2024-05-09T08:33:27.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:27 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:33:27.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:27 smithi028 ceph-mon[30119]: pgmap v224: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 61 op/s 2024-05-09T08:33:27.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:27 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:33:29.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:29 smithi165 ceph-mon[34540]: pgmap v225: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 64 op/s 2024-05-09T08:33:29.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:29 smithi184 ceph-mon[34509]: pgmap v225: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 64 op/s 2024-05-09T08:33:29.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:29 smithi028 ceph-mon[30119]: pgmap v225: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 64 op/s 2024-05-09T08:33:30.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.572+0000 7fad7726b640 1 -- 172.21.15.28:0/3485600690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 msgr2=0x7fad70100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.572+0000 7fad7726b640 1 --2- 172.21.15.28:0/3485600690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 0x7fad70100110 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009930 tx=0x7fad6c02a420 comp rx=0 tx=0).stop 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.573+0000 7fad7726b640 1 -- 172.21.15.28:0/3485600690 shutdown_connections 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.573+0000 7fad7726b640 1 --2- 172.21.15.28:0/3485600690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad70104b00 0x7fad70104f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.573+0000 7fad7726b640 1 --2- 172.21.15.28:0/3485600690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad70069c40 0x7fad70100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.573+0000 7fad7726b640 1 --2- 172.21.15.28:0/3485600690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 0x7fad70100110 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.573+0000 7fad7726b640 1 -- 172.21.15.28:0/3485600690 >> 172.21.15.28:0/3485600690 conn(0x7fad70076cc0 msgr2=0x7fad700fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.573+0000 7fad7726b640 1 -- 172.21.15.28:0/3485600690 shutdown_connections 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.573+0000 7fad7726b640 1 -- 172.21.15.28:0/3485600690 wait complete. 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.573+0000 7fad7726b640 1 Processor -- start 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.573+0000 7fad7726b640 1 -- start start 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad7726b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 0x7fad70198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad7726b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad70069c40 0x7fad70198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad7726b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad70104b00 0x7fad7019c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad7726b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad70110b80 con 0x7fad700697d0 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad7726b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad70110a00 con 0x7fad70104b00 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad7726b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad70110d00 con 0x7fad70069c40 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75007640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 0x7fad70198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75007640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 0x7fad70198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41852/0 (socket says 172.21.15.28:41852) 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75007640 1 -- 172.21.15.28:0/785630166 learned_addr learned my addr 172.21.15.28:0/785630166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad74806640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad70069c40 0x7fad70198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75808640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad70104b00 0x7fad7019c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75007640 1 -- 172.21.15.28:0/785630166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad70069c40 msgr2=0x7fad70198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75007640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad70069c40 0x7fad70198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75007640 1 -- 172.21.15.28:0/785630166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad70104b00 msgr2=0x7fad7019c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75007640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad70104b00 0x7fad7019c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75007640 1 -- 172.21.15.28:0/785630166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7019d0f0 con 0x7fad700697d0 2024-05-09T08:33:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad74806640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad70069c40 0x7fad70198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad75007640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 0x7fad70198100 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fad6c02a940 tx=0x7fad6c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad667fc640 1 -- 172.21.15.28:0/785630166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c03f070 con 0x7fad700697d0 2024-05-09T08:33:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.574+0000 7fad667fc640 1 -- 172.21.15.28:0/785630166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad6c031c90 con 0x7fad700697d0 2024-05-09T08:33:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.575+0000 7fad7726b640 1 -- 172.21.15.28:0/785630166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad7019d380 con 0x7fad700697d0 2024-05-09T08:33:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.575+0000 7fad667fc640 1 -- 172.21.15.28:0/785630166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c035030 con 0x7fad700697d0 2024-05-09T08:33:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.575+0000 7fad7726b640 1 -- 172.21.15.28:0/785630166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad701a4c70 con 0x7fad700697d0 2024-05-09T08:33:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.577+0000 7fad667fc640 1 -- 172.21.15.28:0/785630166 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fad6c042070 con 0x7fad700697d0 2024-05-09T08:33:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad79f7b740 1 Processor -- start 2024-05-09T08:33:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad79f7b740 1 -- start start 2024-05-09T08:33:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87450 0x55778ba87830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87d70 0x55778ba920f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba92630 0x55778ba94a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad79f7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55778b9ec340 con 0x55778ba87450 2024-05-09T08:33:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad79f7b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55778b994290 con 0x55778ba92630 2024-05-09T08:33:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad79f7b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55778b9e8300 con 0x55778ba87d70 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad75808640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87450 0x55778ba87830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad75808640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87450 0x55778ba87830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41864/0 (socket says 172.21.15.28:41864) 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad667fc640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad5c0a2e20 0x7fad5c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad75808640 1 -- 172.21.15.28:0/2185674725 learned_addr learned my addr 172.21.15.28:0/2185674725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad75007640 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87d70 0x55778ba920f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.578+0000 7fad74806640 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba92630 0x55778ba94a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad667fc640 1 -- 172.21.15.28:0/785630166 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9583+0+0 (secure 0 0 0) 0x7fad6c0f0040 con 0x7fad700697d0 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad75808640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad5c0a2e20 0x7fad5c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad74806640 1 -- 172.21.15.28:0/2185674725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87d70 msgr2=0x55778ba920f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad74806640 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87d70 0x55778ba920f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad74806640 1 -- 172.21.15.28:0/2185674725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87450 msgr2=0x55778ba87830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad74806640 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87450 0x55778ba87830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad74806640 1 -- 172.21.15.28:0/2185674725 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55778b9b88e0 con 0x55778ba92630 2024-05-09T08:33:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad75808640 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87450 0x55778ba87830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad74806640 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba92630 0x55778ba94a20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fad700fed10 tx=0x7fad6000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db56fca8ea72f554 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad75808640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad5c0a2e20 0x7fad5c0a52e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fad68005c40 tx=0x7fad68004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad5b7fe640 1 -- 172.21.15.28:0/2185674725 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60015070 con 0x55778ba92630 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad5b7fe640 1 -- 172.21.15.28:0/2185674725 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad60011920 con 0x55778ba92630 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.579+0000 7fad5b7fe640 1 -- 172.21.15.28:0/2185674725 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60010900 con 0x55778ba92630 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 -- 172.21.15.28:0/2185674725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba92630 msgr2=0x55778ba94a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba92630 0x55778ba94a20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fad700fed10 tx=0x7fad6000f730 comp rx=0 tx=0).stop 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 -- 172.21.15.28:0/2185674725 shutdown_connections 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba92630 0x55778ba94a20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87d70 0x55778ba920f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 --2- 172.21.15.28:0/2185674725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87450 0x55778ba87830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 -- 172.21.15.28:0/2185674725 >> 172.21.15.28:0/2185674725 conn(0x55778ba83960 msgr2=0x55778ba83d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 -- 172.21.15.28:0/2185674725 shutdown_connections 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 -- 172.21.15.28:0/2185674725 wait complete. 2024-05-09T08:33:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.580+0000 7fad79f7b740 1 -- start start 2024-05-09T08:33:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad79f7b740 1 Processor -- start 2024-05-09T08:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 0x55778ba8f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba92630 0x55778ba8ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad79f7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55778ba95260 con 0x55778ba87d70 2024-05-09T08:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad79f7b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55778b9a7790 con 0x55778ba87450 2024-05-09T08:33:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad79f7b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55778ba953e0 con 0x55778ba92630 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad79f7b740 10 client.0 _ll_register_callbacks cb 0x55778b9e2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad74806640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad74806640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41876/0 (socket says 172.21.15.28:41876) 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad74806640 1 -- 172.21.15.28:0/1042618570 learned_addr learned my addr 172.21.15.28:0/1042618570 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad75007640 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 0x55778ba8f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad74806640 1 -- 172.21.15.28:0/1042618570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba92630 msgr2=0x55778ba8ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad74806640 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba92630 0x55778ba8ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad74806640 1 -- 172.21.15.28:0/1042618570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 msgr2=0x55778ba8f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad74806640 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 0x55778ba8f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.581+0000 7fad74806640 1 -- 172.21.15.28:0/1042618570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55778b700210 con 0x55778ba87d70 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad74806640 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8fa90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fad6000fc30 tx=0x7fad6000bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933c5a2c719938ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:33:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad58ff9640 1 -- 172.21.15.28:0/1042618570 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60029060 con 0x55778ba87d70 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad79f7b740 1 -- 172.21.15.28:0/1042618570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad60011920 con 0x55778ba87d70 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad58ff9640 1 -- 172.21.15.28:0/1042618570 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9583+0+0 (secure 0 0 0) 0x7fad60011da0 con 0x55778ba87d70 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad58ff9640 1 -- 172.21.15.28:0/1042618570 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fad60030050 con 0x55778ba87d70 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad79f7b740 10 client.16740 fetch_fsmap learned FSMap version 61 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad79f7b740 1 -- 172.21.15.28:0/1042618570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x55778b7cc620 con 0x55778ba87d70 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad58ff9640 1 -- 172.21.15.28:0/1042618570 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad60011920 con 0x55778ba87d70 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.582+0000 7fad58ff9640 1 -- 172.21.15.28:0/1042618570 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 61) ==== 8426+0+0 (secure 0 0 0) 0x7fad60004850 con 0x55778ba87d70 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.583+0000 7fad79f7b740 10 client.16740 fetch_fsmap finished waiting for FSMap version 61 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.583+0000 7fad79f7b740 10 client.16740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.583+0000 7fad79f7b740 20 client.16740 populate_metadata read hostname 'smithi028' 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.583+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55778baa6660 0x55778baa8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.583+0000 7fad79f7b740 4 client.16740 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.583+0000 7fad79f7b740 1 -- 172.21.15.28:0/1042618570 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55778b9f9a70 con 0x55778baa6660 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.583+0000 7fad75007640 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55778baa6660 0x55778baa8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.583+0000 7fad75007640 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55778baa6660 0x55778baa8a50 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d91707a2099389ff server_cookie=a4f986c8e01c45de in_seq=0 out_seq=0 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.583+0000 7fad58ff9640 10 client.16740 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad58ff9640 1 -- 172.21.15.28:0/1042618570 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55778b9f9a70 con 0x55778baa6660 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad58ff9640 10 client.16740 handle_command_reply: tid=0 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad79f7b740 1 client.16740 shutdown 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad79f7b740 1 -- 172.21.15.28:0/1042618570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 msgr2=0x55778ba8fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8fa90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fad6000fc30 tx=0x7fad6000bf70 comp rx=0 tx=0).stop 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad79f7b740 1 -- 172.21.15.28:0/1042618570 shutdown_connections 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55778baa6660 0x55778baa8a50 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba92630 0x55778ba8ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad58ff9640 0 client.16740 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8fa90 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1042618570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 0x55778ba8f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.584+0000 7fad79f7b740 1 -- 172.21.15.28:0/1042618570 >> 172.21.15.28:0/1042618570 conn(0x55778ba83960 msgr2=0x55778ba896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.585+0000 7fad79f7b740 1 -- 172.21.15.28:0/1042618570 shutdown_connections 2024-05-09T08:33:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.585+0000 7fad79f7b740 1 -- 172.21.15.28:0/1042618570 wait complete. 2024-05-09T08:33:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.585+0000 7fad79f7b740 20 client.16740 trim_cache size 0 max 16384 2024-05-09T08:33:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad79f7b740 1 Processor -- start 2024-05-09T08:33:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad79f7b740 1 -- start start 2024-05-09T08:33:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87450 0x55778ba91c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87d70 0x55778ba8e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba92630 0x55778ba89e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad79f7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad701acd00 con 0x55778ba92630 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad79f7b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad701aea50 con 0x55778ba87d70 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad79f7b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad701ad490 con 0x55778ba87450 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75808640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba92630 0x55778ba89e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75808640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba92630 0x55778ba89e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41886/0 (socket says 172.21.15.28:41886) 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75808640 1 -- 172.21.15.28:0/2051537486 learned_addr learned my addr 172.21.15.28:0/2051537486 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad74806640 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87d70 0x55778ba8e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75007640 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87450 0x55778ba91c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75808640 1 -- 172.21.15.28:0/2051537486 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87450 msgr2=0x55778ba91c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75808640 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87450 0x55778ba91c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75808640 1 -- 172.21.15.28:0/2051537486 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87d70 msgr2=0x55778ba8e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75808640 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87d70 0x55778ba8e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75808640 1 -- 172.21.15.28:0/2051537486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55778ba81f00 con 0x55778ba92630 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75007640 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87450 0x55778ba91c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad74806640 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87d70 0x55778ba8e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.600+0000 7fad75808640 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba92630 0x55778ba89e10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fad68009840 tx=0x7fad68009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12eada34a58c45bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad5affd640 1 -- 172.21.15.28:0/2051537486 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68025070 con 0x55778ba92630 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad5affd640 1 -- 172.21.15.28:0/2051537486 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad68012e10 con 0x55778ba92630 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad5affd640 1 -- 172.21.15.28:0/2051537486 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6801a070 con 0x55778ba92630 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad79f7b740 1 -- 172.21.15.28:0/2051537486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba92630 msgr2=0x55778ba89e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad79f7b740 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba92630 0x55778ba89e10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fad68009840 tx=0x7fad68009650 comp rx=0 tx=0).stop 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad79f7b740 1 -- 172.21.15.28:0/2051537486 shutdown_connections 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad79f7b740 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba92630 0x55778ba89e10 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad79f7b740 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87d70 0x55778ba8e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad79f7b740 1 --2- 172.21.15.28:0/2051537486 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba87450 0x55778ba91c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad79f7b740 1 -- 172.21.15.28:0/2051537486 >> 172.21.15.28:0/2051537486 conn(0x55778ba83960 msgr2=0x55778ba82170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad79f7b740 1 -- 172.21.15.28:0/2051537486 shutdown_connections 2024-05-09T08:33:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.601+0000 7fad79f7b740 1 -- 172.21.15.28:0/2051537486 wait complete. 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad79f7b740 1 -- start start 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad79f7b740 1 Processor -- start 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 0x55778ba9ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad79f7b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba92630 0x55778ba8c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad79f7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad701ae380 con 0x55778ba87d70 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad79f7b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad701adc20 con 0x55778ba87450 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad79f7b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad701aad00 con 0x55778ba92630 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad79f7b740 10 client.0 _ll_register_callbacks cb 0x7fad680096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad74806640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad74806640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41892/0 (socket says 172.21.15.28:41892) 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.602+0000 7fad74806640 1 -- 172.21.15.28:0/1151168095 learned_addr learned my addr 172.21.15.28:0/1151168095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad75007640 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 0x55778ba9ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad75808640 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba92630 0x55778ba8c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad74806640 1 -- 172.21.15.28:0/1151168095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba92630 msgr2=0x55778ba8c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad74806640 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba92630 0x55778ba8c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad74806640 1 -- 172.21.15.28:0/1151168095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 msgr2=0x55778ba9ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad74806640 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 0x55778ba9ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad74806640 1 -- 172.21.15.28:0/1151168095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55778ba8cc10 con 0x55778ba87d70 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad74806640 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8bea0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fad60001eb0 tx=0x7fad6000bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f12dabe218ad7a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad58ff9640 1 -- 172.21.15.28:0/1151168095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60029070 con 0x55778ba87d70 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad58ff9640 1 -- 172.21.15.28:0/1151168095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9583+0+0 (secure 0 0 0) 0x7fad60004680 con 0x55778ba87d70 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad79f7b740 1 -- 172.21.15.28:0/1151168095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad68012e10 con 0x55778ba87d70 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad58ff9640 1 -- 172.21.15.28:0/1151168095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fad6003c050 con 0x55778ba87d70 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad79f7b740 10 client.16746 fetch_fsmap learned FSMap version 61 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad79f7b740 1 -- 172.21.15.28:0/1151168095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x55778bb0c880 con 0x55778ba87d70 2024-05-09T08:33:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.603+0000 7fad58ff9640 1 -- 172.21.15.28:0/1151168095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad68012e10 con 0x55778ba87d70 2024-05-09T08:33:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad58ff9640 1 -- 172.21.15.28:0/1151168095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 61) ==== 8426+0+0 (secure 0 0 0) 0x7fad60010b50 con 0x55778ba87d70 2024-05-09T08:33:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad79f7b740 10 client.16746 fetch_fsmap finished waiting for FSMap version 61 2024-05-09T08:33:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad79f7b740 10 client.16746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:33:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad79f7b740 20 client.16746 populate_metadata read hostname 'smithi028' 2024-05-09T08:33:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55778bb0d000 0x55778bb0f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad79f7b740 4 client.16746 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:33:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad79f7b740 1 -- 172.21.15.28:0/1151168095 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55778b9f3dc0 con 0x55778bb0d000 2024-05-09T08:33:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad75007640 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55778bb0d000 0x55778bb0f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad75007640 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55778bb0d000 0x55778bb0f420 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a137a5fc1413e9f server_cookie=86ec09f449551278 in_seq=0 out_seq=0 2024-05-09T08:33:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:30.604+0000 7fad58ff9640 10 client.16746 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:31.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.414+0000 7fad58ff9640 1 -- 172.21.15.28:0/1151168095 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55778b9f3dc0 con 0x55778bb0d000 2024-05-09T08:33:31.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.414+0000 7fad58ff9640 10 client.16746 handle_command_reply: tid=0 2024-05-09T08:33:31.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.414+0000 7fad79f7b740 1 client.16746 shutdown 2024-05-09T08:33:31.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 -- 172.21.15.28:0/1151168095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 msgr2=0x55778ba8bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:31.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8bea0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fad60001eb0 tx=0x7fad6000bb00 comp rx=0 tx=0).stop 2024-05-09T08:33:31.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 -- 172.21.15.28:0/1151168095 shutdown_connections 2024-05-09T08:33:31.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55778bb0d000 0x55778bb0f420 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:31.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55778ba92630 0x55778ba8c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:31.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad58ff9640 0 client.16746 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:31.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55778ba87d70 0x55778ba8bea0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:31.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 --2- 172.21.15.28:0/1151168095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55778ba87450 0x55778ba9ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:31.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 -- 172.21.15.28:0/1151168095 >> 172.21.15.28:0/1151168095 conn(0x55778ba83960 msgr2=0x55778ba82a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:31.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 -- 172.21.15.28:0/1151168095 shutdown_connections 2024-05-09T08:33:31.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 1 -- 172.21.15.28:0/1151168095 wait complete. 2024-05-09T08:33:31.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad79f7b740 20 client.16746 trim_cache size 0 max 16384 2024-05-09T08:33:31.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:33:31.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:33:31.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:33:31.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:33:31.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 -- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad5c0a2e20 msgr2=0x7fad5c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad5c0a2e20 0x7fad5c0a52e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fad68005c40 tx=0x7fad68004ab0 comp rx=0 tx=0).stop 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 -- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 msgr2=0x7fad70198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 0x7fad70198100 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fad6c02a940 tx=0x7fad6c003b90 comp rx=0 tx=0).stop 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 -- 172.21.15.28:0/785630166 shutdown_connections 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad70104b00 0x7fad7019c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad5c0a2e20 0x7fad5c0a52e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad70069c40 0x7fad70198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 --2- 172.21.15.28:0/785630166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad700697d0 0x7fad70198100 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.415+0000 7fad7726b640 1 -- 172.21.15.28:0/785630166 >> 172.21.15.28:0/785630166 conn(0x7fad70076cc0 msgr2=0x7fad700fc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.416+0000 7fad7726b640 1 -- 172.21.15.28:0/785630166 shutdown_connections 2024-05-09T08:33:31.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:31.416+0000 7fad7726b640 1 -- 172.21.15.28:0/785630166 wait complete. 2024-05-09T08:33:31.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:31 smithi165 ceph-mon[34540]: pgmap v226: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.4 MiB/s wr, 56 op/s 2024-05-09T08:33:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:31 smithi184 ceph-mon[34509]: pgmap v226: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.4 MiB/s wr, 56 op/s 2024-05-09T08:33:31.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:31 smithi028 ceph-mon[30119]: pgmap v226: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.4 MiB/s wr, 56 op/s 2024-05-09T08:33:32.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:32 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:33:32.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:32 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:33:32.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:32 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:33:33.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:33 smithi165 ceph-mon[34540]: pgmap v227: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 115 op/s 2024-05-09T08:33:33.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:33 smithi184 ceph-mon[34509]: pgmap v227: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 115 op/s 2024-05-09T08:33:33.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:33 smithi028 ceph-mon[30119]: pgmap v227: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 115 op/s 2024-05-09T08:33:35.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:35 smithi165 ceph-mon[34540]: pgmap v228: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 234 op/s 2024-05-09T08:33:35.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:35 smithi184 ceph-mon[34509]: pgmap v228: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 234 op/s 2024-05-09T08:33:35.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:35 smithi028 ceph-mon[30119]: pgmap v228: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 234 op/s 2024-05-09T08:33:37.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:37 smithi165 ceph-mon[34540]: pgmap v229: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 322 op/s 2024-05-09T08:33:37.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:37 smithi184 ceph-mon[34509]: pgmap v229: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 322 op/s 2024-05-09T08:33:37.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:37 smithi028 ceph-mon[30119]: pgmap v229: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 322 op/s 2024-05-09T08:33:37.991 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-05-09T08:33:37.991 INFO:tasks.workunit.client.0.smithi028.stdout:Final score for writes: 616 2024-05-09T08:33:37.991 INFO:tasks.workunit.client.0.smithi028.stdout:Final score for reads : 750362 2024-05-09T08:33:37.991 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-05-09T08:33:37.991 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-09T08:33:37.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-09T08:33:39.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:39 smithi165 ceph-mon[34540]: pgmap v230: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 298 op/s 2024-05-09T08:33:39.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:39 smithi184 ceph-mon[34509]: pgmap v230: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 298 op/s 2024-05-09T08:33:39.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:39 smithi028 ceph-mon[30119]: pgmap v230: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 298 op/s 2024-05-09T08:33:41.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:41 smithi165 ceph-mon[34540]: pgmap v231: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 333 op/s 2024-05-09T08:33:41.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:41 smithi184 ceph-mon[34509]: pgmap v231: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 333 op/s 2024-05-09T08:33:41.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:41 smithi028 ceph-mon[30119]: pgmap v231: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 333 op/s 2024-05-09T08:33:43.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:43 smithi165 ceph-mon[34540]: pgmap v232: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 339 op/s 2024-05-09T08:33:43.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:43 smithi184 ceph-mon[34509]: pgmap v232: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 339 op/s 2024-05-09T08:33:43.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:43 smithi028 ceph-mon[30119]: pgmap v232: 129 pgs: 129 active+clean; 2.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 339 op/s 2024-05-09T08:33:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.507+0000 7fc86d49a640 1 -- 172.21.15.28:0/3677302675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8681033e0 msgr2=0x7fc86810b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.507+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3677302675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8681033e0 0x7fc86810b8c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc850009900 tx=0x7fc85002a510 comp rx=0 tx=0).stop 2024-05-09T08:33:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.507+0000 7fc86d49a640 1 -- 172.21.15.28:0/3677302675 shutdown_connections 2024-05-09T08:33:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.507+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3677302675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc86810be00 0x7fc86810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.507+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3677302675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8681033e0 0x7fc86810b8c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.507+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3677302675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc868102ac0 0x7fc868102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.507+0000 7fc86d49a640 1 -- 172.21.15.28:0/3677302675 >> 172.21.15.28:0/3677302675 conn(0x7fc8680778b0 msgr2=0x7fc868077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 -- 172.21.15.28:0/3677302675 shutdown_connections 2024-05-09T08:33:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 -- 172.21.15.28:0/3677302675 wait complete. 2024-05-09T08:33:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 Processor -- start 2024-05-09T08:33:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 -- start start 2024-05-09T08:33:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc868102ac0 0x7fc868198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8681033e0 0x7fc8681986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc86810be00 0x7fc86819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc868111c00 con 0x7fc868102ac0 2024-05-09T08:33:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc868111a80 con 0x7fc8681033e0 2024-05-09T08:33:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc86d49a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc868111d80 con 0x7fc86810be00 2024-05-09T08:33:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc866ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc868102ac0 0x7fc868198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc866ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc868102ac0 0x7fc868198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37216/0 (socket says 172.21.15.28:37216) 2024-05-09T08:33:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.508+0000 7fc866ffd640 1 -- 172.21.15.28:0/3188516488 learned_addr learned my addr 172.21.15.28:0/3188516488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc85ffff640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8681033e0 0x7fc8681986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc8677fe640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc86810be00 0x7fc86819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc866ffd640 1 -- 172.21.15.28:0/3188516488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc86810be00 msgr2=0x7fc86819c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc866ffd640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc86810be00 0x7fc86819c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc866ffd640 1 -- 172.21.15.28:0/3188516488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8681033e0 msgr2=0x7fc8681986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc866ffd640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8681033e0 0x7fc8681986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc866ffd640 1 -- 172.21.15.28:0/3188516488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86819d120 con 0x7fc868102ac0 2024-05-09T08:33:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc866ffd640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc868102ac0 0x7fc868198180 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fc85800e9f0 tx=0x7fc85800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc864ff9640 1 -- 172.21.15.28:0/3188516488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85800cd80 con 0x7fc868102ac0 2024-05-09T08:33:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc86d49a640 1 -- 172.21.15.28:0/3188516488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86819d410 con 0x7fc868102ac0 2024-05-09T08:33:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc864ff9640 1 -- 172.21.15.28:0/3188516488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc858010910 con 0x7fc868102ac0 2024-05-09T08:33:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc864ff9640 1 -- 172.21.15.28:0/3188516488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8580188f0 con 0x7fc868102ac0 2024-05-09T08:33:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.509+0000 7fc86d49a640 1 -- 172.21.15.28:0/3188516488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8681a4ca0 con 0x7fc868102ac0 2024-05-09T08:33:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.511+0000 7fc864ff9640 1 -- 172.21.15.28:0/3188516488 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc858010ac0 con 0x7fc868102ac0 2024-05-09T08:33:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc8701a1740 1 Processor -- start 2024-05-09T08:33:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc8701a1740 1 -- start start 2024-05-09T08:33:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc454f860 0x560fc454fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 0x560fc455ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc8701a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560fc44b0340 con 0x560fc4550180 2024-05-09T08:33:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc8701a1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560fc4458290 con 0x560fc454f860 2024-05-09T08:33:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc8701a1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560fc44ac300 con 0x560fc455aa40 2024-05-09T08:33:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc866ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc8677fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc454f860 0x560fc454fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc866ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37222/0 (socket says 172.21.15.28:37222) 2024-05-09T08:33:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc866ffd640 1 -- 172.21.15.28:0/1763550003 learned_addr learned my addr 172.21.15.28:0/1763550003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc85ffff640 1 --2- 172.21.15.28:0/1763550003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 0x560fc455ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.512+0000 7fc864ff9640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc8540a2da0 0x7fc8540a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc8677fe640 1 -- 172.21.15.28:0/1763550003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 msgr2=0x560fc455ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc8677fe640 1 --2- 172.21.15.28:0/1763550003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 0x560fc455ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc8677fe640 1 -- 172.21.15.28:0/1763550003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 msgr2=0x560fc455a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc8677fe640 1 --2- 172.21.15.28:0/1763550003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc8677fe640 1 -- 172.21.15.28:0/1763550003 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fc447c8e0 con 0x560fc454f860 2024-05-09T08:33:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc864ff9640 1 -- 172.21.15.28:0/3188516488 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9583+0+0 (secure 0 0 0) 0x7fc8580cfc20 con 0x7fc868102ac0 2024-05-09T08:33:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc8677fe640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc8540a2da0 0x7fc8540a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc8677fe640 1 --2- 172.21.15.28:0/1763550003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc454f860 0x560fc454fc40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc860009fa0 tx=0x7fc8600118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=861a1ce6a058a33a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc8677fe640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc8540a2da0 0x7fc8540a5260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc860022000 tx=0x7fc860023000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc85d7fa640 1 -- 172.21.15.28:0/1763550003 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860017050 con 0x560fc454f860 2024-05-09T08:33:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc85d7fa640 1 -- 172.21.15.28:0/1763550003 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc860003b90 con 0x560fc454f860 2024-05-09T08:33:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.513+0000 7fc85d7fa640 1 -- 172.21.15.28:0/1763550003 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86001b970 con 0x560fc454f860 2024-05-09T08:33:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 -- 172.21.15.28:0/1763550003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc454f860 msgr2=0x560fc454fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 --2- 172.21.15.28:0/1763550003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc454f860 0x560fc454fc40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc860009fa0 tx=0x7fc8600118d0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 -- 172.21.15.28:0/1763550003 shutdown_connections 2024-05-09T08:33:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 --2- 172.21.15.28:0/1763550003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 0x560fc455ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 --2- 172.21.15.28:0/1763550003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 --2- 172.21.15.28:0/1763550003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc454f860 0x560fc454fc40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 -- 172.21.15.28:0/1763550003 >> 172.21.15.28:0/1763550003 conn(0x560fc454bd70 msgr2=0x560fc454c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 -- 172.21.15.28:0/1763550003 shutdown_connections 2024-05-09T08:33:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 -- 172.21.15.28:0/1763550003 wait complete. 2024-05-09T08:33:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 -- start start 2024-05-09T08:33:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.514+0000 7fc8701a1740 1 Processor -- start 2024-05-09T08:33:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc454f860 0x560fc4557a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 0x560fc4557f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 0x560fc45584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc8701a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560fc455d670 con 0x560fc454f860 2024-05-09T08:33:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc8701a1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560fc446b790 con 0x560fc4550180 2024-05-09T08:33:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc8701a1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560fc455d7f0 con 0x560fc455aa40 2024-05-09T08:33:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc8701a1740 10 client.0 _ll_register_callbacks cb 0x560fc44a6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc866ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc454f860 0x560fc4557a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc866ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc454f860 0x560fc4557a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37224/0 (socket says 172.21.15.28:37224) 2024-05-09T08:33:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc866ffd640 1 -- 172.21.15.28:0/3581602427 learned_addr learned my addr 172.21.15.28:0/3581602427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc85ffff640 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 0x560fc4557f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc8677fe640 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 0x560fc45584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc866ffd640 1 -- 172.21.15.28:0/3581602427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 msgr2=0x560fc45584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc866ffd640 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 0x560fc45584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc866ffd640 1 -- 172.21.15.28:0/3581602427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 msgr2=0x560fc4557f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc866ffd640 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 0x560fc4557f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc866ffd640 1 -- 172.21.15.28:0/3581602427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fc41c4210 con 0x560fc454f860 2024-05-09T08:33:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc866ffd640 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc454f860 0x560fc4557a30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fc85800c6a0 tx=0x7fc8580d3360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18dd19f4d4323afa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:44.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc84e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:33:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc84e7fc640 1 -- 172.21.15.28:0/3581602427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85809b730 con 0x560fc454f860 2024-05-09T08:33:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc84e7fc640 1 -- 172.21.15.28:0/3581602427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9583+0+0 (secure 0 0 0) 0x7fc85809b420 con 0x560fc454f860 2024-05-09T08:33:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.515+0000 7fc8701a1740 1 -- 172.21.15.28:0/3581602427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc860003b90 con 0x560fc454f860 2024-05-09T08:33:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc84e7fc640 1 -- 172.21.15.28:0/3581602427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fc8580a4990 con 0x560fc454f860 2024-05-09T08:33:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc8701a1740 10 client.16758 fetch_fsmap learned FSMap version 62 2024-05-09T08:33:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc8701a1740 1 -- 172.21.15.28:0/3581602427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x560fc4290620 con 0x560fc454f860 2024-05-09T08:33:44.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc84e7fc640 1 -- 172.21.15.28:0/3581602427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc860003b90 con 0x560fc454f860 2024-05-09T08:33:44.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc84e7fc640 1 -- 172.21.15.28:0/3581602427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 62) ==== 8438+0+0 (secure 0 0 0) 0x7fc8580d5020 con 0x560fc454f860 2024-05-09T08:33:44.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc8701a1740 10 client.16758 fetch_fsmap finished waiting for FSMap version 62 2024-05-09T08:33:44.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc8701a1740 10 client.16758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:33:44.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc8701a1740 20 client.16758 populate_metadata read hostname 'smithi028' 2024-05-09T08:33:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560fc456a890 0x560fc456cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc8701a1740 4 client.16758 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc8701a1740 1 -- 172.21.15.28:0/3581602427 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fc44bda70 con 0x560fc456a890 2024-05-09T08:33:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.516+0000 7fc85ffff640 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560fc456a890 0x560fc456cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.517+0000 7fc84e7fc640 10 client.16758 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:44.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.517+0000 7fc85ffff640 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560fc456a890 0x560fc456cd00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=668caa709fe28774 server_cookie=8aa5dfe975898fc2 in_seq=0 out_seq=0 2024-05-09T08:33:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc84e7fc640 1 -- 172.21.15.28:0/3581602427 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560fc44bda70 con 0x560fc456a890 2024-05-09T08:33:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc84e7fc640 10 client.16758 handle_command_reply: tid=0 2024-05-09T08:33:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 client.16758 shutdown 2024-05-09T08:33:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 -- 172.21.15.28:0/3581602427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc454f860 msgr2=0x560fc4557a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc454f860 0x560fc4557a30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fc85800c6a0 tx=0x7fc8580d3360 comp rx=0 tx=0).stop 2024-05-09T08:33:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 -- 172.21.15.28:0/3581602427 shutdown_connections 2024-05-09T08:33:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560fc456a890 0x560fc456cd00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc455aa40 0x560fc45584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 0x560fc4557f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc84e7fc640 0 client.16758 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3581602427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc454f860 0x560fc4557a30 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 -- 172.21.15.28:0/3581602427 >> 172.21.15.28:0/3581602427 conn(0x560fc454bd70 msgr2=0x560fc4551af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:44.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 -- 172.21.15.28:0/3581602427 shutdown_connections 2024-05-09T08:33:44.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 1 -- 172.21.15.28:0/3581602427 wait complete. 2024-05-09T08:33:44.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.518+0000 7fc8701a1740 20 client.16758 trim_cache size 0 max 16384 2024-05-09T08:33:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.533+0000 7fc8701a1740 1 Processor -- start 2024-05-09T08:33:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.533+0000 7fc8701a1740 1 -- start start 2024-05-09T08:33:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 0x560fc455a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 0x560fc4551970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc455aa40 0x560fc4572610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8701a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8681acd00 con 0x560fc455aa40 2024-05-09T08:33:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8701a1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc8681aea50 con 0x560fc4550180 2024-05-09T08:33:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8701a1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8681ad490 con 0x560fc454f860 2024-05-09T08:33:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc455aa40 0x560fc4572610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc85ffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 0x560fc4551970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc455aa40 0x560fc4572610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37236/0 (socket says 172.21.15.28:37236) 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8677fe640 1 -- 172.21.15.28:0/3610072203 learned_addr learned my addr 172.21.15.28:0/3610072203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc866ffd640 1 --2- 172.21.15.28:0/3610072203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 0x560fc455a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8677fe640 1 -- 172.21.15.28:0/3610072203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 msgr2=0x560fc455a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8677fe640 1 --2- 172.21.15.28:0/3610072203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 0x560fc455a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8677fe640 1 -- 172.21.15.28:0/3610072203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 msgr2=0x560fc4551970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8677fe640 1 --2- 172.21.15.28:0/3610072203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 0x560fc4551970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8677fe640 1 -- 172.21.15.28:0/3610072203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fc4555f80 con 0x560fc455aa40 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc8677fe640 1 --2- 172.21.15.28:0/3610072203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc455aa40 0x560fc4572610 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc86001b5e0 tx=0x7fc860028040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16651af438a7c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc85cff9640 1 -- 172.21.15.28:0/3610072203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860003b90 con 0x560fc455aa40 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc85cff9640 1 -- 172.21.15.28:0/3610072203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc860009b30 con 0x560fc455aa40 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.534+0000 7fc85cff9640 1 -- 172.21.15.28:0/3610072203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86000fa00 con 0x560fc455aa40 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.535+0000 7fc8701a1740 1 -- 172.21.15.28:0/3610072203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc455aa40 msgr2=0x560fc4572610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.535+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3610072203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc455aa40 0x560fc4572610 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc86001b5e0 tx=0x7fc860028040 comp rx=0 tx=0).stop 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.535+0000 7fc8701a1740 1 -- 172.21.15.28:0/3610072203 shutdown_connections 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.535+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3610072203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc455aa40 0x560fc4572610 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.535+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3610072203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc4550180 0x560fc4551970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.535+0000 7fc8701a1740 1 --2- 172.21.15.28:0/3610072203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 0x560fc455a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.535+0000 7fc8701a1740 1 -- 172.21.15.28:0/3610072203 >> 172.21.15.28:0/3610072203 conn(0x560fc454bd70 msgr2=0x560fc4551260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.535+0000 7fc8701a1740 1 -- 172.21.15.28:0/3610072203 shutdown_connections 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.535+0000 7fc8701a1740 1 -- 172.21.15.28:0/3610072203 wait complete. 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8701a1740 1 -- start start 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8701a1740 1 Processor -- start 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 0x560fc4562ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8701a1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc455aa40 0x560fc455f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8701a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8681ae380 con 0x560fc4550180 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8701a1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc8681adc20 con 0x560fc455aa40 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8701a1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8681aad00 con 0x560fc454f860 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8701a1740 10 client.0 _ll_register_callbacks cb 0x7fc858009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc85ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc85ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37248/0 (socket says 172.21.15.28:37248) 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc85ffff640 1 -- 172.21.15.28:0/156180279 learned_addr learned my addr 172.21.15.28:0/156180279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc8677fe640 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc455aa40 0x560fc455f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc866ffd640 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 0x560fc4562ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc85ffff640 1 -- 172.21.15.28:0/156180279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 msgr2=0x560fc4562ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc85ffff640 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 0x560fc4562ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc85ffff640 1 -- 172.21.15.28:0/156180279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc455aa40 msgr2=0x560fc455f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc85ffff640 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc455aa40 0x560fc455f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:44.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.536+0000 7fc85ffff640 1 -- 172.21.15.28:0/156180279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fc455f8b0 con 0x560fc4550180 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc85ffff640 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455ee30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc850009900 tx=0x7fc850004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cee1084af463b5e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc84e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc84e7fc640 1 -- 172.21.15.28:0/156180279 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850035040 con 0x560fc4550180 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc84e7fc640 1 -- 172.21.15.28:0/156180279 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9583+0+0 (secure 0 0 0) 0x7fc8500319c0 con 0x560fc4550180 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc8701a1740 1 -- 172.21.15.28:0/156180279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc860009b30 con 0x560fc4550180 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc84e7fc640 1 -- 172.21.15.28:0/156180279 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fc850058050 con 0x560fc4550180 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc8701a1740 10 client.16764 fetch_fsmap learned FSMap version 62 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc8701a1740 1 -- 172.21.15.28:0/156180279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x560fc45d1be0 con 0x560fc4550180 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc84e7fc640 1 -- 172.21.15.28:0/156180279 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc860009b30 con 0x560fc4550180 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc84e7fc640 1 -- 172.21.15.28:0/156180279 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 62) ==== 8438+0+0 (secure 0 0 0) 0x7fc850003c20 con 0x560fc4550180 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc8701a1740 10 client.16764 fetch_fsmap finished waiting for FSMap version 62 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc8701a1740 10 client.16764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:33:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc8701a1740 20 client.16764 populate_metadata read hostname 'smithi028' 2024-05-09T08:33:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc8701a1740 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560fc456a890 0x560fc45d44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc8701a1740 4 client.16764 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:33:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc8701a1740 1 -- 172.21.15.28:0/156180279 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560fc44b7dc0 con 0x560fc456a890 2024-05-09T08:33:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.537+0000 7fc866ffd640 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560fc456a890 0x560fc45d44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.538+0000 7fc84e7fc640 10 client.16764 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:44.538+0000 7fc866ffd640 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560fc456a890 0x560fc45d44e0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f00a094f05ade631 server_cookie=d43c6830c2ca9b5b in_seq=0 out_seq=0 2024-05-09T08:33:45.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:45 smithi165 ceph-mon[34540]: pgmap v233: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 326 op/s 2024-05-09T08:33:45.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:45 smithi184 ceph-mon[34509]: pgmap v233: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 326 op/s 2024-05-09T08:33:45.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:45 smithi028 ceph-mon[30119]: pgmap v233: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 326 op/s 2024-05-09T08:33:47.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:47 smithi165 ceph-mon[34540]: pgmap v234: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 268 op/s 2024-05-09T08:33:47.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:47 smithi184 ceph-mon[34509]: pgmap v234: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 268 op/s 2024-05-09T08:33:47.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:47 smithi028 ceph-mon[30119]: pgmap v234: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 268 op/s 2024-05-09T08:33:49.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:49 smithi165 ceph-mon[34540]: pgmap v235: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 566 op/s 2024-05-09T08:33:49.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:49 smithi184 ceph-mon[34509]: pgmap v235: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 566 op/s 2024-05-09T08:33:49.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:49 smithi028 ceph-mon[30119]: pgmap v235: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 566 op/s 2024-05-09T08:33:51.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:51 smithi165 ceph-mon[34540]: pgmap v236: 129 pgs: 129 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.47k op/s 2024-05-09T08:33:51.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:51 smithi184 ceph-mon[34509]: pgmap v236: 129 pgs: 129 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.47k op/s 2024-05-09T08:33:51.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:51 smithi028 ceph-mon[30119]: pgmap v236: 129 pgs: 129 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 1.47k op/s 2024-05-09T08:33:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.989+0000 7fc84e7fc640 1 -- 172.21.15.28:0/156180279 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560fc44b7dc0 con 0x560fc456a890 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.989+0000 7fc84e7fc640 10 client.16764 handle_command_reply: tid=0 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.989+0000 7fc8701a1740 1 client.16764 shutdown 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.989+0000 7fc8701a1740 1 -- 172.21.15.28:0/156180279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 msgr2=0x560fc455ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc8701a1740 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455ee30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc850009900 tx=0x7fc850004400 comp rx=0 tx=0).stop 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc8701a1740 1 -- 172.21.15.28:0/156180279 shutdown_connections 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc8701a1740 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560fc456a890 0x560fc45d44e0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc8701a1740 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560fc455aa40 0x560fc455f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc84e7fc640 0 client.16764 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc8701a1740 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fc4550180 0x560fc455ee30 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc8701a1740 1 --2- 172.21.15.28:0/156180279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560fc454f860 0x560fc4562ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc8701a1740 1 -- 172.21.15.28:0/156180279 >> 172.21.15.28:0/156180279 conn(0x560fc454bd70 msgr2=0x560fc4558280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc8701a1740 1 -- 172.21.15.28:0/156180279 shutdown_connections 2024-05-09T08:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.990+0000 7fc8701a1740 1 -- 172.21.15.28:0/156180279 wait complete. 2024-05-09T08:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc8701a1740 20 client.16764 trim_cache size 0 max 16384 2024-05-09T08:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:33:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 -- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc8540a2da0 msgr2=0x7fc8540a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc8540a2da0 0x7fc8540a5260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc860022000 tx=0x7fc860023000 comp rx=0 tx=0).stop 2024-05-09T08:33:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 -- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc868102ac0 msgr2=0x7fc868198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc868102ac0 0x7fc868198180 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fc85800e9f0 tx=0x7fc85800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:33:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 -- 172.21.15.28:0/3188516488 shutdown_connections 2024-05-09T08:33:52.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc8540a2da0 0x7fc8540a5260 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:52.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc86810be00 0x7fc86819c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:52.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8681033e0 0x7fc8681986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:52.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 --2- 172.21.15.28:0/3188516488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc868102ac0 0x7fc868198180 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:52.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.991+0000 7fc86d49a640 1 -- 172.21.15.28:0/3188516488 >> 172.21.15.28:0/3188516488 conn(0x7fc8680778b0 msgr2=0x7fc8680fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:52.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.992+0000 7fc86d49a640 1 -- 172.21.15.28:0/3188516488 shutdown_connections 2024-05-09T08:33:52.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:33:52.992+0000 7fc86d49a640 1 -- 172.21.15.28:0/3188516488 wait complete. 2024-05-09T08:33:53.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:53 smithi165 ceph-mon[34540]: pgmap v237: 129 pgs: 129 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 1.68k op/s 2024-05-09T08:33:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:53 smithi184 ceph-mon[34509]: pgmap v237: 129 pgs: 129 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 1.68k op/s 2024-05-09T08:33:53.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:53 smithi028 ceph-mon[30119]: pgmap v237: 129 pgs: 129 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 1.68k op/s 2024-05-09T08:33:55.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:55 smithi165 ceph-mon[34540]: pgmap v238: 129 pgs: 129 active+clean; 2.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 2.09k op/s 2024-05-09T08:33:55.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:55 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:33:55.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:55 smithi184 ceph-mon[34509]: pgmap v238: 129 pgs: 129 active+clean; 2.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 2.09k op/s 2024-05-09T08:33:55.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:55 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:33:55.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:55 smithi028 ceph-mon[30119]: pgmap v238: 129 pgs: 129 active+clean; 2.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 2.09k op/s 2024-05-09T08:33:55.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:55 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:33:56.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:56 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:33:56.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:56 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:33:56.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:56 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:33:56.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:56 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:33:56.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:56 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:33:56.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:56 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:33:56.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:56 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:33:56.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:56 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:33:56.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:56 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:33:57.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:57 smithi028 ceph-mon[30119]: pgmap v239: 129 pgs: 129 active+clean; 2.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.2 MiB/s wr, 2.27k op/s 2024-05-09T08:33:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:57 smithi165 ceph-mon[34540]: pgmap v239: 129 pgs: 129 active+clean; 2.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.2 MiB/s wr, 2.27k op/s 2024-05-09T08:33:58.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:57 smithi184 ceph-mon[34509]: pgmap v239: 129 pgs: 129 active+clean; 2.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.2 MiB/s wr, 2.27k op/s 2024-05-09T08:33:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:33:59 smithi028 ceph-mon[30119]: pgmap v240: 129 pgs: 129 active+clean; 2.3 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1.6 MiB/s wr, 2.34k op/s 2024-05-09T08:34:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:33:59 smithi165 ceph-mon[34540]: pgmap v240: 129 pgs: 129 active+clean; 2.3 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1.6 MiB/s wr, 2.34k op/s 2024-05-09T08:34:00.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:33:59 smithi184 ceph-mon[34509]: pgmap v240: 129 pgs: 129 active+clean; 2.3 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1.6 MiB/s wr, 2.34k op/s 2024-05-09T08:34:01.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:01 smithi028 ceph-mon[30119]: pgmap v241: 129 pgs: 129 active+clean; 2.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.2 MiB/s wr, 2.07k op/s 2024-05-09T08:34:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:01 smithi165 ceph-mon[34540]: pgmap v241: 129 pgs: 129 active+clean; 2.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.2 MiB/s wr, 2.07k op/s 2024-05-09T08:34:02.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:01 smithi184 ceph-mon[34509]: pgmap v241: 129 pgs: 129 active+clean; 2.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.2 MiB/s wr, 2.07k op/s 2024-05-09T08:34:03.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:03 smithi165 ceph-mon[34540]: pgmap v242: 129 pgs: 129 active+clean; 2.3 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 1.19k op/s 2024-05-09T08:34:03.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:03 smithi184 ceph-mon[34509]: pgmap v242: 129 pgs: 129 active+clean; 2.3 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 1.19k op/s 2024-05-09T08:34:03.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:03 smithi028 ceph-mon[30119]: pgmap v242: 129 pgs: 129 active+clean; 2.3 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 1.19k op/s 2024-05-09T08:34:05.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:05 smithi028 ceph-mon[30119]: pgmap v243: 129 pgs: 129 active+clean; 2.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 1.03k op/s 2024-05-09T08:34:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:05 smithi165 ceph-mon[34540]: pgmap v243: 129 pgs: 129 active+clean; 2.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 1.03k op/s 2024-05-09T08:34:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:05 smithi184 ceph-mon[34509]: pgmap v243: 129 pgs: 129 active+clean; 2.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 1.03k op/s 2024-05-09T08:34:06.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.158+0000 7ff9a91d5640 1 -- 172.21.15.28:0/508829538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a411b010 msgr2=0x7ff9a411d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:06.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.158+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/508829538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a411b010 0x7ff9a411d400 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff99c01a940 tx=0x7ff99c03b520 comp rx=0 tx=0).stop 2024-05-09T08:34:06.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.159+0000 7ff9a91d5640 1 -- 172.21.15.28:0/508829538 shutdown_connections 2024-05-09T08:34:06.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.159+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/508829538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a411b010 0x7ff9a411d400 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.159+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/508829538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9a4110400 0x7ff9a4108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.159+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/508829538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a4110020 0x7ff9a4108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.159+0000 7ff9a91d5640 1 -- 172.21.15.28:0/508829538 >> 172.21.15.28:0/508829538 conn(0x7ff9a406c900 msgr2=0x7ff9a406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:06.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.161+0000 7ff9a91d5640 1 -- 172.21.15.28:0/508829538 shutdown_connections 2024-05-09T08:34:06.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.161+0000 7ff9a91d5640 1 -- 172.21.15.28:0/508829538 wait complete. 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.162+0000 7ff9a91d5640 1 Processor -- start 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.162+0000 7ff9a91d5640 1 -- start start 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.162+0000 7ff9a91d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9a4110020 0x7ff9a411a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.162+0000 7ff9a91d5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a411b010 0x7ff9a41169c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.162+0000 7ff9a91d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a4116f00 0x7ff9a41173b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.162+0000 7ff9a91d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a41132d0 con 0x7ff9a4116f00 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.162+0000 7ff9a91d5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a4113150 con 0x7ff9a411b010 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.162+0000 7ff9a91d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a4113450 con 0x7ff9a4110020 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.163+0000 7ff9a359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a4116f00 0x7ff9a41173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.163+0000 7ff9a359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a4116f00 0x7ff9a41173b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43214/0 (socket says 172.21.15.28:43214) 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.163+0000 7ff9a359e640 1 -- 172.21.15.28:0/2894800011 learned_addr learned my addr 172.21.15.28:0/2894800011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.163+0000 7ff9a259c640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a411b010 0x7ff9a41169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.163+0000 7ff9a2d9d640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9a4110020 0x7ff9a411a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.164+0000 7ff9a2d9d640 1 -- 172.21.15.28:0/2894800011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a411b010 msgr2=0x7ff9a41169c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.164+0000 7ff9a2d9d640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a411b010 0x7ff9a41169c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.164+0000 7ff9a2d9d640 1 -- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a4116f00 msgr2=0x7ff9a41173b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.164+0000 7ff9a2d9d640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a4116f00 0x7ff9a41173b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.164+0000 7ff9a2d9d640 1 -- 172.21.15.28:0/2894800011 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a4079c80 con 0x7ff9a4110020 2024-05-09T08:34:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.164+0000 7ff9a259c640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a411b010 0x7ff9a41169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.164+0000 7ff9a359e640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a4116f00 0x7ff9a41173b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.164+0000 7ff9a2d9d640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9a4110020 0x7ff9a411a8e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009fd0 tx=0x7ff98c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.166+0000 7ff99bfff640 1 -- 172.21.15.28:0/2894800011 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c015020 con 0x7ff9a4110020 2024-05-09T08:34:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.166+0000 7ff99bfff640 1 -- 172.21.15.28:0/2894800011 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff98c011d10 con 0x7ff9a4110020 2024-05-09T08:34:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.166+0000 7ff99bfff640 1 -- 172.21.15.28:0/2894800011 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00f2e0 con 0x7ff9a4110020 2024-05-09T08:34:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.166+0000 7ff9a91d5640 1 -- 172.21.15.28:0/2894800011 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a4079f70 con 0x7ff9a4110020 2024-05-09T08:34:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.166+0000 7ff9a91d5640 1 -- 172.21.15.28:0/2894800011 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a407a470 con 0x7ff9a4110020 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.170+0000 7ff9abfbe740 1 Processor -- start 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.171+0000 7ff9abfbe740 1 -- start start 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.172+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971450 0x55df3c971830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.172+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c971d70 0x55df3c97c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.172+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c97c630 0x55df3c97ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.172+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55df3c8d6340 con 0x55df3c97c630 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.172+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55df3c87e290 con 0x55df3c971450 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.172+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55df3c8d2300 con 0x55df3c971d70 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.173+0000 7ff9a359e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971450 0x55df3c971830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.173+0000 7ff9a359e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971450 0x55df3c971830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:49490/0 (socket says 172.21.15.28:49490) 2024-05-09T08:34:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.173+0000 7ff9a359e640 1 -- 172.21.15.28:0/2949563724 learned_addr learned my addr 172.21.15.28:0/2949563724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.173+0000 7ff9a359e640 1 -- 172.21.15.28:0/2949563724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c971d70 msgr2=0x55df3c97c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.173+0000 7ff9a359e640 1 --2- 172.21.15.28:0/2949563724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c971d70 0x55df3c97c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.173+0000 7ff9a359e640 1 -- 172.21.15.28:0/2949563724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c97c630 msgr2=0x55df3c97ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.173+0000 7ff9a359e640 1 --2- 172.21.15.28:0/2949563724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c97c630 0x55df3c97ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.173+0000 7ff9a359e640 1 -- 172.21.15.28:0/2949563724 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df3c8a28e0 con 0x55df3c971450 2024-05-09T08:34:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.174+0000 7ff9a359e640 1 --2- 172.21.15.28:0/2949563724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971450 0x55df3c971830 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff99c005600 tx=0x7ff99c005630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da13b3519e783be3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.174+0000 7ff99bfff640 1 -- 172.21.15.28:0/2894800011 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff98c018020 con 0x7ff9a4110020 2024-05-09T08:34:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.175+0000 7ff99bfff640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff9900a2f80 0x7ff9900a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.175+0000 7ff99bfff640 1 -- 172.21.15.28:0/2894800011 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9671+0+0 (secure 0 0 0) 0x7ff98c0cee70 con 0x7ff9a4110020 2024-05-09T08:34:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.177+0000 7ff9a2d9d640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff9900a2f80 0x7ff9900a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.178+0000 7ff9997fa640 1 -- 172.21.15.28:0/2949563724 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99c046030 con 0x55df3c971450 2024-05-09T08:34:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.178+0000 7ff9997fa640 1 -- 172.21.15.28:0/2949563724 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff99c004d60 con 0x55df3c971450 2024-05-09T08:34:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.178+0000 7ff9997fa640 1 -- 172.21.15.28:0/2949563724 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99c04d440 con 0x55df3c971450 2024-05-09T08:34:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.181+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2949563724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971450 msgr2=0x55df3c971830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.181+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2949563724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971450 0x55df3c971830 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff99c005600 tx=0x7ff99c005630 comp rx=0 tx=0).stop 2024-05-09T08:34:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.182+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2949563724 shutdown_connections 2024-05-09T08:34:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.182+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2949563724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c97c630 0x55df3c97ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.182+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2949563724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c971d70 0x55df3c97c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.182+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2949563724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971450 0x55df3c971830 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.182+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2949563724 >> 172.21.15.28:0/2949563724 conn(0x55df3c96d960 msgr2=0x55df3c96dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:06.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.185+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2949563724 shutdown_connections 2024-05-09T08:34:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.186+0000 7ff9a2d9d640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff9900a2f80 0x7ff9900a5440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55df3c9729f0 tx=0x7ff98c00bf20 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.186+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2949563724 wait complete. 2024-05-09T08:34:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.186+0000 7ff9abfbe740 1 -- start start 2024-05-09T08:34:06.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.186+0000 7ff9abfbe740 1 Processor -- start 2024-05-09T08:34:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 0x55df3c979510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c979a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c97c630 0x55df3c979f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55df3c97f200 con 0x55df3c971450 2024-05-09T08:34:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55df3c891790 con 0x55df3c971d70 2024-05-09T08:34:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55df3c97f380 con 0x55df3c97c630 2024-05-09T08:34:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9abfbe740 10 client.0 _ll_register_callbacks cb 0x55df3c8cca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9a259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c979a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9a259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c979a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:49502/0 (socket says 172.21.15.28:49502) 2024-05-09T08:34:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.188+0000 7ff9a259c640 1 -- 172.21.15.28:0/2313396761 learned_addr learned my addr 172.21.15.28:0/2313396761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.189+0000 7ff9a259c640 1 -- 172.21.15.28:0/2313396761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c97c630 msgr2=0x55df3c979f90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:34:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.189+0000 7ff9a259c640 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c97c630 0x55df3c979f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.189+0000 7ff9a259c640 1 -- 172.21.15.28:0/2313396761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 msgr2=0x55df3c979510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.189+0000 7ff9a259c640 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 0x55df3c979510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.189+0000 7ff9a259c640 1 -- 172.21.15.28:0/2313396761 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df3c5ea210 con 0x55df3c971d70 2024-05-09T08:34:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.190+0000 7ff9a259c640 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c979a50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff9a406af00 tx=0x7ff99400e490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=212e40ca4286bdb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.190+0000 7ff98a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:34:06.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.190+0000 7ff98a7fc640 1 -- 172.21.15.28:0/2313396761 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994018020 con 0x55df3c971d70 2024-05-09T08:34:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.190+0000 7ff98a7fc640 1 -- 172.21.15.28:0/2313396761 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9671+0+0 (secure 0 0 0) 0x7ff994014400 con 0x55df3c971d70 2024-05-09T08:34:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.192+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2313396761 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55df3c8e3a70 con 0x55df3c971d70 2024-05-09T08:34:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.192+0000 7ff98a7fc640 1 -- 172.21.15.28:0/2313396761 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7ff994014600 con 0x55df3c971d70 2024-05-09T08:34:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.195+0000 7ff9abfbe740 10 client.26267 fetch_fsmap learned FSMap version 63 2024-05-09T08:34:06.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.195+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2313396761 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55df3c6b6620 con 0x55df3c971d70 2024-05-09T08:34:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.196+0000 7ff98a7fc640 1 -- 172.21.15.28:0/2313396761 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55df3c8e3a70 con 0x55df3c971d70 2024-05-09T08:34:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.196+0000 7ff98a7fc640 1 -- 172.21.15.28:0/2313396761 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 63) ==== 8442+0+0 (secure 0 0 0) 0x7ff994025020 con 0x55df3c971d70 2024-05-09T08:34:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.196+0000 7ff9abfbe740 10 client.26267 fetch_fsmap finished waiting for FSMap version 63 2024-05-09T08:34:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.196+0000 7ff9abfbe740 10 client.26267 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:34:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.196+0000 7ff9abfbe740 20 client.26267 populate_metadata read hostname 'smithi028' 2024-05-09T08:34:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.196+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55df3c98c930 0x55df3c98edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.196+0000 7ff9abfbe740 4 client.26267 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:06.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.196+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2313396761 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df3c8dddc0 con 0x55df3c98c930 2024-05-09T08:34:06.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.205+0000 7ff9a359e640 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55df3c98c930 0x55df3c98edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.218+0000 7ff9a359e640 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55df3c98c930 0x55df3c98edb0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b70a6e9fbfa2a332 server_cookie=5c3078e2f3e07fc7 in_seq=0 out_seq=0 2024-05-09T08:34:06.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.218+0000 7ff98a7fc640 10 client.26267 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:06.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.222+0000 7ff98a7fc640 1 -- 172.21.15.28:0/2313396761 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df3c8dddc0 con 0x55df3c98c930 2024-05-09T08:34:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.222+0000 7ff98a7fc640 10 client.26267 handle_command_reply: tid=0 2024-05-09T08:34:06.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.223+0000 7ff9abfbe740 1 client.26267 shutdown 2024-05-09T08:34:06.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.224+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2313396761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 msgr2=0x55df3c979a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:06.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.224+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c979a50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff9a406af00 tx=0x7ff99400e490 comp rx=0 tx=0).stop 2024-05-09T08:34:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.226+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2313396761 shutdown_connections 2024-05-09T08:34:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.226+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55df3c98c930 0x55df3c98edb0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.226+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c97c630 0x55df3c979f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.226+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c979a50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.226+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/2313396761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 0x55df3c979510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.226+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2313396761 >> 172.21.15.28:0/2313396761 conn(0x55df3c96d960 msgr2=0x55df3c973690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:06.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.226+0000 7ff98a7fc640 0 client.26267 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.229+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2313396761 shutdown_connections 2024-05-09T08:34:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.231+0000 7ff9abfbe740 1 -- 172.21.15.28:0/2313396761 wait complete. 2024-05-09T08:34:06.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.232+0000 7ff9abfbe740 20 client.26267 trim_cache size 0 max 16384 2024-05-09T08:34:06.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.269+0000 7ff9abfbe740 1 Processor -- start 2024-05-09T08:34:06.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.270+0000 7ff9abfbe740 1 -- start start 2024-05-09T08:34:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.271+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 0x55df3c97bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.271+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c973e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.271+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c97c630 0x55df3c978670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.271+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff984010350 con 0x55df3c971450 2024-05-09T08:34:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.271+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff9840120d0 con 0x55df3c971d70 2024-05-09T08:34:06.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.271+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff984010ae0 con 0x55df3c97c630 2024-05-09T08:34:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.274+0000 7ff9a359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 0x55df3c97bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.274+0000 7ff9a359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 0x55df3c97bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43254/0 (socket says 172.21.15.28:43254) 2024-05-09T08:34:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.274+0000 7ff9a359e640 1 -- 172.21.15.28:0/3492391344 learned_addr learned my addr 172.21.15.28:0/3492391344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.274+0000 7ff9a359e640 1 --2- 172.21.15.28:0/3492391344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c97c630 0x55df3c978670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.274+0000 7ff9a259c640 1 --2- 172.21.15.28:0/3492391344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c973e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.277+0000 7ff9a259c640 1 -- 172.21.15.28:0/3492391344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c97c630 msgr2=0x55df3c978670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:06.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.277+0000 7ff9a259c640 1 --2- 172.21.15.28:0/3492391344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c97c630 0x55df3c978670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.277+0000 7ff9a259c640 1 -- 172.21.15.28:0/3492391344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 msgr2=0x55df3c97bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.277+0000 7ff9a259c640 1 --2- 172.21.15.28:0/3492391344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 0x55df3c97bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.277+0000 7ff9a259c640 1 -- 172.21.15.28:0/3492391344 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df3c98aee0 con 0x55df3c971d70 2024-05-09T08:34:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.280+0000 7ff9a259c640 1 --2- 172.21.15.28:0/3492391344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c973e10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff99401dcd0 tx=0x7ff994009390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f5561a39fa63bbc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.281+0000 7ff998ff9640 1 -- 172.21.15.28:0/3492391344 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994018040 con 0x55df3c971d70 2024-05-09T08:34:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.281+0000 7ff998ff9640 1 -- 172.21.15.28:0/3492391344 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff99401d8d0 con 0x55df3c971d70 2024-05-09T08:34:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.281+0000 7ff998ff9640 1 -- 172.21.15.28:0/3492391344 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994014d70 con 0x55df3c971d70 2024-05-09T08:34:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.282+0000 7ff9abfbe740 1 -- 172.21.15.28:0/3492391344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 msgr2=0x55df3c973e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.282+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/3492391344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c973e10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff99401dcd0 tx=0x7ff994009390 comp rx=0 tx=0).stop 2024-05-09T08:34:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.286+0000 7ff9abfbe740 1 -- 172.21.15.28:0/3492391344 shutdown_connections 2024-05-09T08:34:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.286+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/3492391344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c97c630 0x55df3c978670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.286+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/3492391344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c971d70 0x55df3c973e10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.286+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/3492391344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971450 0x55df3c97bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.286+0000 7ff9abfbe740 1 -- 172.21.15.28:0/3492391344 >> 172.21.15.28:0/3492391344 conn(0x55df3c96d960 msgr2=0x55df3c96c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:06.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.288+0000 7ff9abfbe740 1 -- 172.21.15.28:0/3492391344 shutdown_connections 2024-05-09T08:34:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.290+0000 7ff9abfbe740 1 -- 172.21.15.28:0/3492391344 wait complete. 2024-05-09T08:34:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.290+0000 7ff9abfbe740 1 -- start start 2024-05-09T08:34:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.290+0000 7ff9abfbe740 1 Processor -- start 2024-05-09T08:34:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c971450 0x55df3c984b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971d70 0x55df3c976a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9abfbe740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c97c630 0x55df3c9770a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff984011a00 con 0x55df3c971d70 2024-05-09T08:34:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff984011270 con 0x55df3c97c630 2024-05-09T08:34:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9abfbe740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff98400e350 con 0x55df3c971450 2024-05-09T08:34:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9abfbe740 10 client.0 _ll_register_callbacks cb 0x7ff994009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9a359e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c971450 0x55df3c984b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971d70 0x55df3c976a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971d70 0x55df3c976a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43264/0 (socket says 172.21.15.28:43264) 2024-05-09T08:34:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9a259c640 1 -- 172.21.15.28:0/339757479 learned_addr learned my addr 172.21.15.28:0/339757479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.291+0000 7ff9a359e640 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c97c630 0x55df3c9770a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.292+0000 7ff9a259c640 1 -- 172.21.15.28:0/339757479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c971450 msgr2=0x55df3c984b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.292+0000 7ff9a259c640 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c971450 0x55df3c984b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.292+0000 7ff9a259c640 1 -- 172.21.15.28:0/339757479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c97c630 msgr2=0x55df3c9770a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.292+0000 7ff9a259c640 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c97c630 0x55df3c9770a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.292+0000 7ff9a259c640 1 -- 172.21.15.28:0/339757479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df3c9758b0 con 0x55df3c971d70 2024-05-09T08:34:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.292+0000 7ff9a259c640 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971d70 0x55df3c976a90 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff99401db60 tx=0x7ff994007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51b253d1de9b12e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.292+0000 7ff98a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:34:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.296+0000 7ff98a7fc640 1 -- 172.21.15.28:0/339757479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994018040 con 0x55df3c971d70 2024-05-09T08:34:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.296+0000 7ff98a7fc640 1 -- 172.21.15.28:0/339757479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9671+0+0 (secure 0 0 0) 0x7ff9940149b0 con 0x55df3c971d70 2024-05-09T08:34:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.296+0000 7ff9abfbe740 1 -- 172.21.15.28:0/339757479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55df3c85b8e0 con 0x55df3c971d70 2024-05-09T08:34:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.297+0000 7ff98a7fc640 1 -- 172.21.15.28:0/339757479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7ff99403c050 con 0x55df3c971d70 2024-05-09T08:34:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.297+0000 7ff9abfbe740 10 client.16776 fetch_fsmap learned FSMap version 63 2024-05-09T08:34:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.297+0000 7ff9abfbe740 1 -- 172.21.15.28:0/339757479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55df3c9f23a0 con 0x55df3c971d70 2024-05-09T08:34:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.297+0000 7ff98a7fc640 1 -- 172.21.15.28:0/339757479 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55df3c85b8e0 con 0x55df3c971d70 2024-05-09T08:34:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.298+0000 7ff98a7fc640 1 -- 172.21.15.28:0/339757479 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 63) ==== 8442+0+0 (secure 0 0 0) 0x7ff994025070 con 0x55df3c971d70 2024-05-09T08:34:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.298+0000 7ff9abfbe740 10 client.16776 fetch_fsmap finished waiting for FSMap version 63 2024-05-09T08:34:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.298+0000 7ff9abfbe740 10 client.16776 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:34:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.298+0000 7ff9abfbe740 20 client.16776 populate_metadata read hostname 'smithi028' 2024-05-09T08:34:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.298+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55df3c9f2af0 0x55df3c9f4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.298+0000 7ff9abfbe740 4 client.16776 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:34:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.298+0000 7ff9abfbe740 1 -- 172.21.15.28:0/339757479 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55df3c8a7870 con 0x55df3c9f2af0 2024-05-09T08:34:06.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.306+0000 7ff9a359e640 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55df3c9f2af0 0x55df3c9f4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.316+0000 7ff9a359e640 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55df3c9f2af0 0x55df3c9f4f10 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d704d899a247827 server_cookie=c7abfa8d38eff10f in_seq=0 out_seq=0 2024-05-09T08:34:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:06.317+0000 7ff98a7fc640 10 client.16776 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:06.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:06 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:34:06.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:06 smithi028 ceph-mon[30119]: osdmap e102: 12 total, 12 up, 12 in 2024-05-09T08:34:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.035+0000 7ff98a7fc640 1 -- 172.21.15.28:0/339757479 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55df3c8a7870 con 0x55df3c9f2af0 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.035+0000 7ff98a7fc640 10 client.16776 handle_command_reply: tid=0 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 client.16776 shutdown 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 -- 172.21.15.28:0/339757479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971d70 msgr2=0x55df3c976a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971d70 0x55df3c976a90 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff99401db60 tx=0x7ff994007000 comp rx=0 tx=0).stop 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 -- 172.21.15.28:0/339757479 shutdown_connections 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55df3c9f2af0 0x55df3c9f4f10 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df3c97c630 0x55df3c9770a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df3c971d70 0x55df3c976a90 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 --2- 172.21.15.28:0/339757479 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55df3c971450 0x55df3c984b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff98a7fc640 0 client.16776 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 -- 172.21.15.28:0/339757479 >> 172.21.15.28:0/339757479 conn(0x55df3c96d960 msgr2=0x55df3c96ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:07.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 -- 172.21.15.28:0/339757479 shutdown_connections 2024-05-09T08:34:07.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 1 -- 172.21.15.28:0/339757479 wait complete. 2024-05-09T08:34:07.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.036+0000 7ff9abfbe740 20 client.16776 trim_cache size 0 max 16384 2024-05-09T08:34:07.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:34:07.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:34:07.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:34:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:34:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.038+0000 7ff9a91d5640 1 -- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff9900a2f80 msgr2=0x7ff9900a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.038+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff9900a2f80 0x7ff9900a5440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55df3c9729f0 tx=0x7ff98c00bf20 comp rx=0 tx=0).stop 2024-05-09T08:34:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.038+0000 7ff9a91d5640 1 -- 172.21.15.28:0/2894800011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9a4110020 msgr2=0x7ff9a411a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.038+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9a4110020 0x7ff9a411a8e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009fd0 tx=0x7ff98c00b870 comp rx=0 tx=0).stop 2024-05-09T08:34:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.039+0000 7ff9a91d5640 1 -- 172.21.15.28:0/2894800011 shutdown_connections 2024-05-09T08:34:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.039+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9a4116f00 0x7ff9a41173b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.039+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a411b010 0x7ff9a41169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.039+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff9900a2f80 0x7ff9900a5440 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.039+0000 7ff9a91d5640 1 --2- 172.21.15.28:0/2894800011 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9a4110020 0x7ff9a411a8e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.039+0000 7ff9a91d5640 1 -- 172.21.15.28:0/2894800011 >> 172.21.15.28:0/2894800011 conn(0x7ff9a406c900 msgr2=0x7ff9a411b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.042+0000 7ff9a91d5640 1 -- 172.21.15.28:0/2894800011 shutdown_connections 2024-05-09T08:34:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:07.042+0000 7ff9a91d5640 1 -- 172.21.15.28:0/2894800011 wait complete. 2024-05-09T08:34:07.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:06 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:34:07.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:06 smithi165 ceph-mon[34540]: osdmap e102: 12 total, 12 up, 12 in 2024-05-09T08:34:07.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:06 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:34:07.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:06 smithi184 ceph-mon[34509]: osdmap e102: 12 total, 12 up, 12 in 2024-05-09T08:34:07.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:07 smithi028 ceph-mon[30119]: pgmap v245: 129 pgs: 129 active+clean; 2.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 527 op/s 2024-05-09T08:34:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:07 smithi165 ceph-mon[34540]: pgmap v245: 129 pgs: 129 active+clean; 2.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 527 op/s 2024-05-09T08:34:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:07 smithi184 ceph-mon[34509]: pgmap v245: 129 pgs: 129 active+clean; 2.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 527 op/s 2024-05-09T08:34:09.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:09 smithi028 ceph-mon[30119]: pgmap v246: 129 pgs: 8 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 94 active+clean; 2.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 457 op/s 2024-05-09T08:34:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:09 smithi165 ceph-mon[34540]: pgmap v246: 129 pgs: 8 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 94 active+clean; 2.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 457 op/s 2024-05-09T08:34:10.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:09 smithi184 ceph-mon[34509]: pgmap v246: 129 pgs: 8 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 94 active+clean; 2.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 457 op/s 2024-05-09T08:34:11.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:11 smithi028 ceph-mon[30119]: pgmap v247: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 456 op/s 2024-05-09T08:34:12.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:11 smithi165 ceph-mon[34540]: pgmap v247: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 456 op/s 2024-05-09T08:34:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:11 smithi184 ceph-mon[34509]: pgmap v247: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 456 op/s 2024-05-09T08:34:13.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:13 smithi028 ceph-mon[30119]: pgmap v248: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 442 op/s 2024-05-09T08:34:14.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:13 smithi165 ceph-mon[34540]: pgmap v248: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 442 op/s 2024-05-09T08:34:14.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:13 smithi184 ceph-mon[34509]: pgmap v248: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 442 op/s 2024-05-09T08:34:15.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:15 smithi028 ceph-mon[30119]: pgmap v249: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 371 op/s 2024-05-09T08:34:16.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:15 smithi165 ceph-mon[34540]: pgmap v249: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 371 op/s 2024-05-09T08:34:16.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:15 smithi184 ceph-mon[34509]: pgmap v249: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 371 op/s 2024-05-09T08:34:17.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:17 smithi028 ceph-mon[30119]: pgmap v250: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 2.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-05-09T08:34:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:17 smithi165 ceph-mon[34540]: pgmap v250: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 2.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-05-09T08:34:18.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:17 smithi184 ceph-mon[34509]: pgmap v250: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 2.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-05-09T08:34:19.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:19 smithi028 ceph-mon[30119]: pgmap v251: 129 pgs: 15 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 87 active+clean; 2.4 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 306 op/s 2024-05-09T08:34:20.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:19 smithi165 ceph-mon[34540]: pgmap v251: 129 pgs: 15 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 87 active+clean; 2.4 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 306 op/s 2024-05-09T08:34:20.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:19 smithi184 ceph-mon[34509]: pgmap v251: 129 pgs: 15 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 87 active+clean; 2.4 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 306 op/s 2024-05-09T08:34:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.180+0000 7f7df8eee640 1 -- 172.21.15.28:0/3164887164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df410d160 msgr2=0x7f7df4114d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.180+0000 7f7df8eee640 1 --2- 172.21.15.28:0/3164887164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df410d160 0x7f7df4114d30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009ca0 tx=0x7f7dec02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.180+0000 7f7df8eee640 1 -- 172.21.15.28:0/3164887164 shutdown_connections 2024-05-09T08:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.180+0000 7f7df8eee640 1 --2- 172.21.15.28:0/3164887164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df410d160 0x7f7df4114d30 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.180+0000 7f7df8eee640 1 --2- 172.21.15.28:0/3164887164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7df4108a40 0x7f7df410ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.180+0000 7f7df8eee640 1 --2- 172.21.15.28:0/3164887164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7df4108090 0x7f7df4108470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.180+0000 7f7df8eee640 1 -- 172.21.15.28:0/3164887164 >> 172.21.15.28:0/3164887164 conn(0x7f7df4075dc0 msgr2=0x7f7df40781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.180+0000 7f7df8eee640 1 -- 172.21.15.28:0/3164887164 shutdown_connections 2024-05-09T08:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.180+0000 7f7df8eee640 1 -- 172.21.15.28:0/3164887164 wait complete. 2024-05-09T08:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.181+0000 7f7df8eee640 1 Processor -- start 2024-05-09T08:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.181+0000 7f7df8eee640 1 -- start start 2024-05-09T08:34:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.181+0000 7f7df8eee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7df4108090 0x7f7df419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.181+0000 7f7df8eee640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7df410d160 0x7f7df419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.181+0000 7f7df8eee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df41a2e70 0x7f7df41a3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.181+0000 7f7df8eee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7df4118270 con 0x7f7df41a2e70 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.181+0000 7f7df8eee640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41180f0 con 0x7f7df410d160 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.181+0000 7f7df8eee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41183f0 con 0x7f7df4108090 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7df4108090 0x7f7df419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7df4108090 0x7f7df419e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:47942/0 (socket says 172.21.15.28:47942) 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df259c640 1 -- 172.21.15.28:0/4243229498 learned_addr learned my addr 172.21.15.28:0/4243229498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df41a2e70 0x7f7df41a3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df1d9b640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7df410d160 0x7f7df419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df259c640 1 -- 172.21.15.28:0/4243229498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7df410d160 msgr2=0x7f7df419eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df259c640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7df410d160 0x7f7df419eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df259c640 1 -- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df41a2e70 msgr2=0x7f7df41a3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df259c640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df41a2e70 0x7f7df41a3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df259c640 1 -- 172.21.15.28:0/4243229498 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df41a3960 con 0x7f7df4108090 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.182+0000 7f7df259c640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7df4108090 0x7f7df419e5b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7de400e9f0 tx=0x7f7de400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.183+0000 7f7de37fe640 1 -- 172.21.15.28:0/4243229498 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de400cd80 con 0x7f7df4108090 2024-05-09T08:34:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.183+0000 7f7de37fe640 1 -- 172.21.15.28:0/4243229498 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de4010910 con 0x7f7df4108090 2024-05-09T08:34:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.183+0000 7f7df8eee640 1 -- 172.21.15.28:0/4243229498 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df41a3c50 con 0x7f7df4108090 2024-05-09T08:34:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.183+0000 7f7de37fe640 1 -- 172.21.15.28:0/4243229498 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de40188f0 con 0x7f7df4108090 2024-05-09T08:34:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.183+0000 7f7df8eee640 1 -- 172.21.15.28:0/4243229498 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df41ab330 con 0x7f7df4108090 2024-05-09T08:34:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7dfbbfa740 1 Processor -- start 2024-05-09T08:34:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7dfbbfa740 1 -- start start 2024-05-09T08:34:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7de37fe640 1 -- 172.21.15.28:0/4243229498 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7de4010ac0 con 0x7f7df4108090 2024-05-09T08:34:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 0x5612eb1d9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5612eb131340 con 0x5612eb1d7630 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5612eb0d9290 con 0x5612eb1cc450 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5612eb12d300 con 0x5612eb1ccd70 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7df2d9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7df2d9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:34192/0 (socket says 172.21.15.28:34192) 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.188+0000 7f7df2d9d640 1 -- 172.21.15.28:0/988653257 learned_addr learned my addr 172.21.15.28:0/988653257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.189+0000 7f7df259c640 1 --2- 172.21.15.28:0/988653257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.189+0000 7f7df2d9d640 1 -- 172.21.15.28:0/988653257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 msgr2=0x5612eb1d70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.189+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/988653257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.189+0000 7f7df2d9d640 1 -- 172.21.15.28:0/988653257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 msgr2=0x5612eb1d9a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:34:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.189+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/988653257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 0x5612eb1d9a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.189+0000 7f7df2d9d640 1 -- 172.21.15.28:0/988653257 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612eb0fd8e0 con 0x5612eb1cc450 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.189+0000 7f7df259c640 1 --2- 172.21.15.28:0/988653257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.189+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/988653257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1cc830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7dec005cb0 tx=0x7f7dec02adc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64bf54748787b6c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.190+0000 7f7de0ff9640 1 -- 172.21.15.28:0/988653257 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec031db0 con 0x5612eb1cc450 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.190+0000 7f7de0ff9640 1 -- 172.21.15.28:0/988653257 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dec036040 con 0x5612eb1cc450 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.190+0000 7f7de0ff9640 1 -- 172.21.15.28:0/988653257 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec004370 con 0x5612eb1cc450 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7de37fe640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7ddc0a2e70 0x7f7ddc0a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/988653257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 msgr2=0x5612eb1cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/988653257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1cc830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7dec005cb0 tx=0x7f7dec02adc0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7de37fe640 1 -- 172.21.15.28:0/4243229498 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9671+0+0 (secure 0 0 0) 0x7f7de40cfcc0 con 0x7f7df4108090 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/988653257 shutdown_connections 2024-05-09T08:34:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/988653257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 0x5612eb1d9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/988653257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/988653257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1cc830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/988653257 >> 172.21.15.28:0/988653257 conn(0x5612eb1c8960 msgr2=0x5612eb1c8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.193+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/988653257 shutdown_connections 2024-05-09T08:34:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.194+0000 7f7df259c640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7ddc0a2e70 0x7f7ddc0a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.194+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/988653257 wait complete. 2024-05-09T08:34:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.194+0000 7f7dfbbfa740 1 -- start start 2024-05-09T08:34:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.195+0000 7f7dfbbfa740 1 Processor -- start 2024-05-09T08:34:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.197+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.197+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.197+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.197+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5612eb1da080 con 0x5612eb1cc450 2024-05-09T08:34:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.197+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5612eb0ec790 con 0x5612eb1d7630 2024-05-09T08:34:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.197+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5612eb1da200 con 0x5612eb1ccd70 2024-05-09T08:34:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.197+0000 7f7dfbbfa740 10 client.0 _ll_register_callbacks cb 0x5612eb127a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df2d9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df2d9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:34200/0 (socket says 172.21.15.28:34200) 2024-05-09T08:34:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df2d9d640 1 -- 172.21.15.28:0/4234375622 learned_addr learned my addr 172.21.15.28:0/4234375622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df1d9b640 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df2d9d640 1 -- 172.21.15.28:0/4234375622 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 msgr2=0x5612eb1d4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df2d9d640 1 -- 172.21.15.28:0/4234375622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 msgr2=0x5612eb1d45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.198+0000 7f7df2d9d640 1 -- 172.21.15.28:0/4234375622 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612eae45210 con 0x5612eb1d7630 2024-05-09T08:34:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.199+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d5040 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7dec033040 tx=0x7f7dec031a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8409295cdb0f224 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.199+0000 7f7dd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.199+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/4234375622 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec003a30 con 0x5612eb1d7630 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.199+0000 7f7df259c640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7ddc0a2e70 0x7f7ddc0a5330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7de40cd530 tx=0x7f7de401f000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.199+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4234375622 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dec036040 con 0x5612eb1d7630 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.199+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/4234375622 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9671+0+0 (secure 0 0 0) 0x7f7dec003bd0 con 0x5612eb1d7630 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.199+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/4234375622 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f7dec05d050 con 0x5612eb1d7630 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.200+0000 7f7dfbbfa740 10 client.26285 fetch_fsmap learned FSMap version 63 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.200+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4234375622 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x5612eaf11620 con 0x5612eb1d7630 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.200+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/4234375622 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dec036040 con 0x5612eb1d7630 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.201+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/4234375622 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 63) ==== 8442+0+0 (secure 0 0 0) 0x7f7dec043070 con 0x5612eb1d7630 2024-05-09T08:34:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.201+0000 7f7dfbbfa740 10 client.26285 fetch_fsmap finished waiting for FSMap version 63 2024-05-09T08:34:20.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.201+0000 7f7dfbbfa740 10 client.26285 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:34:20.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.201+0000 7f7dfbbfa740 20 client.26285 populate_metadata read hostname 'smithi028' 2024-05-09T08:34:20.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.201+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5612eb1f8220 0x5612eb1fa640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.201+0000 7f7dfbbfa740 4 client.26285 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:20.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.201+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4234375622 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612eb13ea70 con 0x5612eb1f8220 2024-05-09T08:34:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.201+0000 7f7df1d9b640 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5612eb1f8220 0x5612eb1fa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.202+0000 7f7df1d9b640 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5612eb1f8220 0x5612eb1fa640 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b64fdfcf92c3ca9 server_cookie=825eff2e4c0c9e91 in_seq=0 out_seq=0 2024-05-09T08:34:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.202+0000 7f7dd9ffb640 10 client.26285 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.204+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/4234375622 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612eb13ea70 con 0x5612eb1f8220 2024-05-09T08:34:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.204+0000 7f7dd9ffb640 10 client.26285 handle_command_reply: tid=0 2024-05-09T08:34:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.204+0000 7f7dfbbfa740 1 client.26285 shutdown 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4234375622 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 msgr2=0x5612eb1d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d5040 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7dec033040 tx=0x7f7dec031a10 comp rx=0 tx=0).stop 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4234375622 shutdown_connections 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5612eb1f8220 0x5612eb1fa640 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d5040 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4234375622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4234375622 >> 172.21.15.28:0/4234375622 conn(0x5612eb1c8960 msgr2=0x5612eb1ce690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dd9ffb640 0 client.26285 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4234375622 shutdown_connections 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4234375622 wait complete. 2024-05-09T08:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.205+0000 7f7dfbbfa740 20 client.26285 trim_cache size 0 max 16384 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.230+0000 7f7dfbbfa740 1 Processor -- start 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.230+0000 7f7dfbbfa740 1 -- start start 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.230+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1d6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.230+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.230+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 0x5612eb1ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.230+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41acd10 con 0x5612eb1d7630 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.230+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41aea90 con 0x5612eb1cc450 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.230+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41ad4a0 con 0x5612eb1ccd70 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.230+0000 7f7df2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 0x5612eb1ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 0x5612eb1ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39344/0 (socket says 172.21.15.28:39344) 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df2d9d640 1 -- 172.21.15.28:0/4009066304 learned_addr learned my addr 172.21.15.28:0/4009066304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df1d9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/4009066304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1d6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df2d9d640 1 -- 172.21.15.28:0/4009066304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 msgr2=0x5612eb1d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/4009066304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df2d9d640 1 -- 172.21.15.28:0/4009066304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 msgr2=0x5612eb1d6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/4009066304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1d6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df2d9d640 1 -- 172.21.15.28:0/4009066304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612eb1c6f00 con 0x5612eb1d7630 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/4009066304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 0x5612eb1ce510 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009a60 tx=0x7f7dec047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3974eb7f704f0f97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7ddbfff640 1 -- 172.21.15.28:0/4009066304 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec03f070 con 0x5612eb1d7630 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7ddbfff640 1 -- 172.21.15.28:0/4009066304 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dec036030 con 0x5612eb1d7630 2024-05-09T08:34:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.231+0000 7f7ddbfff640 1 -- 172.21.15.28:0/4009066304 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec030a10 con 0x5612eb1d7630 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.232+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4009066304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 msgr2=0x5612eb1ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.232+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4009066304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 0x5612eb1ce510 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009a60 tx=0x7f7dec047040 comp rx=0 tx=0).stop 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.232+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4009066304 shutdown_connections 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.232+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4009066304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1d7630 0x5612eb1ce510 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.232+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4009066304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.232+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/4009066304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1cc450 0x5612eb1d6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.232+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4009066304 >> 172.21.15.28:0/4009066304 conn(0x5612eb1c8960 msgr2=0x5612eb1cde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.232+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4009066304 shutdown_connections 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.232+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/4009066304 wait complete. 2024-05-09T08:34:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7dfbbfa740 1 -- start start 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7dfbbfa740 1 Processor -- start 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7dfbbfa740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7df2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41ae3c0 con 0x5612eb1cc450 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41adc30 con 0x5612eb1d7630 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7dfbbfa740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7df4073260 con 0x5612eb1ccd70 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7df2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39350/0 (socket says 172.21.15.28:39350) 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7df2d9d640 1 -- 172.21.15.28:0/2722809908 learned_addr learned my addr 172.21.15.28:0/2722809908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.233+0000 7f7dfbbfa740 10 client.0 _ll_register_callbacks cb 0x7f7dec009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7df1d9b640 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7df2d9d640 1 -- 172.21.15.28:0/2722809908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 msgr2=0x5612eb1d1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7df2d9d640 1 -- 172.21.15.28:0/2722809908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 msgr2=0x5612eb1d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7df2d9d640 1 -- 172.21.15.28:0/2722809908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612eb25d8d0 con 0x5612eb1cc450 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7df2d9d640 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d1450 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7dec03e040 tx=0x7f7dec009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537b10959b10a7f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7dd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/2722809908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec03f040 con 0x5612eb1cc450 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/2722809908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9671+0+0 (secure 0 0 0) 0x7f7dec04a070 con 0x5612eb1cc450 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/2722809908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dec036030 con 0x5612eb1cc450 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/2722809908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f7dec0319b0 con 0x5612eb1cc450 2024-05-09T08:34:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7dfbbfa740 10 client.16788 fetch_fsmap learned FSMap version 63 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.234+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/2722809908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x5612eb25eb60 con 0x5612eb1cc450 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.235+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/2722809908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dec040040 con 0x5612eb1cc450 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.235+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/2722809908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 63) ==== 8442+0+0 (secure 0 0 0) 0x7f7dec043070 con 0x5612eb1cc450 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.236+0000 7f7dfbbfa740 10 client.16788 fetch_fsmap finished waiting for FSMap version 63 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.236+0000 7f7dfbbfa740 10 client.16788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.236+0000 7f7dfbbfa740 20 client.16788 populate_metadata read hostname 'smithi028' 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.236+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5612eb25f220 0x5612eb261640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.236+0000 7f7dfbbfa740 4 client.16788 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.236+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/2722809908 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7dec036030 con 0x5612eb25f220 2024-05-09T08:34:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.236+0000 7f7df1d9b640 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5612eb25f220 0x5612eb261640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.237+0000 7f7df1d9b640 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5612eb25f220 0x5612eb261640 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bb6bf1a46a11e4 server_cookie=1cce63ecf0cb88b5 in_seq=0 out_seq=0 2024-05-09T08:34:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:20.237+0000 7f7dd9ffb640 10 client.16788 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:21.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:21 smithi028 ceph-mon[30119]: pgmap v252: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 2.4 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 283 op/s 2024-05-09T08:34:22.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:21 smithi165 ceph-mon[34540]: pgmap v252: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 2.4 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 283 op/s 2024-05-09T08:34:22.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:21 smithi184 ceph-mon[34509]: pgmap v252: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 2.4 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 283 op/s 2024-05-09T08:34:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.627+0000 7f7dd9ffb640 1 -- 172.21.15.28:0/2722809908 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7dec036030 con 0x5612eb25f220 2024-05-09T08:34:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.627+0000 7f7dd9ffb640 10 client.16788 handle_command_reply: tid=0 2024-05-09T08:34:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.627+0000 7f7dfbbfa740 1 client.16788 shutdown 2024-05-09T08:34:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.627+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/2722809908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 msgr2=0x5612eb1d1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.627+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d1450 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7dec03e040 tx=0x7f7dec009540 comp rx=0 tx=0).stop 2024-05-09T08:34:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/2722809908 shutdown_connections 2024-05-09T08:34:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5612eb25f220 0x5612eb261640 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dd9ffb640 0 client.16788 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5612eb1d7630 0x5612eb1d1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5612eb1ccd70 0x5612eb1d1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dfbbfa740 1 --2- 172.21.15.28:0/2722809908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612eb1cc450 0x5612eb1d1450 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/2722809908 >> 172.21.15.28:0/2722809908 conn(0x5612eb1c8960 msgr2=0x5612eb1ce690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/2722809908 shutdown_connections 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dfbbfa740 1 -- 172.21.15.28:0/2722809908 wait complete. 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.628+0000 7f7dfbbfa740 20 client.16788 trim_cache size 0 max 16384 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:34:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:34:22.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 -- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7ddc0a2e70 msgr2=0x7f7ddc0a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7ddc0a2e70 0x7f7ddc0a5330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7de40cd530 tx=0x7f7de401f000 comp rx=0 tx=0).stop 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 -- 172.21.15.28:0/4243229498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7df4108090 msgr2=0x7f7df419e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7df4108090 0x7f7df419e5b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7de400e9f0 tx=0x7f7de400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 -- 172.21.15.28:0/4243229498 shutdown_connections 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df41a2e70 0x7f7df41a3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7df410d160 0x7f7df419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7ddc0a2e70 0x7f7ddc0a5330 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 --2- 172.21.15.28:0/4243229498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7df4108090 0x7f7df419e5b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 -- 172.21.15.28:0/4243229498 >> 172.21.15.28:0/4243229498 conn(0x7f7df4075dc0 msgr2=0x7f7df4076730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:22.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 -- 172.21.15.28:0/4243229498 shutdown_connections 2024-05-09T08:34:22.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:22.630+0000 7f7df8eee640 1 -- 172.21.15.28:0/4243229498 wait complete. 2024-05-09T08:34:24.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:23 smithi165 ceph-mon[34540]: pgmap v253: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 2.4 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 175 op/s 2024-05-09T08:34:24.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:23 smithi184 ceph-mon[34509]: pgmap v253: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 2.4 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 175 op/s 2024-05-09T08:34:24.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:23 smithi028 ceph-mon[30119]: pgmap v253: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 2.4 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 175 op/s 2024-05-09T08:34:26.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:25 smithi165 ceph-mon[34540]: pgmap v254: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 2.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.4 MiB/s wr, 199 op/s 2024-05-09T08:34:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:25 smithi184 ceph-mon[34509]: pgmap v254: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 2.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.4 MiB/s wr, 199 op/s 2024-05-09T08:34:26.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:25 smithi028 ceph-mon[30119]: pgmap v254: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 2.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.4 MiB/s wr, 199 op/s 2024-05-09T08:34:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:27 smithi165 ceph-mon[34540]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 2.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.4 MiB/s wr, 210 op/s 2024-05-09T08:34:28.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:27 smithi184 ceph-mon[34509]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 2.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.4 MiB/s wr, 210 op/s 2024-05-09T08:34:28.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:27 smithi028 ceph-mon[30119]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 2.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.4 MiB/s wr, 210 op/s 2024-05-09T08:34:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:29 smithi165 ceph-mon[34540]: pgmap v256: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 2.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.6 MiB/s wr, 200 op/s 2024-05-09T08:34:30.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:29 smithi184 ceph-mon[34509]: pgmap v256: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 2.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.6 MiB/s wr, 200 op/s 2024-05-09T08:34:30.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:29 smithi028 ceph-mon[30119]: pgmap v256: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 2.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.6 MiB/s wr, 200 op/s 2024-05-09T08:34:31.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:31 smithi165 ceph-mon[34540]: pgmap v257: 129 pgs: 129 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.0 MiB/s wr, 195 op/s 2024-05-09T08:34:31.853 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:31 smithi165 ceph-mon[34540]: osdmap e103: 12 total, 12 up, 12 in 2024-05-09T08:34:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:31 smithi184 ceph-mon[34509]: pgmap v257: 129 pgs: 129 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.0 MiB/s wr, 195 op/s 2024-05-09T08:34:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:31 smithi184 ceph-mon[34509]: osdmap e103: 12 total, 12 up, 12 in 2024-05-09T08:34:31.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:31 smithi028 ceph-mon[30119]: pgmap v257: 129 pgs: 129 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.0 MiB/s wr, 195 op/s 2024-05-09T08:34:31.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:31 smithi028 ceph-mon[30119]: osdmap e103: 12 total, 12 up, 12 in 2024-05-09T08:34:33.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:33 smithi165 ceph-mon[34540]: pgmap v259: 129 pgs: 129 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.2 MiB/s wr, 173 op/s 2024-05-09T08:34:33.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:33 smithi184 ceph-mon[34509]: pgmap v259: 129 pgs: 129 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.2 MiB/s wr, 173 op/s 2024-05-09T08:34:33.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:33 smithi028 ceph-mon[30119]: pgmap v259: 129 pgs: 129 active+clean; 2.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.2 MiB/s wr, 173 op/s 2024-05-09T08:34:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 -- 172.21.15.28:0/1627847341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8101120 msgr2=0x7fcfe810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/1627847341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8101120 0x7fcfe810d5e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009cd0 tx=0x7fcfe002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 -- 172.21.15.28:0/1627847341 shutdown_connections 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/1627847341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8101120 0x7fcfe810d5e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/1627847341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfe8100760 0x7fcfe8100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/1627847341 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfe8106720 0x7fcfe8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 -- 172.21.15.28:0/1627847341 >> 172.21.15.28:0/1627847341 conn(0x7fcfe80778b0 msgr2=0x7fcfe8077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 -- 172.21.15.28:0/1627847341 shutdown_connections 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 -- 172.21.15.28:0/1627847341 wait complete. 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.741+0000 7fcfedcb2640 1 Processor -- start 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfedcb2640 1 -- start start 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfedcb2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8100760 0x7fcfe8196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfedcb2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfe8101120 0x7fcfe8197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfedcb2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfe8106720 0x7fcfe819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfedcb2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe8110b80 con 0x7fcfe8100760 2024-05-09T08:34:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfedcb2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe8110a00 con 0x7fcfe8106720 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfedcb2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe8110d00 con 0x7fcfe8101120 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8100760 0x7fcfe8196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8100760 0x7fcfe8196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42474/0 (socket says 172.21.15.28:42474) 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe77fe640 1 -- 172.21.15.28:0/2425538117 learned_addr learned my addr 172.21.15.28:0/2425538117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfe8101120 0x7fcfe8197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe7fff640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfe8106720 0x7fcfe819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe77fe640 1 -- 172.21.15.28:0/2425538117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfe8101120 msgr2=0x7fcfe8197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfe8101120 0x7fcfe8197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe77fe640 1 -- 172.21.15.28:0/2425538117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfe8106720 msgr2=0x7fcfe819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfe8106720 0x7fcfe819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.742+0000 7fcfe77fe640 1 -- 172.21.15.28:0/2425538117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe819bfd0 con 0x7fcfe8100760 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.743+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8100760 0x7fcfe8196fe0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b4a0 tx=0x7fcfd800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.743+0000 7fcfe4ff9640 1 -- 172.21.15.28:0/2425538117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8014070 con 0x7fcfe8100760 2024-05-09T08:34:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.743+0000 7fcfedcb2640 1 -- 172.21.15.28:0/2425538117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe819c2c0 con 0x7fcfe8100760 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.743+0000 7fcfe4ff9640 1 -- 172.21.15.28:0/2425538117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd8010af0 con 0x7fcfe8100760 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.743+0000 7fcfe4ff9640 1 -- 172.21.15.28:0/2425538117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8017d10 con 0x7fcfe8100760 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.743+0000 7fcfedcb2640 1 -- 172.21.15.28:0/2425538117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe81a3b50 con 0x7fcfe8100760 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.745+0000 7fcfe4ff9640 1 -- 172.21.15.28:0/2425538117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcfd8017070 con 0x7fcfe8100760 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.745+0000 7fcff09bd740 1 Processor -- start 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcff09bd740 1 -- start start 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bcc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5bcec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcff09bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e2b5b24340 con 0x55e2b5bc1f80 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcff09bd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e2b5acc290 con 0x55e2b5bc1660 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcff09bd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e2b5b20300 con 0x55e2b5bcc840 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe4ff9640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcfd00a2da0 0x7fcfd00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bcc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bcc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42484/0 (socket says 172.21.15.28:42484) 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe7fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe77fe640 1 -- 172.21.15.28:0/3854803609 learned_addr learned my addr 172.21.15.28:0/3854803609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe7fff640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcfd00a2da0 0x7fcfd00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/3854803609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5bcec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe4ff9640 1 -- 172.21.15.28:0/2425538117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9567+0+0 (secure 0 0 0) 0x7fcfd80cea10 con 0x7fcfe8100760 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe77fe640 1 -- 172.21.15.28:0/3854803609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 msgr2=0x55e2b5bcec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/3854803609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5bcec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe77fe640 1 -- 172.21.15.28:0/3854803609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 msgr2=0x55e2b5bc1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/3854803609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe77fe640 1 -- 172.21.15.28:0/3854803609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2b5af08e0 con 0x55e2b5bc1f80 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.746+0000 7fcfe7fff640 1 --2- 172.21.15.28:0/3854803609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:35.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/3854803609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bcc300 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80176e0 tx=0x7fcfd8017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcfd752ed450f75 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcfe7fff640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcfd00a2da0 0x7fcfd00a5260 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0005e30 tx=0x7fcfe0032fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcfcdffb640 1 -- 172.21.15.28:0/3854803609 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd801e070 con 0x55e2b5bc1f80 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcfcdffb640 1 -- 172.21.15.28:0/3854803609 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd809efe0 con 0x55e2b5bc1f80 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcfcdffb640 1 -- 172.21.15.28:0/3854803609 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8017430 con 0x55e2b5bc1f80 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcff09bd740 1 -- 172.21.15.28:0/3854803609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 msgr2=0x55e2b5bcc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcff09bd740 1 --2- 172.21.15.28:0/3854803609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bcc300 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80176e0 tx=0x7fcfd8017730 comp rx=0 tx=0).stop 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcff09bd740 1 -- 172.21.15.28:0/3854803609 shutdown_connections 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcff09bd740 1 --2- 172.21.15.28:0/3854803609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5bcec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.747+0000 7fcff09bd740 1 --2- 172.21.15.28:0/3854803609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bcc300 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 --2- 172.21.15.28:0/3854803609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 -- 172.21.15.28:0/3854803609 >> 172.21.15.28:0/3854803609 conn(0x55e2b5bbdb70 msgr2=0x55e2b5bbdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 -- 172.21.15.28:0/3854803609 shutdown_connections 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 -- 172.21.15.28:0/3854803609 wait complete. 2024-05-09T08:34:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 -- start start 2024-05-09T08:34:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 Processor -- start 2024-05-09T08:34:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bc9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5bca280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e2b5bcf470 con 0x55e2b5bc1f80 2024-05-09T08:34:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e2b5adf790 con 0x55e2b5bc1660 2024-05-09T08:34:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e2b5bcf5f0 con 0x55e2b5bcc840 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcff09bd740 10 client.0 _ll_register_callbacks cb 0x55e2b5b1aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcfe6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bc9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcfe6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bc9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42486/0 (socket says 172.21.15.28:42486) 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.748+0000 7fcfe6ffd640 1 -- 172.21.15.28:0/1525155607 learned_addr learned my addr 172.21.15.28:0/1525155607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfe7fff640 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5bca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfe77fe640 1 -- 172.21.15.28:0/1525155607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 msgr2=0x55e2b5bca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5bca280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfe77fe640 1 -- 172.21.15.28:0/1525155607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 msgr2=0x55e2b5bc9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bc9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfe77fe640 1 -- 172.21.15.28:0/1525155607 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2b5838210 con 0x55e2b5bc1660 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bc9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc9800 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55e2b5bc2c00 tx=0x7fcfd80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=413659a92998af57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1525155607 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd801e050 con 0x55e2b5bc1660 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcff09bd740 1 -- 172.21.15.28:0/1525155607 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfd809efe0 con 0x55e2b5bc1660 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1525155607 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9567+0+0 (secure 0 0 0) 0x7fcfd80cbfd0 con 0x55e2b5bc1660 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1525155607 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fcfd80a3200 con 0x55e2b5bc1660 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcff09bd740 10 client.26303 fetch_fsmap learned FSMap version 63 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcff09bd740 1 -- 172.21.15.28:0/1525155607 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55e2b5904620 con 0x55e2b5bc1660 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.749+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1525155607 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd809efe0 con 0x55e2b5bc1660 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1525155607 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 63) ==== 8442+0+0 (secure 0 0 0) 0x7fcfd80a3410 con 0x55e2b5bc1660 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcff09bd740 10 client.26303 fetch_fsmap finished waiting for FSMap version 63 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcff09bd740 10 client.26303 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcff09bd740 20 client.26303 populate_metadata read hostname 'smithi028' 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e2b5bdc4a0 0x55e2b5bde890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcff09bd740 4 client.26303 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcff09bd740 1 -- 172.21.15.28:0/1525155607 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2b5b31a70 con 0x55e2b5bdc4a0 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e2b5bdc4a0 0x55e2b5bde890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcfc6ffd640 10 client.26303 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.750+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e2b5bdc4a0 0x55e2b5bde890 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cb9da51a872d9e3 server_cookie=2aa0e73ba8d6f8b9 in_seq=0 out_seq=0 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1525155607 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e2b5b31a70 con 0x55e2b5bdc4a0 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcfc6ffd640 10 client.26303 handle_command_reply: tid=0 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 client.26303 shutdown 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 -- 172.21.15.28:0/1525155607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 msgr2=0x55e2b5bc9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc9800 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55e2b5bc2c00 tx=0x7fcfd80046c0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 -- 172.21.15.28:0/1525155607 shutdown_connections 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e2b5bdc4a0 0x55e2b5bde890 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5bca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bc9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1525155607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bc9800 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcfc6ffd640 0 client.26303 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 -- 172.21.15.28:0/1525155607 >> 172.21.15.28:0/1525155607 conn(0x55e2b5bbdb70 msgr2=0x55e2b5bc38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 -- 172.21.15.28:0/1525155607 shutdown_connections 2024-05-09T08:34:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 1 -- 172.21.15.28:0/1525155607 wait complete. 2024-05-09T08:34:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.753+0000 7fcff09bd740 20 client.26303 trim_cache size 0 max 16384 2024-05-09T08:34:35.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.785+0000 7fcff09bd740 1 Processor -- start 2024-05-09T08:34:35.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.786+0000 7fcff09bd740 1 -- start start 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.786+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bcbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.786+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.786+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5be3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.786+0000 7fcff09bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81acce0 con 0x55e2b5bc1f80 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.786+0000 7fcff09bd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81aea60 con 0x55e2b5bc1660 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.786+0000 7fcff09bd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81ad470 con 0x55e2b5bcc840 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.786+0000 7fcfe6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.787+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bcbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.787+0000 7fcfe6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42492/0 (socket says 172.21.15.28:42492) 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.787+0000 7fcfe6ffd640 1 -- 172.21.15.28:0/2199771814 learned_addr learned my addr 172.21.15.28:0/2199771814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.787+0000 7fcfe6ffd640 1 -- 172.21.15.28:0/2199771814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 msgr2=0x55e2b5be3e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.787+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/2199771814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5be3e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.787+0000 7fcfe6ffd640 1 -- 172.21.15.28:0/2199771814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 msgr2=0x55e2b5bcbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.787+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/2199771814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bcbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.787+0000 7fcfe6ffd640 1 -- 172.21.15.28:0/2199771814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2b5bc26f0 con 0x55e2b5bc1f80 2024-05-09T08:34:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.787+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/2199771814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd8330 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x55e2b5bc9060 tx=0x7fcfdc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec68dca4ff3bfb54 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.788+0000 7fcfcd7fa640 1 -- 172.21.15.28:0/2199771814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc0202f0 con 0x55e2b5bc1f80 2024-05-09T08:34:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.788+0000 7fcfcd7fa640 1 -- 172.21.15.28:0/2199771814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc010430 con 0x55e2b5bc1f80 2024-05-09T08:34:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.788+0000 7fcfcd7fa640 1 -- 172.21.15.28:0/2199771814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc01e650 con 0x55e2b5bc1f80 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 -- 172.21.15.28:0/2199771814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 msgr2=0x55e2b5bd8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 --2- 172.21.15.28:0/2199771814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd8330 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x55e2b5bc9060 tx=0x7fcfdc005f90 comp rx=0 tx=0).stop 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 -- 172.21.15.28:0/2199771814 shutdown_connections 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 --2- 172.21.15.28:0/2199771814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bcc840 0x55e2b5be3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 --2- 172.21.15.28:0/2199771814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd8330 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 --2- 172.21.15.28:0/2199771814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bc1660 0x55e2b5bcbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 -- 172.21.15.28:0/2199771814 >> 172.21.15.28:0/2199771814 conn(0x55e2b5bbdb70 msgr2=0x55e2b5bbaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 -- 172.21.15.28:0/2199771814 shutdown_connections 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 -- 172.21.15.28:0/2199771814 wait complete. 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.789+0000 7fcff09bd740 1 -- start start 2024-05-09T08:34:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcff09bd740 1 Processor -- start 2024-05-09T08:34:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1660 0x55e2b5bd3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcff09bd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bcc840 0x55e2b5bd45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcff09bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81ae390 con 0x55e2b5bc1660 2024-05-09T08:34:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcff09bd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81adc00 con 0x55e2b5bcc840 2024-05-09T08:34:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcff09bd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81aace0 con 0x55e2b5bc1f80 2024-05-09T08:34:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcff09bd740 10 client.0 _ll_register_callbacks cb 0x7fcfe0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1660 0x55e2b5bd3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1660 0x55e2b5bd3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42506/0 (socket says 172.21.15.28:42506) 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe77fe640 1 -- 172.21.15.28:0/1635978954 learned_addr learned my addr 172.21.15.28:0/1635978954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe7fff640 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bcc840 0x55e2b5bd45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe77fe640 1 -- 172.21.15.28:0/1635978954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bc1f80 msgr2=0x55e2b5bd3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe77fe640 1 -- 172.21.15.28:0/1635978954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bcc840 msgr2=0x55e2b5bd45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bcc840 0x55e2b5bd45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe77fe640 1 -- 172.21.15.28:0/1635978954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2b5bd4d70 con 0x55e2b5bc1660 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe7fff640 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bcc840 0x55e2b5bd45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.790+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.791+0000 7fcfe77fe640 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1660 0x55e2b5bd3a30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80976b0 tx=0x7fcfd8017360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74d463c274a5e44 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.791+0000 7fcfc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.791+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1635978954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd801e070 con 0x55e2b5bc1660 2024-05-09T08:34:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.791+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1635978954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9567+0+0 (secure 0 0 0) 0x7fcfd809a130 con 0x55e2b5bc1660 2024-05-09T08:34:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.791+0000 7fcff09bd740 1 -- 172.21.15.28:0/1635978954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfdc010430 con 0x55e2b5bc1660 2024-05-09T08:34:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.791+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1635978954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fcfd809efe0 con 0x55e2b5bc1660 2024-05-09T08:34:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.791+0000 7fcff09bd740 10 client.16806 fetch_fsmap learned FSMap version 63 2024-05-09T08:34:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.791+0000 7fcff09bd740 1 -- 172.21.15.28:0/1635978954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x55e2b5bd4d70 con 0x55e2b5bc1660 2024-05-09T08:34:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.793+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1635978954 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc010430 con 0x55e2b5bc1660 2024-05-09T08:34:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.793+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1635978954 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 63) ==== 8442+0+0 (secure 0 0 0) 0x7fcfd8079b00 con 0x55e2b5bc1660 2024-05-09T08:34:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.794+0000 7fcff09bd740 10 client.16806 fetch_fsmap finished waiting for FSMap version 63 2024-05-09T08:34:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.794+0000 7fcff09bd740 10 client.16806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:34:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.794+0000 7fcff09bd740 20 client.16806 populate_metadata read hostname 'smithi028' 2024-05-09T08:34:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.794+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e2b5bbc4e0 0x55e2b5bbc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.794+0000 7fcff09bd740 4 client.16806 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:34:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.794+0000 7fcff09bd740 1 -- 172.21.15.28:0/1635978954 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e2b5b2bdc0 con 0x55e2b5bbc4e0 2024-05-09T08:34:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.797+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e2b5bbc4e0 0x55e2b5bbc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:35.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.803+0000 7fcfe6ffd640 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e2b5bbc4e0 0x55e2b5bbc8f0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79a8fd2d24693cb1 server_cookie=da36014a3648ef5b in_seq=0 out_seq=0 2024-05-09T08:34:35.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:35.803+0000 7fcfc6ffd640 10 client.16806 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:35.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:35 smithi165 ceph-mon[34540]: pgmap v260: 129 pgs: 129 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.6 MiB/s wr, 164 op/s 2024-05-09T08:34:35.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:35 smithi184 ceph-mon[34509]: pgmap v260: 129 pgs: 129 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.6 MiB/s wr, 164 op/s 2024-05-09T08:34:35.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:35 smithi028 ceph-mon[30119]: pgmap v260: 129 pgs: 129 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.6 MiB/s wr, 164 op/s 2024-05-09T08:34:36.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:36 smithi165 ceph-mon[34540]: osdmap e104: 12 total, 12 up, 12 in 2024-05-09T08:34:36.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:36 smithi184 ceph-mon[34509]: osdmap e104: 12 total, 12 up, 12 in 2024-05-09T08:34:36.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:36 smithi028 ceph-mon[30119]: osdmap e104: 12 total, 12 up, 12 in 2024-05-09T08:34:37.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:37 smithi165 ceph-mon[34540]: pgmap v262: 129 pgs: 129 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.1 MiB/s wr, 177 op/s 2024-05-09T08:34:37.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:37 smithi184 ceph-mon[34509]: pgmap v262: 129 pgs: 129 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.1 MiB/s wr, 177 op/s 2024-05-09T08:34:37.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:37 smithi028 ceph-mon[30119]: pgmap v262: 129 pgs: 129 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.1 MiB/s wr, 177 op/s 2024-05-09T08:34:39.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:39 smithi165 ceph-mon[34540]: pgmap v263: 129 pgs: 129 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 364 op/s 2024-05-09T08:34:39.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:39 smithi184 ceph-mon[34509]: pgmap v263: 129 pgs: 129 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 364 op/s 2024-05-09T08:34:39.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:39 smithi028 ceph-mon[30119]: pgmap v263: 129 pgs: 129 active+clean; 2.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 364 op/s 2024-05-09T08:34:41.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:41 smithi165 ceph-mon[34540]: pgmap v264: 129 pgs: 129 active+clean; 2.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 1.06k op/s 2024-05-09T08:34:41.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:41 smithi184 ceph-mon[34509]: pgmap v264: 129 pgs: 129 active+clean; 2.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 1.06k op/s 2024-05-09T08:34:41.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:41 smithi028 ceph-mon[30119]: pgmap v264: 129 pgs: 129 active+clean; 2.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 1.06k op/s 2024-05-09T08:34:43.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:43 smithi165 ceph-mon[34540]: pgmap v265: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 1.29k op/s 2024-05-09T08:34:43.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:43 smithi184 ceph-mon[34509]: pgmap v265: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 1.29k op/s 2024-05-09T08:34:43.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:43 smithi028 ceph-mon[30119]: pgmap v265: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 1.29k op/s 2024-05-09T08:34:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcfc6ffd640 1 -- 172.21.15.28:0/1635978954 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e2b5b2bdc0 con 0x55e2b5bbc4e0 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcfc6ffd640 10 client.16806 handle_command_reply: tid=0 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcff09bd740 1 client.16806 shutdown 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcff09bd740 1 -- 172.21.15.28:0/1635978954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1660 msgr2=0x55e2b5bd3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1660 0x55e2b5bd3a30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80976b0 tx=0x7fcfd8017360 comp rx=0 tx=0).stop 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcff09bd740 1 -- 172.21.15.28:0/1635978954 shutdown_connections 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e2b5bbc4e0 0x55e2b5bbc8f0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e2b5bcc840 0x55e2b5bd45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e2b5bc1f80 0x55e2b5bd3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcff09bd740 1 --2- 172.21.15.28:0/1635978954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2b5bc1660 0x55e2b5bd3a30 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcff09bd740 1 -- 172.21.15.28:0/1635978954 >> 172.21.15.28:0/1635978954 conn(0x55e2b5bbdb70 msgr2=0x55e2b5bc2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.154+0000 7fcfc6ffd640 0 client.16806 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcff09bd740 1 -- 172.21.15.28:0/1635978954 shutdown_connections 2024-05-09T08:34:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcff09bd740 1 -- 172.21.15.28:0/1635978954 wait complete. 2024-05-09T08:34:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcff09bd740 20 client.16806 trim_cache size 0 max 16384 2024-05-09T08:34:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:34:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:34:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:34:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 -- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcfd00a2da0 msgr2=0x7fcfd00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcfd00a2da0 0x7fcfd00a5260 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0005e30 tx=0x7fcfe0032fd0 comp rx=0 tx=0).stop 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 -- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8100760 msgr2=0x7fcfe8196fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8100760 0x7fcfe8196fe0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b4a0 tx=0x7fcfd800b960 comp rx=0 tx=0).stop 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 -- 172.21.15.28:0/2425538117 shutdown_connections 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfe8106720 0x7fcfe819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcfd00a2da0 0x7fcfd00a5260 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfe8101120 0x7fcfe8197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 --2- 172.21.15.28:0/2425538117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe8100760 0x7fcfe8196fe0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.155+0000 7fcfedcb2640 1 -- 172.21.15.28:0/2425538117 >> 172.21.15.28:0/2425538117 conn(0x7fcfe80778b0 msgr2=0x7fcfe8103160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.156+0000 7fcfedcb2640 1 -- 172.21.15.28:0/2425538117 shutdown_connections 2024-05-09T08:34:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:45.156+0000 7fcfedcb2640 1 -- 172.21.15.28:0/2425538117 wait complete. 2024-05-09T08:34:45.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:45 smithi165 ceph-mon[34540]: pgmap v266: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 1.99k op/s 2024-05-09T08:34:45.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:45 smithi184 ceph-mon[34509]: pgmap v266: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 1.99k op/s 2024-05-09T08:34:45.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:45 smithi028 ceph-mon[30119]: pgmap v266: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 1.99k op/s 2024-05-09T08:34:47.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:47 smithi165 ceph-mon[34540]: pgmap v267: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 2.32k op/s 2024-05-09T08:34:47.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:47 smithi184 ceph-mon[34509]: pgmap v267: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 2.32k op/s 2024-05-09T08:34:47.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:47 smithi028 ceph-mon[30119]: pgmap v267: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 2.32k op/s 2024-05-09T08:34:49.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:49 smithi165 ceph-mon[34540]: pgmap v268: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 2.18k op/s 2024-05-09T08:34:49.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:49 smithi184 ceph-mon[34509]: pgmap v268: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 2.18k op/s 2024-05-09T08:34:49.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:49 smithi028 ceph-mon[30119]: pgmap v268: 129 pgs: 129 active+clean; 2.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 2.18k op/s 2024-05-09T08:34:51.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:51 smithi165 ceph-mon[34540]: pgmap v269: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 2.21k op/s 2024-05-09T08:34:51.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:51 smithi184 ceph-mon[34509]: pgmap v269: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 2.21k op/s 2024-05-09T08:34:51.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:51 smithi028 ceph-mon[30119]: pgmap v269: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 2.21k op/s 2024-05-09T08:34:53.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:53 smithi165 ceph-mon[34540]: pgmap v270: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 1.65k op/s 2024-05-09T08:34:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:53 smithi184 ceph-mon[34509]: pgmap v270: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 1.65k op/s 2024-05-09T08:34:53.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:53 smithi028 ceph-mon[30119]: pgmap v270: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 1.65k op/s 2024-05-09T08:34:55.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:55 smithi165 ceph-mon[34540]: pgmap v271: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 1.53k op/s 2024-05-09T08:34:55.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:55 smithi184 ceph-mon[34509]: pgmap v271: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 1.53k op/s 2024-05-09T08:34:55.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:55 smithi028 ceph-mon[30119]: pgmap v271: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 1.53k op/s 2024-05-09T08:34:56.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:56 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:34:56.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:56 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:34:57.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:56 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:34:57.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:57 smithi184 ceph-mon[34509]: pgmap v272: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 977 op/s 2024-05-09T08:34:57.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:57 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:34:57.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:57 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:34:57.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:57 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:34:57.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:57 smithi028 ceph-mon[30119]: pgmap v272: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 977 op/s 2024-05-09T08:34:57.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:57 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:34:57.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:57 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:34:57.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:57 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:34:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:57 smithi165 ceph-mon[34540]: pgmap v272: 129 pgs: 129 active+clean; 2.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 977 op/s 2024-05-09T08:34:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:57 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:34:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:57 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:34:58.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:57 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:34:58.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.296+0000 7f7790241640 1 -- 172.21.15.28:0/4020484242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788068ac0 msgr2=0x7f7788100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:58.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.296+0000 7f7790241640 1 --2- 172.21.15.28:0/4020484242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788068ac0 0x7f7788100630 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7778013080 tx=0x7f777803aee0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.297+0000 7f7790241640 1 -- 172.21.15.28:0/4020484242 shutdown_connections 2024-05-09T08:34:58.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.297+0000 7f7790241640 1 --2- 172.21.15.28:0/4020484242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7788104ae0 0x7f7788104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.297+0000 7f7790241640 1 --2- 172.21.15.28:0/4020484242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788068ac0 0x7f7788100630 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.297+0000 7f7790241640 1 --2- 172.21.15.28:0/4020484242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7788068650 0x7f77881000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.297+0000 7f7790241640 1 -- 172.21.15.28:0/4020484242 >> 172.21.15.28:0/4020484242 conn(0x7f7788076cc0 msgr2=0x7f77880fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:58.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.297+0000 7f7790241640 1 -- 172.21.15.28:0/4020484242 shutdown_connections 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.297+0000 7f7790241640 1 -- 172.21.15.28:0/4020484242 wait complete. 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.298+0000 7f7790241640 1 Processor -- start 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.298+0000 7f7790241640 1 -- start start 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.298+0000 7f7790241640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7788068650 0x7f7788195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.298+0000 7f7790241640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788068ac0 0x7f7788196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.298+0000 7f7790241640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7788104ae0 0x7f778819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f778dfdd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7788068650 0x7f7788195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f778dfdd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7788068650 0x7f7788195df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:35630/0 (socket says 172.21.15.28:35630) 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f778dfdd640 1 -- 172.21.15.28:0/877387306 learned_addr learned my addr 172.21.15.28:0/877387306 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f778810b970 con 0x7f7788068ac0 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f778810b7f0 con 0x7f7788104ae0 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f778810baf0 con 0x7f7788068650 2024-05-09T08:34:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f778e7de640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7788104ae0 0x7f778819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f778dfdd640 1 -- 172.21.15.28:0/877387306 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7788104ae0 msgr2=0x7f778819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f778dfdd640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7788104ae0 0x7f778819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f778dfdd640 1 -- 172.21.15.28:0/877387306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788068ac0 msgr2=0x7f7788196330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:34:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f778dfdd640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788068ac0 0x7f7788196330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.299+0000 7f778dfdd640 1 -- 172.21.15.28:0/877387306 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f778819ade0 con 0x7f7788068650 2024-05-09T08:34:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.300+0000 7f778dfdd640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7788068650 0x7f7788195df0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f778400d950 tx=0x7f778400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.301+0000 7f777effd640 1 -- 172.21.15.28:0/877387306 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784017070 con 0x7f7788068650 2024-05-09T08:34:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.301+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f778819b0d0 con 0x7f7788068650 2024-05-09T08:34:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.302+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77881a2960 con 0x7f7788068650 2024-05-09T08:34:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.302+0000 7f777effd640 1 -- 172.21.15.28:0/877387306 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7784009c30 con 0x7f7788068650 2024-05-09T08:34:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.303+0000 7f777effd640 1 -- 172.21.15.28:0/877387306 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778400f690 con 0x7f7788068650 2024-05-09T08:34:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.305+0000 7f779301f740 1 Processor -- start 2024-05-09T08:34:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f779301f740 1 -- start start 2024-05-09T08:34:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f779301f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f779301f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 0x560dc56b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f779301f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56b75e0 0x560dc56b99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f779301f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560dc5611340 con 0x560dc56ac400 2024-05-09T08:34:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f779301f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560dc55b9290 con 0x560dc56b75e0 2024-05-09T08:34:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f779301f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560dc560d300 con 0x560dc56acd20 2024-05-09T08:34:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f778e7de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f778e7de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35664/0 (socket says 172.21.15.28:35664) 2024-05-09T08:34:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.306+0000 7f778e7de640 1 -- 172.21.15.28:0/1854502421 learned_addr learned my addr 172.21.15.28:0/1854502421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.307+0000 7f778e7de640 1 -- 172.21.15.28:0/1854502421 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 msgr2=0x560dc56b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.307+0000 7f778e7de640 1 --2- 172.21.15.28:0/1854502421 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 0x560dc56b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.307+0000 7f778e7de640 1 -- 172.21.15.28:0/1854502421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56b75e0 msgr2=0x560dc56b99d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.307+0000 7f778e7de640 1 --2- 172.21.15.28:0/1854502421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56b75e0 0x560dc56b99d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.307+0000 7f778e7de640 1 -- 172.21.15.28:0/1854502421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dc55dd8e0 con 0x560dc56ac400 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.307+0000 7f777effd640 1 -- 172.21.15.28:0/877387306 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f778400f830 con 0x7f7788068650 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.307+0000 7f778e7de640 1 --2- 172.21.15.28:0/1854502421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56ac7e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7780009f50 tx=0x7f77800118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75360bd289ade2d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.307+0000 7f7773fff640 1 -- 172.21.15.28:0/1854502421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780004d20 con 0x560dc56ac400 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.308+0000 7f7773fff640 1 -- 172.21.15.28:0/1854502421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7780004050 con 0x560dc56ac400 2024-05-09T08:34:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.308+0000 7f7773fff640 1 -- 172.21.15.28:0/1854502421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780005280 con 0x560dc56ac400 2024-05-09T08:34:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f779301f740 1 -- 172.21.15.28:0/1854502421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 msgr2=0x560dc56ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f779301f740 1 --2- 172.21.15.28:0/1854502421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56ac7e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7780009f50 tx=0x7f77800118d0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f779301f740 1 -- 172.21.15.28:0/1854502421 shutdown_connections 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f779301f740 1 --2- 172.21.15.28:0/1854502421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56b75e0 0x560dc56b99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f779301f740 1 --2- 172.21.15.28:0/1854502421 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 0x560dc56b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f779301f740 1 --2- 172.21.15.28:0/1854502421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56ac7e0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f779301f740 1 -- 172.21.15.28:0/1854502421 >> 172.21.15.28:0/1854502421 conn(0x560dc56a8910 msgr2=0x560dc56a8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f779301f740 1 -- 172.21.15.28:0/1854502421 shutdown_connections 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f779301f740 1 -- 172.21.15.28:0/1854502421 wait complete. 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.309+0000 7f777effd640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77740a2e90 0x7f77740a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.310+0000 7f777effd640 1 -- 172.21.15.28:0/877387306 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9567+0+0 (secure 0 0 0) 0x7f77840cf3f0 con 0x7f7788068650 2024-05-09T08:34:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.310+0000 7f779301f740 1 -- start start 2024-05-09T08:34:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.310+0000 7f779301f740 1 Processor -- start 2024-05-09T08:34:58.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.310+0000 7f778d7dc640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77740a2e90 0x7f77740a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f779301f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f779301f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f779301f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 0x560dc56b5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f779301f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560dc56ba210 con 0x560dc56ac400 2024-05-09T08:34:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f779301f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x560dc55cc790 con 0x560dc56acd20 2024-05-09T08:34:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f779301f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560dc56ba390 con 0x560dc56b75e0 2024-05-09T08:34:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f779301f740 10 client.0 _ll_register_callbacks cb 0x560dc5607a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f778d7dc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f778d7dc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:54752/0 (socket says 172.21.15.28:54752) 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f778d7dc640 1 -- 172.21.15.28:0/888513148 learned_addr learned my addr 172.21.15.28:0/888513148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f778e7de640 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 0x560dc56b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.311+0000 7f778d7dc640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77740a2e90 0x7f77740a5350 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x560dc56a9d30 tx=0x7f7778000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.312+0000 7f778dfdd640 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.312+0000 7f778d7dc640 1 -- 172.21.15.28:0/888513148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 msgr2=0x560dc56b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.312+0000 7f778d7dc640 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 0x560dc56b5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.312+0000 7f778d7dc640 1 -- 172.21.15.28:0/888513148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 msgr2=0x560dc56b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.312+0000 7f778d7dc640 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.312+0000 7f778d7dc640 1 -- 172.21.15.28:0/888513148 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dc5325210 con 0x560dc56acd20 2024-05-09T08:34:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.312+0000 7f778dfdd640 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.312+0000 7f778d7dc640 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b4af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7778011a50 tx=0x7f777803aee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66d71b4c4d2779b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.312+0000 7f77717fa640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:34:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.313+0000 7f77717fa640 1 -- 172.21.15.28:0/888513148 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778053040 con 0x560dc56acd20 2024-05-09T08:34:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.313+0000 7f77717fa640 1 -- 172.21.15.28:0/888513148 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9567+0+0 (secure 0 0 0) 0x7f7778004240 con 0x560dc56acd20 2024-05-09T08:34:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.313+0000 7f779301f740 1 -- 172.21.15.28:0/888513148 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7780004050 con 0x560dc56acd20 2024-05-09T08:34:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.313+0000 7f77717fa640 1 -- 172.21.15.28:0/888513148 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f7778004440 con 0x560dc56acd20 2024-05-09T08:34:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.313+0000 7f779301f740 10 client.26315 fetch_fsmap learned FSMap version 63 2024-05-09T08:34:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.313+0000 7f779301f740 1 -- 172.21.15.28:0/888513148 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x560dc53f1620 con 0x560dc56acd20 2024-05-09T08:34:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.314+0000 7f77717fa640 1 -- 172.21.15.28:0/888513148 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7780004050 con 0x560dc56acd20 2024-05-09T08:34:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.314+0000 7f77717fa640 1 -- 172.21.15.28:0/888513148 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 63) ==== 8442+0+0 (secure 0 0 0) 0x7f777803b7e0 con 0x560dc56acd20 2024-05-09T08:34:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.314+0000 7f779301f740 10 client.26315 fetch_fsmap finished waiting for FSMap version 63 2024-05-09T08:34:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.314+0000 7f779301f740 10 client.26315 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:34:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.314+0000 7f779301f740 20 client.26315 populate_metadata read hostname 'smithi028' 2024-05-09T08:34:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.314+0000 7f779301f740 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560dc56c74a0 0x560dc56c9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.314+0000 7f779301f740 4 client.26315 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.314+0000 7f779301f740 1 -- 172.21.15.28:0/888513148 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560dc561ea70 con 0x560dc56c74a0 2024-05-09T08:34:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.315+0000 7f778dfdd640 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560dc56c74a0 0x560dc56c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.317+0000 7f778dfdd640 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560dc56c74a0 0x560dc56c9890 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8405bed11a506e61 server_cookie=655ea73dbcf28080 in_seq=0 out_seq=0 2024-05-09T08:34:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.318+0000 7f77717fa640 10 client.26315 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.320+0000 7f77717fa640 1 -- 172.21.15.28:0/888513148 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560dc561ea70 con 0x560dc56c74a0 2024-05-09T08:34:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.320+0000 7f77717fa640 10 client.26315 handle_command_reply: tid=0 2024-05-09T08:34:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.320+0000 7f779301f740 1 client.26315 shutdown 2024-05-09T08:34:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 -- 172.21.15.28:0/888513148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 msgr2=0x560dc56b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b4af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7778011a50 tx=0x7f777803aee0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 -- 172.21.15.28:0/888513148 shutdown_connections 2024-05-09T08:34:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560dc56c74a0 0x560dc56c9890 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 0x560dc56b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b4af0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 --2- 172.21.15.28:0/888513148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 -- 172.21.15.28:0/888513148 >> 172.21.15.28:0/888513148 conn(0x560dc56a8910 msgr2=0x560dc56ae690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f77717fa640 0 client.26315 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 -- 172.21.15.28:0/888513148 shutdown_connections 2024-05-09T08:34:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.321+0000 7f779301f740 1 -- 172.21.15.28:0/888513148 wait complete. 2024-05-09T08:34:58.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.322+0000 7f779301f740 20 client.26315 trim_cache size 0 max 16384 2024-05-09T08:34:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.349+0000 7f779301f740 1 Processor -- start 2024-05-09T08:34:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.349+0000 7f779301f740 1 -- start start 2024-05-09T08:34:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f779301f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f779301f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 0x560dc56b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f779301f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56b75e0 0x560dc56aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f779301f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77881acd80 con 0x560dc56ac400 2024-05-09T08:34:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f779301f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f77881aeb00 con 0x560dc56b75e0 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f779301f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f77881ad510 con 0x560dc56acd20 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778dfdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778dfdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35678/0 (socket says 172.21.15.28:35678) 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778dfdd640 1 -- 172.21.15.28:0/4060008393 learned_addr learned my addr 172.21.15.28:0/4060008393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778d7dc640 1 --2- 172.21.15.28:0/4060008393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 0x560dc56b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778e7de640 1 --2- 172.21.15.28:0/4060008393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56b75e0 0x560dc56aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778d7dc640 1 -- 172.21.15.28:0/4060008393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56b75e0 msgr2=0x560dc56aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778d7dc640 1 --2- 172.21.15.28:0/4060008393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56b75e0 0x560dc56aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778d7dc640 1 -- 172.21.15.28:0/4060008393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 msgr2=0x560dc56b6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778d7dc640 1 --2- 172.21.15.28:0/4060008393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.350+0000 7f778d7dc640 1 -- 172.21.15.28:0/4060008393 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dc56b9d40 con 0x560dc56acd20 2024-05-09T08:34:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.351+0000 7f778d7dc640 1 --2- 172.21.15.28:0/4060008393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 0x560dc56b4e90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f777803bc30 tx=0x7f7778014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15a9be291f604ab9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.351+0000 7f77737fe640 1 -- 172.21.15.28:0/4060008393 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777806e070 con 0x560dc56acd20 2024-05-09T08:34:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.351+0000 7f77737fe640 1 -- 172.21.15.28:0/4060008393 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f777805c070 con 0x560dc56acd20 2024-05-09T08:34:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.351+0000 7f77737fe640 1 -- 172.21.15.28:0/4060008393 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778011070 con 0x560dc56acd20 2024-05-09T08:34:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.353+0000 7f779301f740 1 -- 172.21.15.28:0/4060008393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 msgr2=0x560dc56b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.353+0000 7f779301f740 1 --2- 172.21.15.28:0/4060008393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 0x560dc56b4e90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f777803bc30 tx=0x7f7778014000 comp rx=0 tx=0).stop 2024-05-09T08:34:58.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.353+0000 7f779301f740 1 -- 172.21.15.28:0/4060008393 shutdown_connections 2024-05-09T08:34:58.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.353+0000 7f779301f740 1 --2- 172.21.15.28:0/4060008393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56b75e0 0x560dc56aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.353+0000 7f779301f740 1 --2- 172.21.15.28:0/4060008393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56acd20 0x560dc56b4e90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.353+0000 7f779301f740 1 --2- 172.21.15.28:0/4060008393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.353+0000 7f779301f740 1 -- 172.21.15.28:0/4060008393 >> 172.21.15.28:0/4060008393 conn(0x560dc56a8910 msgr2=0x560dc56a79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.354+0000 7f779301f740 1 -- 172.21.15.28:0/4060008393 shutdown_connections 2024-05-09T08:34:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.354+0000 7f779301f740 1 -- 172.21.15.28:0/4060008393 wait complete. 2024-05-09T08:34:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.355+0000 7f779301f740 1 -- start start 2024-05-09T08:34:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.355+0000 7f779301f740 1 Processor -- start 2024-05-09T08:34:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.355+0000 7f779301f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f779301f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f779301f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 0x560dc56b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f778dfdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f779301f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77881ae430 con 0x560dc56ac400 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f779301f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f77881adca0 con 0x560dc56acd20 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f779301f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f77881aad80 con 0x560dc56b75e0 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f778dfdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35686/0 (socket says 172.21.15.28:35686) 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f778dfdd640 1 -- 172.21.15.28:0/1852121990 learned_addr learned my addr 172.21.15.28:0/1852121990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f779301f740 10 client.0 _ll_register_callbacks cb 0x7f77800093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f778d7dc640 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f778e7de640 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 0x560dc56b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f778e7de640 1 -- 172.21.15.28:0/1852121990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 msgr2=0x560dc56b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f778e7de640 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f778e7de640 1 -- 172.21.15.28:0/1852121990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 msgr2=0x560dc56b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.356+0000 7f778e7de640 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.357+0000 7f778e7de640 1 -- 172.21.15.28:0/1852121990 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dc572cba0 con 0x560dc56b75e0 2024-05-09T08:34:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.357+0000 7f778e7de640 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 0x560dc56b1f40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7780004740 tx=0x7f7780009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a55ee0005a0f76d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.357+0000 7f77717fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:34:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.357+0000 7f77717fa640 1 -- 172.21.15.28:0/1852121990 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780004560 con 0x560dc56b75e0 2024-05-09T08:34:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.357+0000 7f77717fa640 1 -- 172.21.15.28:0/1852121990 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9567+0+0 (secure 0 0 0) 0x7f7780004820 con 0x560dc56b75e0 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.357+0000 7f779301f740 1 -- 172.21.15.28:0/1852121990 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f777805c070 con 0x560dc56b75e0 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.357+0000 7f77717fa640 1 -- 172.21.15.28:0/1852121990 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f7780036050 con 0x560dc56b75e0 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f779301f740 10 client.26188 fetch_fsmap learned FSMap version 63 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f779301f740 1 -- 172.21.15.28:0/1852121990 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=4+}) -- 0x560dc572de50 con 0x560dc56b75e0 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f77717fa640 1 -- 172.21.15.28:0/1852121990 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f777805c070 con 0x560dc56b75e0 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f77717fa640 1 -- 172.21.15.28:0/1852121990 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 63) ==== 8442+0+0 (secure 0 0 0) 0x7f77800056b0 con 0x560dc56b75e0 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f779301f740 10 client.26188 fetch_fsmap finished waiting for FSMap version 63 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f779301f740 10 client.26188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f779301f740 20 client.26188 populate_metadata read hostname 'smithi028' 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f779301f740 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560dc572e460 0x560dc5730880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f779301f740 4 client.26188 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:34:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.358+0000 7f779301f740 1 -- 172.21.15.28:0/1852121990 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560dc5618dc0 con 0x560dc572e460 2024-05-09T08:34:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.362+0000 7f778dfdd640 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560dc572e460 0x560dc5730880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:58.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.366+0000 7f778dfdd640 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560dc572e460 0x560dc5730880 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5892ca8d381656db server_cookie=1ce156e9a89d3ceb in_seq=0 out_seq=0 2024-05-09T08:34:58.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:58.367+0000 7f77717fa640 10 client.26188 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.656+0000 7f77717fa640 1 -- 172.21.15.28:0/1852121990 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560dc5618dc0 con 0x560dc572e460 2024-05-09T08:34:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.656+0000 7f77717fa640 10 client.26188 handle_command_reply: tid=0 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.656+0000 7f779301f740 1 client.26188 shutdown 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.656+0000 7f779301f740 1 -- 172.21.15.28:0/1852121990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 msgr2=0x560dc56b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.656+0000 7f779301f740 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 0x560dc56b1f40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7780004740 tx=0x7f7780009780 comp rx=0 tx=0).stop 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f779301f740 1 -- 172.21.15.28:0/1852121990 shutdown_connections 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f779301f740 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x560dc572e460 0x560dc5730880 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f779301f740 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560dc56b75e0 0x560dc56b1f40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f779301f740 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560dc56acd20 0x560dc56b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f779301f740 1 --2- 172.21.15.28:0/1852121990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dc56ac400 0x560dc56b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f779301f740 1 -- 172.21.15.28:0/1852121990 >> 172.21.15.28:0/1852121990 conn(0x560dc56a8910 msgr2=0x560dc56c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f77717fa640 0 client.26188 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f779301f740 1 -- 172.21.15.28:0/1852121990 shutdown_connections 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f779301f740 1 -- 172.21.15.28:0/1852121990 wait complete. 2024-05-09T08:34:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.657+0000 7f779301f740 20 client.26188 trim_cache size 0 max 16384 2024-05-09T08:34:59.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:34:59.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:34:59.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:34:59.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:34:59.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77740a2e90 msgr2=0x7f77740a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:59.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77740a2e90 0x7f77740a5350 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x560dc56a9d30 tx=0x7f7778000bc0 comp rx=0 tx=0).stop 2024-05-09T08:34:59.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7788068650 msgr2=0x7f7788195df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:59.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7788068650 0x7f7788195df0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f778400d950 tx=0x7f778400de10 comp rx=0 tx=0).stop 2024-05-09T08:34:59.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 shutdown_connections 2024-05-09T08:34:59.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7788104ae0 0x7f778819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:59.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788068ac0 0x7f7788196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:59.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f77740a2e90 0x7f77740a5350 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:59.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 --2- 172.21.15.28:0/877387306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7788068650 0x7f7788195df0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 >> 172.21.15.28:0/877387306 conn(0x7f7788076cc0 msgr2=0x7f77881089c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 shutdown_connections 2024-05-09T08:34:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:34:59.658+0000 7f7790241640 1 -- 172.21.15.28:0/877387306 wait complete. 2024-05-09T08:34:59.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:34:59 smithi184 ceph-mon[34509]: pgmap v273: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 583 op/s 2024-05-09T08:34:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:34:59 smithi028 ceph-mon[30119]: pgmap v273: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 583 op/s 2024-05-09T08:35:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:34:59 smithi165 ceph-mon[34540]: pgmap v273: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 583 op/s 2024-05-09T08:35:01.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:01 smithi028 ceph-mon[30119]: pgmap v274: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 636 op/s 2024-05-09T08:35:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:01 smithi165 ceph-mon[34540]: pgmap v274: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 636 op/s 2024-05-09T08:35:02.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:01 smithi184 ceph-mon[34509]: pgmap v274: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 636 op/s 2024-05-09T08:35:03.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:03 smithi165 ceph-mon[34540]: pgmap v275: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 514 op/s 2024-05-09T08:35:03.907 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:03 smithi184 ceph-mon[34509]: pgmap v275: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 514 op/s 2024-05-09T08:35:03.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:03 smithi028 ceph-mon[30119]: pgmap v275: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 514 op/s 2024-05-09T08:35:05.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:05 smithi028 ceph-mon[30119]: pgmap v276: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 565 op/s 2024-05-09T08:35:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:05 smithi165 ceph-mon[34540]: pgmap v276: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 565 op/s 2024-05-09T08:35:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:05 smithi184 ceph-mon[34509]: pgmap v276: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 565 op/s 2024-05-09T08:35:07.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:07 smithi028 ceph-mon[30119]: pgmap v277: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 514 op/s 2024-05-09T08:35:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:07 smithi165 ceph-mon[34540]: pgmap v277: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 514 op/s 2024-05-09T08:35:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:07 smithi184 ceph-mon[34509]: pgmap v277: 129 pgs: 129 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 514 op/s 2024-05-09T08:35:08.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:08 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:35:08.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:08 smithi028 ceph-mon[30119]: osdmap e105: 12 total, 12 up, 12 in 2024-05-09T08:35:09.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:08 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:35:09.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:08 smithi165 ceph-mon[34540]: osdmap e105: 12 total, 12 up, 12 in 2024-05-09T08:35:09.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:08 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:35:09.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:08 smithi184 ceph-mon[34509]: osdmap e105: 12 total, 12 up, 12 in 2024-05-09T08:35:09.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:09 smithi028 ceph-mon[30119]: pgmap v279: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 495 op/s 2024-05-09T08:35:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:09 smithi165 ceph-mon[34540]: pgmap v279: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 495 op/s 2024-05-09T08:35:10.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:09 smithi184 ceph-mon[34509]: pgmap v279: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 495 op/s 2024-05-09T08:35:11.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:11 smithi028 ceph-mon[30119]: pgmap v280: 129 pgs: 16 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 65 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 347 op/s 2024-05-09T08:35:12.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:11 smithi165 ceph-mon[34540]: pgmap v280: 129 pgs: 16 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 65 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 347 op/s 2024-05-09T08:35:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:11 smithi184 ceph-mon[34509]: pgmap v280: 129 pgs: 16 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 65 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 347 op/s 2024-05-09T08:35:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.813+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/3393807901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc110020 msgr2=0x7fc1cc108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.813+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/3393807901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc110020 0x7fc1cc108830 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc1c001a940 tx=0x7fc1c003b520 comp rx=0 tx=0).stop 2024-05-09T08:35:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.814+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/3393807901 shutdown_connections 2024-05-09T08:35:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.814+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/3393807901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1cc11b010 0x7fc1cc11d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.814+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/3393807901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc110400 0x7fc1cc108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.814+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/3393807901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc110020 0x7fc1cc108830 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.814+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/3393807901 >> 172.21.15.28:0/3393807901 conn(0x7fc1cc06c900 msgr2=0x7fc1cc06cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.818+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/3393807901 shutdown_connections 2024-05-09T08:35:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.818+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/3393807901 wait complete. 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.818+0000 7fc1d1ee7640 1 Processor -- start 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.819+0000 7fc1d1ee7640 1 -- start start 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.819+0000 7fc1d1ee7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc110020 0x7fc1cc11a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.819+0000 7fc1d1ee7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1cc11b010 0x7fc1cc116860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.819+0000 7fc1d1ee7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc116da0 0x7fc1cc117250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.819+0000 7fc1d1ee7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc113250 con 0x7fc1cc116da0 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.819+0000 7fc1d1ee7640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1130d0 con 0x7fc1cc11b010 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.819+0000 7fc1d1ee7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1133d0 con 0x7fc1cc110020 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.819+0000 7fc1caffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1cc11b010 0x7fc1cc116860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.820+0000 7fc1caffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1cc11b010 0x7fc1cc116860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:52998/0 (socket says 172.21.15.28:52998) 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.820+0000 7fc1caffd640 1 -- 172.21.15.28:0/2416050076 learned_addr learned my addr 172.21.15.28:0/2416050076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.820+0000 7fc1caffd640 1 -- 172.21.15.28:0/2416050076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc110020 msgr2=0x7fc1cc11a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.820+0000 7fc1caffd640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc110020 0x7fc1cc11a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.820+0000 7fc1caffd640 1 -- 172.21.15.28:0/2416050076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc116da0 msgr2=0x7fc1cc117250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.820+0000 7fc1caffd640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc116da0 0x7fc1cc117250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.820+0000 7fc1caffd640 1 -- 172.21.15.28:0/2416050076 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc1cce60 con 0x7fc1cc11b010 2024-05-09T08:35:12.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.823+0000 7fc1caffd640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1cc11b010 0x7fc1cc116860 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00b470 tx=0x7fc1bc00b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.824+0000 7fc1c8ff9640 1 -- 172.21.15.28:0/2416050076 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc015020 con 0x7fc1cc11b010 2024-05-09T08:35:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.825+0000 7fc1c8ff9640 1 -- 172.21.15.28:0/2416050076 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc011d70 con 0x7fc1cc11b010 2024-05-09T08:35:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.825+0000 7fc1c8ff9640 1 -- 172.21.15.28:0/2416050076 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc00f340 con 0x7fc1cc11b010 2024-05-09T08:35:12.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.825+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/2416050076 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1cc1cd150 con 0x7fc1cc11b010 2024-05-09T08:35:12.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.825+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/2416050076 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1cc1cd620 con 0x7fc1cc11b010 2024-05-09T08:35:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.829+0000 7fc1d4bf0740 1 Processor -- start 2024-05-09T08:35:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.829+0000 7fc1d4bf0740 1 -- start start 2024-05-09T08:35:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.829+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c1d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.829+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c280f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.829+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c2aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.829+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c42b82340 con 0x558c42c1d450 2024-05-09T08:35:12.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.829+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558c42b2a290 con 0x558c42c1dd70 2024-05-09T08:35:12.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.829+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558c42b7e300 con 0x558c42c28630 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1caffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c2aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1cb7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c280f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1cb7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c280f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:53002/0 (socket says 172.21.15.28:53002) 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1cb7fe640 1 -- 172.21.15.28:0/3379305784 learned_addr learned my addr 172.21.15.28:0/3379305784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1cbfff640 1 --2- 172.21.15.28:0/3379305784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c1d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1cb7fe640 1 -- 172.21.15.28:0/3379305784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 msgr2=0x558c42c2aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1cb7fe640 1 --2- 172.21.15.28:0/3379305784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c2aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1cb7fe640 1 -- 172.21.15.28:0/3379305784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 msgr2=0x558c42c1d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1cb7fe640 1 --2- 172.21.15.28:0/3379305784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c1d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.830+0000 7fc1cb7fe640 1 -- 172.21.15.28:0/3379305784 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c42b4e8e0 con 0x558c42c1dd70 2024-05-09T08:35:12.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.831+0000 7fc1cb7fe640 1 --2- 172.21.15.28:0/3379305784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c280f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc1c003bb10 tx=0x7fc1c0004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2088c89b666b63c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:12.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.834+0000 7fc1c8ff9640 1 -- 172.21.15.28:0/2416050076 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc1bc018020 con 0x7fc1cc11b010 2024-05-09T08:35:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.835+0000 7fc1c8ff9640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b40a2ef0 0x7fc1b40a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.835+0000 7fc1c8ff9640 1 -- 172.21.15.28:0/2416050076 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9671+0+0 (secure 0 0 0) 0x7fc1bc0d2540 con 0x7fc1cc11b010 2024-05-09T08:35:12.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.836+0000 7fc1b1ffb640 1 -- 172.21.15.28:0/3379305784 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0046070 con 0x558c42c1dd70 2024-05-09T08:35:12.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.836+0000 7fc1b1ffb640 1 -- 172.21.15.28:0/3379305784 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c0013070 con 0x558c42c1dd70 2024-05-09T08:35:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.837+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/3379305784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 msgr2=0x558c42c280f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.837+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/3379305784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c280f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc1c003bb10 tx=0x7fc1c0004800 comp rx=0 tx=0).stop 2024-05-09T08:35:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.840+0000 7fc1cbfff640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b40a2ef0 0x7fc1b40a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.840+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/3379305784 shutdown_connections 2024-05-09T08:35:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.840+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/3379305784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c2aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.840+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/3379305784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c280f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.840+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/3379305784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c1d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.840+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/3379305784 >> 172.21.15.28:0/3379305784 conn(0x558c42c19960 msgr2=0x558c42c19d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.844+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/3379305784 shutdown_connections 2024-05-09T08:35:12.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.847+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/3379305784 wait complete. 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.847+0000 7fc1d4bf0740 1 -- start start 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.847+0000 7fc1d4bf0740 1 Processor -- start 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.848+0000 7fc1cbfff640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b40a2ef0 0x7fc1b40a53b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4007750 tx=0x7fc1c40076e0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c25590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c25ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c26010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c42c2b200 con 0x558c42c1d450 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558c42b3d790 con 0x558c42c1dd70 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558c42c2b380 con 0x558c42c28630 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1d4bf0740 10 client.0 _ll_register_callbacks cb 0x558c42b78a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c26010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1cbfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c26010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51538/0 (socket says 172.21.15.28:51538) 2024-05-09T08:35:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.849+0000 7fc1cbfff640 1 -- 172.21.15.28:0/2965201462 learned_addr learned my addr 172.21.15.28:0/2965201462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.850+0000 7fc1cbfff640 1 -- 172.21.15.28:0/2965201462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 msgr2=0x558c42c25ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.850+0000 7fc1cbfff640 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c25ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.850+0000 7fc1cbfff640 1 -- 172.21.15.28:0/2965201462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 msgr2=0x558c42c25590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.850+0000 7fc1cbfff640 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c25590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.850+0000 7fc1cbfff640 1 -- 172.21.15.28:0/2965201462 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c42896210 con 0x558c42c28630 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.850+0000 7fc1cbfff640 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c26010 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4010db0 tx=0x7fc1c40074a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16d875a2dec4c334 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.851+0000 7fc1aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.851+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2965201462 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4026480 con 0x558c42c28630 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.851+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2965201462 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558c42b8fa70 con 0x558c42c28630 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.851+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2965201462 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9671+0+0 (secure 0 0 0) 0x7fc1c4026620 con 0x558c42c28630 2024-05-09T08:35:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.851+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2965201462 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fc1c402f050 con 0x558c42c28630 2024-05-09T08:35:12.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.852+0000 7fc1d4bf0740 10 client.26191 fetch_fsmap learned FSMap version 65 2024-05-09T08:35:12.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.852+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2965201462 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x558c42962620 con 0x558c42c28630 2024-05-09T08:35:12.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.856+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2965201462 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558c42b8fa70 con 0x558c42c28630 2024-05-09T08:35:12.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.857+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2965201462 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7fc1c401d900 con 0x558c42c28630 2024-05-09T08:35:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.857+0000 7fc1d4bf0740 10 client.26191 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:35:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.857+0000 7fc1d4bf0740 10 client.26191 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:35:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.857+0000 7fc1d4bf0740 20 client.26191 populate_metadata read hostname 'smithi028' 2024-05-09T08:35:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.857+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558c42c38aa0 0x558c42c3af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.857+0000 7fc1d4bf0740 4 client.26191 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.857+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2965201462 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c42b89dc0 con 0x558c42c38aa0 2024-05-09T08:35:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.857+0000 7fc1cb7fe640 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558c42c38aa0 0x558c42c3af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.858+0000 7fc1cb7fe640 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558c42c38aa0 0x558c42c3af60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cb2fc1662bcc118 server_cookie=48dc2d1e3abcb601 in_seq=0 out_seq=0 2024-05-09T08:35:12.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.859+0000 7fc1aaffd640 10 client.26191 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:12.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.861+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2965201462 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c42b89dc0 con 0x558c42c38aa0 2024-05-09T08:35:12.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.861+0000 7fc1aaffd640 10 client.26191 handle_command_reply: tid=0 2024-05-09T08:35:12.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.861+0000 7fc1d4bf0740 1 client.26191 shutdown 2024-05-09T08:35:12.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.864+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2965201462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 msgr2=0x558c42c26010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.864+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c26010 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4010db0 tx=0x7fc1c40074a0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.868+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2965201462 shutdown_connections 2024-05-09T08:35:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.868+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558c42c38aa0 0x558c42c3af60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.868+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c26010 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.868+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c25ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.868+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2965201462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c25590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.868+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2965201462 >> 172.21.15.28:0/2965201462 conn(0x558c42c19960 msgr2=0x558c42c1f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.868+0000 7fc1aaffd640 0 client.26191 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.869+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2965201462 shutdown_connections 2024-05-09T08:35:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.869+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2965201462 wait complete. 2024-05-09T08:35:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.869+0000 7fc1d4bf0740 20 client.26191 trim_cache size 0 max 16384 2024-05-09T08:35:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.898+0000 7fc1d4bf0740 1 Processor -- start 2024-05-09T08:35:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1d4bf0740 1 -- start start 2024-05-09T08:35:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1d450 0x558c42c34170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c1dd70 0x558c42c1fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c28630 0x558c42c18880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b8010390 con 0x558c42c28630 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b8012110 con 0x558c42c1d450 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b8010b20 con 0x558c42c1dd70 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1caffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c1dd70 0x558c42c1fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1caffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c1dd70 0x558c42c1fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51550/0 (socket says 172.21.15.28:51550) 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.899+0000 7fc1caffd640 1 -- 172.21.15.28:0/1146888655 learned_addr learned my addr 172.21.15.28:0/1146888655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.900+0000 7fc1caffd640 1 -- 172.21.15.28:0/1146888655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1d450 msgr2=0x558c42c34170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.900+0000 7fc1caffd640 1 --2- 172.21.15.28:0/1146888655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1d450 0x558c42c34170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.900+0000 7fc1caffd640 1 -- 172.21.15.28:0/1146888655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c28630 msgr2=0x558c42c18880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.900+0000 7fc1caffd640 1 --2- 172.21.15.28:0/1146888655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c28630 0x558c42c18880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.900+0000 7fc1caffd640 1 -- 172.21.15.28:0/1146888655 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c42c37040 con 0x558c42c1dd70 2024-05-09T08:35:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.900+0000 7fc1caffd640 1 --2- 172.21.15.28:0/1146888655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c1dd70 0x558c42c1fe10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc006fd0 tx=0x7fc1bc09e2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f95119da11afd8e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.903+0000 7fc1b17fa640 1 -- 172.21.15.28:0/1146888655 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc09e690 con 0x558c42c1dd70 2024-05-09T08:35:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.903+0000 7fc1b17fa640 1 -- 172.21.15.28:0/1146888655 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc0a2a40 con 0x558c42c1dd70 2024-05-09T08:35:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.904+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/1146888655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c1dd70 msgr2=0x558c42c1fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.904+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/1146888655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c1dd70 0x558c42c1fe10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc006fd0 tx=0x7fc1bc09e2d0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.910+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/1146888655 shutdown_connections 2024-05-09T08:35:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.910+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/1146888655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c28630 0x558c42c18880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.910+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/1146888655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c1dd70 0x558c42c1fe10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.910+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/1146888655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1d450 0x558c42c34170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.910+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/1146888655 >> 172.21.15.28:0/1146888655 conn(0x558c42c19960 msgr2=0x558c42c18170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.914+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/1146888655 shutdown_connections 2024-05-09T08:35:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.915+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/1146888655 wait complete. 2024-05-09T08:35:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.915+0000 7fc1d4bf0740 1 -- start start 2024-05-09T08:35:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.916+0000 7fc1d4bf0740 1 Processor -- start 2024-05-09T08:35:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.916+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c30b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.916+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c229f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.916+0000 7fc1d4bf0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c22f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.916+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b8011a40 con 0x558c42c1d450 2024-05-09T08:35:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.916+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b80112b0 con 0x558c42c1dd70 2024-05-09T08:35:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.916+0000 7fc1d4bf0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b800e390 con 0x558c42c28630 2024-05-09T08:35:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.916+0000 7fc1d4bf0740 10 client.0 _ll_register_callbacks cb 0x7fc1c400fee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.917+0000 7fc1caffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c229f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.917+0000 7fc1caffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c229f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:53018/0 (socket says 172.21.15.28:53018) 2024-05-09T08:35:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.917+0000 7fc1caffd640 1 -- 172.21.15.28:0/2372172275 learned_addr learned my addr 172.21.15.28:0/2372172275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.917+0000 7fc1caffd640 1 -- 172.21.15.28:0/2372172275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 msgr2=0x558c42c22f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.917+0000 7fc1caffd640 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c22f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.917+0000 7fc1caffd640 1 -- 172.21.15.28:0/2372172275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 msgr2=0x558c42c30b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.917+0000 7fc1caffd640 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c30b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.917+0000 7fc1caffd640 1 -- 172.21.15.28:0/2372172275 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c42c21820 con 0x558c42c1dd70 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.917+0000 7fc1caffd640 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c229f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc09e390 tx=0x7fc1bc0d21e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db38a12f58dd5796 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.918+0000 7fc1aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.918+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2372172275 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc0a7300 con 0x558c42c1dd70 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.918+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2372172275 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9671+0+0 (secure 0 0 0) 0x7fc1bc0b2bb0 con 0x558c42c1dd70 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.920+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2372172275 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1bc0a2a40 con 0x558c42c1dd70 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.920+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2372172275 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fc1bc09e690 con 0x558c42c1dd70 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.920+0000 7fc1d4bf0740 10 client.26330 fetch_fsmap learned FSMap version 65 2024-05-09T08:35:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.920+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2372172275 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x558c42c9e3a0 con 0x558c42c1dd70 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.921+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2372172275 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc0a2a40 con 0x558c42c1dd70 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.921+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2372172275 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7fc1bc09db70 con 0x558c42c1dd70 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.921+0000 7fc1d4bf0740 10 client.26330 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.921+0000 7fc1d4bf0740 10 client.26330 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.921+0000 7fc1d4bf0740 20 client.26330 populate_metadata read hostname 'smithi028' 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.921+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558c42c9eab0 0x558c42ca0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.921+0000 7fc1d4bf0740 4 client.26330 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.921+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2372172275 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c42b078e0 con 0x558c42c9eab0 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.922+0000 7fc1cb7fe640 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558c42c9eab0 0x558c42ca0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.923+0000 7fc1cb7fe640 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558c42c9eab0 0x558c42ca0f30 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f378f34345806d2 server_cookie=6e020ef3da40e79e in_seq=0 out_seq=0 2024-05-09T08:35:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:12.923+0000 7fc1aaffd640 10 client.26330 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:13.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.441+0000 7fc1aaffd640 1 -- 172.21.15.28:0/2372172275 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558c42b078e0 con 0x558c42c9eab0 2024-05-09T08:35:13.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.441+0000 7fc1aaffd640 10 client.26330 handle_command_reply: tid=0 2024-05-09T08:35:13.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.441+0000 7fc1d4bf0740 1 client.26330 shutdown 2024-05-09T08:35:13.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2372172275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 msgr2=0x558c42c229f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c229f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc09e390 tx=0x7fc1bc0d21e0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2372172275 shutdown_connections 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558c42c9eab0 0x558c42ca0f30 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558c42c28630 0x558c42c22f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558c42c1dd70 0x558c42c229f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 --2- 172.21.15.28:0/2372172275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c42c1d450 0x558c42c30b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2372172275 >> 172.21.15.28:0/2372172275 conn(0x558c42c19960 msgr2=0x558c42c18a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1aaffd640 0 client.26330 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2372172275 shutdown_connections 2024-05-09T08:35:13.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 1 -- 172.21.15.28:0/2372172275 wait complete. 2024-05-09T08:35:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.442+0000 7fc1d4bf0740 20 client.26330 trim_cache size 0 max 16384 2024-05-09T08:35:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:35:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:35:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:35:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:35:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/2416050076 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b40a2ef0 msgr2=0x7fc1b40a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b40a2ef0 0x7fc1b40a53b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4007750 tx=0x7fc1c40076e0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/2416050076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1cc11b010 msgr2=0x7fc1cc116860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1cc11b010 0x7fc1cc116860 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00b470 tx=0x7fc1bc00b930 comp rx=0 tx=0).stop 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/2416050076 shutdown_connections 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1cc116da0 0x7fc1cc117250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1cc11b010 0x7fc1cc116860 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc1b40a2ef0 0x7fc1b40a53b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 --2- 172.21.15.28:0/2416050076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1cc110020 0x7fc1cc11a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.443+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/2416050076 >> 172.21.15.28:0/2416050076 conn(0x7fc1cc06c900 msgr2=0x7fc1cc11b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.444+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/2416050076 shutdown_connections 2024-05-09T08:35:13.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:13.444+0000 7fc1d1ee7640 1 -- 172.21.15.28:0/2416050076 wait complete. 2024-05-09T08:35:13.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:13 smithi028 ceph-mon[30119]: pgmap v281: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 297 op/s 2024-05-09T08:35:13.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:13 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:35:14.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:13 smithi165 ceph-mon[34540]: pgmap v281: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 297 op/s 2024-05-09T08:35:14.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:13 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:35:14.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:13 smithi184 ceph-mon[34509]: pgmap v281: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 2.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 297 op/s 2024-05-09T08:35:14.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:13 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:35:15.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:15 smithi028 ceph-mon[30119]: pgmap v282: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 2.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-05-09T08:35:16.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:15 smithi165 ceph-mon[34540]: pgmap v282: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 2.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-05-09T08:35:16.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:15 smithi184 ceph-mon[34509]: pgmap v282: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 2.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 209 op/s 2024-05-09T08:35:17.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:17 smithi028 ceph-mon[30119]: pgmap v283: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.5 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-05-09T08:35:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:17 smithi165 ceph-mon[34540]: pgmap v283: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.5 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-05-09T08:35:18.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:17 smithi184 ceph-mon[34509]: pgmap v283: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.5 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 194 op/s 2024-05-09T08:35:18.681 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:35:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:35:18.425+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:35:20.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:19 smithi165 ceph-mon[34540]: pgmap v284: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.5 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-05-09T08:35:20.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:19 smithi184 ceph-mon[34509]: pgmap v284: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.5 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-05-09T08:35:20.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:19 smithi028 ceph-mon[30119]: pgmap v284: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.5 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-05-09T08:35:22.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:21 smithi165 ceph-mon[34540]: pgmap v285: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 2.5 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 194 op/s 2024-05-09T08:35:22.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:21 smithi184 ceph-mon[34509]: pgmap v285: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 2.5 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 194 op/s 2024-05-09T08:35:22.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:21 smithi028 ceph-mon[30119]: pgmap v285: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 2.5 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 194 op/s 2024-05-09T08:35:24.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:23 smithi165 ceph-mon[34540]: pgmap v286: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 2.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-05-09T08:35:24.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:23 smithi184 ceph-mon[34509]: pgmap v286: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 2.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-05-09T08:35:24.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:23 smithi028 ceph-mon[30119]: pgmap v286: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 2.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 175 op/s 2024-05-09T08:35:26.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:25 smithi165 ceph-mon[34540]: pgmap v287: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 2.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 206 op/s 2024-05-09T08:35:26.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:25 smithi165 ceph-mon[34540]: osdmap e106: 12 total, 12 up, 12 in 2024-05-09T08:35:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:25 smithi184 ceph-mon[34509]: pgmap v287: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 2.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 206 op/s 2024-05-09T08:35:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:25 smithi184 ceph-mon[34509]: osdmap e106: 12 total, 12 up, 12 in 2024-05-09T08:35:26.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:25 smithi028 ceph-mon[30119]: pgmap v287: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 2.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 206 op/s 2024-05-09T08:35:26.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:25 smithi028 ceph-mon[30119]: osdmap e106: 12 total, 12 up, 12 in 2024-05-09T08:35:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.597+0000 7fd2a204c640 1 -- 172.21.15.28:0/1652735691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c101340 msgr2=0x7fd29c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.597+0000 7fd2a204c640 1 --2- 172.21.15.28:0/1652735691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c101340 0x7fd29c10d570 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd294009ca0 tx=0x7fd29402a8b0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.598+0000 7fd2a204c640 1 -- 172.21.15.28:0/1652735691 shutdown_connections 2024-05-09T08:35:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.598+0000 7fd2a204c640 1 --2- 172.21.15.28:0/1652735691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c101340 0x7fd29c10d570 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.598+0000 7fd2a204c640 1 --2- 172.21.15.28:0/1652735691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd29c1009a0 0x7fd29c100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.598+0000 7fd2a204c640 1 --2- 172.21.15.28:0/1652735691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd29c1067d0 0x7fd29c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.598+0000 7fd2a204c640 1 -- 172.21.15.28:0/1652735691 >> 172.21.15.28:0/1652735691 conn(0x7fd29c077610 msgr2=0x7fd29c077a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.598+0000 7fd2a204c640 1 -- 172.21.15.28:0/1652735691 shutdown_connections 2024-05-09T08:35:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.598+0000 7fd2a204c640 1 -- 172.21.15.28:0/1652735691 wait complete. 2024-05-09T08:35:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.599+0000 7fd2a204c640 1 Processor -- start 2024-05-09T08:35:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.599+0000 7fd2a204c640 1 -- start start 2024-05-09T08:35:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.599+0000 7fd2a204c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c1009a0 0x7fd29c10d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.599+0000 7fd2a204c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd29c101340 0x7fd29c10a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.599+0000 7fd2a204c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd29c1067d0 0x7fd29c10aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.599+0000 7fd2a204c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd29c10fa90 con 0x7fd29c101340 2024-05-09T08:35:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.599+0000 7fd2a204c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd29c10f910 con 0x7fd29c1009a0 2024-05-09T08:35:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.599+0000 7fd2a204c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd29c10fc10 con 0x7fd29c1067d0 2024-05-09T08:35:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.600+0000 7fd29bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd29c1067d0 0x7fd29c10aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.600+0000 7fd29b7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c1009a0 0x7fd29c10d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.600+0000 7fd29b7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c1009a0 0x7fd29c10d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33936/0 (socket says 172.21.15.28:33936) 2024-05-09T08:35:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.600+0000 7fd29b7fe640 1 -- 172.21.15.28:0/2077059174 learned_addr learned my addr 172.21.15.28:0/2077059174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.601+0000 7fd29b7fe640 1 -- 172.21.15.28:0/2077059174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd29c1067d0 msgr2=0x7fd29c10aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.601+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd29c1067d0 0x7fd29c10aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.601+0000 7fd29b7fe640 1 -- 172.21.15.28:0/2077059174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd29c101340 msgr2=0x7fd29c10a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.601+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd29c101340 0x7fd29c10a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.601+0000 7fd29b7fe640 1 -- 172.21.15.28:0/2077059174 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd29c10b2e0 con 0x7fd29c1009a0 2024-05-09T08:35:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.601+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c1009a0 0x7fd29c10d420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00e9f0 tx=0x7fd28c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.602+0000 7fd298ff9640 1 -- 172.21.15.28:0/2077059174 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00cd80 con 0x7fd29c1009a0 2024-05-09T08:35:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.602+0000 7fd298ff9640 1 -- 172.21.15.28:0/2077059174 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd28c010910 con 0x7fd29c1009a0 2024-05-09T08:35:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.602+0000 7fd298ff9640 1 -- 172.21.15.28:0/2077059174 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c0188b0 con 0x7fd29c1009a0 2024-05-09T08:35:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.602+0000 7fd2a204c640 1 -- 172.21.15.28:0/2077059174 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd29c0723c0 con 0x7fd29c1009a0 2024-05-09T08:35:26.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.602+0000 7fd2a204c640 1 -- 172.21.15.28:0/2077059174 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd29c072890 con 0x7fd29c1009a0 2024-05-09T08:35:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.608+0000 7fd298ff9640 1 -- 172.21.15.28:0/2077059174 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd28c010ac0 con 0x7fd29c1009a0 2024-05-09T08:35:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.609+0000 7fd2a4e31740 1 Processor -- start 2024-05-09T08:35:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.611+0000 7fd298ff9640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2840b3f90 0x7fd2840b6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.611+0000 7fd298ff9640 1 -- 172.21.15.28:0/2077059174 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9595+0+0 (secure 0 0 0) 0x7fd28c0cfc90 con 0x7fd29c1009a0 2024-05-09T08:35:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.611+0000 7fd29bfff640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2840b3f90 0x7fd2840b6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.612+0000 7fd2a4e31740 1 -- start start 2024-05-09T08:35:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.613+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df20d450 0x5597df20d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.613+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 0x5597df2180f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.613+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df218630 0x5597df21aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.613+0000 7fd2a4e31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5597df172340 con 0x5597df20d450 2024-05-09T08:35:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.613+0000 7fd2a4e31740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5597df11a290 con 0x5597df218630 2024-05-09T08:35:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.613+0000 7fd2a4e31740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5597df16e300 con 0x5597df20dd70 2024-05-09T08:35:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.613+0000 7fd29bfff640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2840b3f90 0x7fd2840b6450 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd294006010 tx=0x7fd294005d80 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.613+0000 7fd29bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df218630 0x5597df21aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.614+0000 7fd29bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df218630 0x5597df21aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33950/0 (socket says 172.21.15.28:33950) 2024-05-09T08:35:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.614+0000 7fd29bfff640 1 -- 172.21.15.28:0/4239468016 learned_addr learned my addr 172.21.15.28:0/4239468016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.614+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/4239468016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df20d450 0x5597df20d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.614+0000 7fd29affd640 1 --2- 172.21.15.28:0/4239468016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 0x5597df2180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.615+0000 7fd29bfff640 1 -- 172.21.15.28:0/4239468016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 msgr2=0x5597df2180f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.615+0000 7fd29bfff640 1 --2- 172.21.15.28:0/4239468016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 0x5597df2180f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.615+0000 7fd29bfff640 1 -- 172.21.15.28:0/4239468016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df20d450 msgr2=0x5597df20d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.615+0000 7fd29bfff640 1 --2- 172.21.15.28:0/4239468016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df20d450 0x5597df20d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.615+0000 7fd29bfff640 1 -- 172.21.15.28:0/4239468016 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597df13e8e0 con 0x5597df218630 2024-05-09T08:35:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.616+0000 7fd29bfff640 1 --2- 172.21.15.28:0/4239468016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df218630 0x5597df21aa20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd294035000 tx=0x7fd29403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f375979f8cb0597 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.616+0000 7fd281ffb640 1 -- 172.21.15.28:0/4239468016 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294030b30 con 0x5597df218630 2024-05-09T08:35:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.616+0000 7fd281ffb640 1 -- 172.21.15.28:0/4239468016 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd294030cd0 con 0x5597df218630 2024-05-09T08:35:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.617+0000 7fd281ffb640 1 -- 172.21.15.28:0/4239468016 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29403c830 con 0x5597df218630 2024-05-09T08:35:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 -- 172.21.15.28:0/4239468016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df218630 msgr2=0x5597df21aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/4239468016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df218630 0x5597df21aa20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd294035000 tx=0x7fd29403f000 comp rx=0 tx=0).stop 2024-05-09T08:35:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 -- 172.21.15.28:0/4239468016 shutdown_connections 2024-05-09T08:35:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/4239468016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df218630 0x5597df21aa20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/4239468016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 0x5597df2180f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/4239468016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df20d450 0x5597df20d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 -- 172.21.15.28:0/4239468016 >> 172.21.15.28:0/4239468016 conn(0x5597df209960 msgr2=0x5597df209d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 -- 172.21.15.28:0/4239468016 shutdown_connections 2024-05-09T08:35:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 -- 172.21.15.28:0/4239468016 wait complete. 2024-05-09T08:35:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.618+0000 7fd2a4e31740 1 -- start start 2024-05-09T08:35:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.619+0000 7fd2a4e31740 1 Processor -- start 2024-05-09T08:35:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.619+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df215580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.619+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df20dd70 0x5597df215ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.619+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df218630 0x5597df216000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.619+0000 7fd2a4e31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5597df21b080 con 0x5597df20dd70 2024-05-09T08:35:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.619+0000 7fd2a4e31740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5597df12d790 con 0x5597df20d450 2024-05-09T08:35:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.619+0000 7fd2a4e31740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5597df21b200 con 0x5597df218630 2024-05-09T08:35:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.619+0000 7fd2a4e31740 10 client.0 _ll_register_callbacks cb 0x5597df168a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29b7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df215580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29b7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df215580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33960/0 (socket says 172.21.15.28:33960) 2024-05-09T08:35:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29b7fe640 1 -- 172.21.15.28:0/1826437147 learned_addr learned my addr 172.21.15.28:0/1826437147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29bfff640 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df218630 0x5597df216000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29b7fe640 1 -- 172.21.15.28:0/1826437147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df218630 msgr2=0x5597df216000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df218630 0x5597df216000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29b7fe640 1 -- 172.21.15.28:0/1826437147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df20dd70 msgr2=0x5597df215ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:35:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df20dd70 0x5597df215ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29b7fe640 1 -- 172.21.15.28:0/1826437147 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597dee86210 con 0x5597df20d450 2024-05-09T08:35:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.620+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df215580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd28c09ba20 tx=0x7fd28c09bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5300e7cc74e2025a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.621+0000 7fd27affd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:35:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.621+0000 7fd27affd640 1 -- 172.21.15.28:0/1826437147 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c0cd3e0 con 0x5597df20d450 2024-05-09T08:35:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.621+0000 7fd27affd640 1 -- 172.21.15.28:0/1826437147 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9595+0+0 (secure 0 0 0) 0x7fd28c09b370 con 0x5597df20d450 2024-05-09T08:35:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.621+0000 7fd2a4e31740 1 -- 172.21.15.28:0/1826437147 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd294030cd0 con 0x5597df20d450 2024-05-09T08:35:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.623+0000 7fd27affd640 1 -- 172.21.15.28:0/1826437147 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fd28c0a4be0 con 0x5597df20d450 2024-05-09T08:35:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.623+0000 7fd2a4e31740 10 client.26342 fetch_fsmap learned FSMap version 65 2024-05-09T08:35:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.623+0000 7fd2a4e31740 1 -- 172.21.15.28:0/1826437147 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5597def52620 con 0x5597df20d450 2024-05-09T08:35:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.625+0000 7fd27affd640 1 -- 172.21.15.28:0/1826437147 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd294030cd0 con 0x5597df20d450 2024-05-09T08:35:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.625+0000 7fd27affd640 1 -- 172.21.15.28:0/1826437147 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7fd28c0a1260 con 0x5597df20d450 2024-05-09T08:35:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.625+0000 7fd2a4e31740 10 client.26342 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:35:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.625+0000 7fd2a4e31740 10 client.26342 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:35:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.625+0000 7fd2a4e31740 20 client.26342 populate_metadata read hostname 'smithi028' 2024-05-09T08:35:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.625+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5597df228680 0x5597df22ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.625+0000 7fd2a4e31740 4 client.26342 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.625+0000 7fd2a4e31740 1 -- 172.21.15.28:0/1826437147 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597df17fa70 con 0x5597df228680 2024-05-09T08:35:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.627+0000 7fd29affd640 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5597df228680 0x5597df22ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.627+0000 7fd29affd640 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5597df228680 0x5597df22ab00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7647003485d67c server_cookie=2788211f964d5d74 in_seq=0 out_seq=0 2024-05-09T08:35:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.627+0000 7fd27affd640 10 client.26342 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.628+0000 7fd27affd640 1 -- 172.21.15.28:0/1826437147 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5597df17fa70 con 0x5597df228680 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.628+0000 7fd27affd640 10 client.26342 handle_command_reply: tid=0 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 client.26342 shutdown 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 -- 172.21.15.28:0/1826437147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 msgr2=0x5597df215580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df215580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd28c09ba20 tx=0x7fd28c09bde0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 -- 172.21.15.28:0/1826437147 shutdown_connections 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5597df228680 0x5597df22ab00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df218630 0x5597df216000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df20dd70 0x5597df215ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd27affd640 0 client.26342 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/1826437147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df215580 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 -- 172.21.15.28:0/1826437147 >> 172.21.15.28:0/1826437147 conn(0x5597df209960 msgr2=0x5597df20f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 -- 172.21.15.28:0/1826437147 shutdown_connections 2024-05-09T08:35:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 1 -- 172.21.15.28:0/1826437147 wait complete. 2024-05-09T08:35:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.629+0000 7fd2a4e31740 20 client.26342 trim_cache size 0 max 16384 2024-05-09T08:35:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.656+0000 7fd2a4e31740 1 Processor -- start 2024-05-09T08:35:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.656+0000 7fd2a4e31740 1 -- start start 2024-05-09T08:35:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df217cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 0x5597df20fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 0x5597df215e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd29b7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df217cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd2a4e31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd29c1b5610 con 0x5597df218630 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd2a4e31740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd29c1b7390 con 0x5597df20d450 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd29b7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df217cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33968/0 (socket says 172.21.15.28:33968) 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd29b7fe640 1 -- 172.21.15.28:0/3519027798 learned_addr learned my addr 172.21.15.28:0/3519027798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd2a4e31740 1 -- 172.21.15.28:0/3519027798 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd29c1b5da0 con 0x5597df20dd70 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd29bfff640 1 --2- 172.21.15.28:0/3519027798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 0x5597df215e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.657+0000 7fd29affd640 1 --2- 172.21.15.28:0/3519027798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 0x5597df20fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.658+0000 7fd29b7fe640 1 -- 172.21.15.28:0/3519027798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 msgr2=0x5597df20fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.658+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/3519027798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 0x5597df20fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.658+0000 7fd29b7fe640 1 -- 172.21.15.28:0/3519027798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 msgr2=0x5597df215e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.658+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/3519027798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 0x5597df215e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.658+0000 7fd29b7fe640 1 -- 172.21.15.28:0/3519027798 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597df210350 con 0x5597df20d450 2024-05-09T08:35:26.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.658+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/3519027798 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df217cb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0a4460 tx=0x7fd28c0980f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f51e96ec021631c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.658+0000 7fd2817fa640 1 -- 172.21.15.28:0/3519027798 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c09c420 con 0x5597df20d450 2024-05-09T08:35:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.658+0000 7fd2817fa640 1 -- 172.21.15.28:0/3519027798 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd28c0a1260 con 0x5597df20d450 2024-05-09T08:35:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.658+0000 7fd2817fa640 1 -- 172.21.15.28:0/3519027798 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c018e00 con 0x5597df20d450 2024-05-09T08:35:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.659+0000 7fd2a4e31740 1 -- 172.21.15.28:0/3519027798 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 msgr2=0x5597df217cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.659+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/3519027798 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df217cb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0a4460 tx=0x7fd28c0980f0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.660+0000 7fd2a4e31740 1 -- 172.21.15.28:0/3519027798 shutdown_connections 2024-05-09T08:35:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.660+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/3519027798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 0x5597df215e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.660+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/3519027798 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20dd70 0x5597df20fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.660+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/3519027798 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20d450 0x5597df217cb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.660+0000 7fd2a4e31740 1 -- 172.21.15.28:0/3519027798 >> 172.21.15.28:0/3519027798 conn(0x5597df209960 msgr2=0x5597df208170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.662+0000 7fd2a4e31740 1 -- 172.21.15.28:0/3519027798 shutdown_connections 2024-05-09T08:35:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.662+0000 7fd2a4e31740 1 -- 172.21.15.28:0/3519027798 wait complete. 2024-05-09T08:35:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.663+0000 7fd2a4e31740 1 -- start start 2024-05-09T08:35:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.663+0000 7fd2a4e31740 1 Processor -- start 2024-05-09T08:35:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.664+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20d450 0x5597df21ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.664+0000 7fd2a4e31740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20dd70 0x5597df21bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.664+0000 7fd29b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20d450 0x5597df21ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.665+0000 7fd29b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20d450 0x5597df21ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38002/0 (socket says 172.21.15.28:38002) 2024-05-09T08:35:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.665+0000 7fd29b7fe640 1 -- 172.21.15.28:0/2263027807 learned_addr learned my addr 172.21.15.28:0/2263027807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.665+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 0x5597df21fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.665+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd29c1b6cc0 con 0x5597df218630 2024-05-09T08:35:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.665+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd29c1b6530 con 0x5597df20dd70 2024-05-09T08:35:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.665+0000 7fd29bfff640 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 0x5597df21fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.665+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd29c11b8b0 con 0x5597df20d450 2024-05-09T08:35:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.665+0000 7fd29affd640 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20dd70 0x5597df21bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.665+0000 7fd2a4e31740 10 client.0 _ll_register_callbacks cb 0x7fd28c098040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd29b7fe640 1 -- 172.21.15.28:0/2263027807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20dd70 msgr2=0x5597df21bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20dd70 0x5597df21bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd29b7fe640 1 -- 172.21.15.28:0/2263027807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 msgr2=0x5597df21fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 0x5597df21fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd29b7fe640 1 -- 172.21.15.28:0/2263027807 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597df220510 con 0x5597df20d450 2024-05-09T08:35:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd29b7fe640 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20d450 0x5597df21ba30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0986a0 tx=0x7fd28c0a01d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5c465120a8034d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:26.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd27affd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:35:26.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd27affd640 1 -- 172.21.15.28:0/2263027807 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c0a1410 con 0x5597df20d450 2024-05-09T08:35:26.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd27affd640 1 -- 172.21.15.28:0/2263027807 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9595+0+0 (secure 0 0 0) 0x7fd28c07abc0 con 0x5597df20d450 2024-05-09T08:35:26.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.666+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd28c0a1260 con 0x5597df20d450 2024-05-09T08:35:26.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.667+0000 7fd27affd640 1 -- 172.21.15.28:0/2263027807 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fd28c09c420 con 0x5597df20d450 2024-05-09T08:35:26.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.668+0000 7fd2a4e31740 10 client.26209 fetch_fsmap learned FSMap version 65 2024-05-09T08:35:26.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.668+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5597df28e7c0 con 0x5597df20d450 2024-05-09T08:35:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.669+0000 7fd27affd640 1 -- 172.21.15.28:0/2263027807 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd28c07adc0 con 0x5597df20d450 2024-05-09T08:35:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.669+0000 7fd27affd640 1 -- 172.21.15.28:0/2263027807 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7fd28c0ba3c0 con 0x5597df20d450 2024-05-09T08:35:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.669+0000 7fd2a4e31740 10 client.26209 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:35:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.669+0000 7fd2a4e31740 10 client.26209 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:35:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.669+0000 7fd2a4e31740 20 client.26209 populate_metadata read hostname 'smithi028' 2024-05-09T08:35:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.669+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5597df28eed0 0x5597df2912f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.669+0000 7fd2a4e31740 4 client.26209 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:35:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.669+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd28c0a1260 con 0x5597df28eed0 2024-05-09T08:35:26.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.670+0000 7fd29affd640 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5597df28eed0 0x5597df2912f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:26.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.671+0000 7fd27affd640 10 client.26209 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:26.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.671+0000 7fd29affd640 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5597df28eed0 0x5597df2912f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=800b8b2b6b8f8a50 server_cookie=74c03d631efd279e in_seq=0 out_seq=0 2024-05-09T08:35:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.748+0000 7fd27affd640 1 -- 172.21.15.28:0/2263027807 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd28c0a1260 con 0x5597df28eed0 2024-05-09T08:35:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.748+0000 7fd27affd640 10 client.26209 handle_command_reply: tid=0 2024-05-09T08:35:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.748+0000 7fd2a4e31740 1 client.26209 shutdown 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20d450 msgr2=0x5597df21ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20d450 0x5597df21ba30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0986a0 tx=0x7fd28c0a01d0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 shutdown_connections 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5597df28eed0 0x5597df2912f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd27affd640 0 client.26209 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597df218630 0x5597df21fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5597df20dd70 0x5597df21bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 --2- 172.21.15.28:0/2263027807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5597df20d450 0x5597df21ba30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 >> 172.21.15.28:0/2263027807 conn(0x5597df209960 msgr2=0x5597df208a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 shutdown_connections 2024-05-09T08:35:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 1 -- 172.21.15.28:0/2263027807 wait complete. 2024-05-09T08:35:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.749+0000 7fd2a4e31740 20 client.26209 trim_cache size 0 max 16384 2024-05-09T08:35:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:35:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:35:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:35:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:35:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 -- 172.21.15.28:0/2077059174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2840b3f90 msgr2=0x7fd2840b6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2840b3f90 0x7fd2840b6450 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd294006010 tx=0x7fd294005d80 comp rx=0 tx=0).stop 2024-05-09T08:35:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 -- 172.21.15.28:0/2077059174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c1009a0 msgr2=0x7fd29c10d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c1009a0 0x7fd29c10d420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00e9f0 tx=0x7fd28c00eeb0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 -- 172.21.15.28:0/2077059174 shutdown_connections 2024-05-09T08:35:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2840b3f90 0x7fd2840b6450 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd29c1067d0 0x7fd29c10aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd29c101340 0x7fd29c10a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 --2- 172.21.15.28:0/2077059174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c1009a0 0x7fd29c10d420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 -- 172.21.15.28:0/2077059174 >> 172.21.15.28:0/2077059174 conn(0x7fd29c077610 msgr2=0x7fd29c0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 -- 172.21.15.28:0/2077059174 shutdown_connections 2024-05-09T08:35:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:26.750+0000 7fd2a204c640 1 -- 172.21.15.28:0/2077059174 wait complete. 2024-05-09T08:35:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:27 smithi165 ceph-mon[34540]: pgmap v289: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 2.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 222 op/s 2024-05-09T08:35:28.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:27 smithi184 ceph-mon[34509]: pgmap v289: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 2.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 222 op/s 2024-05-09T08:35:28.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:27 smithi028 ceph-mon[30119]: pgmap v289: 129 pgs: 9 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 101 active+clean; 2.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 222 op/s 2024-05-09T08:35:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:29 smithi165 ceph-mon[34540]: pgmap v290: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 224 op/s 2024-05-09T08:35:30.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:29 smithi184 ceph-mon[34509]: pgmap v290: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 224 op/s 2024-05-09T08:35:30.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:29 smithi028 ceph-mon[30119]: pgmap v290: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 224 op/s 2024-05-09T08:35:31.983 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:31 smithi028 ceph-mon[30119]: pgmap v291: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 250 op/s 2024-05-09T08:35:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:31 smithi165 ceph-mon[34540]: pgmap v291: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 250 op/s 2024-05-09T08:35:32.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:31 smithi184 ceph-mon[34509]: pgmap v291: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 250 op/s 2024-05-09T08:35:34.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:33 smithi165 ceph-mon[34540]: pgmap v292: 129 pgs: 129 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 257 op/s 2024-05-09T08:35:34.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:33 smithi184 ceph-mon[34509]: pgmap v292: 129 pgs: 129 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 257 op/s 2024-05-09T08:35:34.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:33 smithi028 ceph-mon[30119]: pgmap v292: 129 pgs: 129 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 257 op/s 2024-05-09T08:35:36.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:35 smithi165 ceph-mon[34540]: pgmap v293: 129 pgs: 129 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 272 op/s 2024-05-09T08:35:36.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:35 smithi165 ceph-mon[34540]: osdmap e107: 12 total, 12 up, 12 in 2024-05-09T08:35:36.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:35 smithi184 ceph-mon[34509]: pgmap v293: 129 pgs: 129 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 272 op/s 2024-05-09T08:35:36.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:35 smithi184 ceph-mon[34509]: osdmap e107: 12 total, 12 up, 12 in 2024-05-09T08:35:36.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:35 smithi028 ceph-mon[30119]: pgmap v293: 129 pgs: 129 active+clean; 2.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 272 op/s 2024-05-09T08:35:36.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:35 smithi028 ceph-mon[30119]: osdmap e107: 12 total, 12 up, 12 in 2024-05-09T08:35:38.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:37 smithi165 ceph-mon[34540]: pgmap v295: 129 pgs: 129 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 273 op/s 2024-05-09T08:35:38.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:37 smithi184 ceph-mon[34509]: pgmap v295: 129 pgs: 129 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 273 op/s 2024-05-09T08:35:38.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:37 smithi028 ceph-mon[30119]: pgmap v295: 129 pgs: 129 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 273 op/s 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.850+0000 7f76ec7ee640 1 -- 172.21.15.28:0/2349478966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e4101120 msgr2=0x7f76e410d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.850+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/2349478966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e4101120 0x7f76e410d5e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f76dc009cd0 tx=0x7f76dc02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.851+0000 7f76ec7ee640 1 -- 172.21.15.28:0/2349478966 shutdown_connections 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.851+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/2349478966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e4101120 0x7f76e410d5e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.851+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/2349478966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e4100760 0x7f76e4100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.851+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/2349478966 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76e4106720 0x7f76e4106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.851+0000 7f76ec7ee640 1 -- 172.21.15.28:0/2349478966 >> 172.21.15.28:0/2349478966 conn(0x7f76e40778b0 msgr2=0x7f76e4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.851+0000 7f76ec7ee640 1 -- 172.21.15.28:0/2349478966 shutdown_connections 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.851+0000 7f76ec7ee640 1 -- 172.21.15.28:0/2349478966 wait complete. 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.851+0000 7f76ec7ee640 1 Processor -- start 2024-05-09T08:35:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.851+0000 7f76ec7ee640 1 -- start start 2024-05-09T08:35:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76ec7ee640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e4100760 0x7f76e419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76ec7ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76e4101120 0x7f76e419f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76ec7ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e4106720 0x7f76e419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76ec7ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76e410faa0 con 0x7f76e4106720 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76ec7ee640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f76e410f920 con 0x7f76e4100760 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76ec7ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f76e410fc20 con 0x7f76e4101120 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76ead8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e4106720 0x7f76e419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76e9d89640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76e4101120 0x7f76e419f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76e9d89640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76e4101120 0x7f76e419f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50810/0 (socket says 172.21.15.28:50810) 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76ead8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e4106720 0x7f76e419ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43202/0 (socket says 172.21.15.28:43202) 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76ea58a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e4100760 0x7f76e419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76e9d89640 1 -- 172.21.15.28:0/458230601 learned_addr learned my addr 172.21.15.28:0/458230601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76e9d89640 1 -- 172.21.15.28:0/458230601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e4100760 msgr2=0x7f76e419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76e9d89640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e4100760 0x7f76e419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76e9d89640 1 -- 172.21.15.28:0/458230601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e4106720 msgr2=0x7f76e419ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76e9d89640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e4106720 0x7f76e419ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.852+0000 7f76e9d89640 1 -- 172.21.15.28:0/458230601 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e419b420 con 0x7f76e4101120 2024-05-09T08:35:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.853+0000 7f76e9d89640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76e4101120 0x7f76e419f450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f76d400e9f0 tx=0x7f76d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.853+0000 7f76db7fe640 1 -- 172.21.15.28:0/458230601 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d400cd80 con 0x7f76e4101120 2024-05-09T08:35:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.853+0000 7f76ec7ee640 1 -- 172.21.15.28:0/458230601 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76e419b710 con 0x7f76e4101120 2024-05-09T08:35:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.853+0000 7f76db7fe640 1 -- 172.21.15.28:0/458230601 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d4010910 con 0x7f76e4101120 2024-05-09T08:35:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.853+0000 7f76db7fe640 1 -- 172.21.15.28:0/458230601 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d40188f0 con 0x7f76e4101120 2024-05-09T08:35:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.853+0000 7f76ec7ee640 1 -- 172.21.15.28:0/458230601 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76e41a6e40 con 0x7f76e4101120 2024-05-09T08:35:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.855+0000 7f76db7fe640 1 -- 172.21.15.28:0/458230601 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f76d4010ac0 con 0x7f76e4101120 2024-05-09T08:35:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.856+0000 7f76db7fe640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76d00a2da0 0x7f76d00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.856+0000 7f76ea58a640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76d00a2da0 0x7f76d00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.856+0000 7f76ef5de740 1 Processor -- start 2024-05-09T08:35:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76db7fe640 1 -- 172.21.15.28:0/458230601 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9567+0+0 (secure 0 0 0) 0x7f76d40cfc70 con 0x7f76e4101120 2024-05-09T08:35:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76ef5de740 1 -- start start 2024-05-09T08:35:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76ea58a640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76d00a2da0 0x7f76d00a5260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009b50 tx=0x7f76e0005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec29fea830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29ff5630 0x55ec29ff7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76ef5de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ec29f4f340 con 0x55ec29fead70 2024-05-09T08:35:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76ef5de740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ec29ef7290 con 0x55ec29ff5630 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76ef5de740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ec29f4b300 con 0x55ec29fea450 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76e9d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76ead8b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29ff5630 0x55ec29ff7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76e9d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43208/0 (socket says 172.21.15.28:43208) 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.857+0000 7f76e9d89640 1 -- 172.21.15.28:0/1690465784 learned_addr learned my addr 172.21.15.28:0/1690465784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76e9d89640 1 -- 172.21.15.28:0/1690465784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 msgr2=0x55ec29fea830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76e9d89640 1 --2- 172.21.15.28:0/1690465784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec29fea830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76e9d89640 1 -- 172.21.15.28:0/1690465784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29ff5630 msgr2=0x55ec29ff7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76e9d89640 1 --2- 172.21.15.28:0/1690465784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29ff5630 0x55ec29ff7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76e9d89640 1 -- 172.21.15.28:0/1690465784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec29f1b8e0 con 0x55ec29fead70 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76ead8b640 1 --2- 172.21.15.28:0/1690465784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29ff5630 0x55ec29ff7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76e9d89640 1 --2- 172.21.15.28:0/1690465784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff50f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f76d4098230 tx=0x7f76d40a00d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fedebfb7a895232 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76d8ff9640 1 -- 172.21.15.28:0/1690465784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d4098440 con 0x55ec29fead70 2024-05-09T08:35:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76d8ff9640 1 -- 172.21.15.28:0/1690465784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d409b220 con 0x55ec29fead70 2024-05-09T08:35:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.858+0000 7f76d8ff9640 1 -- 172.21.15.28:0/1690465784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d4097e80 con 0x55ec29fead70 2024-05-09T08:35:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.859+0000 7f76ef5de740 1 -- 172.21.15.28:0/1690465784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 msgr2=0x55ec29ff50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.859+0000 7f76ef5de740 1 --2- 172.21.15.28:0/1690465784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff50f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f76d4098230 tx=0x7f76d40a00d0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.859+0000 7f76ef5de740 1 -- 172.21.15.28:0/1690465784 shutdown_connections 2024-05-09T08:35:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.859+0000 7f76ef5de740 1 --2- 172.21.15.28:0/1690465784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29ff5630 0x55ec29ff7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.859+0000 7f76ef5de740 1 --2- 172.21.15.28:0/1690465784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff50f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.859+0000 7f76ef5de740 1 --2- 172.21.15.28:0/1690465784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec29fea830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.859+0000 7f76ef5de740 1 -- 172.21.15.28:0/1690465784 >> 172.21.15.28:0/1690465784 conn(0x55ec29fe6960 msgr2=0x55ec29fe6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.859+0000 7f76ef5de740 1 -- 172.21.15.28:0/1690465784 shutdown_connections 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.859+0000 7f76ef5de740 1 -- 172.21.15.28:0/1690465784 wait complete. 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76ef5de740 1 -- start start 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76ef5de740 1 Processor -- start 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fea450 0x55ec29ff25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29ff5630 0x55ec29ff3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76ef5de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ec29ff8260 con 0x55ec29fead70 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76ef5de740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ec29f0a790 con 0x55ec29fea450 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76ef5de740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ec29ff83e0 con 0x55ec29ff5630 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76e9d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76ef5de740 10 client.0 _ll_register_callbacks cb 0x55ec29f45a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.860+0000 7f76e9d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43214/0 (socket says 172.21.15.28:43214) 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76e9d89640 1 -- 172.21.15.28:0/3866597864 learned_addr learned my addr 172.21.15.28:0/3866597864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76ead8b640 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29ff5630 0x55ec29ff3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76ea58a640 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fea450 0x55ec29ff25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76e9d89640 1 -- 172.21.15.28:0/3866597864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29ff5630 msgr2=0x55ec29ff3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76e9d89640 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29ff5630 0x55ec29ff3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76e9d89640 1 -- 172.21.15.28:0/3866597864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fea450 msgr2=0x55ec29ff25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76e9d89640 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fea450 0x55ec29ff25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76e9d89640 1 -- 172.21.15.28:0/3866597864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec29c63210 con 0x55ec29fead70 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76e9d89640 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff2b30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f76d409bad0 tx=0x7f76d4009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ec8e870154197e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76cdffb640 1 -- 172.21.15.28:0/3866597864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d40a9060 con 0x55ec29fead70 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76cdffb640 1 -- 172.21.15.28:0/3866597864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9567+0+0 (secure 0 0 0) 0x7f76d40a4de0 con 0x55ec29fead70 2024-05-09T08:35:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76ef5de740 1 -- 172.21.15.28:0/3866597864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76d409b220 con 0x55ec29fead70 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76cdffb640 1 -- 172.21.15.28:0/3866597864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f76d4097c20 con 0x55ec29fead70 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76ef5de740 10 client.16851 fetch_fsmap learned FSMap version 65 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.861+0000 7f76ef5de740 1 -- 172.21.15.28:0/3866597864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55ec29d2f620 con 0x55ec29fead70 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.862+0000 7f76cdffb640 1 -- 172.21.15.28:0/3866597864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d409b220 con 0x55ec29fead70 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.862+0000 7f76cdffb640 1 -- 172.21.15.28:0/3866597864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f76d40b3070 con 0x55ec29fead70 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.862+0000 7f76ef5de740 10 client.16851 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.862+0000 7f76ef5de740 10 client.16851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.862+0000 7f76ef5de740 20 client.16851 populate_metadata read hostname 'smithi028' 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.862+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ec2a005250 0x55ec2a007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.862+0000 7f76ef5de740 4 client.16851 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.862+0000 7f76ef5de740 1 -- 172.21.15.28:0/3866597864 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec29f5ca70 con 0x55ec2a005250 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.862+0000 7f76ea58a640 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ec2a005250 0x55ec2a007640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.863+0000 7f76ea58a640 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ec2a005250 0x55ec2a007640 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e56990b155a8731 server_cookie=ebfa738edf8b5887 in_seq=0 out_seq=0 2024-05-09T08:35:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.863+0000 7f76cdffb640 10 client.16851 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.863+0000 7f76cdffb640 1 -- 172.21.15.28:0/3866597864 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec29f5ca70 con 0x55ec2a005250 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.863+0000 7f76cdffb640 10 client.16851 handle_command_reply: tid=0 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.863+0000 7f76ef5de740 1 client.16851 shutdown 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.863+0000 7f76ef5de740 1 -- 172.21.15.28:0/3866597864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 msgr2=0x55ec29ff2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.863+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff2b30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f76d409bad0 tx=0x7f76d4009700 comp rx=0 tx=0).stop 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.863+0000 7f76ef5de740 1 -- 172.21.15.28:0/3866597864 shutdown_connections 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.863+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ec2a005250 0x55ec2a007640 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.864+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29ff5630 0x55ec29ff3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.864+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29fead70 0x55ec29ff2b30 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.864+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3866597864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fea450 0x55ec29ff25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.864+0000 7f76ef5de740 1 -- 172.21.15.28:0/3866597864 >> 172.21.15.28:0/3866597864 conn(0x55ec29fe6960 msgr2=0x55ec29fec690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.864+0000 7f76cdffb640 0 client.16851 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.864+0000 7f76ef5de740 1 -- 172.21.15.28:0/3866597864 shutdown_connections 2024-05-09T08:35:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.864+0000 7f76ef5de740 1 -- 172.21.15.28:0/3866597864 wait complete. 2024-05-09T08:35:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.864+0000 7f76ef5de740 20 client.16851 trim_cache size 0 max 16384 2024-05-09T08:35:39.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.895+0000 7f76ef5de740 1 Processor -- start 2024-05-09T08:35:39.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ef5de740 1 -- start start 2024-05-09T08:35:39.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec2a001120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 0x55ec29ff2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec2a00cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ef5de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41b1080 con 0x55ec29ff5630 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ef5de740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41b2e00 con 0x55ec29fead70 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ef5de740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41b1810 con 0x55ec29fea450 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ea58a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec2a001120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ead8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec2a00cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ea58a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec2a001120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50836/0 (socket says 172.21.15.28:50836) 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ea58a640 1 -- 172.21.15.28:0/3252769279 learned_addr learned my addr 172.21.15.28:0/3252769279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76e9d89640 1 --2- 172.21.15.28:0/3252769279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 0x55ec29ff2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76e9d89640 1 -- 172.21.15.28:0/3252769279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 msgr2=0x55ec2a001120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76e9d89640 1 --2- 172.21.15.28:0/3252769279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec2a001120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76e9d89640 1 -- 172.21.15.28:0/3252769279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 msgr2=0x55ec2a00cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76e9d89640 1 --2- 172.21.15.28:0/3252769279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec2a00cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76e9d89640 1 -- 172.21.15.28:0/3252769279 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec29feb920 con 0x55ec29fead70 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.896+0000 7f76ea58a640 1 --2- 172.21.15.28:0/3252769279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec2a001120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.897+0000 7f76e9d89640 1 --2- 172.21.15.28:0/3252769279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 0x55ec29ff2e90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f76d40987a0 tx=0x7f76d4009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95ff167ce440dc89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.897+0000 7f76cffff640 1 -- 172.21.15.28:0/3252769279 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d40b8070 con 0x55ec29fead70 2024-05-09T08:35:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.897+0000 7f76cffff640 1 -- 172.21.15.28:0/3252769279 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d40a4be0 con 0x55ec29fead70 2024-05-09T08:35:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.897+0000 7f76cffff640 1 -- 172.21.15.28:0/3252769279 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d40a9070 con 0x55ec29fead70 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 -- 172.21.15.28:0/3252769279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 msgr2=0x55ec29ff2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3252769279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 0x55ec29ff2e90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f76d40987a0 tx=0x7f76d4009770 comp rx=0 tx=0).stop 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 -- 172.21.15.28:0/3252769279 shutdown_connections 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3252769279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec2a00cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3252769279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 0x55ec29ff2e90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3252769279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec2a001120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 -- 172.21.15.28:0/3252769279 >> 172.21.15.28:0/3252769279 conn(0x55ec29fe6960 msgr2=0x55ec29fe5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 -- 172.21.15.28:0/3252769279 shutdown_connections 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 -- 172.21.15.28:0/3252769279 wait complete. 2024-05-09T08:35:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 -- start start 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 Processor -- start 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec29ff8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 0x55ec29ff9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec29ffd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41b2730 con 0x55ec29ff5630 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41b1fa0 con 0x55ec29fead70 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41af080 con 0x55ec29fea450 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.898+0000 7f76ef5de740 10 client.0 _ll_register_callbacks cb 0x7f76e0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ead8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec29ffd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ea58a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec29ff8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ead8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec29ffd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43234/0 (socket says 172.21.15.28:43234) 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ead8b640 1 -- 172.21.15.28:0/3759859681 learned_addr learned my addr 172.21.15.28:0/3759859681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76e9d89640 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 0x55ec29ff9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ead8b640 1 -- 172.21.15.28:0/3759859681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 msgr2=0x55ec29ff8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ead8b640 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec29ff8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ead8b640 1 -- 172.21.15.28:0/3759859681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 msgr2=0x55ec29ff9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ead8b640 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 0x55ec29ff9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ead8b640 1 -- 172.21.15.28:0/3759859681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec29ffd6f0 con 0x55ec29ff5630 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ead8b640 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec29ffd080 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f76dc009cd0 tx=0x7f76dc031c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3643094dfd245581 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76cdffb640 1 -- 172.21.15.28:0/3759859681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc003b80 con 0x55ec29ff5630 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76cdffb640 1 -- 172.21.15.28:0/3759859681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9567+0+0 (secure 0 0 0) 0x7f76dc003d20 con 0x55ec29ff5630 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ef5de740 1 -- 172.21.15.28:0/3759859681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76d40a4be0 con 0x55ec29ff5630 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76cdffb640 1 -- 172.21.15.28:0/3759859681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f76dc055050 con 0x55ec29ff5630 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ef5de740 10 client.16857 fetch_fsmap learned FSMap version 65 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.899+0000 7f76ef5de740 1 -- 172.21.15.28:0/3759859681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55ec2a06afe0 con 0x55ec29ff5630 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.900+0000 7f76cdffb640 1 -- 172.21.15.28:0/3759859681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d40a4be0 con 0x55ec29ff5630 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.900+0000 7f76cdffb640 1 -- 172.21.15.28:0/3759859681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f76dc030070 con 0x55ec29ff5630 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.900+0000 7f76ef5de740 10 client.16857 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:35:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.900+0000 7f76ef5de740 10 client.16857 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:35:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.900+0000 7f76ef5de740 20 client.16857 populate_metadata read hostname 'smithi028' 2024-05-09T08:35:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.900+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ec2a06b8d0 0x55ec2a06dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.900+0000 7f76ef5de740 4 client.16857 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:35:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.900+0000 7f76ef5de740 1 -- 172.21.15.28:0/3759859681 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ec29f56dc0 con 0x55ec2a06b8d0 2024-05-09T08:35:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.900+0000 7f76ea58a640 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ec2a06b8d0 0x55ec2a06dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.901+0000 7f76ea58a640 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ec2a06b8d0 0x55ec2a06dcf0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67a97316ace17a94 server_cookie=ce69b444c1d08d84 in_seq=0 out_seq=0 2024-05-09T08:35:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:39.901+0000 7f76cdffb640 10 client.16857 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:40.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:39 smithi165 ceph-mon[34540]: pgmap v296: 129 pgs: 129 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 258 op/s 2024-05-09T08:35:40.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:39 smithi184 ceph-mon[34509]: pgmap v296: 129 pgs: 129 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 258 op/s 2024-05-09T08:35:40.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:39 smithi028 ceph-mon[30119]: pgmap v296: 129 pgs: 129 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 258 op/s 2024-05-09T08:35:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.273+0000 7f76cdffb640 1 -- 172.21.15.28:0/3759859681 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ec29f56dc0 con 0x55ec2a06b8d0 2024-05-09T08:35:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.273+0000 7f76cdffb640 10 client.16857 handle_command_reply: tid=0 2024-05-09T08:35:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.273+0000 7f76ef5de740 1 client.16857 shutdown 2024-05-09T08:35:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.273+0000 7f76ef5de740 1 -- 172.21.15.28:0/3759859681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 msgr2=0x55ec29ffd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec29ffd080 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f76dc009cd0 tx=0x7f76dc031c30 comp rx=0 tx=0).stop 2024-05-09T08:35:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 1 -- 172.21.15.28:0/3759859681 shutdown_connections 2024-05-09T08:35:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ec2a06b8d0 0x55ec2a06dcf0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec29ff5630 0x55ec29ffd080 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76cdffb640 0 client.16857 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ec29fead70 0x55ec29ff9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 1 --2- 172.21.15.28:0/3759859681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ec29fea450 0x55ec29ff8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 1 -- 172.21.15.28:0/3759859681 >> 172.21.15.28:0/3759859681 conn(0x55ec29fe6960 msgr2=0x55ec29febda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 1 -- 172.21.15.28:0/3759859681 shutdown_connections 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 1 -- 172.21.15.28:0/3759859681 wait complete. 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.274+0000 7f76ef5de740 20 client.16857 trim_cache size 0 max 16384 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:35:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:35:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:35:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.275+0000 7f76ec7ee640 1 -- 172.21.15.28:0/458230601 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76d00a2da0 msgr2=0x7f76d00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.275+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76d00a2da0 0x7f76d00a5260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009b50 tx=0x7f76e0005f90 comp rx=0 tx=0).stop 2024-05-09T08:35:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.275+0000 7f76ec7ee640 1 -- 172.21.15.28:0/458230601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76e4101120 msgr2=0x7f76e419f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.275+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76e4101120 0x7f76e419f450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f76d400e9f0 tx=0x7f76d400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:35:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.276+0000 7f76ec7ee640 1 -- 172.21.15.28:0/458230601 shutdown_connections 2024-05-09T08:35:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.276+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e4106720 0x7f76e419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.276+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76d00a2da0 0x7f76d00a5260 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.276+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76e4101120 0x7f76e419f450 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.276+0000 7f76ec7ee640 1 --2- 172.21.15.28:0/458230601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76e4100760 0x7f76e419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.276+0000 7f76ec7ee640 1 -- 172.21.15.28:0/458230601 >> 172.21.15.28:0/458230601 conn(0x7f76e40778b0 msgr2=0x7f76e40fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.276+0000 7f76ec7ee640 1 -- 172.21.15.28:0/458230601 shutdown_connections 2024-05-09T08:35:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:40.276+0000 7f76ec7ee640 1 -- 172.21.15.28:0/458230601 wait complete. 2024-05-09T08:35:42.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:41 smithi165 ceph-mon[34540]: pgmap v297: 129 pgs: 129 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.9 MiB/s wr, 220 op/s 2024-05-09T08:35:42.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:41 smithi184 ceph-mon[34509]: pgmap v297: 129 pgs: 129 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.9 MiB/s wr, 220 op/s 2024-05-09T08:35:42.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:41 smithi028 ceph-mon[30119]: pgmap v297: 129 pgs: 129 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.9 MiB/s wr, 220 op/s 2024-05-09T08:35:44.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:43 smithi165 ceph-mon[34540]: pgmap v298: 129 pgs: 129 active+clean; 2.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.2 MiB/s wr, 225 op/s 2024-05-09T08:35:44.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:43 smithi184 ceph-mon[34509]: pgmap v298: 129 pgs: 129 active+clean; 2.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.2 MiB/s wr, 225 op/s 2024-05-09T08:35:44.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:43 smithi028 ceph-mon[30119]: pgmap v298: 129 pgs: 129 active+clean; 2.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.2 MiB/s wr, 225 op/s 2024-05-09T08:35:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:45 smithi165 ceph-mon[34540]: pgmap v299: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.6 MiB/s wr, 279 op/s 2024-05-09T08:35:46.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:45 smithi184 ceph-mon[34509]: pgmap v299: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.6 MiB/s wr, 279 op/s 2024-05-09T08:35:46.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:45 smithi028 ceph-mon[30119]: pgmap v299: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.6 MiB/s wr, 279 op/s 2024-05-09T08:35:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:47 smithi165 ceph-mon[34540]: pgmap v300: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 292 op/s 2024-05-09T08:35:48.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:47 smithi184 ceph-mon[34509]: pgmap v300: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 292 op/s 2024-05-09T08:35:48.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:47 smithi028 ceph-mon[30119]: pgmap v300: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 292 op/s 2024-05-09T08:35:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:49 smithi165 ceph-mon[34540]: pgmap v301: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.9 MiB/s wr, 301 op/s 2024-05-09T08:35:50.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:49 smithi184 ceph-mon[34509]: pgmap v301: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.9 MiB/s wr, 301 op/s 2024-05-09T08:35:50.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:49 smithi028 ceph-mon[30119]: pgmap v301: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.9 MiB/s wr, 301 op/s 2024-05-09T08:35:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:51 smithi165 ceph-mon[34540]: pgmap v302: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.7 MiB/s wr, 348 op/s 2024-05-09T08:35:52.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:51 smithi184 ceph-mon[34509]: pgmap v302: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.7 MiB/s wr, 348 op/s 2024-05-09T08:35:52.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:51 smithi028 ceph-mon[30119]: pgmap v302: 129 pgs: 129 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.7 MiB/s wr, 348 op/s 2024-05-09T08:35:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.409+0000 7f04fa431640 1 -- 172.21.15.28:0/3306778873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04f411b010 msgr2=0x7f04f411d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.409+0000 7f04fa431640 1 --2- 172.21.15.28:0/3306778873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04f411b010 0x7f04f411d400 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009890 tx=0x7f04e402a440 comp rx=0 tx=0).stop 2024-05-09T08:35:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.410+0000 7f04fa431640 1 -- 172.21.15.28:0/3306778873 shutdown_connections 2024-05-09T08:35:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.410+0000 7f04fa431640 1 --2- 172.21.15.28:0/3306778873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04f411b010 0x7f04f411d400 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.410+0000 7f04fa431640 1 --2- 172.21.15.28:0/3306778873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04f4110400 0x7f04f4108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.410+0000 7f04fa431640 1 --2- 172.21.15.28:0/3306778873 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04f4110020 0x7f04f4108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.410+0000 7f04fa431640 1 -- 172.21.15.28:0/3306778873 >> 172.21.15.28:0/3306778873 conn(0x7f04f406c900 msgr2=0x7f04f406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.410+0000 7f04fa431640 1 -- 172.21.15.28:0/3306778873 shutdown_connections 2024-05-09T08:35:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.411+0000 7f04fa431640 1 -- 172.21.15.28:0/3306778873 wait complete. 2024-05-09T08:35:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.411+0000 7f04fa431640 1 Processor -- start 2024-05-09T08:35:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.411+0000 7f04fa431640 1 -- start start 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.412+0000 7f04fa431640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04f4110020 0x7f04f411a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.412+0000 7f04fa431640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04f411b010 0x7f04f4116720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.412+0000 7f04fa431640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04f4116c60 0x7f04f4117110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.412+0000 7f04fa431640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04f4113230 con 0x7f04f411b010 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.412+0000 7f04fa431640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f04f41130b0 con 0x7f04f4116c60 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.412+0000 7f04fa431640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04f41133b0 con 0x7f04f4110020 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.412+0000 7f04f89ce640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04f4116c60 0x7f04f4117110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.412+0000 7f04f89ce640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04f4116c60 0x7f04f4117110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:47198/0 (socket says 172.21.15.28:47198) 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.412+0000 7f04f89ce640 1 -- 172.21.15.28:0/3378988216 learned_addr learned my addr 172.21.15.28:0/3378988216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.413+0000 7f04f89ce640 1 -- 172.21.15.28:0/3378988216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04f4110020 msgr2=0x7f04f411a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.413+0000 7f04f89ce640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04f4110020 0x7f04f411a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.413+0000 7f04f89ce640 1 -- 172.21.15.28:0/3378988216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04f411b010 msgr2=0x7f04f4116720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.413+0000 7f04f89ce640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04f411b010 0x7f04f4116720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.413+0000 7f04f89ce640 1 -- 172.21.15.28:0/3378988216 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f41aae00 con 0x7f04f4116c60 2024-05-09T08:35:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.414+0000 7f04f89ce640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04f4116c60 0x7f04f4117110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04e402a9b0 tx=0x7f04e40055e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.415+0000 7f04f17fa640 1 -- 172.21.15.28:0/3378988216 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4040070 con 0x7f04f4116c60 2024-05-09T08:35:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.415+0000 7f04fa431640 1 -- 172.21.15.28:0/3378988216 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04f41ab030 con 0x7f04f4116c60 2024-05-09T08:35:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.415+0000 7f04fa431640 1 -- 172.21.15.28:0/3378988216 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04f41ab4f0 con 0x7f04f4116c60 2024-05-09T08:35:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.415+0000 7f04f17fa640 1 -- 172.21.15.28:0/3378988216 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e4004460 con 0x7f04f4116c60 2024-05-09T08:35:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.416+0000 7f04f17fa640 1 -- 172.21.15.28:0/3378988216 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4004bc0 con 0x7f04f4116c60 2024-05-09T08:35:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.418+0000 7f04f17fa640 1 -- 172.21.15.28:0/3378988216 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f04e4043070 con 0x7f04f4116c60 2024-05-09T08:35:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.421+0000 7f04f17fa640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04dc0a2d20 0x7f04dc0a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.421+0000 7f04f17fa640 1 -- 172.21.15.28:0/3378988216 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9567+0+0 (secure 0 0 0) 0x7f04e40f0720 con 0x7f04f4116c60 2024-05-09T08:35:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.421+0000 7f04f3fff640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04dc0a2d20 0x7f04dc0a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.423+0000 7f04fd21a740 1 Processor -- start 2024-05-09T08:35:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.424+0000 7f04fd21a740 1 -- start start 2024-05-09T08:35:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.424+0000 7f04f3fff640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04dc0a2d20 0x7f04dc0a51e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04e80058c0 tx=0x7f04e8005850 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:53.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.424+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f1010a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.424+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f101150f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.425+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10117a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.425+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f1010a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.425+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f1010a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51444/0 (socket says 172.21.15.28:51444) 2024-05-09T08:35:53.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.425+0000 7f04f37fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f101150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.425+0000 7f04f37fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f101150f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:47210/0 (socket says 172.21.15.28:47210) 2024-05-09T08:35:53.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.425+0000 7f04f3fff640 1 -- 172.21.15.28:0/3119193430 learned_addr learned my addr 172.21.15.28:0/3119193430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:53.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.425+0000 7f04fd21a740 1 -- 172.21.15.28:0/3119193430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564f1006f340 con 0x564f10115630 2024-05-09T08:35:53.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.425+0000 7f04fd21a740 1 -- 172.21.15.28:0/3119193430 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x564f10017290 con 0x564f1010ad70 2024-05-09T08:35:53.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.425+0000 7f04fd21a740 1 -- 172.21.15.28:0/3119193430 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564f1006b300 con 0x564f1010a450 2024-05-09T08:35:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.427+0000 7f04f3fff640 1 -- 172.21.15.28:0/3119193430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 msgr2=0x564f101150f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.427+0000 7f04f3fff640 1 --2- 172.21.15.28:0/3119193430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f101150f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.427+0000 7f04f3fff640 1 -- 172.21.15.28:0/3119193430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 msgr2=0x564f10117a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.427+0000 7f04f3fff640 1 --2- 172.21.15.28:0/3119193430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10117a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.427+0000 7f04f3fff640 1 -- 172.21.15.28:0/3119193430 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f1003b8e0 con 0x564f1010a450 2024-05-09T08:35:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.428+0000 7f04f3fff640 1 --2- 172.21.15.28:0/3119193430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f1010a830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f04e801fd70 tx=0x7f04e8022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ebf14f54af7694f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.428+0000 7f04da7fc640 1 -- 172.21.15.28:0/3119193430 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e8031070 con 0x564f1010a450 2024-05-09T08:35:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.429+0000 7f04da7fc640 1 -- 172.21.15.28:0/3119193430 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e801a790 con 0x564f1010a450 2024-05-09T08:35:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.429+0000 7f04da7fc640 1 -- 172.21.15.28:0/3119193430 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e802dbf0 con 0x564f1010a450 2024-05-09T08:35:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.430+0000 7f04fd21a740 1 -- 172.21.15.28:0/3119193430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 msgr2=0x564f1010a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.431+0000 7f04fd21a740 1 --2- 172.21.15.28:0/3119193430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f1010a830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f04e801fd70 tx=0x7f04e8022000 comp rx=0 tx=0).stop 2024-05-09T08:35:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.431+0000 7f04fd21a740 1 -- 172.21.15.28:0/3119193430 shutdown_connections 2024-05-09T08:35:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.431+0000 7f04fd21a740 1 --2- 172.21.15.28:0/3119193430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10117a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.431+0000 7f04fd21a740 1 --2- 172.21.15.28:0/3119193430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f101150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.431+0000 7f04fd21a740 1 --2- 172.21.15.28:0/3119193430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f1010a830 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.431+0000 7f04fd21a740 1 -- 172.21.15.28:0/3119193430 >> 172.21.15.28:0/3119193430 conn(0x564f10106960 msgr2=0x564f10106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.431+0000 7f04fd21a740 1 -- 172.21.15.28:0/3119193430 shutdown_connections 2024-05-09T08:35:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.431+0000 7f04fd21a740 1 -- 172.21.15.28:0/3119193430 wait complete. 2024-05-09T08:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.432+0000 7f04fd21a740 1 -- start start 2024-05-09T08:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.432+0000 7f04fd21a740 1 Processor -- start 2024-05-09T08:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.433+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f1010a450 0x564f10112590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.433+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10112ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.433+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f1010a450 0x564f10112590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.433+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f1010a450 0x564f10112590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34034/0 (socket says 172.21.15.28:34034) 2024-05-09T08:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.433+0000 7f04f3fff640 1 -- 172.21.15.28:0/279492621 learned_addr learned my addr 172.21.15.28:0/279492621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.434+0000 7f04fd21a740 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f10115630 0x564f10113010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.434+0000 7f04f37fe640 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10112ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.434+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564f10118260 con 0x564f1010a450 2024-05-09T08:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.434+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x564f1002a790 con 0x564f1010ad70 2024-05-09T08:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.434+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564f101183e0 con 0x564f10115630 2024-05-09T08:35:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.434+0000 7f04fd21a740 10 client.0 _ll_register_callbacks cb 0x564f10065a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.435+0000 7f04f37fe640 1 -- 172.21.15.28:0/279492621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f10115630 msgr2=0x564f10113010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.435+0000 7f04f37fe640 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f10115630 0x564f10113010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.435+0000 7f04f37fe640 1 -- 172.21.15.28:0/279492621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f1010a450 msgr2=0x564f10112590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.435+0000 7f04f37fe640 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f1010a450 0x564f10112590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.435+0000 7f04f37fe640 1 -- 172.21.15.28:0/279492621 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f0fd83210 con 0x564f1010ad70 2024-05-09T08:35:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.435+0000 7f04f37fe640 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10112ad0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f04f4117f70 tx=0x7f04ec00cdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbd141b07f1db013 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.435+0000 7f04d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:35:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.435+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564f1007ca70 con 0x564f1010ad70 2024-05-09T08:35:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.435+0000 7f04d37fe640 1 -- 172.21.15.28:0/279492621 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec01a060 con 0x564f1010ad70 2024-05-09T08:35:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.436+0000 7f04d37fe640 1 -- 172.21.15.28:0/279492621 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9567+0+0 (secure 0 0 0) 0x7f04ec007920 con 0x564f1010ad70 2024-05-09T08:35:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.436+0000 7f04d37fe640 1 -- 172.21.15.28:0/279492621 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f04ec021050 con 0x564f1010ad70 2024-05-09T08:35:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.436+0000 7f04fd21a740 10 client.26375 fetch_fsmap learned FSMap version 65 2024-05-09T08:35:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.436+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({fsmap=65}) -- 0x564f0fe4f620 con 0x564f1010ad70 2024-05-09T08:35:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.439+0000 7f04d37fe640 1 -- 172.21.15.28:0/279492621 <== mon.2 v2:172.21.15.165:3300/0 4 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f04ec007b20 con 0x564f1010ad70 2024-05-09T08:35:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.439+0000 7f04fd21a740 10 client.26375 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:35:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.439+0000 7f04fd21a740 10 client.26375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:35:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.439+0000 7f04fd21a740 20 client.26375 populate_metadata read hostname 'smithi028' 2024-05-09T08:35:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.439+0000 7f04fd21a740 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564f101250e0 0x564f10127580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.439+0000 7f04fd21a740 4 client.26375 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.439+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f1007ca70 con 0x564f101250e0 2024-05-09T08:35:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.441+0000 7f04f3fff640 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564f101250e0 0x564f10127580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.442+0000 7f04d37fe640 10 client.26375 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.442+0000 7f04f3fff640 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564f101250e0 0x564f10127580 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9305a0aa60da1323 server_cookie=c93d2d5af9e80c37 in_seq=0 out_seq=0 2024-05-09T08:35:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04d37fe640 1 -- 172.21.15.28:0/279492621 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f1007ca70 con 0x564f101250e0 2024-05-09T08:35:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04d37fe640 10 client.26375 handle_command_reply: tid=0 2024-05-09T08:35:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04fd21a740 1 client.26375 shutdown 2024-05-09T08:35:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 msgr2=0x564f10112ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04fd21a740 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10112ad0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f04f4117f70 tx=0x7f04ec00cdf0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 shutdown_connections 2024-05-09T08:35:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04fd21a740 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564f101250e0 0x564f10127580 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04fd21a740 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f10115630 0x564f10113010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04fd21a740 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10112ad0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04fd21a740 1 --2- 172.21.15.28:0/279492621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f1010a450 0x564f10112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 >> 172.21.15.28:0/279492621 conn(0x564f10106960 msgr2=0x564f1010c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.443+0000 7f04d37fe640 0 client.26375 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.444+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 shutdown_connections 2024-05-09T08:35:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.444+0000 7f04fd21a740 1 -- 172.21.15.28:0/279492621 wait complete. 2024-05-09T08:35:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.444+0000 7f04fd21a740 20 client.26375 trim_cache size 0 max 16384 2024-05-09T08:35:53.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.480+0000 7f04fd21a740 1 Processor -- start 2024-05-09T08:35:53.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.480+0000 7f04fd21a740 1 -- start start 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.480+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10114cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.480+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10121150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10106d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04fd21a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04e00100f0 con 0x564f10115630 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f37fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10121150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f37fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10121150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:47224/0 (socket says 172.21.15.28:47224) 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f37fe640 1 -- 172.21.15.28:0/2402970053 learned_addr learned my addr 172.21.15.28:0/2402970053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04fd21a740 1 -- 172.21.15.28:0/2402970053 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f04e0011e70 con 0x564f1010ad70 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04fd21a740 1 -- 172.21.15.28:0/2402970053 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04e0010880 con 0x564f1010a450 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f3fff640 1 --2- 172.21.15.28:0/2402970053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10114cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f89ce640 1 --2- 172.21.15.28:0/2402970053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10106d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f3fff640 1 -- 172.21.15.28:0/2402970053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 msgr2=0x564f10121150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f3fff640 1 --2- 172.21.15.28:0/2402970053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10121150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f3fff640 1 -- 172.21.15.28:0/2402970053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 msgr2=0x564f10106d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f3fff640 1 --2- 172.21.15.28:0/2402970053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10106d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.481+0000 7f04f3fff640 1 -- 172.21.15.28:0/2402970053 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f10111540 con 0x564f1010a450 2024-05-09T08:35:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.482+0000 7f04f3fff640 1 --2- 172.21.15.28:0/2402970053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10114cc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x564f10127dd0 tx=0x7f04e802d750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5b0661e29429c69 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.483+0000 7f04d9ffb640 1 -- 172.21.15.28:0/2402970053 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e8025040 con 0x564f1010a450 2024-05-09T08:35:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.483+0000 7f04d9ffb640 1 -- 172.21.15.28:0/2402970053 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e802de50 con 0x564f1010a450 2024-05-09T08:35:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.483+0000 7f04d9ffb640 1 -- 172.21.15.28:0/2402970053 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e8031040 con 0x564f1010a450 2024-05-09T08:35:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.484+0000 7f04fd21a740 1 -- 172.21.15.28:0/2402970053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 msgr2=0x564f10114cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.484+0000 7f04fd21a740 1 --2- 172.21.15.28:0/2402970053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10114cc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x564f10127dd0 tx=0x7f04e802d750 comp rx=0 tx=0).stop 2024-05-09T08:35:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.484+0000 7f04fd21a740 1 -- 172.21.15.28:0/2402970053 shutdown_connections 2024-05-09T08:35:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.484+0000 7f04fd21a740 1 --2- 172.21.15.28:0/2402970053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10106d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.484+0000 7f04fd21a740 1 --2- 172.21.15.28:0/2402970053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f10121150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.484+0000 7f04fd21a740 1 --2- 172.21.15.28:0/2402970053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10114cc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.484+0000 7f04fd21a740 1 -- 172.21.15.28:0/2402970053 >> 172.21.15.28:0/2402970053 conn(0x564f10106960 msgr2=0x564f10105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:53.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.485+0000 7f04fd21a740 1 -- 172.21.15.28:0/2402970053 shutdown_connections 2024-05-09T08:35:53.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.486+0000 7f04fd21a740 1 -- 172.21.15.28:0/2402970053 wait complete. 2024-05-09T08:35:53.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.486+0000 7f04fd21a740 1 -- start start 2024-05-09T08:35:53.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.487+0000 7f04fd21a740 1 Processor -- start 2024-05-09T08:35:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.487+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10119370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.487+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f101198b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10119370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10119370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51482/0 (socket says 172.21.15.28:51482) 2024-05-09T08:35:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04fd21a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10119e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04f37fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f101198b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04f3fff640 1 -- 172.21.15.28:0/69569713 learned_addr learned my addr 172.21.15.28:0/69569713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:35:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04e00117a0 con 0x564f10115630 2024-05-09T08:35:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f04e0011010 con 0x564f1010ad70 2024-05-09T08:35:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04e000e0f0 con 0x564f1010a450 2024-05-09T08:35:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04fd21a740 10 client.0 _ll_register_callbacks cb 0x7f04ec0095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.488+0000 7f04f89ce640 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10119e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04f89ce640 1 -- 172.21.15.28:0/69569713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 msgr2=0x564f10119370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04f89ce640 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10119370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04f89ce640 1 -- 172.21.15.28:0/69569713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 msgr2=0x564f101198b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04f89ce640 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f101198b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04f89ce640 1 -- 172.21.15.28:0/69569713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f1018a9f0 con 0x564f10115630 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04f89ce640 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10119e30 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f04e40eda60 tx=0x7f04e40bc190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5911f6fff097c7f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04d37fe640 1 -- 172.21.15.28:0/69569713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e404a040 con 0x564f10115630 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04e802de50 con 0x564f10115630 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.489+0000 7f04d37fe640 1 -- 172.21.15.28:0/69569713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9567+0+0 (secure 0 0 0) 0x7f04e40b8ea0 con 0x564f10115630 2024-05-09T08:35:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04d37fe640 1 -- 172.21.15.28:0/69569713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f04e40bcf30 con 0x564f10115630 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04fd21a740 10 client.16869 fetch_fsmap learned FSMap version 65 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x564f1018bd00 con 0x564f10115630 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04d37fe640 1 -- 172.21.15.28:0/69569713 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e802de50 con 0x564f10115630 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04d37fe640 1 -- 172.21.15.28:0/69569713 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f04e40bca90 con 0x564f10115630 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04fd21a740 10 client.16869 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04fd21a740 10 client.16869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04fd21a740 20 client.16869 populate_metadata read hostname 'smithi028' 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04fd21a740 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564f1018c440 0x564f1018e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04fd21a740 4 client.16869 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.490+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564f10076dc0 con 0x564f1018c440 2024-05-09T08:35:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.491+0000 7f04f3fff640 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564f1018c440 0x564f1018e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.491+0000 7f04d37fe640 10 client.16869 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.491+0000 7f04f3fff640 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564f1018c440 0x564f1018e860 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1e9e9e900539238 server_cookie=77069d9bca92256e in_seq=0 out_seq=0 2024-05-09T08:35:53.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04d37fe640 1 -- 172.21.15.28:0/69569713 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564f10076dc0 con 0x564f1018c440 2024-05-09T08:35:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04d37fe640 10 client.16869 handle_command_reply: tid=0 2024-05-09T08:35:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 client.16869 shutdown 2024-05-09T08:35:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 msgr2=0x564f10119e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10119e30 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f04e40eda60 tx=0x7f04e40bc190 comp rx=0 tx=0).stop 2024-05-09T08:35:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 shutdown_connections 2024-05-09T08:35:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x564f1018c440 0x564f1018e860 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04d37fe640 0 client.16869 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:35:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f10115630 0x564f10119e30 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564f1010ad70 0x564f101198b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 --2- 172.21.15.28:0/69569713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564f1010a450 0x564f10119370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 >> 172.21.15.28:0/69569713 conn(0x564f10106960 msgr2=0x564f10105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 shutdown_connections 2024-05-09T08:35:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 1 -- 172.21.15.28:0/69569713 wait complete. 2024-05-09T08:35:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.543+0000 7f04fd21a740 20 client.16869 trim_cache size 0 max 16384 2024-05-09T08:35:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:35:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:35:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:35:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:35:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 -- 172.21.15.28:0/3378988216 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04dc0a2d20 msgr2=0x7f04dc0a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04dc0a2d20 0x7f04dc0a51e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04e80058c0 tx=0x7f04e8005850 comp rx=0 tx=0).stop 2024-05-09T08:35:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 -- 172.21.15.28:0/3378988216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04f4116c60 msgr2=0x7f04f4117110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04f4116c60 0x7f04f4117110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04e402a9b0 tx=0x7f04e40055e0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 -- 172.21.15.28:0/3378988216 shutdown_connections 2024-05-09T08:35:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04f4116c60 0x7f04f4117110 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04f411b010 0x7f04f4116720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04dc0a2d20 0x7f04dc0a51e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 --2- 172.21.15.28:0/3378988216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04f4110020 0x7f04f411a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.544+0000 7f04fa431640 1 -- 172.21.15.28:0/3378988216 >> 172.21.15.28:0/3378988216 conn(0x7f04f406c900 msgr2=0x7f04f411b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.545+0000 7f04fa431640 1 -- 172.21.15.28:0/3378988216 shutdown_connections 2024-05-09T08:35:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:35:53.545+0000 7f04fa431640 1 -- 172.21.15.28:0/3378988216 wait complete. 2024-05-09T08:35:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:53 smithi165 ceph-mon[34540]: pgmap v303: 129 pgs: 129 active+clean; 2.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 360 op/s 2024-05-09T08:35:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:53 smithi184 ceph-mon[34509]: pgmap v303: 129 pgs: 129 active+clean; 2.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 360 op/s 2024-05-09T08:35:54.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:53 smithi028 ceph-mon[30119]: pgmap v303: 129 pgs: 129 active+clean; 2.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 360 op/s 2024-05-09T08:35:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:55 smithi165 ceph-mon[34540]: pgmap v304: 129 pgs: 129 active+clean; 2.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 407 op/s 2024-05-09T08:35:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:55 smithi184 ceph-mon[34509]: pgmap v304: 129 pgs: 129 active+clean; 2.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 407 op/s 2024-05-09T08:35:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:55 smithi028 ceph-mon[30119]: pgmap v304: 129 pgs: 129 active+clean; 2.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 407 op/s 2024-05-09T08:35:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:57 smithi165 ceph-mon[34540]: pgmap v305: 129 pgs: 129 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 367 op/s 2024-05-09T08:35:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:57 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:35:58.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:57 smithi184 ceph-mon[34509]: pgmap v305: 129 pgs: 129 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 367 op/s 2024-05-09T08:35:58.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:57 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:35:58.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:57 smithi028 ceph-mon[30119]: pgmap v305: 129 pgs: 129 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 367 op/s 2024-05-09T08:35:58.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:57 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:35:59.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:58 smithi165 ceph-mon[34540]: pgmap v306: 129 pgs: 129 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 333 op/s 2024-05-09T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:58 smithi184 ceph-mon[34509]: pgmap v306: 129 pgs: 129 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 333 op/s 2024-05-09T08:35:59.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:58 smithi028 ceph-mon[30119]: pgmap v306: 129 pgs: 129 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 333 op/s 2024-05-09T08:36:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:59 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:36:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:59 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:36:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:35:59 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:36:00.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:59 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:36:00.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:59 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:36:00.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:35:59 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:36:00.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:59 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:36:00.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:59 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:36:00.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:35:59 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:36:01.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:00 smithi028 ceph-mon[30119]: pgmap v307: 129 pgs: 129 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 348 op/s 2024-05-09T08:36:01.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:00 smithi165 ceph-mon[34540]: pgmap v307: 129 pgs: 129 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 348 op/s 2024-05-09T08:36:01.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:00 smithi184 ceph-mon[34509]: pgmap v307: 129 pgs: 129 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 348 op/s 2024-05-09T08:36:03.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:03 smithi165 ceph-mon[34540]: pgmap v308: 129 pgs: 129 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 324 op/s 2024-05-09T08:36:03.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:03 smithi184 ceph-mon[34509]: pgmap v308: 129 pgs: 129 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 324 op/s 2024-05-09T08:36:03.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:03 smithi028 ceph-mon[30119]: pgmap v308: 129 pgs: 129 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 324 op/s 2024-05-09T08:36:05.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:05 smithi165 ceph-mon[34540]: pgmap v309: 129 pgs: 129 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 363 op/s 2024-05-09T08:36:05.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:05 smithi184 ceph-mon[34509]: pgmap v309: 129 pgs: 129 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 363 op/s 2024-05-09T08:36:05.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:05 smithi028 ceph-mon[30119]: pgmap v309: 129 pgs: 129 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 363 op/s 2024-05-09T08:36:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.682+0000 7f7a766a5640 1 -- 172.21.15.28:0/1594101625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a70101120 msgr2=0x7f7a7010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.682+0000 7f7a766a5640 1 --2- 172.21.15.28:0/1594101625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a70101120 0x7f7a7010d5e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009c10 tx=0x7f7a6802a710 comp rx=0 tx=0).stop 2024-05-09T08:36:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.684+0000 7f7a766a5640 1 -- 172.21.15.28:0/1594101625 shutdown_connections 2024-05-09T08:36:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.684+0000 7f7a766a5640 1 --2- 172.21.15.28:0/1594101625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a70101120 0x7f7a7010d5e0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.684+0000 7f7a766a5640 1 --2- 172.21.15.28:0/1594101625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a70100760 0x7f7a70100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.684+0000 7f7a766a5640 1 --2- 172.21.15.28:0/1594101625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a70106720 0x7f7a70106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.684+0000 7f7a766a5640 1 -- 172.21.15.28:0/1594101625 >> 172.21.15.28:0/1594101625 conn(0x7f7a700778b0 msgr2=0x7f7a70077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.684+0000 7f7a766a5640 1 -- 172.21.15.28:0/1594101625 shutdown_connections 2024-05-09T08:36:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.685+0000 7f7a766a5640 1 -- 172.21.15.28:0/1594101625 wait complete. 2024-05-09T08:36:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.686+0000 7f7a766a5640 1 Processor -- start 2024-05-09T08:36:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.686+0000 7f7a766a5640 1 -- start start 2024-05-09T08:36:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.687+0000 7f7a766a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a70100760 0x7f7a701980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.687+0000 7f7a766a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a70101120 0x7f7a70198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.687+0000 7f7a766a5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a70106720 0x7f7a7019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.687+0000 7f7a766a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a70110b00 con 0x7f7a70101120 2024-05-09T08:36:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.687+0000 7f7a766a5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7a70110980 con 0x7f7a70106720 2024-05-09T08:36:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.687+0000 7f7a74c42640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a70106720 0x7f7a7019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.687+0000 7f7a74c42640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a70106720 0x7f7a7019c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:46894/0 (socket says 172.21.15.28:46894) 2024-05-09T08:36:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.687+0000 7f7a74c42640 1 -- 172.21.15.28:0/2511059371 learned_addr learned my addr 172.21.15.28:0/2511059371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.687+0000 7f7a766a5640 1 -- 172.21.15.28:0/2511059371 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a70110c80 con 0x7f7a70100760 2024-05-09T08:36:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.689+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a70100760 0x7f7a701980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.690+0000 7f7a74c42640 1 -- 172.21.15.28:0/2511059371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a70100760 msgr2=0x7f7a701980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.690+0000 7f7a74c42640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a70100760 0x7f7a701980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.690+0000 7f7a74c42640 1 -- 172.21.15.28:0/2511059371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a70101120 msgr2=0x7f7a70198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.690+0000 7f7a74c42640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a70101120 0x7f7a70198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.690+0000 7f7a74c42640 1 -- 172.21.15.28:0/2511059371 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a7019d0b0 con 0x7f7a70106720 2024-05-09T08:36:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.690+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a70100760 0x7f7a701980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.691+0000 7f7a74c42640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a70106720 0x7f7a7019c980 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7a680305b0 tx=0x7f7a680305e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.691+0000 7f7a6dffb640 1 -- 172.21.15.28:0/2511059371 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68041020 con 0x7f7a70106720 2024-05-09T08:36:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.691+0000 7f7a6dffb640 1 -- 172.21.15.28:0/2511059371 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a68037040 con 0x7f7a70106720 2024-05-09T08:36:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.691+0000 7f7a6dffb640 1 -- 172.21.15.28:0/2511059371 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68036070 con 0x7f7a70106720 2024-05-09T08:36:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.692+0000 7f7a766a5640 1 -- 172.21.15.28:0/2511059371 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a7019d340 con 0x7f7a70106720 2024-05-09T08:36:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.692+0000 7f7a766a5640 1 -- 172.21.15.28:0/2511059371 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a701a4c30 con 0x7f7a70106720 2024-05-09T08:36:06.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.695+0000 7f7a793b8740 1 Processor -- start 2024-05-09T08:36:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a793b8740 1 -- start start 2024-05-09T08:36:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a6dffb640 1 -- 172.21.15.28:0/2511059371 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7a68044020 con 0x7f7a70106720 2024-05-09T08:36:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 0x55f065a1c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a1ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a2fcd0 0x55f065a320c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a793b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f065981340 con 0x55f065a1d990 2024-05-09T08:36:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a793b8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f065929290 con 0x55f065a1c850 2024-05-09T08:36:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a793b8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f06597d300 con 0x55f065a2fcd0 2024-05-09T08:36:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a1ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a1ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:46900/0 (socket says 172.21.15.28:46900) 2024-05-09T08:36:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.698+0000 7f7a6ffff640 1 -- 172.21.15.28:0/1218593877 learned_addr learned my addr 172.21.15.28:0/1218593877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.699+0000 7f7a6dffb640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7a5c0a2ea0 0x7f7a5c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.699+0000 7f7a6ffff640 1 -- 172.21.15.28:0/1218593877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a2fcd0 msgr2=0x55f065a320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.699+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/1218593877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a2fcd0 0x55f065a320c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.699+0000 7f7a6ffff640 1 -- 172.21.15.28:0/1218593877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 msgr2=0x55f065a1c310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.699+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/1218593877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 0x55f065a1c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.699+0000 7f7a6ffff640 1 -- 172.21.15.28:0/1218593877 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f06594d8e0 con 0x55f065a1c850 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.699+0000 7f7a6dffb640 1 -- 172.21.15.28:0/2511059371 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9567+0+0 (secure 0 0 0) 0x7f7a680f0370 con 0x7f7a70106720 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.700+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/1218593877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a1ccd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7a580114f0 tx=0x7f7a580119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=350b0cd77cc4b680 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.700+0000 7f7a667fc640 1 -- 172.21.15.28:0/1218593877 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a58005020 con 0x55f065a1c850 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.700+0000 7f7a667fc640 1 -- 172.21.15.28:0/1218593877 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a580047a0 con 0x55f065a1c850 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.700+0000 7f7a667fc640 1 -- 172.21.15.28:0/1218593877 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5800fae0 con 0x55f065a1c850 2024-05-09T08:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.700+0000 7f7a74c42640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7a5c0a2ea0 0x7f7a5c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a793b8740 1 -- 172.21.15.28:0/1218593877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 msgr2=0x55f065a1ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1218593877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a1ccd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7a580114f0 tx=0x7f7a580119b0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a74c42640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7a5c0a2ea0 0x7f7a5c0a5360 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7a68006010 tx=0x7f7a680094c0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a793b8740 1 -- 172.21.15.28:0/1218593877 shutdown_connections 2024-05-09T08:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1218593877 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a2fcd0 0x55f065a320c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1218593877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a1ccd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1218593877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 0x55f065a1c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a793b8740 1 -- 172.21.15.28:0/1218593877 >> 172.21.15.28:0/1218593877 conn(0x55f065a18960 msgr2=0x55f065a18d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a793b8740 1 -- 172.21.15.28:0/1218593877 shutdown_connections 2024-05-09T08:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.702+0000 7f7a793b8740 1 -- 172.21.15.28:0/1218593877 wait complete. 2024-05-09T08:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.703+0000 7f7a793b8740 1 -- start start 2024-05-09T08:36:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.703+0000 7f7a793b8740 1 Processor -- start 2024-05-09T08:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.704+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a29960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.704+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1d990 0x55f065a2f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.704+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 0x55f065a29f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.704+0000 7f7a793b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f065a238a0 con 0x55f065a2fcd0 2024-05-09T08:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.704+0000 7f7a793b8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f06593c790 con 0x55f065a1c850 2024-05-09T08:36:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.704+0000 7f7a793b8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f065a23a20 con 0x55f065a1d990 2024-05-09T08:36:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.704+0000 7f7a793b8740 10 client.0 _ll_register_callbacks cb 0x55f065977a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a677fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a29960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a677fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a29960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:46902/0 (socket says 172.21.15.28:46902) 2024-05-09T08:36:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a677fe640 1 -- 172.21.15.28:0/1597343754 learned_addr learned my addr 172.21.15.28:0/1597343754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a677fe640 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 0x55f065a29f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1d990 0x55f065a2f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a6ffff640 1 -- 172.21.15.28:0/1597343754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 msgr2=0x55f065a29960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a29960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a6ffff640 1 -- 172.21.15.28:0/1597343754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 msgr2=0x55f065a29f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 0x55f065a29f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.705+0000 7f7a6ffff640 1 -- 172.21.15.28:0/1597343754 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f065695210 con 0x55f065a1d990 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.706+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1d990 0x55f065a2f680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7a58006010 tx=0x7f7a580051d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d919303024f42eb3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.706+0000 7f7a4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.706+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/1597343754 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a58028070 con 0x55f065a1d990 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.706+0000 7f7a793b8740 1 -- 172.21.15.28:0/1597343754 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a580047a0 con 0x55f065a1d990 2024-05-09T08:36:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.706+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/1597343754 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9567+0+0 (secure 0 0 0) 0x7f7a5800f820 con 0x55f065a1d990 2024-05-09T08:36:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.708+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/1597343754 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f7a58032050 con 0x55f065a1d990 2024-05-09T08:36:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.708+0000 7f7a793b8740 10 client.26242 fetch_fsmap learned FSMap version 65 2024-05-09T08:36:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.708+0000 7f7a793b8740 1 -- 172.21.15.28:0/1597343754 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55f065761620 con 0x55f065a1d990 2024-05-09T08:36:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.708+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/1597343754 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a580047a0 con 0x55f065a1d990 2024-05-09T08:36:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.709+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/1597343754 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f7a580247b0 con 0x55f065a1d990 2024-05-09T08:36:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.709+0000 7f7a793b8740 10 client.26242 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:36:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.709+0000 7f7a793b8740 10 client.26242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:36:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.709+0000 7f7a793b8740 20 client.26242 populate_metadata read hostname 'smithi028' 2024-05-09T08:36:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.709+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f065a48920 0x55f065a4adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.709+0000 7f7a793b8740 4 client.26242 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.709+0000 7f7a793b8740 1 -- 172.21.15.28:0/1597343754 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f06598ea70 con 0x55f065a48920 2024-05-09T08:36:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.709+0000 7f7a677fe640 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f065a48920 0x55f065a4adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.710+0000 7f7a4f7fe640 10 client.26242 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.710+0000 7f7a677fe640 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f065a48920 0x55f065a4adc0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca5fedb1165fa83d server_cookie=cbf5bc19b44d929 in_seq=0 out_seq=0 2024-05-09T08:36:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.711+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/1597343754 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f06598ea70 con 0x55f065a48920 2024-05-09T08:36:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.711+0000 7f7a4f7fe640 10 client.26242 handle_command_reply: tid=0 2024-05-09T08:36:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.711+0000 7f7a793b8740 1 client.26242 shutdown 2024-05-09T08:36:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.711+0000 7f7a793b8740 1 -- 172.21.15.28:0/1597343754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1d990 msgr2=0x55f065a2f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.711+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1d990 0x55f065a2f680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7a58006010 tx=0x7f7a580051d0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.712+0000 7f7a793b8740 1 -- 172.21.15.28:0/1597343754 shutdown_connections 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.712+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f065a48920 0x55f065a4adc0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.712+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 0x55f065a29f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.712+0000 7f7a4f7fe640 0 client.26242 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.712+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1d990 0x55f065a2f680 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.712+0000 7f7a793b8740 1 --2- 172.21.15.28:0/1597343754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1c850 0x55f065a29960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.712+0000 7f7a793b8740 1 -- 172.21.15.28:0/1597343754 >> 172.21.15.28:0/1597343754 conn(0x55f065a18960 msgr2=0x55f065a1e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.712+0000 7f7a793b8740 1 -- 172.21.15.28:0/1597343754 shutdown_connections 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.712+0000 7f7a793b8740 1 -- 172.21.15.28:0/1597343754 wait complete. 2024-05-09T08:36:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.713+0000 7f7a793b8740 20 client.26242 trim_cache size 0 max 16384 2024-05-09T08:36:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.742+0000 7f7a793b8740 1 Processor -- start 2024-05-09T08:36:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.742+0000 7f7a793b8740 1 -- start start 2024-05-09T08:36:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.743+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a18180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.743+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 0x55f065a50690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.743+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a2fcd0 0x55f065a50bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.743+0000 7f7a793b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a701acd90 con 0x55f065a1d990 2024-05-09T08:36:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.743+0000 7f7a793b8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7a701aeb10 con 0x55f065a2fcd0 2024-05-09T08:36:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.743+0000 7f7a793b8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a701ad520 con 0x55f065a1c850 2024-05-09T08:36:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.743+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 0x55f065a50690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.744+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 0x55f065a50690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48996/0 (socket says 172.21.15.28:48996) 2024-05-09T08:36:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.744+0000 7f7a6ffff640 1 -- 172.21.15.28:0/2928656518 learned_addr learned my addr 172.21.15.28:0/2928656518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.744+0000 7f7a677fe640 1 --2- 172.21.15.28:0/2928656518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a18180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.744+0000 7f7a677fe640 1 --2- 172.21.15.28:0/2928656518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a2fcd0 0x55f065a50bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.744+0000 7f7a6ffff640 1 -- 172.21.15.28:0/2928656518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 msgr2=0x55f065a18180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.744+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/2928656518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a18180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.744+0000 7f7a6ffff640 1 -- 172.21.15.28:0/2928656518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a2fcd0 msgr2=0x55f065a50bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.744+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/2928656518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a2fcd0 0x55f065a50bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.744+0000 7f7a6ffff640 1 -- 172.21.15.28:0/2928656518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f065a16f00 con 0x55f065a1d990 2024-05-09T08:36:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.745+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/2928656518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 0x55f065a50690 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f7a580092a0 tx=0x7f7a58005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9855b92de6b7cfb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.746+0000 7f7a65ffb640 1 -- 172.21.15.28:0/2928656518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a58028040 con 0x55f065a1d990 2024-05-09T08:36:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.746+0000 7f7a65ffb640 1 -- 172.21.15.28:0/2928656518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a58005550 con 0x55f065a1d990 2024-05-09T08:36:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.746+0000 7f7a65ffb640 1 -- 172.21.15.28:0/2928656518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a580246a0 con 0x55f065a1d990 2024-05-09T08:36:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.747+0000 7f7a793b8740 1 -- 172.21.15.28:0/2928656518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 msgr2=0x55f065a50690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.747+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2928656518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 0x55f065a50690 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f7a580092a0 tx=0x7f7a58005df0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.748+0000 7f7a793b8740 1 -- 172.21.15.28:0/2928656518 shutdown_connections 2024-05-09T08:36:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.748+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2928656518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a2fcd0 0x55f065a50bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.748+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2928656518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a1d990 0x55f065a50690 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.748+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2928656518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a18180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.748+0000 7f7a793b8740 1 -- 172.21.15.28:0/2928656518 >> 172.21.15.28:0/2928656518 conn(0x55f065a18960 msgr2=0x55f065a18d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.753+0000 7f7a793b8740 1 -- 172.21.15.28:0/2928656518 shutdown_connections 2024-05-09T08:36:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.753+0000 7f7a793b8740 1 -- 172.21.15.28:0/2928656518 wait complete. 2024-05-09T08:36:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.753+0000 7f7a793b8740 1 -- start start 2024-05-09T08:36:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.754+0000 7f7a793b8740 1 Processor -- start 2024-05-09T08:36:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.754+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a5c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.754+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1d990 0x55f065a57550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.754+0000 7f7a793b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 0x55f065a57c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.754+0000 7f7a793b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a701ae440 con 0x55f065a2fcd0 2024-05-09T08:36:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.754+0000 7f7a793b8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7a701adcb0 con 0x55f065a1d990 2024-05-09T08:36:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.754+0000 7f7a793b8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a701aadf0 con 0x55f065a1c850 2024-05-09T08:36:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.754+0000 7f7a793b8740 10 client.0 _ll_register_callbacks cb 0x7f7a58009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a677fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a5c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a677fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a5c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50798/0 (socket says 172.21.15.28:50798) 2024-05-09T08:36:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a677fe640 1 -- 172.21.15.28:0/2635820190 learned_addr learned my addr 172.21.15.28:0/2635820190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a677fe640 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 0x55f065a57c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1d990 0x55f065a57550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a677fe640 1 -- 172.21.15.28:0/2635820190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1d990 msgr2=0x55f065a57550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a677fe640 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1d990 0x55f065a57550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a677fe640 1 -- 172.21.15.28:0/2635820190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 msgr2=0x55f065a57c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a677fe640 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 0x55f065a57c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.755+0000 7f7a677fe640 1 -- 172.21.15.28:0/2635820190 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f065aad9e0 con 0x55f065a1c850 2024-05-09T08:36:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.756+0000 7f7a677fe640 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a5c1e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55f065a4b5d0 tx=0x7f7a5800f680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca75636317cdc3c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.756+0000 7f7a4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:36:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.756+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/2635820190 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60012040 con 0x55f065a1c850 2024-05-09T08:36:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.756+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/2635820190 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9567+0+0 (secure 0 0 0) 0x7f7a60004b60 con 0x55f065a1c850 2024-05-09T08:36:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.758+0000 7f7a793b8740 1 -- 172.21.15.28:0/2635820190 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a58005550 con 0x55f065a1c850 2024-05-09T08:36:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.761+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/2635820190 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f7a6001a910 con 0x55f065a1c850 2024-05-09T08:36:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.761+0000 7f7a793b8740 10 client.26248 fetch_fsmap learned FSMap version 65 2024-05-09T08:36:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.761+0000 7f7a793b8740 1 -- 172.21.15.28:0/2635820190 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55f065aaebf0 con 0x55f065a1c850 2024-05-09T08:36:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.762+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/2635820190 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a58005550 con 0x55f065a1c850 2024-05-09T08:36:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.762+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/2635820190 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f7a60018040 con 0x55f065a1c850 2024-05-09T08:36:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.762+0000 7f7a793b8740 10 client.26248 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:36:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.762+0000 7f7a793b8740 10 client.26248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:36:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.762+0000 7f7a793b8740 20 client.26248 populate_metadata read hostname 'smithi028' 2024-05-09T08:36:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.762+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f065aaf3b0 0x55f065ab17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.762+0000 7f7a793b8740 4 client.26248 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:36:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.762+0000 7f7a793b8740 1 -- 172.21.15.28:0/2635820190 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f065a24cd0 con 0x55f065aaf3b0 2024-05-09T08:36:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.763+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f065aaf3b0 0x55f065ab17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.764+0000 7f7a6ffff640 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f065aaf3b0 0x55f065ab17d0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e0057eed91a1f1b server_cookie=c12aef094ebe0a3b in_seq=0 out_seq=0 2024-05-09T08:36:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.764+0000 7f7a4f7fe640 10 client.26248 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:06.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.850+0000 7f7a4f7fe640 1 -- 172.21.15.28:0/2635820190 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f065a24cd0 con 0x55f065aaf3b0 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.850+0000 7f7a4f7fe640 10 client.26248 handle_command_reply: tid=0 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.850+0000 7f7a793b8740 1 client.26248 shutdown 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 -- 172.21.15.28:0/2635820190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 msgr2=0x55f065a5c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a5c1e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55f065a4b5d0 tx=0x7f7a5800f680 comp rx=0 tx=0).stop 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 -- 172.21.15.28:0/2635820190 shutdown_connections 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f065aaf3b0 0x55f065ab17d0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f065a2fcd0 0x55f065a57c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f065a1d990 0x55f065a57550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 --2- 172.21.15.28:0/2635820190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f065a1c850 0x55f065a5c1e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 -- 172.21.15.28:0/2635820190 >> 172.21.15.28:0/2635820190 conn(0x55f065a18960 msgr2=0x55f065a1def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a4f7fe640 0 client.26248 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 -- 172.21.15.28:0/2635820190 shutdown_connections 2024-05-09T08:36:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 1 -- 172.21.15.28:0/2635820190 wait complete. 2024-05-09T08:36:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.851+0000 7f7a793b8740 20 client.26248 trim_cache size 0 max 16384 2024-05-09T08:36:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:36:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:36:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:36:06.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:36:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.854+0000 7f7a766a5640 1 -- 172.21.15.28:0/2511059371 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7a5c0a2ea0 msgr2=0x7f7a5c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.854+0000 7f7a766a5640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7a5c0a2ea0 0x7f7a5c0a5360 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7a68006010 tx=0x7f7a680094c0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.854+0000 7f7a766a5640 1 -- 172.21.15.28:0/2511059371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a70106720 msgr2=0x7f7a7019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.854+0000 7f7a766a5640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a70106720 0x7f7a7019c980 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7a680305b0 tx=0x7f7a680305e0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.855+0000 7f7a766a5640 1 -- 172.21.15.28:0/2511059371 shutdown_connections 2024-05-09T08:36:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.855+0000 7f7a766a5640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a70106720 0x7f7a7019c980 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.855+0000 7f7a766a5640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a70101120 0x7f7a70198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.855+0000 7f7a766a5640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f7a5c0a2ea0 0x7f7a5c0a5360 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.855+0000 7f7a766a5640 1 --2- 172.21.15.28:0/2511059371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a70100760 0x7f7a701980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.855+0000 7f7a766a5640 1 -- 172.21.15.28:0/2511059371 >> 172.21.15.28:0/2511059371 conn(0x7f7a700778b0 msgr2=0x7f7a700ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.856+0000 7f7a766a5640 1 -- 172.21.15.28:0/2511059371 shutdown_connections 2024-05-09T08:36:06.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:06.856+0000 7f7a766a5640 1 -- 172.21.15.28:0/2511059371 wait complete. 2024-05-09T08:36:07.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:07 smithi165 ceph-mon[34540]: pgmap v310: 129 pgs: 129 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 345 op/s 2024-05-09T08:36:07.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:07 smithi184 ceph-mon[34509]: pgmap v310: 129 pgs: 129 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 345 op/s 2024-05-09T08:36:07.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:07 smithi028 ceph-mon[30119]: pgmap v310: 129 pgs: 129 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 345 op/s 2024-05-09T08:36:08.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:08 smithi165 ceph-mon[34540]: osdmap e108: 12 total, 12 up, 12 in 2024-05-09T08:36:08.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:08 smithi184 ceph-mon[34509]: osdmap e108: 12 total, 12 up, 12 in 2024-05-09T08:36:08.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:08 smithi028 ceph-mon[30119]: osdmap e108: 12 total, 12 up, 12 in 2024-05-09T08:36:09.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:09 smithi165 ceph-mon[34540]: pgmap v312: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 354 op/s 2024-05-09T08:36:09.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:09 smithi184 ceph-mon[34509]: pgmap v312: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 354 op/s 2024-05-09T08:36:09.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:09 smithi028 ceph-mon[30119]: pgmap v312: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 354 op/s 2024-05-09T08:36:11.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:11 smithi165 ceph-mon[34540]: pgmap v313: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 374 op/s 2024-05-09T08:36:11.605 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:11 smithi184 ceph-mon[34509]: pgmap v313: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 374 op/s 2024-05-09T08:36:11.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:11 smithi028 ceph-mon[30119]: pgmap v313: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 374 op/s 2024-05-09T08:36:13.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:13 smithi184 ceph-mon[34509]: pgmap v314: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 2.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-05-09T08:36:13.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:13 smithi028 ceph-mon[30119]: pgmap v314: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 2.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-05-09T08:36:13.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:13 smithi165 ceph-mon[34540]: pgmap v314: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 2.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-05-09T08:36:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:15 smithi028 ceph-mon[30119]: pgmap v315: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 306 op/s 2024-05-09T08:36:15.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:15 smithi165 ceph-mon[34540]: pgmap v315: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 306 op/s 2024-05-09T08:36:15.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:15 smithi184 ceph-mon[34509]: pgmap v315: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 306 op/s 2024-05-09T08:36:17.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:17 smithi028 ceph-mon[30119]: pgmap v316: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 270 op/s 2024-05-09T08:36:17.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:17 smithi165 ceph-mon[34540]: pgmap v316: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 270 op/s 2024-05-09T08:36:17.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:17 smithi184 ceph-mon[34509]: pgmap v316: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 270 op/s 2024-05-09T08:36:18.931 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:36:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:36:18.460+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:36:19.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:19 smithi028 ceph-mon[30119]: pgmap v317: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 256 op/s 2024-05-09T08:36:19.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:19 smithi165 ceph-mon[34540]: pgmap v317: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 256 op/s 2024-05-09T08:36:19.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:19 smithi184 ceph-mon[34509]: pgmap v317: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 256 op/s 2024-05-09T08:36:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.014+0000 7efcf4e15640 1 -- 172.21.15.28:0/3041931317 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf0108830 msgr2=0x7efcf0108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.014+0000 7efcf4e15640 1 --2- 172.21.15.28:0/3041931317 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf0108830 0x7efcf0108cb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efce8009930 tx=0x7efce802c800 comp rx=0 tx=0).stop 2024-05-09T08:36:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.017+0000 7efcf4e15640 1 -- 172.21.15.28:0/3041931317 shutdown_connections 2024-05-09T08:36:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.017+0000 7efcf4e15640 1 --2- 172.21.15.28:0/3041931317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf01091f0 0x7efcf0115120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.017+0000 7efcf4e15640 1 --2- 172.21.15.28:0/3041931317 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf0108830 0x7efcf0108cb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.017+0000 7efcf4e15640 1 --2- 172.21.15.28:0/3041931317 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcf010d290 0x7efcf010d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.017+0000 7efcf4e15640 1 -- 172.21.15.28:0/3041931317 >> 172.21.15.28:0/3041931317 conn(0x7efcf006cd90 msgr2=0x7efcf006d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.017+0000 7efcf4e15640 1 -- 172.21.15.28:0/3041931317 shutdown_connections 2024-05-09T08:36:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.020+0000 7efcf4e15640 1 -- 172.21.15.28:0/3041931317 wait complete. 2024-05-09T08:36:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.021+0000 7efcf4e15640 1 Processor -- start 2024-05-09T08:36:20.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.026+0000 7efcf4e15640 1 -- start start 2024-05-09T08:36:20.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.026+0000 7efcf4e15640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf0108830 0x7efcf0113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.026+0000 7efcf4e15640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcf01091f0 0x7efcf01140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.026+0000 7efcf4e15640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf010d290 0x7efcf01145f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.026+0000 7efcf4e15640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcf0117940 con 0x7efcf010d290 2024-05-09T08:36:20.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.026+0000 7efcf4e15640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efcf01177c0 con 0x7efcf0108830 2024-05-09T08:36:20.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.026+0000 7efcf4e15640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efcf0117ac0 con 0x7efcf01091f0 2024-05-09T08:36:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.028+0000 7efcedd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcf01091f0 0x7efcf01140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.028+0000 7efcedd9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcf01091f0 0x7efcf01140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42896/0 (socket says 172.21.15.28:42896) 2024-05-09T08:36:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.028+0000 7efcedd9b640 1 -- 172.21.15.28:0/2836597274 learned_addr learned my addr 172.21.15.28:0/2836597274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:20.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.029+0000 7efceed9d640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf010d290 0x7efcf01145f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.030+0000 7efcedd9b640 1 -- 172.21.15.28:0/2836597274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf0108830 msgr2=0x7efcf0113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.030+0000 7efcedd9b640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf0108830 0x7efcf0113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.030+0000 7efcedd9b640 1 -- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf010d290 msgr2=0x7efcf01145f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.030+0000 7efcedd9b640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf010d290 0x7efcf01145f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.030+0000 7efcedd9b640 1 -- 172.21.15.28:0/2836597274 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf01bb990 con 0x7efcf01091f0 2024-05-09T08:36:20.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.031+0000 7efceed9d640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf010d290 0x7efcf01145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.033+0000 7efcedd9b640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcf01091f0 0x7efcf01140b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efce8009900 tx=0x7efce8033a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.034+0000 7efcdf7fe640 1 -- 172.21.15.28:0/2836597274 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8041070 con 0x7efcf01091f0 2024-05-09T08:36:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.034+0000 7efcf4e15640 1 -- 172.21.15.28:0/2836597274 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf01bbc20 con 0x7efcf01091f0 2024-05-09T08:36:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.034+0000 7efcf4e15640 1 -- 172.21.15.28:0/2836597274 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf01bc110 con 0x7efcf01091f0 2024-05-09T08:36:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.034+0000 7efcdf7fe640 1 -- 172.21.15.28:0/2836597274 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce8010080 con 0x7efcf01091f0 2024-05-09T08:36:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.034+0000 7efcdf7fe640 1 -- 172.21.15.28:0/2836597274 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8032860 con 0x7efcf01091f0 2024-05-09T08:36:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.037+0000 7efcf7c02740 1 Processor -- start 2024-05-09T08:36:20.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.039+0000 7efcf7c02740 1 -- start start 2024-05-09T08:36:20.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.043+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f5e450 0x556ac5f5e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.043+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f690f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.044+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f69630 0x556ac5f6ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.044+0000 7efcf7c02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556ac5ec3340 con 0x556ac5f5ed70 2024-05-09T08:36:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.044+0000 7efcf7c02740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x556ac5e6b290 con 0x556ac5f5e450 2024-05-09T08:36:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.044+0000 7efcf7c02740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556ac5ebf300 con 0x556ac5f69630 2024-05-09T08:36:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.044+0000 7efcee59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.044+0000 7efcee59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f690f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55280/0 (socket says 172.21.15.28:55280) 2024-05-09T08:36:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.044+0000 7efcee59c640 1 -- 172.21.15.28:0/2269138556 learned_addr learned my addr 172.21.15.28:0/2269138556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.045+0000 7efcee59c640 1 -- 172.21.15.28:0/2269138556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f69630 msgr2=0x556ac5f6ba20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:36:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.045+0000 7efcee59c640 1 --2- 172.21.15.28:0/2269138556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f69630 0x556ac5f6ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.045+0000 7efcee59c640 1 -- 172.21.15.28:0/2269138556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f5e450 msgr2=0x556ac5f5e830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:36:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.045+0000 7efcee59c640 1 --2- 172.21.15.28:0/2269138556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f5e450 0x556ac5f5e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.045+0000 7efcee59c640 1 -- 172.21.15.28:0/2269138556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac5e8f8e0 con 0x556ac5f5ed70 2024-05-09T08:36:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.045+0000 7efcdf7fe640 1 -- 172.21.15.28:0/2836597274 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7efce8044070 con 0x7efcf01091f0 2024-05-09T08:36:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.045+0000 7efcee59c640 1 --2- 172.21.15.28:0/2269138556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f690f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7efce0009820 tx=0x7efce0009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998d959a567db736 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.046+0000 7efcdf7fe640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efcd80a2fc0 0x7efcd80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.047+0000 7efcdf7fe640 1 -- 172.21.15.28:0/2836597274 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9671+0+0 (secure 0 0 0) 0x7efce80f07e0 con 0x7efcf01091f0 2024-05-09T08:36:20.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.047+0000 7efcdcff9640 1 -- 172.21.15.28:0/2269138556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce0015020 con 0x556ac5f5ed70 2024-05-09T08:36:20.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.047+0000 7efcdcff9640 1 -- 172.21.15.28:0/2269138556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce0011420 con 0x556ac5f5ed70 2024-05-09T08:36:20.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.047+0000 7efcdcff9640 1 -- 172.21.15.28:0/2269138556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce00105e0 con 0x556ac5f5ed70 2024-05-09T08:36:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.051+0000 7efcf7c02740 1 -- 172.21.15.28:0/2269138556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 msgr2=0x556ac5f690f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.051+0000 7efcf7c02740 1 --2- 172.21.15.28:0/2269138556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f690f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7efce0009820 tx=0x7efce0009ce0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.054+0000 7efceed9d640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efcd80a2fc0 0x7efcd80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.056+0000 7efcf7c02740 1 -- 172.21.15.28:0/2269138556 shutdown_connections 2024-05-09T08:36:20.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.056+0000 7efcf7c02740 1 --2- 172.21.15.28:0/2269138556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f69630 0x556ac5f6ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.056+0000 7efcf7c02740 1 --2- 172.21.15.28:0/2269138556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f690f0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.056+0000 7efcf7c02740 1 --2- 172.21.15.28:0/2269138556 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f5e450 0x556ac5f5e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.056+0000 7efcf7c02740 1 -- 172.21.15.28:0/2269138556 >> 172.21.15.28:0/2269138556 conn(0x556ac5f5a960 msgr2=0x556ac5f5ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:20.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.060+0000 7efcf7c02740 1 -- 172.21.15.28:0/2269138556 shutdown_connections 2024-05-09T08:36:20.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.060+0000 7efcf7c02740 1 -- 172.21.15.28:0/2269138556 wait complete. 2024-05-09T08:36:20.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.061+0000 7efcf7c02740 1 -- start start 2024-05-09T08:36:20.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.061+0000 7efcf7c02740 1 Processor -- start 2024-05-09T08:36:20.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.065+0000 7efceed9d640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efcd80a2fc0 0x7efcd80a5480 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efce400b830 tx=0x7efce4008040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:20.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.065+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f66530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.066+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f5ed70 0x556ac5f66a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.066+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f69630 0x556ac5f66fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.066+0000 7efcf7c02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556ac5f6c260 con 0x556ac5f69630 2024-05-09T08:36:20.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.066+0000 7efcf7c02740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x556ac5e7e790 con 0x556ac5f5ed70 2024-05-09T08:36:20.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.066+0000 7efcf7c02740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556ac5f6c3e0 con 0x556ac5f5e450 2024-05-09T08:36:20.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.066+0000 7efcf7c02740 10 client.0 _ll_register_callbacks cb 0x556ac5eb9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.066+0000 7efcee59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f66530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.066+0000 7efcee59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f66530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42906/0 (socket says 172.21.15.28:42906) 2024-05-09T08:36:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.066+0000 7efcee59c640 1 -- 172.21.15.28:0/810616277 learned_addr learned my addr 172.21.15.28:0/810616277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.067+0000 7efcedd9b640 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f5ed70 0x556ac5f66a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.067+0000 7efcee59c640 1 -- 172.21.15.28:0/810616277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f5ed70 msgr2=0x556ac5f66a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.067+0000 7efcee59c640 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f5ed70 0x556ac5f66a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.067+0000 7efcee59c640 1 -- 172.21.15.28:0/810616277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f69630 msgr2=0x556ac5f66fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:36:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.067+0000 7efcee59c640 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f69630 0x556ac5f66fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.067+0000 7efcee59c640 1 -- 172.21.15.28:0/810616277 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac5bd7210 con 0x556ac5f5e450 2024-05-09T08:36:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.067+0000 7efcee59c640 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f66530 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efce0011e40 tx=0x7efce0011800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b358a13db223d96e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.067+0000 7efcd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:36:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.067+0000 7efcd5ffb640 1 -- 172.21.15.28:0/810616277 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce000bad0 con 0x556ac5f5e450 2024-05-09T08:36:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.068+0000 7efcd5ffb640 1 -- 172.21.15.28:0/810616277 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9671+0+0 (secure 0 0 0) 0x7efce000bc70 con 0x556ac5f5e450 2024-05-09T08:36:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.068+0000 7efcf7c02740 1 -- 172.21.15.28:0/810616277 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efce0011420 con 0x556ac5f5e450 2024-05-09T08:36:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.068+0000 7efcd5ffb640 1 -- 172.21.15.28:0/810616277 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7efce002f050 con 0x556ac5f5e450 2024-05-09T08:36:20.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.068+0000 7efcf7c02740 10 client.26254 fetch_fsmap learned FSMap version 65 2024-05-09T08:36:20.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.069+0000 7efcf7c02740 1 -- 172.21.15.28:0/810616277 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x556ac5ca3620 con 0x556ac5f5e450 2024-05-09T08:36:20.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.069+0000 7efcd5ffb640 1 -- 172.21.15.28:0/810616277 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce0011420 con 0x556ac5f5e450 2024-05-09T08:36:20.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.069+0000 7efcd5ffb640 1 -- 172.21.15.28:0/810616277 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7efce001a450 con 0x556ac5f5e450 2024-05-09T08:36:20.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.071+0000 7efcf7c02740 10 client.26254 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:36:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.071+0000 7efcf7c02740 10 client.26254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:36:20.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.071+0000 7efcf7c02740 20 client.26254 populate_metadata read hostname 'smithi028' 2024-05-09T08:36:20.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.071+0000 7efcf7c02740 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556ac5f797a0 0x556ac5f7bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.071+0000 7efcf7c02740 4 client.26254 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:20.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.071+0000 7efcf7c02740 1 -- 172.21.15.28:0/810616277 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ac5ed0a70 con 0x556ac5f797a0 2024-05-09T08:36:20.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.072+0000 7efcedd9b640 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556ac5f797a0 0x556ac5f7bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.077+0000 7efcedd9b640 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556ac5f797a0 0x556ac5f7bc60 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa4d085025881300 server_cookie=8af96e604f9761d8 in_seq=0 out_seq=0 2024-05-09T08:36:20.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.077+0000 7efcd5ffb640 10 client.26254 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.438+0000 7efcd5ffb640 1 -- 172.21.15.28:0/810616277 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556ac5ed0a70 con 0x556ac5f797a0 2024-05-09T08:36:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.438+0000 7efcd5ffb640 10 client.26254 handle_command_reply: tid=0 2024-05-09T08:36:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.438+0000 7efcf7c02740 1 client.26254 shutdown 2024-05-09T08:36:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.438+0000 7efcf7c02740 1 -- 172.21.15.28:0/810616277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 msgr2=0x556ac5f66530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.438+0000 7efcf7c02740 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f66530 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efce0011e40 tx=0x7efce0011800 comp rx=0 tx=0).stop 2024-05-09T08:36:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcf7c02740 1 -- 172.21.15.28:0/810616277 shutdown_connections 2024-05-09T08:36:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcf7c02740 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556ac5f797a0 0x556ac5f7bc60 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcf7c02740 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f69630 0x556ac5f66fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcf7c02740 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f5ed70 0x556ac5f66a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcf7c02740 1 --2- 172.21.15.28:0/810616277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f66530 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcf7c02740 1 -- 172.21.15.28:0/810616277 >> 172.21.15.28:0/810616277 conn(0x556ac5f5a960 msgr2=0x556ac5f606e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcd5ffb640 0 client.26254 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcf7c02740 1 -- 172.21.15.28:0/810616277 shutdown_connections 2024-05-09T08:36:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcf7c02740 1 -- 172.21.15.28:0/810616277 wait complete. 2024-05-09T08:36:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.439+0000 7efcf7c02740 20 client.26254 trim_cache size 0 max 16384 2024-05-09T08:36:20.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.472+0000 7efcf7c02740 1 Processor -- start 2024-05-09T08:36:20.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.473+0000 7efcf7c02740 1 -- start start 2024-05-09T08:36:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.473+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f75150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.473+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f65e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.474+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 0x556ac5f81570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.474+0000 7efcf7c02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcd00101e0 con 0x556ac5f5ed70 2024-05-09T08:36:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.474+0000 7efcf7c02740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efcd0011f60 con 0x556ac5f69630 2024-05-09T08:36:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.474+0000 7efcee59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f75150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.474+0000 7efcedd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f65e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.474+0000 7efcf7c02740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efcd0010970 con 0x556ac5f5e450 2024-05-09T08:36:20.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.474+0000 7efcedd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f65e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55294/0 (socket says 172.21.15.28:55294) 2024-05-09T08:36:20.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.474+0000 7efcedd9b640 1 -- 172.21.15.28:0/3442566795 learned_addr learned my addr 172.21.15.28:0/3442566795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:20.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.474+0000 7efceed9d640 1 --2- 172.21.15.28:0/3442566795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 0x556ac5f81570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.475+0000 7efceed9d640 1 -- 172.21.15.28:0/3442566795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 msgr2=0x556ac5f75150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:20.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.475+0000 7efceed9d640 1 --2- 172.21.15.28:0/3442566795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f75150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.475+0000 7efceed9d640 1 -- 172.21.15.28:0/3442566795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 msgr2=0x556ac5f65e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:20.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.475+0000 7efceed9d640 1 --2- 172.21.15.28:0/3442566795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f65e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.475+0000 7efceed9d640 1 -- 172.21.15.28:0/3442566795 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac5f58f00 con 0x556ac5f69630 2024-05-09T08:36:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.475+0000 7efceed9d640 1 --2- 172.21.15.28:0/3442566795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 0x556ac5f81570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efce400bfd0 tx=0x7efce4010000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b92009f0e861a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.476+0000 7efcd7fff640 1 -- 172.21.15.28:0/3442566795 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce4024070 con 0x556ac5f69630 2024-05-09T08:36:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.476+0000 7efcd7fff640 1 -- 172.21.15.28:0/3442566795 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce4014de0 con 0x556ac5f69630 2024-05-09T08:36:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.476+0000 7efcd7fff640 1 -- 172.21.15.28:0/3442566795 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce40123e0 con 0x556ac5f69630 2024-05-09T08:36:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.477+0000 7efcf7c02740 1 -- 172.21.15.28:0/3442566795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 msgr2=0x556ac5f81570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:20.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.477+0000 7efcf7c02740 1 --2- 172.21.15.28:0/3442566795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 0x556ac5f81570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efce400bfd0 tx=0x7efce4010000 comp rx=0 tx=0).stop 2024-05-09T08:36:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.483+0000 7efcf7c02740 1 -- 172.21.15.28:0/3442566795 shutdown_connections 2024-05-09T08:36:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.483+0000 7efcf7c02740 1 --2- 172.21.15.28:0/3442566795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 0x556ac5f81570 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.483+0000 7efcf7c02740 1 --2- 172.21.15.28:0/3442566795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f65e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.483+0000 7efcf7c02740 1 --2- 172.21.15.28:0/3442566795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f75150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.483+0000 7efcf7c02740 1 -- 172.21.15.28:0/3442566795 >> 172.21.15.28:0/3442566795 conn(0x556ac5f5a960 msgr2=0x556ac5f5fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:20.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.486+0000 7efcf7c02740 1 -- 172.21.15.28:0/3442566795 shutdown_connections 2024-05-09T08:36:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.487+0000 7efcf7c02740 1 -- 172.21.15.28:0/3442566795 wait complete. 2024-05-09T08:36:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.487+0000 7efcf7c02740 1 -- start start 2024-05-09T08:36:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.487+0000 7efcf7c02740 1 Processor -- start 2024-05-09T08:36:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.488+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f6c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.488+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f6cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.488+0000 7efcf7c02740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 0x556ac5fdddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.488+0000 7efcf7c02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcd0011890 con 0x556ac5f5ed70 2024-05-09T08:36:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.488+0000 7efcf7c02740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efcd0011100 con 0x556ac5f69630 2024-05-09T08:36:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.488+0000 7efcf7c02740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efcd000e1e0 con 0x556ac5f5e450 2024-05-09T08:36:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.488+0000 7efcf7c02740 10 client.0 _ll_register_callbacks cb 0x7efce0009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.489+0000 7efcee59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f6c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.489+0000 7efcee59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f6c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:42928/0 (socket says 172.21.15.28:42928) 2024-05-09T08:36:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.489+0000 7efcee59c640 1 -- 172.21.15.28:0/529642559 learned_addr learned my addr 172.21.15.28:0/529642559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.489+0000 7efcedd9b640 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f6cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.489+0000 7efceed9d640 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 0x556ac5fdddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.490+0000 7efcedd9b640 1 -- 172.21.15.28:0/529642559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 msgr2=0x556ac5f6c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.490+0000 7efcedd9b640 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f6c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.490+0000 7efcedd9b640 1 -- 172.21.15.28:0/529642559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 msgr2=0x556ac5fdddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.490+0000 7efcedd9b640 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 0x556ac5fdddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.490+0000 7efcedd9b640 1 -- 172.21.15.28:0/529642559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac5f6d400 con 0x556ac5f5ed70 2024-05-09T08:36:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.490+0000 7efcedd9b640 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f6cb60 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7efce80b8c80 tx=0x7efce8031fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1be8b43425ac0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.491+0000 7efcd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:36:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.491+0000 7efcd5ffb640 1 -- 172.21.15.28:0/529642559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce80c0bd0 con 0x556ac5f5ed70 2024-05-09T08:36:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.491+0000 7efcf7c02740 1 -- 172.21.15.28:0/529642559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efce4014de0 con 0x556ac5f5ed70 2024-05-09T08:36:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.492+0000 7efcd5ffb640 1 -- 172.21.15.28:0/529642559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9671+0+0 (secure 0 0 0) 0x7efce80b8e90 con 0x556ac5f5ed70 2024-05-09T08:36:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.493+0000 7efcd5ffb640 1 -- 172.21.15.28:0/529642559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7efce80c5490 con 0x556ac5f5ed70 2024-05-09T08:36:20.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.496+0000 7efcf7c02740 10 client.16893 fetch_fsmap learned FSMap version 65 2024-05-09T08:36:20.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.496+0000 7efcf7c02740 1 -- 172.21.15.28:0/529642559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x556ac5fdf3f0 con 0x556ac5f5ed70 2024-05-09T08:36:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.500+0000 7efcd5ffb640 1 -- 172.21.15.28:0/529642559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce4014de0 con 0x556ac5f5ed70 2024-05-09T08:36:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.500+0000 7efcd5ffb640 1 -- 172.21.15.28:0/529642559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7efce80c5090 con 0x556ac5f5ed70 2024-05-09T08:36:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.504+0000 7efcf7c02740 10 client.16893 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:36:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.504+0000 7efcf7c02740 10 client.16893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:36:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.504+0000 7efcf7c02740 20 client.16893 populate_metadata read hostname 'smithi028' 2024-05-09T08:36:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.504+0000 7efcf7c02740 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556ac5fdfc30 0x556ac5fe2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.504+0000 7efcf7c02740 4 client.16893 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:36:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.504+0000 7efcf7c02740 1 -- 172.21.15.28:0/529642559 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ac5ecadc0 con 0x556ac5fdfc30 2024-05-09T08:36:20.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.517+0000 7efcee59c640 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556ac5fdfc30 0x556ac5fe2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.523+0000 7efcee59c640 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556ac5fdfc30 0x556ac5fe2050 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f5dc0222a6caa4e server_cookie=c5bc6b63136e0eef in_seq=0 out_seq=0 2024-05-09T08:36:20.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:20.523+0000 7efcd5ffb640 10 client.16893 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:21.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:21 smithi028 ceph-mon[30119]: pgmap v318: 129 pgs: 21 active+clean+snaptrim, 65 active+clean+snaptrim_wait, 43 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 236 op/s 2024-05-09T08:36:21.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:21 smithi165 ceph-mon[34540]: pgmap v318: 129 pgs: 21 active+clean+snaptrim, 65 active+clean+snaptrim_wait, 43 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 236 op/s 2024-05-09T08:36:21.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:21 smithi184 ceph-mon[34509]: pgmap v318: 129 pgs: 21 active+clean+snaptrim, 65 active+clean+snaptrim_wait, 43 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 236 op/s 2024-05-09T08:36:23.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:23 smithi028 ceph-mon[30119]: pgmap v319: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 176 op/s 2024-05-09T08:36:23.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:23 smithi165 ceph-mon[34540]: pgmap v319: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 176 op/s 2024-05-09T08:36:23.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:23 smithi184 ceph-mon[34509]: pgmap v319: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 176 op/s 2024-05-09T08:36:25.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:25 smithi028 ceph-mon[30119]: pgmap v320: 129 pgs: 18 active+clean+snaptrim, 44 active+clean+snaptrim_wait, 67 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 231 op/s 2024-05-09T08:36:25.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:25 smithi165 ceph-mon[34540]: pgmap v320: 129 pgs: 18 active+clean+snaptrim, 44 active+clean+snaptrim_wait, 67 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 231 op/s 2024-05-09T08:36:25.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:25 smithi184 ceph-mon[34509]: pgmap v320: 129 pgs: 18 active+clean+snaptrim, 44 active+clean+snaptrim_wait, 67 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 231 op/s 2024-05-09T08:36:27.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:27 smithi028 ceph-mon[30119]: pgmap v321: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 619 op/s 2024-05-09T08:36:27.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:27 smithi165 ceph-mon[34540]: pgmap v321: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 619 op/s 2024-05-09T08:36:27.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:27 smithi184 ceph-mon[34509]: pgmap v321: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 619 op/s 2024-05-09T08:36:29.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:29 smithi165 ceph-mon[34540]: pgmap v322: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 887 op/s 2024-05-09T08:36:29.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:29 smithi184 ceph-mon[34509]: pgmap v322: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 887 op/s 2024-05-09T08:36:29.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:29 smithi028 ceph-mon[30119]: pgmap v322: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 887 op/s 2024-05-09T08:36:31.733 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:31 smithi028 ceph-mon[30119]: pgmap v323: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.6 MiB/s wr, 1.30k op/s 2024-05-09T08:36:31.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:31 smithi165 ceph-mon[34540]: pgmap v323: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.6 MiB/s wr, 1.30k op/s 2024-05-09T08:36:31.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:31 smithi184 ceph-mon[34509]: pgmap v323: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.6 MiB/s wr, 1.30k op/s 2024-05-09T08:36:33.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:33 smithi165 ceph-mon[34540]: pgmap v324: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 1.6 MiB/s wr, 1.65k op/s 2024-05-09T08:36:33.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:33 smithi184 ceph-mon[34509]: pgmap v324: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 1.6 MiB/s wr, 1.65k op/s 2024-05-09T08:36:33.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:33 smithi028 ceph-mon[30119]: pgmap v324: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 1.6 MiB/s wr, 1.65k op/s 2024-05-09T08:36:35.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:35 smithi165 ceph-mon[34540]: pgmap v325: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.3 MiB/s wr, 2.09k op/s 2024-05-09T08:36:35.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:35 smithi184 ceph-mon[34509]: pgmap v325: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.3 MiB/s wr, 2.09k op/s 2024-05-09T08:36:35.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:35 smithi028 ceph-mon[30119]: pgmap v325: 129 pgs: 17 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 72 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.3 MiB/s wr, 2.09k op/s 2024-05-09T08:36:37.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:37 smithi165 ceph-mon[34540]: pgmap v326: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.0 MiB/s wr, 2.37k op/s 2024-05-09T08:36:37.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:37 smithi184 ceph-mon[34509]: pgmap v326: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.0 MiB/s wr, 2.37k op/s 2024-05-09T08:36:37.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:37 smithi028 ceph-mon[30119]: pgmap v326: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.0 MiB/s wr, 2.37k op/s 2024-05-09T08:36:39.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:39 smithi165 ceph-mon[34540]: pgmap v327: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 676 KiB/s wr, 2.05k op/s 2024-05-09T08:36:39.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:39 smithi184 ceph-mon[34509]: pgmap v327: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 676 KiB/s wr, 2.05k op/s 2024-05-09T08:36:39.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:39 smithi028 ceph-mon[30119]: pgmap v327: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 676 KiB/s wr, 2.05k op/s 2024-05-09T08:36:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcd5ffb640 1 -- 172.21.15.28:0/529642559 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556ac5ecadc0 con 0x556ac5fdfc30 2024-05-09T08:36:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcd5ffb640 10 client.16893 handle_command_reply: tid=0 2024-05-09T08:36:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcf7c02740 1 client.16893 shutdown 2024-05-09T08:36:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcf7c02740 1 -- 172.21.15.28:0/529642559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 msgr2=0x556ac5f6cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcf7c02740 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f6cb60 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7efce80b8c80 tx=0x7efce8031fd0 comp rx=0 tx=0).stop 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcf7c02740 1 -- 172.21.15.28:0/529642559 shutdown_connections 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcf7c02740 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556ac5fdfc30 0x556ac5fe2050 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcf7c02740 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ac5f69630 0x556ac5fdddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcf7c02740 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac5f5ed70 0x556ac5f6cb60 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.189+0000 7efcf7c02740 1 --2- 172.21.15.28:0/529642559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556ac5f5e450 0x556ac5f6c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.190+0000 7efcf7c02740 1 -- 172.21.15.28:0/529642559 >> 172.21.15.28:0/529642559 conn(0x556ac5f5a960 msgr2=0x556ac5f66e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.190+0000 7efcd5ffb640 0 client.16893 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.190+0000 7efcf7c02740 1 -- 172.21.15.28:0/529642559 shutdown_connections 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.190+0000 7efcf7c02740 1 -- 172.21.15.28:0/529642559 wait complete. 2024-05-09T08:36:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.190+0000 7efcf7c02740 20 client.16893 trim_cache size 0 max 16384 2024-05-09T08:36:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:36:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:36:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:36:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:36:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.191+0000 7efcf4e15640 1 -- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efcd80a2fc0 msgr2=0x7efcd80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.191+0000 7efcf4e15640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efcd80a2fc0 0x7efcd80a5480 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efce400b830 tx=0x7efce4008040 comp rx=0 tx=0).stop 2024-05-09T08:36:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.191+0000 7efcf4e15640 1 -- 172.21.15.28:0/2836597274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcf01091f0 msgr2=0x7efcf01140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.191+0000 7efcf4e15640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcf01091f0 0x7efcf01140b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efce8009900 tx=0x7efce8033a00 comp rx=0 tx=0).stop 2024-05-09T08:36:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.194+0000 7efcf4e15640 1 -- 172.21.15.28:0/2836597274 shutdown_connections 2024-05-09T08:36:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.194+0000 7efcf4e15640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf010d290 0x7efcf01145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.194+0000 7efcf4e15640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efcd80a2fc0 0x7efcd80a5480 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.194+0000 7efcf4e15640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcf01091f0 0x7efcf01140b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.194+0000 7efcf4e15640 1 --2- 172.21.15.28:0/2836597274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf0108830 0x7efcf0113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.194+0000 7efcf4e15640 1 -- 172.21.15.28:0/2836597274 >> 172.21.15.28:0/2836597274 conn(0x7efcf006cd90 msgr2=0x7efcf0072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.196+0000 7efcf4e15640 1 -- 172.21.15.28:0/2836597274 shutdown_connections 2024-05-09T08:36:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:40.196+0000 7efcf4e15640 1 -- 172.21.15.28:0/2836597274 wait complete. 2024-05-09T08:36:41.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:41 smithi165 ceph-mon[34540]: pgmap v328: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 602 KiB/s wr, 1.94k op/s 2024-05-09T08:36:41.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:41 smithi184 ceph-mon[34509]: pgmap v328: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 602 KiB/s wr, 1.94k op/s 2024-05-09T08:36:41.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:41 smithi028 ceph-mon[30119]: pgmap v328: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 602 KiB/s wr, 1.94k op/s 2024-05-09T08:36:43.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:43 smithi165 ceph-mon[34540]: pgmap v329: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 610 KiB/s wr, 1.52k op/s 2024-05-09T08:36:43.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:43 smithi184 ceph-mon[34509]: pgmap v329: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 610 KiB/s wr, 1.52k op/s 2024-05-09T08:36:43.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:43 smithi028 ceph-mon[30119]: pgmap v329: 129 pgs: 14 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 75 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 610 KiB/s wr, 1.52k op/s 2024-05-09T08:36:45.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:45 smithi165 ceph-mon[34540]: pgmap v330: 129 pgs: 13 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 81 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 1.0 MiB/s wr, 1.20k op/s 2024-05-09T08:36:45.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:45 smithi184 ceph-mon[34509]: pgmap v330: 129 pgs: 13 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 81 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 1.0 MiB/s wr, 1.20k op/s 2024-05-09T08:36:45.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:45 smithi028 ceph-mon[30119]: pgmap v330: 129 pgs: 13 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 81 active+clean; 2.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 1.0 MiB/s wr, 1.20k op/s 2024-05-09T08:36:47.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:47 smithi165 ceph-mon[34540]: pgmap v331: 129 pgs: 12 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 88 active+clean; 2.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 804 op/s 2024-05-09T08:36:47.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:47 smithi184 ceph-mon[34509]: pgmap v331: 129 pgs: 12 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 88 active+clean; 2.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 804 op/s 2024-05-09T08:36:47.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:47 smithi028 ceph-mon[30119]: pgmap v331: 129 pgs: 12 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 88 active+clean; 2.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 804 op/s 2024-05-09T08:36:49.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:49 smithi165 ceph-mon[34540]: pgmap v332: 129 pgs: 9 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 106 active+clean; 2.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 469 op/s 2024-05-09T08:36:49.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:49 smithi184 ceph-mon[34509]: pgmap v332: 129 pgs: 9 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 106 active+clean; 2.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 469 op/s 2024-05-09T08:36:49.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:49 smithi028 ceph-mon[30119]: pgmap v332: 129 pgs: 9 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 106 active+clean; 2.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 469 op/s 2024-05-09T08:36:51.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:51 smithi165 ceph-mon[34540]: pgmap v333: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 2.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 460 op/s 2024-05-09T08:36:51.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:51 smithi165 ceph-mon[34540]: osdmap e109: 12 total, 12 up, 12 in 2024-05-09T08:36:51.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:51 smithi184 ceph-mon[34509]: pgmap v333: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 2.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 460 op/s 2024-05-09T08:36:51.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:51 smithi184 ceph-mon[34509]: osdmap e109: 12 total, 12 up, 12 in 2024-05-09T08:36:51.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:51 smithi028 ceph-mon[30119]: pgmap v333: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 2.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 460 op/s 2024-05-09T08:36:51.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:51 smithi028 ceph-mon[30119]: osdmap e109: 12 total, 12 up, 12 in 2024-05-09T08:36:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.345+0000 7f001425d640 1 -- 172.21.15.28:0/1256849184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000c073bd0 msgr2=0x7f000c073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.345+0000 7f001425d640 1 --2- 172.21.15.28:0/1256849184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000c073bd0 0x7f000c073fb0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f000801a910 tx=0x7f000803b4e0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.345+0000 7f001425d640 1 -- 172.21.15.28:0/1256849184 shutdown_connections 2024-05-09T08:36:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.345+0000 7f001425d640 1 --2- 172.21.15.28:0/1256849184 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c112850 0x7f000c114c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.345+0000 7f001425d640 1 --2- 172.21.15.28:0/1256849184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f000c0744f0 0x7f000c112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.345+0000 7f001425d640 1 --2- 172.21.15.28:0/1256849184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000c073bd0 0x7f000c073fb0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.345+0000 7f001425d640 1 -- 172.21.15.28:0/1256849184 >> 172.21.15.28:0/1256849184 conn(0x7f000c0fd2f0 msgr2=0x7f000c0ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.345+0000 7f001425d640 1 -- 172.21.15.28:0/1256849184 shutdown_connections 2024-05-09T08:36:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.345+0000 7f001425d640 1 -- 172.21.15.28:0/1256849184 wait complete. 2024-05-09T08:36:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.346+0000 7f001425d640 1 Processor -- start 2024-05-09T08:36:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.346+0000 7f001425d640 1 -- start start 2024-05-09T08:36:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.346+0000 7f001425d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f000c073bd0 0x7f000c112090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.346+0000 7f001425d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c0744f0 0x7f000c10e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.346+0000 7f001425d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000c112850 0x7f000c10e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.346+0000 7f001425d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f000c108580 con 0x7f000c112850 2024-05-09T08:36:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.346+0000 7f001425d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f000c108400 con 0x7f000c0744f0 2024-05-09T08:36:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.346+0000 7f001425d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f000c108700 con 0x7f000c073bd0 2024-05-09T08:36:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.347+0000 7f0011ff9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f000c073bd0 0x7f000c112090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.347+0000 7f0011ff9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f000c073bd0 0x7f000c112090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37792/0 (socket says 172.21.15.28:37792) 2024-05-09T08:36:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.347+0000 7f0011ff9640 1 -- 172.21.15.28:0/2776277111 learned_addr learned my addr 172.21.15.28:0/2776277111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.347+0000 7f00127fa640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000c112850 0x7f000c10e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.348+0000 7f00127fa640 1 -- 172.21.15.28:0/2776277111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f000c073bd0 msgr2=0x7f000c112090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.348+0000 7f00127fa640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f000c073bd0 0x7f000c112090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.348+0000 7f00127fa640 1 -- 172.21.15.28:0/2776277111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c0744f0 msgr2=0x7f000c10e1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.348+0000 7f00127fa640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c0744f0 0x7f000c10e1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.348+0000 7f00127fa640 1 -- 172.21.15.28:0/2776277111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f000c10ef70 con 0x7f000c112850 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.349+0000 7f00127fa640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000c112850 0x7f000c10e700 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7efffc00b3e0 tx=0x7efffc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.349+0000 7f0002ffd640 1 -- 172.21.15.28:0/2776277111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc015020 con 0x7f000c112850 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.349+0000 7f001425d640 1 -- 172.21.15.28:0/2776277111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f000c1a2620 con 0x7f000c112850 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.349+0000 7f0002ffd640 1 -- 172.21.15.28:0/2776277111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efffc011d10 con 0x7f000c112850 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.349+0000 7f001425d640 1 -- 172.21.15.28:0/2776277111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f000c1a2af0 con 0x7f000c112850 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.350+0000 7f0002ffd640 1 -- 172.21.15.28:0/2776277111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc015020 con 0x7f000c112850 2024-05-09T08:36:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.353+0000 7f0016f6a740 1 Processor -- start 2024-05-09T08:36:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.354+0000 7f0002ffd640 1 -- 172.21.15.28:0/2776277111 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7efffc018020 con 0x7f000c112850 2024-05-09T08:36:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.355+0000 7f0016f6a740 1 -- start start 2024-05-09T08:36:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.355+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e09830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.355+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e140f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.355+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 0x55d986e16a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.355+0000 7f0016f6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d986d6e340 con 0x55d986e09d70 2024-05-09T08:36:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.355+0000 7f0016f6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55d986d16290 con 0x55d986e09450 2024-05-09T08:36:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.355+0000 7f0016f6a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d986d6a300 con 0x55d986e14630 2024-05-09T08:36:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.355+0000 7f00117f8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e09830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.356+0000 7f00117f8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e09830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:44520/0 (socket says 172.21.15.28:44520) 2024-05-09T08:36:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.356+0000 7f00117f8640 1 -- 172.21.15.28:0/1260353625 learned_addr learned my addr 172.21.15.28:0/1260353625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.356+0000 7f0002ffd640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efff80b4210 0x7efff80b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.356+0000 7f00117f8640 1 --2- 172.21.15.28:0/1260353625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 0x55d986e16a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.356+0000 7f0011ff9640 1 --2- 172.21.15.28:0/1260353625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.356+0000 7f00127fa640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efff80b4210 0x7efff80b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.356+0000 7f0002ffd640 1 -- 172.21.15.28:0/2776277111 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9595+0+0 (secure 0 0 0) 0x7efffc011630 con 0x7f000c112850 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.357+0000 7f00117f8640 1 -- 172.21.15.28:0/1260353625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 msgr2=0x55d986e16a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.357+0000 7f00117f8640 1 --2- 172.21.15.28:0/1260353625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 0x55d986e16a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.357+0000 7f00117f8640 1 -- 172.21.15.28:0/1260353625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 msgr2=0x55d986e140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.357+0000 7f00117f8640 1 --2- 172.21.15.28:0/1260353625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e140f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.357+0000 7f00117f8640 1 -- 172.21.15.28:0/1260353625 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d986d3a8e0 con 0x55d986e09450 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.357+0000 7f00127fa640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efff80b4210 0x7efff80b66d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efffc096ed0 tx=0x7efffc08ff00 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.357+0000 7f00117f8640 1 --2- 172.21.15.28:0/1260353625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e09830 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0004013910 tx=0x7f00040018c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=898c88caaca182ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.358+0000 7efff7fff640 1 -- 172.21.15.28:0/1260353625 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000400d070 con 0x55d986e09450 2024-05-09T08:36:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.358+0000 7efff7fff640 1 -- 172.21.15.28:0/1260353625 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0004013e90 con 0x55d986e09450 2024-05-09T08:36:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.358+0000 7efff7fff640 1 -- 172.21.15.28:0/1260353625 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0004010070 con 0x55d986e09450 2024-05-09T08:36:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.359+0000 7f0016f6a740 1 -- 172.21.15.28:0/1260353625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 msgr2=0x55d986e09830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.359+0000 7f0016f6a740 1 --2- 172.21.15.28:0/1260353625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e09830 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0004013910 tx=0x7f00040018c0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.360+0000 7f0016f6a740 1 -- 172.21.15.28:0/1260353625 shutdown_connections 2024-05-09T08:36:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.360+0000 7f0016f6a740 1 --2- 172.21.15.28:0/1260353625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 0x55d986e16a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.360+0000 7f0016f6a740 1 --2- 172.21.15.28:0/1260353625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.360+0000 7f0016f6a740 1 --2- 172.21.15.28:0/1260353625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e09830 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.360+0000 7f0016f6a740 1 -- 172.21.15.28:0/1260353625 >> 172.21.15.28:0/1260353625 conn(0x55d986e05960 msgr2=0x55d986e05d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.363+0000 7f0016f6a740 1 -- 172.21.15.28:0/1260353625 shutdown_connections 2024-05-09T08:36:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.363+0000 7f0016f6a740 1 -- 172.21.15.28:0/1260353625 wait complete. 2024-05-09T08:36:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.363+0000 7f0016f6a740 1 -- start start 2024-05-09T08:36:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0016f6a740 1 Processor -- start 2024-05-09T08:36:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e115b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e11af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 0x55d986e12030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0016f6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d986e17200 con 0x55d986e09d70 2024-05-09T08:36:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0016f6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55d986d29790 con 0x55d986e09450 2024-05-09T08:36:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0016f6a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55d986e17380 con 0x55d986e14630 2024-05-09T08:36:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0016f6a740 10 client.0 _ll_register_callbacks cb 0x55d986d64a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0011ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e11af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0011ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e11af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53308/0 (socket says 172.21.15.28:53308) 2024-05-09T08:36:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.364+0000 7f0011ff9640 1 -- 172.21.15.28:0/2977684598 learned_addr learned my addr 172.21.15.28:0/2977684598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:53.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.365+0000 7f00117f8640 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 0x55d986e12030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.365+0000 7f00117f8640 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.365+0000 7f0011ff9640 1 -- 172.21.15.28:0/2977684598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 msgr2=0x55d986e12030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.365+0000 7f0011ff9640 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 0x55d986e12030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.365+0000 7f0011ff9640 1 -- 172.21.15.28:0/2977684598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 msgr2=0x55d986e115b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.365+0000 7f0011ff9640 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e115b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.365+0000 7f0011ff9640 1 -- 172.21.15.28:0/2977684598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d986a82210 con 0x55d986e09d70 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.365+0000 7f0011ff9640 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e11af0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0008003850 tx=0x7f0008003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71ec3a4ed0569d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.365+0000 7efff57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.366+0000 7efff57fa640 1 -- 172.21.15.28:0/2977684598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008003c30 con 0x55d986e09d70 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.366+0000 7efff57fa640 1 -- 172.21.15.28:0/2977684598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9595+0+0 (secure 0 0 0) 0x7f0008003dd0 con 0x55d986e09d70 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.366+0000 7f0016f6a740 1 -- 172.21.15.28:0/2977684598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0004013e90 con 0x55d986e09d70 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.366+0000 7efff57fa640 1 -- 172.21.15.28:0/2977684598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f0008040510 con 0x55d986e09d70 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.366+0000 7f0016f6a740 10 client.16905 fetch_fsmap learned FSMap version 65 2024-05-09T08:36:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.366+0000 7f0016f6a740 1 -- 172.21.15.28:0/2977684598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55d986b4e620 con 0x55d986e09d70 2024-05-09T08:36:53.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.367+0000 7efff57fa640 1 -- 172.21.15.28:0/2977684598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0004013e90 con 0x55d986e09d70 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.367+0000 7efff57fa640 1 -- 172.21.15.28:0/2977684598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f0008055070 con 0x55d986e09d70 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.367+0000 7f0016f6a740 10 client.16905 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.367+0000 7f0016f6a740 10 client.16905 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.367+0000 7f0016f6a740 20 client.16905 populate_metadata read hostname 'smithi028' 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.367+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d986e24370 0x55d986e267f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.367+0000 7f0016f6a740 4 client.16905 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.367+0000 7f0016f6a740 1 -- 172.21.15.28:0/2977684598 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d986d7ba70 con 0x55d986e24370 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.367+0000 7f00117f8640 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d986e24370 0x55d986e267f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.368+0000 7f00117f8640 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d986e24370 0x55d986e267f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b73b25dd3b7c13ab server_cookie=b2b6c97831f16931 in_seq=0 out_seq=0 2024-05-09T08:36:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.368+0000 7efff57fa640 10 client.16905 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7efff57fa640 1 -- 172.21.15.28:0/2977684598 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d986d7ba70 con 0x55d986e24370 2024-05-09T08:36:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7efff57fa640 10 client.16905 handle_command_reply: tid=0 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 client.16905 shutdown 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 -- 172.21.15.28:0/2977684598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 msgr2=0x55d986e11af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e11af0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0008003850 tx=0x7f0008003880 comp rx=0 tx=0).stop 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 -- 172.21.15.28:0/2977684598 shutdown_connections 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d986e24370 0x55d986e267f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e14630 0x55d986e12030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e11af0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2977684598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e09450 0x55d986e115b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 -- 172.21.15.28:0/2977684598 >> 172.21.15.28:0/2977684598 conn(0x55d986e05960 msgr2=0x55d986e0b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7efff57fa640 0 client.16905 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 -- 172.21.15.28:0/2977684598 shutdown_connections 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.369+0000 7f0016f6a740 1 -- 172.21.15.28:0/2977684598 wait complete. 2024-05-09T08:36:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.370+0000 7f0016f6a740 20 client.16905 trim_cache size 0 max 16384 2024-05-09T08:36:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.397+0000 7f0016f6a740 1 Processor -- start 2024-05-09T08:36:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.397+0000 7f0016f6a740 1 -- start start 2024-05-09T08:36:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.397+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09450 0x55d986e200c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.397+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e10710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 0x55d986e0b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f00117f8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09450 0x55d986e200c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f0011ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e10710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f00117f8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09450 0x55d986e200c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37820/0 (socket says 172.21.15.28:37820) 2024-05-09T08:36:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f0016f6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f000c1a4650 con 0x55d986e09d70 2024-05-09T08:36:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f0016f6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f000c1a63d0 con 0x55d986e14630 2024-05-09T08:36:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f0016f6a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f000c1a4de0 con 0x55d986e09450 2024-05-09T08:36:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f00117f8640 1 -- 172.21.15.28:0/3817669177 learned_addr learned my addr 172.21.15.28:0/3817669177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f00117f8640 1 -- 172.21.15.28:0/3817669177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 msgr2=0x55d986e0b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f00117f8640 1 --2- 172.21.15.28:0/3817669177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 0x55d986e0b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f00117f8640 1 -- 172.21.15.28:0/3817669177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 msgr2=0x55d986e10710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f00117f8640 1 --2- 172.21.15.28:0/3817669177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e10710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.398+0000 7f00117f8640 1 -- 172.21.15.28:0/3817669177 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d986e03f00 con 0x55d986e09450 2024-05-09T08:36:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.399+0000 7f00117f8640 1 --2- 172.21.15.28:0/3817669177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09450 0x55d986e200c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0004013750 tx=0x7f0004001f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9bb1fc8715f60c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.400+0000 7efff77fe640 1 -- 172.21.15.28:0/3817669177 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0004030020 con 0x55d986e09450 2024-05-09T08:36:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.403+0000 7efff77fe640 1 -- 172.21.15.28:0/3817669177 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0004002000 con 0x55d986e09450 2024-05-09T08:36:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.403+0000 7efff77fe640 1 -- 172.21.15.28:0/3817669177 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000400d040 con 0x55d986e09450 2024-05-09T08:36:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.405+0000 7f0016f6a740 1 -- 172.21.15.28:0/3817669177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09450 msgr2=0x55d986e200c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.405+0000 7f0016f6a740 1 --2- 172.21.15.28:0/3817669177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09450 0x55d986e200c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0004013750 tx=0x7f0004001f90 comp rx=0 tx=0).stop 2024-05-09T08:36:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.405+0000 7f0016f6a740 1 -- 172.21.15.28:0/3817669177 shutdown_connections 2024-05-09T08:36:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.406+0000 7f0016f6a740 1 --2- 172.21.15.28:0/3817669177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 0x55d986e0b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.406+0000 7f0016f6a740 1 --2- 172.21.15.28:0/3817669177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09d70 0x55d986e10710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.406+0000 7f0016f6a740 1 --2- 172.21.15.28:0/3817669177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09450 0x55d986e200c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.406+0000 7f0016f6a740 1 -- 172.21.15.28:0/3817669177 >> 172.21.15.28:0/3817669177 conn(0x55d986e05960 msgr2=0x55d986e0ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.406+0000 7f0016f6a740 1 -- 172.21.15.28:0/3817669177 shutdown_connections 2024-05-09T08:36:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.406+0000 7f0016f6a740 1 -- 172.21.15.28:0/3817669177 wait complete. 2024-05-09T08:36:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.407+0000 7f0016f6a740 1 -- start start 2024-05-09T08:36:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.407+0000 7f0016f6a740 1 Processor -- start 2024-05-09T08:36:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.407+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09450 0x55d986e0e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.407+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09d70 0x55d986e0e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.407+0000 7f0016f6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 0x55d986e0ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.408+0000 7f0016f6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f000c1a5d00 con 0x55d986e09450 2024-05-09T08:36:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.408+0000 7f0016f6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f000c1a5570 con 0x55d986e14630 2024-05-09T08:36:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.408+0000 7f0016f6a740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f000c073280 con 0x55d986e09d70 2024-05-09T08:36:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.408+0000 7f0016f6a740 10 client.0 _ll_register_callbacks cb 0x7f00080047a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.408+0000 7f00117f8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 0x55d986e0ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.408+0000 7f00117f8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 0x55d986e0ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:44536/0 (socket says 172.21.15.28:44536) 2024-05-09T08:36:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.408+0000 7f00117f8640 1 -- 172.21.15.28:0/2549175837 learned_addr learned my addr 172.21.15.28:0/2549175837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:36:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.408+0000 7f00117f8640 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09450 0x55d986e0e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.409+0000 7f00117f8640 1 -- 172.21.15.28:0/2549175837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09d70 msgr2=0x55d986e0e9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:36:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.409+0000 7f00117f8640 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09d70 0x55d986e0e9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.409+0000 7f00117f8640 1 -- 172.21.15.28:0/2549175837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09450 msgr2=0x55d986e0e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.409+0000 7f00117f8640 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09450 0x55d986e0e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.409+0000 7f00117f8640 1 -- 172.21.15.28:0/2549175837 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d986e89b80 con 0x55d986e14630 2024-05-09T08:36:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.409+0000 7f00117f8640 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 0x55d986e0ef00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0004014010 tx=0x7f0004013670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53518cf150b3ae5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.412+0000 7efff57fa640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:36:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.412+0000 7efff57fa640 1 -- 172.21.15.28:0/2549175837 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0004030070 con 0x55d986e14630 2024-05-09T08:36:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.412+0000 7f0016f6a740 1 -- 172.21.15.28:0/2549175837 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0004002000 con 0x55d986e14630 2024-05-09T08:36:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.412+0000 7efff57fa640 1 -- 172.21.15.28:0/2549175837 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9595+0+0 (secure 0 0 0) 0x7f00040189c0 con 0x55d986e14630 2024-05-09T08:36:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.413+0000 7efff57fa640 1 -- 172.21.15.28:0/2549175837 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f000400d040 con 0x55d986e14630 2024-05-09T08:36:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.416+0000 7f0016f6a740 10 client.26417 fetch_fsmap learned FSMap version 65 2024-05-09T08:36:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.416+0000 7f0016f6a740 1 -- 172.21.15.28:0/2549175837 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55d986e8ae50 con 0x55d986e14630 2024-05-09T08:36:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.416+0000 7efff57fa640 1 -- 172.21.15.28:0/2549175837 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0004002000 con 0x55d986e14630 2024-05-09T08:36:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.417+0000 7efff57fa640 1 -- 172.21.15.28:0/2549175837 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f000400a790 con 0x55d986e14630 2024-05-09T08:36:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.417+0000 7f0016f6a740 10 client.26417 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:36:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.417+0000 7f0016f6a740 10 client.26417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:36:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.417+0000 7f0016f6a740 20 client.26417 populate_metadata read hostname 'smithi028' 2024-05-09T08:36:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.417+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d986e8b480 0x55d986e8d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.417+0000 7f0016f6a740 4 client.26417 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:36:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.417+0000 7f0016f6a740 1 -- 172.21.15.28:0/2549175837 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d986d75dc0 con 0x55d986e8b480 2024-05-09T08:36:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.417+0000 7f0011ff9640 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d986e8b480 0x55d986e8d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.418+0000 7f0011ff9640 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d986e8b480 0x55d986e8d8a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59bd6545db937af server_cookie=dafda3db982e62b7 in_seq=0 out_seq=0 2024-05-09T08:36:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.418+0000 7efff57fa640 10 client.26417 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:53.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7efff57fa640 1 -- 172.21.15.28:0/2549175837 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d986d75dc0 con 0x55d986e8b480 2024-05-09T08:36:53.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7efff57fa640 10 client.26417 handle_command_reply: tid=0 2024-05-09T08:36:53.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7f0016f6a740 1 client.26417 shutdown 2024-05-09T08:36:53.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7f0016f6a740 1 -- 172.21.15.28:0/2549175837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 msgr2=0x55d986e0ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:53.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 0x55d986e0ef00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0004014010 tx=0x7f0004013670 comp rx=0 tx=0).stop 2024-05-09T08:36:53.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7f0016f6a740 1 -- 172.21.15.28:0/2549175837 shutdown_connections 2024-05-09T08:36:53.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55d986e8b480 0x55d986e8d8a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d986e14630 0x55d986e0ef00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55d986e09d70 0x55d986e0e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7f0016f6a740 1 --2- 172.21.15.28:0/2549175837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d986e09450 0x55d986e0e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.547+0000 7f0016f6a740 1 -- 172.21.15.28:0/2549175837 >> 172.21.15.28:0/2549175837 conn(0x55d986e05960 msgr2=0x55d986e04a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.548+0000 7efff57fa640 0 client.26417 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:36:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.548+0000 7f0016f6a740 1 -- 172.21.15.28:0/2549175837 shutdown_connections 2024-05-09T08:36:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.548+0000 7f0016f6a740 1 -- 172.21.15.28:0/2549175837 wait complete. 2024-05-09T08:36:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.548+0000 7f0016f6a740 20 client.26417 trim_cache size 0 max 16384 2024-05-09T08:36:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:36:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:36:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:36:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:36:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 -- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efff80b4210 msgr2=0x7efff80b66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efff80b4210 0x7efff80b66d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efffc096ed0 tx=0x7efffc08ff00 comp rx=0 tx=0).stop 2024-05-09T08:36:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 -- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000c112850 msgr2=0x7f000c10e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000c112850 0x7f000c10e700 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7efffc00b3e0 tx=0x7efffc00b8a0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 -- 172.21.15.28:0/2776277111 shutdown_connections 2024-05-09T08:36:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000c112850 0x7f000c10e700 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c0744f0 0x7f000c10e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7efff80b4210 0x7efff80b66d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 --2- 172.21.15.28:0/2776277111 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f000c073bd0 0x7f000c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.549+0000 7f001425d640 1 -- 172.21.15.28:0/2776277111 >> 172.21.15.28:0/2776277111 conn(0x7f000c0fd2f0 msgr2=0x7f000c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.550+0000 7f001425d640 1 -- 172.21.15.28:0/2776277111 shutdown_connections 2024-05-09T08:36:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:36:53.550+0000 7f001425d640 1 -- 172.21.15.28:0/2776277111 wait complete. 2024-05-09T08:36:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:53 smithi184 ceph-mon[34509]: pgmap v335: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 2.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 353 op/s 2024-05-09T08:36:53.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:53 smithi028 ceph-mon[30119]: pgmap v335: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 2.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 353 op/s 2024-05-09T08:36:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:53 smithi165 ceph-mon[34540]: pgmap v335: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 2.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 353 op/s 2024-05-09T08:36:55.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:55 smithi184 ceph-mon[34509]: pgmap v336: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 399 op/s 2024-05-09T08:36:55.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:55 smithi028 ceph-mon[30119]: pgmap v336: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 399 op/s 2024-05-09T08:36:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:55 smithi165 ceph-mon[34540]: pgmap v336: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 399 op/s 2024-05-09T08:36:57.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:57 smithi028 ceph-mon[30119]: pgmap v337: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 496 op/s 2024-05-09T08:36:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:57 smithi165 ceph-mon[34540]: pgmap v337: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 496 op/s 2024-05-09T08:36:58.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:57 smithi184 ceph-mon[34509]: pgmap v337: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 496 op/s 2024-05-09T08:36:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:59 smithi028 ceph-mon[30119]: pgmap v338: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 524 op/s 2024-05-09T08:36:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:36:59 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:37:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:59 smithi165 ceph-mon[34540]: pgmap v338: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 524 op/s 2024-05-09T08:37:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:36:59 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:37:00.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:59 smithi184 ceph-mon[34509]: pgmap v338: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 524 op/s 2024-05-09T08:37:00.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:36:59 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:37:01.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:01 smithi165 ceph-mon[34540]: pgmap v339: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 581 op/s 2024-05-09T08:37:01.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:01 smithi165 ceph-mon[34540]: osdmap e110: 12 total, 12 up, 12 in 2024-05-09T08:37:01.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:01 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:01.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:01 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:01.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:01 smithi184 ceph-mon[34509]: pgmap v339: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 581 op/s 2024-05-09T08:37:01.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:01 smithi184 ceph-mon[34509]: osdmap e110: 12 total, 12 up, 12 in 2024-05-09T08:37:01.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:01 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:01.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:01 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:01.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:01 smithi028 ceph-mon[30119]: pgmap v339: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 581 op/s 2024-05-09T08:37:01.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:01 smithi028 ceph-mon[30119]: osdmap e110: 12 total, 12 up, 12 in 2024-05-09T08:37:01.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:01 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:01.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:01 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:02.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:02.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:02.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:02.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:02.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:02.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:02.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:02.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:02.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:02.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:02 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:02 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:03.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:03.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:02 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:03.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:03 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-05-09T08:37:03.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:03 smithi028 ceph-mon[30119]: Unable to set osd_memory_target on smithi028 to 185879859: error parsing value: Value '185879859' is below minimum 939524096 2024-05-09T08:37:03.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:03 smithi028 ceph-mon[30119]: pgmap v341: 129 pgs: 129 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 612 op/s 2024-05-09T08:37:03.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:03 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi184 to 176.2M 2024-05-09T08:37:03.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:03 smithi028 ceph-mon[30119]: Unable to set osd_memory_target on smithi184 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-05-09T08:37:03.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:03 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:03.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:03 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:04.006 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:03 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-05-09T08:37:04.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:03 smithi184 ceph-mon[34509]: Unable to set osd_memory_target on smithi028 to 185879859: error parsing value: Value '185879859' is below minimum 939524096 2024-05-09T08:37:04.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:03 smithi184 ceph-mon[34509]: pgmap v341: 129 pgs: 129 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 612 op/s 2024-05-09T08:37:04.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:03 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi184 to 176.2M 2024-05-09T08:37:04.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:03 smithi184 ceph-mon[34509]: Unable to set osd_memory_target on smithi184 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-05-09T08:37:04.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:03 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:04.007 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:03 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:04.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:03 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-05-09T08:37:04.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:03 smithi165 ceph-mon[34540]: Unable to set osd_memory_target on smithi028 to 185879859: error parsing value: Value '185879859' is below minimum 939524096 2024-05-09T08:37:04.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:03 smithi165 ceph-mon[34540]: pgmap v341: 129 pgs: 129 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 612 op/s 2024-05-09T08:37:04.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:03 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi184 to 176.2M 2024-05-09T08:37:04.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:03 smithi165 ceph-mon[34540]: Unable to set osd_memory_target on smithi184 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-05-09T08:37:04.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:03 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:04.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:03 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:04.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:04 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:04.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:04 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:04.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:04 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:04.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:04 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:04.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:04 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:37:04.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:04 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:37:04.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:04 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:04.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:04 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:04.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:04 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:04.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:04 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:04.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:04 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:04.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:04 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:37:04.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:04 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:37:04.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:04 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:05.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:04 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:05.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:04 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:05.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:04 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:05.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:04 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:37:05.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:04 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:37:05.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:04 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:37:05.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:04 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:37:05.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:05 smithi028 ceph-mon[30119]: Adjusting osd_memory_target on smithi165 to 176.2M 2024-05-09T08:37:05.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:05 smithi028 ceph-mon[30119]: Unable to set osd_memory_target on smithi165 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-09T08:37:05.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:05 smithi028 ceph-mon[30119]: pgmap v342: 129 pgs: 129 active+clean; 2.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 609 op/s 2024-05-09T08:37:05.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:05 smithi028 ceph-mon[30119]: osdmap e111: 12 total, 12 up, 12 in 2024-05-09T08:37:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:05 smithi165 ceph-mon[34540]: Adjusting osd_memory_target on smithi165 to 176.2M 2024-05-09T08:37:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:05 smithi165 ceph-mon[34540]: Unable to set osd_memory_target on smithi165 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-09T08:37:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:05 smithi165 ceph-mon[34540]: pgmap v342: 129 pgs: 129 active+clean; 2.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 609 op/s 2024-05-09T08:37:06.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:05 smithi165 ceph-mon[34540]: osdmap e111: 12 total, 12 up, 12 in 2024-05-09T08:37:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:05 smithi184 ceph-mon[34509]: Adjusting osd_memory_target on smithi165 to 176.2M 2024-05-09T08:37:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:05 smithi184 ceph-mon[34509]: Unable to set osd_memory_target on smithi165 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-09T08:37:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:05 smithi184 ceph-mon[34509]: pgmap v342: 129 pgs: 129 active+clean; 2.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 609 op/s 2024-05-09T08:37:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:05 smithi184 ceph-mon[34509]: osdmap e111: 12 total, 12 up, 12 in 2024-05-09T08:37:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.717+0000 7f28a5f63640 1 -- 172.21.15.28:0/2329755451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a011b0d0 msgr2=0x7f28a011d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.717+0000 7f28a5f63640 1 --2- 172.21.15.28:0/2329755451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a011b0d0 0x7f28a011d4c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f289801acb0 tx=0x7f289803b9b0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.718+0000 7f28a5f63640 1 -- 172.21.15.28:0/2329755451 shutdown_connections 2024-05-09T08:37:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.718+0000 7f28a5f63640 1 --2- 172.21.15.28:0/2329755451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a011b0d0 0x7f28a011d4c0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.718+0000 7f28a5f63640 1 --2- 172.21.15.28:0/2329755451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28a006bb80 0x7f28a011ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.718+0000 7f28a5f63640 1 --2- 172.21.15.28:0/2329755451 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a006b260 0x7f28a006b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.718+0000 7f28a5f63640 1 -- 172.21.15.28:0/2329755451 >> 172.21.15.28:0/2329755451 conn(0x7f28a006e970 msgr2=0x7f28a0108830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:06.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.720+0000 7f28a5f63640 1 -- 172.21.15.28:0/2329755451 shutdown_connections 2024-05-09T08:37:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.721+0000 7f28a5f63640 1 -- 172.21.15.28:0/2329755451 wait complete. 2024-05-09T08:37:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.722+0000 7f28a5f63640 1 Processor -- start 2024-05-09T08:37:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.725+0000 7f28a5f63640 1 -- start start 2024-05-09T08:37:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f28a5f63640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28a006b260 0x7f28a011a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f28a5f63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a006bb80 0x7f28a0116960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f28a5f63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a011b0d0 0x7f28a0116ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f28a5f63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28a0110df0 con 0x7f28a006bb80 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f28a5f63640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f28a0110c70 con 0x7f28a006b260 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f28a5f63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28a0110f70 con 0x7f28a011b0d0 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f289effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a006bb80 0x7f28a0116960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f289effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a006bb80 0x7f28a0116960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49980/0 (socket says 172.21.15.28:49980) 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f289effd640 1 -- 172.21.15.28:0/3982315400 learned_addr learned my addr 172.21.15.28:0/3982315400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.726+0000 7f289f7fe640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28a006b260 0x7f28a011a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.727+0000 7f289effd640 1 -- 172.21.15.28:0/3982315400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a011b0d0 msgr2=0x7f28a0116ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.727+0000 7f289effd640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a011b0d0 0x7f28a0116ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.727+0000 7f289effd640 1 -- 172.21.15.28:0/3982315400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28a006b260 msgr2=0x7f28a011a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.727+0000 7f289effd640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28a006b260 0x7f28a011a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.727+0000 7f289effd640 1 -- 172.21.15.28:0/3982315400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a0117620 con 0x7f28a006bb80 2024-05-09T08:37:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.727+0000 7f289effd640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a006bb80 0x7f28a0116960 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f289000b450 tx=0x7f289000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.727+0000 7f289cff9640 1 -- 172.21.15.28:0/3982315400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2890015020 con 0x7f28a006bb80 2024-05-09T08:37:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.728+0000 7f289cff9640 1 -- 172.21.15.28:0/3982315400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2890011d10 con 0x7f28a006bb80 2024-05-09T08:37:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.728+0000 7f289cff9640 1 -- 172.21.15.28:0/3982315400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289000f2e0 con 0x7f28a006bb80 2024-05-09T08:37:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.728+0000 7f28a5f63640 1 -- 172.21.15.28:0/3982315400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a01aadc0 con 0x7f28a006bb80 2024-05-09T08:37:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.728+0000 7f28a5f63640 1 -- 172.21.15.28:0/3982315400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a01ab1d0 con 0x7f28a006bb80 2024-05-09T08:37:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.739+0000 7f28a8c75740 1 Processor -- start 2024-05-09T08:37:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.742+0000 7f28a8c75740 1 -- start start 2024-05-09T08:37:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.743+0000 7f289cff9640 1 -- 172.21.15.28:0/3982315400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2890018020 con 0x7f28a006bb80 2024-05-09T08:37:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.743+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.743+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1d70 0x5585924cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.743+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924cc630 0x5585924cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.743+0000 7f28a8c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558592426340 con 0x5585924c1d70 2024-05-09T08:37:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.743+0000 7f28a8c75740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5585923ce290 con 0x5585924cc630 2024-05-09T08:37:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.743+0000 7f28a8c75740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558592422300 con 0x5585924c1450 2024-05-09T08:37:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.744+0000 7f289f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1d70 0x5585924cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.744+0000 7f289f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1d70 0x5585924cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49982/0 (socket says 172.21.15.28:49982) 2024-05-09T08:37:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.744+0000 7f289f7fe640 1 -- 172.21.15.28:0/1349315617 learned_addr learned my addr 172.21.15.28:0/1349315617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.745+0000 7f289ffff640 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.763+0000 7f289cff9640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f288c0a2fd0 0x7f288c0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.764+0000 7f289effd640 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924cc630 0x5585924cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.764+0000 7f289cff9640 1 -- 172.21.15.28:0/3982315400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7f28900cf3d0 con 0x7f28a006bb80 2024-05-09T08:37:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.765+0000 7f289f7fe640 1 -- 172.21.15.28:0/1349315617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 msgr2=0x5585924c1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.765+0000 7f289f7fe640 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.765+0000 7f289f7fe640 1 -- 172.21.15.28:0/1349315617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924cc630 msgr2=0x5585924cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.765+0000 7f289f7fe640 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924cc630 0x5585924cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.765+0000 7f289f7fe640 1 -- 172.21.15.28:0/1349315617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585923f28e0 con 0x5585924c1d70 2024-05-09T08:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.765+0000 7f289f7fe640 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1d70 0x5585924cc0f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f288800b820 tx=0x7f288800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5f0f0f62331f03 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.766+0000 7f289ffff640 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.766+0000 7f2895ffb640 1 -- 172.21.15.28:0/1349315617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288800d950 con 0x5585924c1d70 2024-05-09T08:37:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.766+0000 7f2895ffb640 1 -- 172.21.15.28:0/1349315617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2888019070 con 0x5585924c1d70 2024-05-09T08:37:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.766+0000 7f2895ffb640 1 -- 172.21.15.28:0/1349315617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28880123e0 con 0x5585924c1d70 2024-05-09T08:37:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.766+0000 7f289ffff640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f288c0a2fd0 0x7f288c0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.767+0000 7f289ffff640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f288c0a2fd0 0x7f288c0a5490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f289803bfa0 tx=0x7f289803b9b0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.768+0000 7f28a8c75740 1 -- 172.21.15.28:0/1349315617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1d70 msgr2=0x5585924cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.768+0000 7f28a8c75740 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1d70 0x5585924cc0f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f288800b820 tx=0x7f288800bce0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.768+0000 7f28a8c75740 1 -- 172.21.15.28:0/1349315617 shutdown_connections 2024-05-09T08:37:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.768+0000 7f28a8c75740 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924cc630 0x5585924cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.768+0000 7f28a8c75740 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1d70 0x5585924cc0f0 secure :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f288800b820 tx=0x7f288800bce0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.768+0000 7f28a8c75740 1 --2- 172.21.15.28:0/1349315617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.768+0000 7f28a8c75740 1 -- 172.21.15.28:0/1349315617 >> 172.21.15.28:0/1349315617 conn(0x5585924bd960 msgr2=0x5585924bdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.770+0000 7f28a8c75740 1 -- 172.21.15.28:0/1349315617 shutdown_connections 2024-05-09T08:37:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.774+0000 7f28a8c75740 1 -- 172.21.15.28:0/1349315617 wait complete. 2024-05-09T08:37:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.774+0000 7f28a8c75740 1 -- start start 2024-05-09T08:37:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.774+0000 7f28a8c75740 1 Processor -- start 2024-05-09T08:37:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.775+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.775+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.775+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924d86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.775+0000 7f28a8c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585924cf260 con 0x5585924cc630 2024-05-09T08:37:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.775+0000 7f28a8c75740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5585923e1790 con 0x5585924ca120 2024-05-09T08:37:06.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.775+0000 7f28a8c75740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5585924cf3e0 con 0x5585924c1450 2024-05-09T08:37:06.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.775+0000 7f28a8c75740 10 client.0 _ll_register_callbacks cb 0x55859241ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:06.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.779+0000 7f289f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.779+0000 7f289f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:49010/0 (socket says 172.21.15.28:49010) 2024-05-09T08:37:06.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.779+0000 7f289f7fe640 1 -- 172.21.15.28:0/460455956 learned_addr learned my addr 172.21.15.28:0/460455956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:06.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.779+0000 7f289effd640 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.781+0000 7f289effd640 1 -- 172.21.15.28:0/460455956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 msgr2=0x5585924c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.781+0000 7f289effd640 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.781+0000 7f289effd640 1 -- 172.21.15.28:0/460455956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 msgr2=0x5585924d86d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:37:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.781+0000 7f289effd640 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924d86d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.781+0000 7f289effd640 1 -- 172.21.15.28:0/460455956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55859213a210 con 0x5585924cc630 2024-05-09T08:37:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.782+0000 7f289effd640 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c9be0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f28900cc750 tx=0x7f289009b120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910a8017cc97af42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.782+0000 7f287effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:37:06.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.783+0000 7f287effd640 1 -- 172.21.15.28:0/460455956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289009f8d0 con 0x5585924cc630 2024-05-09T08:37:06.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.783+0000 7f28a8c75740 1 -- 172.21.15.28:0/460455956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2888019070 con 0x5585924cc630 2024-05-09T08:37:06.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.783+0000 7f287effd640 1 -- 172.21.15.28:0/460455956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7f2890010850 con 0x5585924cc630 2024-05-09T08:37:06.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.785+0000 7f287effd640 1 -- 172.21.15.28:0/460455956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f28900a4190 con 0x5585924cc630 2024-05-09T08:37:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.794+0000 7f28a8c75740 10 client.16920 fetch_fsmap learned FSMap version 65 2024-05-09T08:37:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.794+0000 7f28a8c75740 1 -- 172.21.15.28:0/460455956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x558592206620 con 0x5585924cc630 2024-05-09T08:37:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.797+0000 7f287effd640 1 -- 172.21.15.28:0/460455956 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2888019070 con 0x5585924cc630 2024-05-09T08:37:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.799+0000 7f287effd640 1 -- 172.21.15.28:0/460455956 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f2890097b90 con 0x5585924cc630 2024-05-09T08:37:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.799+0000 7f28a8c75740 10 client.16920 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:37:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.799+0000 7f28a8c75740 10 client.16920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:37:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.799+0000 7f28a8c75740 20 client.16920 populate_metadata read hostname 'smithi028' 2024-05-09T08:37:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.799+0000 7f28a8c75740 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5585924dc8f0 0x5585924ded10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.799+0000 7f28a8c75740 4 client.16920 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.799+0000 7f28a8c75740 1 -- 172.21.15.28:0/460455956 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558592433a70 con 0x5585924dc8f0 2024-05-09T08:37:06.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.800+0000 7f289f7fe640 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5585924dc8f0 0x5585924ded10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.809+0000 7f289f7fe640 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5585924dc8f0 0x5585924ded10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83140737237d8ce5 server_cookie=366a8ef9c1f7fd62 in_seq=0 out_seq=0 2024-05-09T08:37:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.811+0000 7f287effd640 10 client.16920 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.812+0000 7f287effd640 1 -- 172.21.15.28:0/460455956 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558592433a70 con 0x5585924dc8f0 2024-05-09T08:37:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.812+0000 7f287effd640 10 client.16920 handle_command_reply: tid=0 2024-05-09T08:37:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.812+0000 7f28a8c75740 1 client.16920 shutdown 2024-05-09T08:37:06.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.815+0000 7f28a8c75740 1 -- 172.21.15.28:0/460455956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 msgr2=0x5585924c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:06.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.815+0000 7f28a8c75740 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c9be0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f28900cc750 tx=0x7f289009b120 comp rx=0 tx=0).stop 2024-05-09T08:37:06.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.819+0000 7f28a8c75740 1 -- 172.21.15.28:0/460455956 shutdown_connections 2024-05-09T08:37:06.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.819+0000 7f28a8c75740 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5585924dc8f0 0x5585924ded10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.819+0000 7f28a8c75740 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924d86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.819+0000 7f28a8c75740 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c9be0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.819+0000 7f28a8c75740 1 --2- 172.21.15.28:0/460455956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.819+0000 7f28a8c75740 1 -- 172.21.15.28:0/460455956 >> 172.21.15.28:0/460455956 conn(0x5585924bd960 msgr2=0x5585924c36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:06.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.819+0000 7f287effd640 0 client.16920 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:06.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.823+0000 7f28a8c75740 1 -- 172.21.15.28:0/460455956 shutdown_connections 2024-05-09T08:37:06.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.824+0000 7f28a8c75740 1 -- 172.21.15.28:0/460455956 wait complete. 2024-05-09T08:37:06.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.824+0000 7f28a8c75740 20 client.16920 trim_cache size 0 max 16384 2024-05-09T08:37:06.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.856+0000 7f28a8c75740 1 Processor -- start 2024-05-09T08:37:06.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.859+0000 7f28a8c75740 1 -- start start 2024-05-09T08:37:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.860+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1450 0x5585924e4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.860+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924e4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.860+0000 7f289f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1450 0x5585924e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.860+0000 7f289f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1450 0x5585924e4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50000/0 (socket says 172.21.15.28:50000) 2024-05-09T08:37:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.860+0000 7f289f7fe640 1 -- 172.21.15.28:0/407064429 learned_addr learned my addr 172.21.15.28:0/407064429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.860+0000 7f28a8c75740 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924cc630 0x5585924d8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.860+0000 7f28a8c75740 1 -- 172.21.15.28:0/407064429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28800100f0 con 0x5585924c1450 2024-05-09T08:37:06.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.860+0000 7f28a8c75740 1 -- 172.21.15.28:0/407064429 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2880011e70 con 0x5585924ca120 2024-05-09T08:37:06.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.860+0000 7f28a8c75740 1 -- 172.21.15.28:0/407064429 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2880010880 con 0x5585924cc630 2024-05-09T08:37:06.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.861+0000 7f289effd640 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924e4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.861+0000 7f289ffff640 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924cc630 0x5585924d8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.862+0000 7f289effd640 1 -- 172.21.15.28:0/407064429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924cc630 msgr2=0x5585924d8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.862+0000 7f289effd640 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924cc630 0x5585924d8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.862+0000 7f289effd640 1 -- 172.21.15.28:0/407064429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1450 msgr2=0x5585924e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.862+0000 7f289effd640 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1450 0x5585924e4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.862+0000 7f289effd640 1 -- 172.21.15.28:0/407064429 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585924c9f40 con 0x5585924ca120 2024-05-09T08:37:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.862+0000 7f289effd640 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924e4b60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f289000be10 tx=0x7f289009b190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23b290e7b1c50155 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.863+0000 7f289ffff640 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924cc630 0x5585924d8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.863+0000 7f28957fa640 1 -- 172.21.15.28:0/407064429 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2890011830 con 0x5585924ca120 2024-05-09T08:37:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.863+0000 7f28957fa640 1 -- 172.21.15.28:0/407064429 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f289009f8d0 con 0x5585924ca120 2024-05-09T08:37:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.863+0000 7f28957fa640 1 -- 172.21.15.28:0/407064429 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2890011a30 con 0x5585924ca120 2024-05-09T08:37:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.864+0000 7f28a8c75740 1 -- 172.21.15.28:0/407064429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 msgr2=0x5585924e4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.864+0000 7f28a8c75740 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924e4b60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f289000be10 tx=0x7f289009b190 comp rx=0 tx=0).stop 2024-05-09T08:37:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.865+0000 7f28a8c75740 1 -- 172.21.15.28:0/407064429 shutdown_connections 2024-05-09T08:37:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.865+0000 7f28a8c75740 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924cc630 0x5585924d8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.865+0000 7f28a8c75740 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924e4b60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.865+0000 7f28a8c75740 1 --2- 172.21.15.28:0/407064429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924c1450 0x5585924e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.866+0000 7f28a8c75740 1 -- 172.21.15.28:0/407064429 >> 172.21.15.28:0/407064429 conn(0x5585924bd960 msgr2=0x5585924bca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.867+0000 7f28a8c75740 1 -- 172.21.15.28:0/407064429 shutdown_connections 2024-05-09T08:37:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.867+0000 7f28a8c75740 1 -- 172.21.15.28:0/407064429 wait complete. 2024-05-09T08:37:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.868+0000 7f28a8c75740 1 -- start start 2024-05-09T08:37:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.868+0000 7f28a8c75740 1 Processor -- start 2024-05-09T08:37:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.868+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924d4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.868+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.869+0000 7f28a8c75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.869+0000 7f28a8c75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28800117a0 con 0x5585924cc630 2024-05-09T08:37:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.869+0000 7f28a8c75740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2880011010 con 0x5585924ca120 2024-05-09T08:37:06.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.869+0000 7f28a8c75740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f288000e180 con 0x5585924c1450 2024-05-09T08:37:06.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.869+0000 7f28a8c75740 10 client.0 _ll_register_callbacks cb 0x7f289009b2f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:06.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.870+0000 7f289ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.870+0000 7f289ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50014/0 (socket says 172.21.15.28:50014) 2024-05-09T08:37:06.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.870+0000 7f289ffff640 1 -- 172.21.15.28:0/3210596065 learned_addr learned my addr 172.21.15.28:0/3210596065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:06.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.873+0000 7f289effd640 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.873+0000 7f289f7fe640 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924d4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.873+0000 7f289effd640 1 -- 172.21.15.28:0/3210596065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 msgr2=0x5585924d4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.873+0000 7f289effd640 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924d4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.873+0000 7f289effd640 1 -- 172.21.15.28:0/3210596065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 msgr2=0x5585924c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.873+0000 7f289effd640 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.873+0000 7f289effd640 1 -- 172.21.15.28:0/3210596065 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558592541e90 con 0x5585924ca120 2024-05-09T08:37:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.873+0000 7f289effd640 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924c61b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f289000be10 tx=0x7f2890004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90c8f89f31539b04 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:06.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.874+0000 7f287effd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:37:06.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.874+0000 7f287effd640 1 -- 172.21.15.28:0/3210596065 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28900a9030 con 0x5585924ca120 2024-05-09T08:37:06.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.874+0000 7f28a8c75740 1 -- 172.21.15.28:0/3210596065 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f289009f8d0 con 0x5585924ca120 2024-05-09T08:37:06.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.874+0000 7f287effd640 1 -- 172.21.15.28:0/3210596065 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7f2890010380 con 0x5585924ca120 2024-05-09T08:37:06.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.875+0000 7f287effd640 1 -- 172.21.15.28:0/3210596065 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f2890011830 con 0x5585924ca120 2024-05-09T08:37:06.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.875+0000 7f28a8c75740 10 client.26426 fetch_fsmap learned FSMap version 65 2024-05-09T08:37:06.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.875+0000 7f28a8c75740 1 -- 172.21.15.28:0/3210596065 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x558592543000 con 0x5585924ca120 2024-05-09T08:37:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.878+0000 7f287effd640 1 -- 172.21.15.28:0/3210596065 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f289009f8d0 con 0x5585924ca120 2024-05-09T08:37:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.878+0000 7f287effd640 1 -- 172.21.15.28:0/3210596065 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f289009a3e0 con 0x5585924ca120 2024-05-09T08:37:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.880+0000 7f28a8c75740 10 client.26426 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:37:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.880+0000 7f28a8c75740 10 client.26426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:37:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.880+0000 7f28a8c75740 20 client.26426 populate_metadata read hostname 'smithi028' 2024-05-09T08:37:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.880+0000 7f28a8c75740 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558592543570 0x558592545990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.880+0000 7f28a8c75740 4 client.26426 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:37:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.880+0000 7f28a8c75740 1 -- 172.21.15.28:0/3210596065 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55859242ddc0 con 0x558592543570 2024-05-09T08:37:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.883+0000 7f289f7fe640 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558592543570 0x558592545990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.886+0000 7f289f7fe640 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558592543570 0x558592545990 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=135e8aa27a458ffe server_cookie=b6eed7ce197b9461 in_seq=0 out_seq=0 2024-05-09T08:37:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:06.886+0000 7f287effd640 10 client.26426 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:07.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.018+0000 7f287effd640 1 -- 172.21.15.28:0/3210596065 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55859242ddc0 con 0x558592543570 2024-05-09T08:37:07.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.018+0000 7f287effd640 10 client.26426 handle_command_reply: tid=0 2024-05-09T08:37:07.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.018+0000 7f28a8c75740 1 client.26426 shutdown 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 -- 172.21.15.28:0/3210596065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 msgr2=0x5585924c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924c61b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f289000be10 tx=0x7f2890004750 comp rx=0 tx=0).stop 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 -- 172.21.15.28:0/3210596065 shutdown_connections 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558592543570 0x558592545990 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585924cc630 0x5585924c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5585924ca120 0x5585924c61b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 --2- 172.21.15.28:0/3210596065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5585924c1450 0x5585924d4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 -- 172.21.15.28:0/3210596065 >> 172.21.15.28:0/3210596065 conn(0x5585924bd960 msgr2=0x5585924c8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f287effd640 0 client.26426 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 -- 172.21.15.28:0/3210596065 shutdown_connections 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 1 -- 172.21.15.28:0/3210596065 wait complete. 2024-05-09T08:37:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.019+0000 7f28a8c75740 20 client.26426 trim_cache size 0 max 16384 2024-05-09T08:37:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:37:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:37:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:37:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:37:07.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.020+0000 7f28a5f63640 1 -- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f288c0a2fd0 msgr2=0x7f288c0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:07.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.020+0000 7f28a5f63640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f288c0a2fd0 0x7f288c0a5490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f289803bfa0 tx=0x7f289803b9b0 comp rx=0 tx=0).stop 2024-05-09T08:37:07.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.020+0000 7f28a5f63640 1 -- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a006bb80 msgr2=0x7f28a0116960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:07.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.020+0000 7f28a5f63640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a006bb80 0x7f28a0116960 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f289000b450 tx=0x7f289000b910 comp rx=0 tx=0).stop 2024-05-09T08:37:07.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.021+0000 7f28a5f63640 1 -- 172.21.15.28:0/3982315400 shutdown_connections 2024-05-09T08:37:07.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.021+0000 7f28a5f63640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f288c0a2fd0 0x7f288c0a5490 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:07.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.021+0000 7f28a5f63640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28a011b0d0 0x7f28a0116ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:07.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.021+0000 7f28a5f63640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28a006bb80 0x7f28a0116960 secure :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f289000b450 tx=0x7f289000b910 comp rx=0 tx=0).stop 2024-05-09T08:37:07.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.021+0000 7f28a5f63640 1 --2- 172.21.15.28:0/3982315400 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28a006b260 0x7f28a011a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.021+0000 7f28a5f63640 1 -- 172.21.15.28:0/3982315400 >> 172.21.15.28:0/3982315400 conn(0x7f28a006e970 msgr2=0x7f28a010a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.021+0000 7f28a5f63640 1 -- 172.21.15.28:0/3982315400 shutdown_connections 2024-05-09T08:37:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:07.021+0000 7f28a5f63640 1 -- 172.21.15.28:0/3982315400 wait complete. 2024-05-09T08:37:07.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:07 smithi165 ceph-mon[34540]: pgmap v344: 129 pgs: 129 active+clean; 2.7 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 641 op/s 2024-05-09T08:37:07.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:07 smithi165 ceph-mon[34540]: osdmap e112: 12 total, 12 up, 12 in 2024-05-09T08:37:07.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:07 smithi184 ceph-mon[34509]: pgmap v344: 129 pgs: 129 active+clean; 2.7 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 641 op/s 2024-05-09T08:37:07.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:07 smithi184 ceph-mon[34509]: osdmap e112: 12 total, 12 up, 12 in 2024-05-09T08:37:07.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:07 smithi028 ceph-mon[30119]: pgmap v344: 129 pgs: 129 active+clean; 2.7 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 641 op/s 2024-05-09T08:37:07.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:07 smithi028 ceph-mon[30119]: osdmap e112: 12 total, 12 up, 12 in 2024-05-09T08:37:09.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:09 smithi184 ceph-mon[34509]: pgmap v346: 129 pgs: 15 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 110 active+clean; 2.7 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 456 op/s 2024-05-09T08:37:09.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:09 smithi028 ceph-mon[30119]: pgmap v346: 129 pgs: 15 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 110 active+clean; 2.7 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 456 op/s 2024-05-09T08:37:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:09 smithi165 ceph-mon[34540]: pgmap v346: 129 pgs: 15 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 110 active+clean; 2.7 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 456 op/s 2024-05-09T08:37:11.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:11 smithi028 ceph-mon[30119]: pgmap v347: 129 pgs: 52 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 59 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 468 op/s 2024-05-09T08:37:12.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:11 smithi165 ceph-mon[34540]: pgmap v347: 129 pgs: 52 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 59 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 468 op/s 2024-05-09T08:37:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:11 smithi184 ceph-mon[34509]: pgmap v347: 129 pgs: 52 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 59 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 468 op/s 2024-05-09T08:37:13.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:13 smithi028 ceph-mon[30119]: pgmap v348: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 376 op/s 2024-05-09T08:37:14.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:13 smithi165 ceph-mon[34540]: pgmap v348: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 376 op/s 2024-05-09T08:37:14.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:13 smithi184 ceph-mon[34509]: pgmap v348: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 376 op/s 2024-05-09T08:37:15.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:15 smithi028 ceph-mon[30119]: pgmap v349: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 279 op/s 2024-05-09T08:37:16.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:15 smithi165 ceph-mon[34540]: pgmap v349: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 279 op/s 2024-05-09T08:37:16.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:15 smithi184 ceph-mon[34509]: pgmap v349: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 279 op/s 2024-05-09T08:37:17.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:17 smithi165 ceph-mon[34540]: pgmap v350: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 951 KiB/s wr, 265 op/s 2024-05-09T08:37:17.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:17 smithi028 ceph-mon[30119]: pgmap v350: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 951 KiB/s wr, 265 op/s 2024-05-09T08:37:18.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:17 smithi184 ceph-mon[34509]: pgmap v350: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 951 KiB/s wr, 265 op/s 2024-05-09T08:37:18.931 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:37:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:37:18.495+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:37:19.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:19 smithi028 ceph-mon[30119]: pgmap v351: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 958 KiB/s wr, 244 op/s 2024-05-09T08:37:20.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:19 smithi165 ceph-mon[34540]: pgmap v351: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 958 KiB/s wr, 244 op/s 2024-05-09T08:37:20.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:19 smithi184 ceph-mon[34509]: pgmap v351: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 958 KiB/s wr, 244 op/s 2024-05-09T08:37:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.460+0000 7ff50a90e640 1 -- 172.21.15.28:0/930436240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4fc0065e0 msgr2=0x7ff4fc006a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.460+0000 7ff50a90e640 1 --2- 172.21.15.28:0/930436240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4fc0065e0 0x7ff4fc006a40 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff504066fa0 tx=0x7ff504096290 comp rx=0 tx=0).stop 2024-05-09T08:37:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.467+0000 7ff50a90e640 1 -- 172.21.15.28:0/930436240 shutdown_connections 2024-05-09T08:37:20.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.467+0000 7ff50a90e640 1 --2- 172.21.15.28:0/930436240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4fc007060 0x7ff4fc098740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.467+0000 7ff50a90e640 1 --2- 172.21.15.28:0/930436240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4fc0065e0 0x7ff4fc006a40 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.467+0000 7ff50a90e640 1 --2- 172.21.15.28:0/930436240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4fc09ac40 0x7ff4fc09b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.467+0000 7ff50a90e640 1 -- 172.21.15.28:0/930436240 >> 172.21.15.28:0/930436240 conn(0x7ff4fc00a6f0 msgr2=0x7ff4fc00ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:20.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.473+0000 7ff50a90e640 1 -- 172.21.15.28:0/930436240 shutdown_connections 2024-05-09T08:37:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.473+0000 7ff50a90e640 1 -- 172.21.15.28:0/930436240 wait complete. 2024-05-09T08:37:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.474+0000 7ff50a90e640 1 Processor -- start 2024-05-09T08:37:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.474+0000 7ff50a90e640 1 -- start start 2024-05-09T08:37:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.474+0000 7ff50a90e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4fc0065e0 0x7ff4fc149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.474+0000 7ff50a90e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4fc007060 0x7ff4fc149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.474+0000 7ff50a90e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4fc09ac40 0x7ff4fc143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50990c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4fc0065e0 0x7ff4fc149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50990c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4fc0065e0 0x7ff4fc149880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:54530/0 (socket says 172.21.15.28:54530) 2024-05-09T08:37:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50a90e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc0bd630 con 0x7ff4fc007060 2024-05-09T08:37:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50a90e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc0bd4b0 con 0x7ff4fc09ac40 2024-05-09T08:37:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50a90e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc0bd7b0 con 0x7ff4fc0065e0 2024-05-09T08:37:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50990c640 1 -- 172.21.15.28:0/2997725304 learned_addr learned my addr 172.21.15.28:0/2997725304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:20.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50910b640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4fc007060 0x7ff4fc149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50a10d640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4fc09ac40 0x7ff4fc143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50a10d640 1 -- 172.21.15.28:0/2997725304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4fc0065e0 msgr2=0x7ff4fc149880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:20.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50a10d640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4fc0065e0 0x7ff4fc149880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50a10d640 1 -- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4fc007060 msgr2=0x7ff4fc149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:20.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50a10d640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4fc007060 0x7ff4fc149dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.475+0000 7ff50a10d640 1 -- 172.21.15.28:0/2997725304 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4fc1441b0 con 0x7ff4fc09ac40 2024-05-09T08:37:20.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.476+0000 7ff50a10d640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4fc09ac40 0x7ff4fc143a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400e970 tx=0x7ff4f400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:20.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.478+0000 7ff4faffd640 1 -- 172.21.15.28:0/2997725304 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f400ccc0 con 0x7ff4fc09ac40 2024-05-09T08:37:20.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.478+0000 7ff4faffd640 1 -- 172.21.15.28:0/2997725304 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4f4005020 con 0x7ff4fc09ac40 2024-05-09T08:37:20.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.478+0000 7ff4faffd640 1 -- 172.21.15.28:0/2997725304 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4004a00 con 0x7ff4fc09ac40 2024-05-09T08:37:20.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.479+0000 7ff50a90e640 1 -- 172.21.15.28:0/2997725304 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4fc1444a0 con 0x7ff4fc09ac40 2024-05-09T08:37:20.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.479+0000 7ff50a90e640 1 -- 172.21.15.28:0/2997725304 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4fc150740 con 0x7ff4fc09ac40 2024-05-09T08:37:20.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.483+0000 7ff50e880740 1 Processor -- start 2024-05-09T08:37:20.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.483+0000 7ff50e880740 1 -- start start 2024-05-09T08:37:20.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.484+0000 7ff50e880740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e942cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.484+0000 7ff50e880740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e94d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.484+0000 7ff50e880740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.484+0000 7ff50e880740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ac7e8a7340 con 0x55ac7e9428e0 2024-05-09T08:37:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.484+0000 7ff50e880740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ac7e84f290 con 0x55ac7e94dac0 2024-05-09T08:37:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.484+0000 7ff50e880740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ac7e8a3300 con 0x55ac7e943200 2024-05-09T08:37:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.487+0000 7ff50990c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e94d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.487+0000 7ff50990c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e94d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:54536/0 (socket says 172.21.15.28:54536) 2024-05-09T08:37:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.487+0000 7ff50990c640 1 -- 172.21.15.28:0/2149903787 learned_addr learned my addr 172.21.15.28:0/2149903787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.487+0000 7ff50910b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.488+0000 7ff4faffd640 1 -- 172.21.15.28:0/2997725304 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff4f40051d0 con 0x7ff4fc09ac40 2024-05-09T08:37:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.488+0000 7ff50990c640 1 -- 172.21.15.28:0/2149903787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 msgr2=0x55ac7e94feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.488+0000 7ff50990c640 1 --2- 172.21.15.28:0/2149903787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.488+0000 7ff50990c640 1 -- 172.21.15.28:0/2149903787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 msgr2=0x55ac7e942cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.488+0000 7ff50990c640 1 --2- 172.21.15.28:0/2149903787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e942cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.488+0000 7ff50990c640 1 -- 172.21.15.28:0/2149903787 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac7e8738e0 con 0x55ac7e943200 2024-05-09T08:37:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.488+0000 7ff50990c640 1 --2- 172.21.15.28:0/2149903787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e94d580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff50000bd40 tx=0x7ff5000107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e9055a6118e0b2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.489+0000 7ff4faffd640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff4f00a2f00 0x7ff4f00a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.489+0000 7ff4ef7fe640 1 -- 172.21.15.28:0/2149903787 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500011440 con 0x55ac7e943200 2024-05-09T08:37:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.489+0000 7ff4ef7fe640 1 -- 172.21.15.28:0/2149903787 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff500011ac0 con 0x55ac7e943200 2024-05-09T08:37:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.490+0000 7ff50e880740 1 -- 172.21.15.28:0/2149903787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 msgr2=0x55ac7e94d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.490+0000 7ff50e880740 1 --2- 172.21.15.28:0/2149903787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e94d580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff50000bd40 tx=0x7ff5000107a0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.491+0000 7ff4faffd640 1 -- 172.21.15.28:0/2997725304 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7ff4f40cf8c0 con 0x7ff4fc09ac40 2024-05-09T08:37:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.491+0000 7ff50e880740 1 -- 172.21.15.28:0/2149903787 shutdown_connections 2024-05-09T08:37:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.491+0000 7ff50e880740 1 --2- 172.21.15.28:0/2149903787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.491+0000 7ff50e880740 1 --2- 172.21.15.28:0/2149903787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e94d580 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.491+0000 7ff50e880740 1 --2- 172.21.15.28:0/2149903787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e942cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.491+0000 7ff50e880740 1 -- 172.21.15.28:0/2149903787 >> 172.21.15.28:0/2149903787 conn(0x55ac7e93edf0 msgr2=0x55ac7e93f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.491+0000 7ff50e880740 1 -- 172.21.15.28:0/2149903787 shutdown_connections 2024-05-09T08:37:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.491+0000 7ff50910b640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff4f00a2f00 0x7ff4f00a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.492+0000 7ff50e880740 1 -- 172.21.15.28:0/2149903787 wait complete. 2024-05-09T08:37:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.492+0000 7ff50e880740 1 -- start start 2024-05-09T08:37:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.492+0000 7ff50e880740 1 Processor -- start 2024-05-09T08:37:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50e880740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e9428e0 0x55ac7e94d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50e880740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e943200 0x55ac7e94a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50e880740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50e880740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ac7e950600 con 0x55ac7e943200 2024-05-09T08:37:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50e880740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ac7e862790 con 0x55ac7e94dac0 2024-05-09T08:37:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50e880740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ac7e950780 con 0x55ac7e9428e0 2024-05-09T08:37:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50e880740 10 client.0 _ll_register_callbacks cb 0x55ac7e89da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50a10d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50a10d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33268/0 (socket says 172.21.15.28:33268) 2024-05-09T08:37:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.493+0000 7ff50a10d640 1 -- 172.21.15.28:0/2476577881 learned_addr learned my addr 172.21.15.28:0/2476577881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.495+0000 7ff50a10d640 1 -- 172.21.15.28:0/2476577881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e9428e0 msgr2=0x55ac7e94d3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:37:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.495+0000 7ff50a10d640 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e9428e0 0x55ac7e94d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.495+0000 7ff50a10d640 1 -- 172.21.15.28:0/2476577881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e943200 msgr2=0x55ac7e94a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.495+0000 7ff50a10d640 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e943200 0x55ac7e94a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.495+0000 7ff50a10d640 1 -- 172.21.15.28:0/2476577881 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac7e5bb210 con 0x55ac7e94dac0 2024-05-09T08:37:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.495+0000 7ff50a10d640 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ac70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40a4040 tx=0x7ff4f409c0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce940e2dfa1107af server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.496+0000 7ff4ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:37:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.496+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/2476577881 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f401c040 con 0x55ac7e94dac0 2024-05-09T08:37:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.497+0000 7ff50e880740 1 -- 172.21.15.28:0/2476577881 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff500011ac0 con 0x55ac7e94dac0 2024-05-09T08:37:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.497+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/2476577881 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7ff4f40a42e0 con 0x55ac7e94dac0 2024-05-09T08:37:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.499+0000 7ff50910b640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff4f00a2f00 0x7ff4f00a53c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff50404fd30 tx=0x7ff5040677a0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.501+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/2476577881 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7ff4f40ccfc0 con 0x55ac7e94dac0 2024-05-09T08:37:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.503+0000 7ff50e880740 10 client.26438 fetch_fsmap learned FSMap version 65 2024-05-09T08:37:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.503+0000 7ff50e880740 1 -- 172.21.15.28:0/2476577881 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55ac7e687620 con 0x55ac7e94dac0 2024-05-09T08:37:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.505+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/2476577881 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff500011ac0 con 0x55ac7e94dac0 2024-05-09T08:37:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.505+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/2476577881 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7ff4f40cf070 con 0x55ac7e94dac0 2024-05-09T08:37:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.505+0000 7ff50e880740 10 client.26438 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:37:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.505+0000 7ff50e880740 10 client.26438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:37:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.505+0000 7ff50e880740 20 client.26438 populate_metadata read hostname 'smithi028' 2024-05-09T08:37:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.505+0000 7ff50e880740 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ac7e96e530 0x55ac7e970980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.505+0000 7ff50e880740 4 client.26438 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.505+0000 7ff50e880740 1 -- 172.21.15.28:0/2476577881 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac7e8b4a70 con 0x55ac7e96e530 2024-05-09T08:37:20.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.508+0000 7ff50990c640 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ac7e96e530 0x55ac7e970980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.518+0000 7ff50990c640 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ac7e96e530 0x55ac7e970980 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=485a7815b59b0787 server_cookie=32288e6f409cf17b in_seq=0 out_seq=0 2024-05-09T08:37:20.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.518+0000 7ff4ed7fa640 10 client.26438 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:20.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.520+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/2476577881 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ac7e8b4a70 con 0x55ac7e96e530 2024-05-09T08:37:20.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.520+0000 7ff4ed7fa640 10 client.26438 handle_command_reply: tid=0 2024-05-09T08:37:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.528+0000 7ff50e880740 1 client.26438 shutdown 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.532+0000 7ff50e880740 1 -- 172.21.15.28:0/2476577881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 msgr2=0x55ac7e94ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.532+0000 7ff50e880740 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ac70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40a4040 tx=0x7ff4f409c0c0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.532+0000 7ff50e880740 1 -- 172.21.15.28:0/2476577881 shutdown_connections 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.532+0000 7ff50e880740 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ac7e96e530 0x55ac7e970980 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.532+0000 7ff50e880740 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ac70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.532+0000 7ff50e880740 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e943200 0x55ac7e94a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.532+0000 7ff50e880740 1 --2- 172.21.15.28:0/2476577881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e9428e0 0x55ac7e94d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.532+0000 7ff50e880740 1 -- 172.21.15.28:0/2476577881 >> 172.21.15.28:0/2476577881 conn(0x55ac7e93edf0 msgr2=0x55ac7e944b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.533+0000 7ff4ed7fa640 0 client.26438 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.533+0000 7ff50e880740 1 -- 172.21.15.28:0/2476577881 shutdown_connections 2024-05-09T08:37:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.533+0000 7ff50e880740 1 -- 172.21.15.28:0/2476577881 wait complete. 2024-05-09T08:37:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.534+0000 7ff50e880740 20 client.26438 trim_cache size 0 max 16384 2024-05-09T08:37:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.564+0000 7ff50e880740 1 Processor -- start 2024-05-09T08:37:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.565+0000 7ff50e880740 1 -- start start 2024-05-09T08:37:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.565+0000 7ff50e880740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e959540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.565+0000 7ff50e880740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e943200 0x55ac7e9449a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.565+0000 7ff50e880740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.565+0000 7ff50e880740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc1523b0 con 0x55ac7e9428e0 2024-05-09T08:37:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.565+0000 7ff50e880740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc154130 con 0x55ac7e943200 2024-05-09T08:37:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.565+0000 7ff50e880740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc152b40 con 0x55ac7e94dac0 2024-05-09T08:37:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.569+0000 7ff50a10d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.569+0000 7ff50a10d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:54544/0 (socket says 172.21.15.28:54544) 2024-05-09T08:37:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.569+0000 7ff50a10d640 1 -- 172.21.15.28:0/960530121 learned_addr learned my addr 172.21.15.28:0/960530121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.569+0000 7ff50910b640 1 --2- 172.21.15.28:0/960530121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e943200 0x55ac7e9449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.569+0000 7ff50990c640 1 --2- 172.21.15.28:0/960530121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e959540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.572+0000 7ff50990c640 1 -- 172.21.15.28:0/960530121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e94dac0 msgr2=0x55ac7e94ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.572+0000 7ff50990c640 1 --2- 172.21.15.28:0/960530121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.572+0000 7ff50990c640 1 -- 172.21.15.28:0/960530121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e943200 msgr2=0x55ac7e9449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.572+0000 7ff50990c640 1 --2- 172.21.15.28:0/960530121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e943200 0x55ac7e9449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.572+0000 7ff50990c640 1 -- 172.21.15.28:0/960530121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac7e94afd0 con 0x55ac7e9428e0 2024-05-09T08:37:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.572+0000 7ff50990c640 1 --2- 172.21.15.28:0/960530121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e959540 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff50001b960 tx=0x7ff50001b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3567f89ba447f52 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.574+0000 7ff4eeffd640 1 -- 172.21.15.28:0/960530121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500005040 con 0x55ac7e9428e0 2024-05-09T08:37:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.575+0000 7ff4eeffd640 1 -- 172.21.15.28:0/960530121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5000035f0 con 0x55ac7e9428e0 2024-05-09T08:37:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.575+0000 7ff4eeffd640 1 -- 172.21.15.28:0/960530121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5000116a0 con 0x55ac7e9428e0 2024-05-09T08:37:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.576+0000 7ff50e880740 1 -- 172.21.15.28:0/960530121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 msgr2=0x55ac7e959540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.576+0000 7ff50e880740 1 --2- 172.21.15.28:0/960530121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e959540 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff50001b960 tx=0x7ff50001b860 comp rx=0 tx=0).stop 2024-05-09T08:37:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.578+0000 7ff50e880740 1 -- 172.21.15.28:0/960530121 shutdown_connections 2024-05-09T08:37:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.578+0000 7ff50e880740 1 --2- 172.21.15.28:0/960530121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e94dac0 0x55ac7e94ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.578+0000 7ff50e880740 1 --2- 172.21.15.28:0/960530121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e943200 0x55ac7e9449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.578+0000 7ff50e880740 1 --2- 172.21.15.28:0/960530121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e959540 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.578+0000 7ff50e880740 1 -- 172.21.15.28:0/960530121 >> 172.21.15.28:0/960530121 conn(0x55ac7e93edf0 msgr2=0x55ac7e944290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.583+0000 7ff50e880740 1 -- 172.21.15.28:0/960530121 shutdown_connections 2024-05-09T08:37:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.584+0000 7ff50e880740 1 -- 172.21.15.28:0/960530121 wait complete. 2024-05-09T08:37:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.584+0000 7ff50e880740 1 -- start start 2024-05-09T08:37:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.584+0000 7ff50e880740 1 Processor -- start 2024-05-09T08:37:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50e880740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e955c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50e880740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e946df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50e880740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e947330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50e880740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc153a60 con 0x55ac7e9428e0 2024-05-09T08:37:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50e880740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc1532d0 con 0x55ac7e94dac0 2024-05-09T08:37:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50e880740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc005a60 con 0x55ac7e943200 2024-05-09T08:37:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50e880740 10 client.0 _ll_register_callbacks cb 0x7ff4f4097c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50a10d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e947330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50910b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e946df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50a10d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e947330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33294/0 (socket says 172.21.15.28:33294) 2024-05-09T08:37:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.585+0000 7ff50a10d640 1 -- 172.21.15.28:0/96698206 learned_addr learned my addr 172.21.15.28:0/96698206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.586+0000 7ff50910b640 1 -- 172.21.15.28:0/96698206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 msgr2=0x55ac7e947330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.586+0000 7ff50910b640 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e947330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.586+0000 7ff50910b640 1 -- 172.21.15.28:0/96698206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 msgr2=0x55ac7e955c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.586+0000 7ff50910b640 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e955c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.586+0000 7ff50910b640 1 -- 172.21.15.28:0/96698206 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac7e947870 con 0x55ac7e943200 2024-05-09T08:37:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.588+0000 7ff50910b640 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e946df0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff504000b60 tx=0x7ff504067140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=549ea0a92212493f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.588+0000 7ff4ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:37:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.590+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/96698206 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50409cbe0 con 0x55ac7e943200 2024-05-09T08:37:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.590+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/96698206 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7ff50409cd80 con 0x55ac7e943200 2024-05-09T08:37:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.590+0000 7ff50e880740 1 -- 172.21.15.28:0/96698206 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5000035f0 con 0x55ac7e943200 2024-05-09T08:37:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.591+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/96698206 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7ff5040a8420 con 0x55ac7e943200 2024-05-09T08:37:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.591+0000 7ff50e880740 10 client.26296 fetch_fsmap learned FSMap version 65 2024-05-09T08:37:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.591+0000 7ff50e880740 1 -- 172.21.15.28:0/96698206 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55ac7e9d4790 con 0x55ac7e943200 2024-05-09T08:37:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.595+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/96698206 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5000035f0 con 0x55ac7e943200 2024-05-09T08:37:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.595+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/96698206 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7ff50406d5f0 con 0x55ac7e943200 2024-05-09T08:37:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.595+0000 7ff50e880740 10 client.26296 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:37:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.595+0000 7ff50e880740 10 client.26296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:37:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.595+0000 7ff50e880740 20 client.26296 populate_metadata read hostname 'smithi028' 2024-05-09T08:37:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.595+0000 7ff50e880740 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ac7e9d4f50 0x55ac7e9d7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.595+0000 7ff50e880740 4 client.26296 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:37:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.595+0000 7ff50e880740 1 -- 172.21.15.28:0/96698206 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ac7e8aedc0 con 0x55ac7e9d4f50 2024-05-09T08:37:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.608+0000 7ff50990c640 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ac7e9d4f50 0x55ac7e9d7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.625+0000 7ff50990c640 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ac7e9d4f50 0x55ac7e9d7370 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a49a9ba4b25dc07 server_cookie=2c6637e724280e9d in_seq=0 out_seq=0 2024-05-09T08:37:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.627+0000 7ff4ed7fa640 10 client.26296 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:20.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff4ed7fa640 1 -- 172.21.15.28:0/96698206 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ac7e8aedc0 con 0x55ac7e9d4f50 2024-05-09T08:37:20.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff4ed7fa640 10 client.26296 handle_command_reply: tid=0 2024-05-09T08:37:20.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff50e880740 1 client.26296 shutdown 2024-05-09T08:37:20.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff50e880740 1 -- 172.21.15.28:0/96698206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 msgr2=0x55ac7e946df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:20.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff50e880740 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e946df0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff504000b60 tx=0x7ff504067140 comp rx=0 tx=0).stop 2024-05-09T08:37:20.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff50e880740 1 -- 172.21.15.28:0/96698206 shutdown_connections 2024-05-09T08:37:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff50e880740 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ac7e9d4f50 0x55ac7e9d7370 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff50e880740 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ac7e94dac0 0x55ac7e947330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff50e880740 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ac7e943200 0x55ac7e946df0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff50e880740 1 --2- 172.21.15.28:0/96698206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac7e9428e0 0x55ac7e955c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.881+0000 7ff50e880740 1 -- 172.21.15.28:0/96698206 >> 172.21.15.28:0/96698206 conn(0x55ac7e93edf0 msgr2=0x55ac7e94a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.882+0000 7ff4ed7fa640 0 client.26296 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.882+0000 7ff50e880740 1 -- 172.21.15.28:0/96698206 shutdown_connections 2024-05-09T08:37:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.882+0000 7ff50e880740 1 -- 172.21.15.28:0/96698206 wait complete. 2024-05-09T08:37:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.882+0000 7ff50e880740 20 client.26296 trim_cache size 0 max 16384 2024-05-09T08:37:20.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:37:20.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:37:20.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:37:20.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:37:20.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 -- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff4f00a2f00 msgr2=0x7ff4f00a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff4f00a2f00 0x7ff4f00a53c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff50404fd30 tx=0x7ff5040677a0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 -- 172.21.15.28:0/2997725304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4fc09ac40 msgr2=0x7ff4fc143a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4fc09ac40 0x7ff4fc143a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400e970 tx=0x7ff4f400ee30 comp rx=0 tx=0).stop 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 -- 172.21.15.28:0/2997725304 shutdown_connections 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4fc09ac40 0x7ff4fc143a30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4fc007060 0x7ff4fc149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff4f00a2f00 0x7ff4f00a53c0 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff50404fd30 tx=0x7ff5040677a0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 --2- 172.21.15.28:0/2997725304 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff4fc0065e0 0x7ff4fc149880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 -- 172.21.15.28:0/2997725304 >> 172.21.15.28:0/2997725304 conn(0x7ff4fc00a6f0 msgr2=0x7ff4fc092590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 -- 172.21.15.28:0/2997725304 shutdown_connections 2024-05-09T08:37:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:20.883+0000 7ff4f8bff640 1 -- 172.21.15.28:0/2997725304 wait complete. 2024-05-09T08:37:21.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:21 smithi028 ceph-mon[30119]: pgmap v352: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 259 op/s 2024-05-09T08:37:22.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:21 smithi165 ceph-mon[34540]: pgmap v352: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 259 op/s 2024-05-09T08:37:22.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:21 smithi184 ceph-mon[34509]: pgmap v352: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 259 op/s 2024-05-09T08:37:23.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:23 smithi028 ceph-mon[30119]: pgmap v353: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 214 op/s 2024-05-09T08:37:24.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:23 smithi165 ceph-mon[34540]: pgmap v353: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 214 op/s 2024-05-09T08:37:24.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:23 smithi184 ceph-mon[34509]: pgmap v353: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 214 op/s 2024-05-09T08:37:25.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:25 smithi028 ceph-mon[30119]: pgmap v354: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 215 op/s 2024-05-09T08:37:26.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:25 smithi165 ceph-mon[34540]: pgmap v354: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 215 op/s 2024-05-09T08:37:26.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:25 smithi184 ceph-mon[34509]: pgmap v354: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 215 op/s 2024-05-09T08:37:27.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:27 smithi028 ceph-mon[30119]: pgmap v355: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 248 op/s 2024-05-09T08:37:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:27 smithi165 ceph-mon[34540]: pgmap v355: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 248 op/s 2024-05-09T08:37:28.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:27 smithi184 ceph-mon[34509]: pgmap v355: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 248 op/s 2024-05-09T08:37:29.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:29 smithi028 ceph-mon[30119]: pgmap v356: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 219 op/s 2024-05-09T08:37:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:29 smithi165 ceph-mon[34540]: pgmap v356: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 219 op/s 2024-05-09T08:37:30.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:29 smithi184 ceph-mon[34509]: pgmap v356: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 219 op/s 2024-05-09T08:37:31.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:31 smithi028 ceph-mon[30119]: pgmap v357: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 256 op/s 2024-05-09T08:37:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:31 smithi165 ceph-mon[34540]: pgmap v357: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 256 op/s 2024-05-09T08:37:32.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:31 smithi184 ceph-mon[34509]: pgmap v357: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 256 op/s 2024-05-09T08:37:33.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:33 smithi028 ceph-mon[30119]: pgmap v358: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 992 KiB/s wr, 230 op/s 2024-05-09T08:37:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.041+0000 7f11f7bdc640 1 -- 172.21.15.28:0/4056192922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11f0112820 msgr2=0x7f11f0114c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:34.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.041+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/4056192922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11f0112820 0x7f11f0114c10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f11e801acb0 tx=0x7f11e803b9b0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.048+0000 7f11f7bdc640 1 -- 172.21.15.28:0/4056192922 shutdown_connections 2024-05-09T08:37:34.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.048+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/4056192922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11f0112820 0x7f11f0114c10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.048+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/4056192922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11f00744f0 0x7f11f01122e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.048+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/4056192922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f0073bd0 0x7f11f0073fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.048+0000 7f11f7bdc640 1 -- 172.21.15.28:0/4056192922 >> 172.21.15.28:0/4056192922 conn(0x7f11f00fd2e0 msgr2=0x7f11f00ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.050+0000 7f11f7bdc640 1 -- 172.21.15.28:0/4056192922 shutdown_connections 2024-05-09T08:37:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.054+0000 7f11f7bdc640 1 -- 172.21.15.28:0/4056192922 wait complete. 2024-05-09T08:37:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.055+0000 7f11f7bdc640 1 Processor -- start 2024-05-09T08:37:34.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.057+0000 7f11f7bdc640 1 -- start start 2024-05-09T08:37:34.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.057+0000 7f11f7bdc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11f0073bd0 0x7f11f0073600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.057+0000 7f11f7bdc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11f00744f0 0x7f11f00706a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.057+0000 7f11f7bdc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f0112820 0x7f11f0070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.058+0000 7f11f7bdc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11f0108480 con 0x7f11f0112820 2024-05-09T08:37:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.058+0000 7f11f6179640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f0112820 0x7f11f0070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.058+0000 7f11f6179640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f0112820 0x7f11f0070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35900/0 (socket says 172.21.15.28:35900) 2024-05-09T08:37:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.058+0000 7f11f6179640 1 -- 172.21.15.28:0/1999410632 learned_addr learned my addr 172.21.15.28:0/1999410632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.058+0000 7f11f5978640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11f0073bd0 0x7f11f0073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.058+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f11f0108300 con 0x7f11f0073bd0 2024-05-09T08:37:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.058+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f11f0108600 con 0x7f11f00744f0 2024-05-09T08:37:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.059+0000 7f11f6179640 1 -- 172.21.15.28:0/1999410632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11f00744f0 msgr2=0x7f11f00706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.059+0000 7f11f6179640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11f00744f0 0x7f11f00706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.059+0000 7f11f6179640 1 -- 172.21.15.28:0/1999410632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11f0073bd0 msgr2=0x7f11f0073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.059+0000 7f11f6179640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11f0073bd0 0x7f11f0073600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.059+0000 7f11f6179640 1 -- 172.21.15.28:0/1999410632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f0111ca0 con 0x7f11f0112820 2024-05-09T08:37:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.059+0000 7f11f6179640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f0112820 0x7f11f0070dc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f11e8040ac0 tx=0x7f11e8047600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.061+0000 7f11e6ffd640 1 -- 172.21.15.28:0/1999410632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e8056070 con 0x7f11f0112820 2024-05-09T08:37:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.061+0000 7f11e6ffd640 1 -- 172.21.15.28:0/1999410632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e8047db0 con 0x7f11f0112820 2024-05-09T08:37:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.061+0000 7f11e6ffd640 1 -- 172.21.15.28:0/1999410632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e8044040 con 0x7f11f0112820 2024-05-09T08:37:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.061+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f0111f30 con 0x7f11f0112820 2024-05-09T08:37:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.061+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f01123a0 con 0x7f11f0112820 2024-05-09T08:37:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.062+0000 7f11f5978640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11f0073bd0 0x7f11f0073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.072+0000 7f11fa9ca740 1 Processor -- start 2024-05-09T08:37:34.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.074+0000 7f11fa9ca740 1 -- start start 2024-05-09T08:37:34.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda79830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79d70 0x55f6fda840f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 0x55f6fda86a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6fd9de340 con 0x55f6fda79450 2024-05-09T08:37:34.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f6fd986290 con 0x55f6fda79d70 2024-05-09T08:37:34.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f6fd9da300 con 0x55f6fda84630 2024-05-09T08:37:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11f5177640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda79830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11f5177640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda79830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35908/0 (socket says 172.21.15.28:35908) 2024-05-09T08:37:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11f5177640 1 -- 172.21.15.28:0/2591331934 learned_addr learned my addr 172.21.15.28:0/2591331934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11f5177640 1 --2- 172.21.15.28:0/2591331934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 0x55f6fda86a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.075+0000 7f11e6ffd640 1 -- 172.21.15.28:0/1999410632 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f11e8047070 con 0x7f11f0112820 2024-05-09T08:37:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.077+0000 7f11e6ffd640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f11dc0a2fd0 0x7f11dc0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.077+0000 7f11e6ffd640 1 -- 172.21.15.28:0/1999410632 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7f11e8100860 con 0x7f11f0112820 2024-05-09T08:37:34.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.077+0000 7f11f5177640 1 -- 172.21.15.28:0/2591331934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 msgr2=0x55f6fda86a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:34.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.077+0000 7f11f5177640 1 --2- 172.21.15.28:0/2591331934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 0x55f6fda86a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.077+0000 7f11f5177640 1 -- 172.21.15.28:0/2591331934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79d70 msgr2=0x55f6fda840f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:37:34.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.077+0000 7f11f5177640 1 --2- 172.21.15.28:0/2591331934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79d70 0x55f6fda840f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.077+0000 7f11f5177640 1 -- 172.21.15.28:0/2591331934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6fd9aa8e0 con 0x55f6fda79450 2024-05-09T08:37:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.078+0000 7f11f5177640 1 --2- 172.21.15.28:0/2591331934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda79830 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0058c0 tx=0x7f11ec0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d17085af229703 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.078+0000 7f11dbfff640 1 -- 172.21.15.28:0/2591331934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec018030 con 0x55f6fda79450 2024-05-09T08:37:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.078+0000 7f11dbfff640 1 -- 172.21.15.28:0/2591331934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11ec005020 con 0x55f6fda79450 2024-05-09T08:37:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.079+0000 7f11dbfff640 1 -- 172.21.15.28:0/2591331934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec004a90 con 0x55f6fda79450 2024-05-09T08:37:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.080+0000 7f11fa9ca740 1 -- 172.21.15.28:0/2591331934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 msgr2=0x55f6fda79830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.080+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/2591331934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda79830 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0058c0 tx=0x7f11ec0096c0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.080+0000 7f11f6179640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f11dc0a2fd0 0x7f11dc0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.082+0000 7f11f6179640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f11dc0a2fd0 0x7f11dc0a5490 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f11e80fdb30 tx=0x7f11e8003f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.082+0000 7f11fa9ca740 1 -- 172.21.15.28:0/2591331934 shutdown_connections 2024-05-09T08:37:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.082+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/2591331934 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 0x55f6fda86a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.082+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/2591331934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79d70 0x55f6fda840f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.082+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/2591331934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda79830 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.082+0000 7f11fa9ca740 1 -- 172.21.15.28:0/2591331934 >> 172.21.15.28:0/2591331934 conn(0x55f6fda75960 msgr2=0x55f6fda75d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.087+0000 7f11fa9ca740 1 -- 172.21.15.28:0/2591331934 shutdown_connections 2024-05-09T08:37:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.087+0000 7f11fa9ca740 1 -- 172.21.15.28:0/2591331934 wait complete. 2024-05-09T08:37:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.087+0000 7f11fa9ca740 1 -- start start 2024-05-09T08:37:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.088+0000 7f11fa9ca740 1 Processor -- start 2024-05-09T08:37:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.088+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda815c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.088+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda79d70 0x55f6fda81b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.088+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda84630 0x55f6fda82040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.088+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6fda87260 con 0x55f6fda79450 2024-05-09T08:37:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.088+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f6fd999790 con 0x55f6fda84630 2024-05-09T08:37:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.088+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f6fda873e0 con 0x55f6fda79d70 2024-05-09T08:37:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.088+0000 7f11fa9ca740 10 client.0 _ll_register_callbacks cb 0x55f6fd9d4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.089+0000 7f11f5177640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda84630 0x55f6fda82040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.089+0000 7f11f5978640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda79d70 0x55f6fda81b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.089+0000 7f11f5177640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda84630 0x55f6fda82040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60216/0 (socket says 172.21.15.28:60216) 2024-05-09T08:37:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.089+0000 7f11f5177640 1 -- 172.21.15.28:0/1224642810 learned_addr learned my addr 172.21.15.28:0/1224642810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.089+0000 7f11f5177640 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda815c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.090+0000 7f11f5177640 1 -- 172.21.15.28:0/1224642810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda79d70 msgr2=0x55f6fda81b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.090+0000 7f11f5177640 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda79d70 0x55f6fda81b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.090+0000 7f11f5177640 1 -- 172.21.15.28:0/1224642810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 msgr2=0x55f6fda815c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.090+0000 7f11f5177640 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda815c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.090+0000 7f11f5177640 1 -- 172.21.15.28:0/1224642810 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6fd6f2210 con 0x55f6fda84630 2024-05-09T08:37:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.090+0000 7f11f5177640 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda84630 0x55f6fda82040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11ec005940 tx=0x7f11ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82e41e2f11651864 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.090+0000 7f11d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:37:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.090+0000 7f11d97fa640 1 -- 172.21.15.28:0/1224642810 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec004c20 con 0x55f6fda84630 2024-05-09T08:37:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.091+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1224642810 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11ec005020 con 0x55f6fda84630 2024-05-09T08:37:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.091+0000 7f11d97fa640 1 -- 172.21.15.28:0/1224642810 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7f11ec004dc0 con 0x55f6fda84630 2024-05-09T08:37:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.091+0000 7f11d97fa640 1 -- 172.21.15.28:0/1224642810 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f11ec0035f0 con 0x55f6fda84630 2024-05-09T08:37:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.092+0000 7f11fa9ca740 10 client.26450 fetch_fsmap learned FSMap version 65 2024-05-09T08:37:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.092+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1224642810 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55f6fd7be620 con 0x55f6fda84630 2024-05-09T08:37:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.093+0000 7f11d97fa640 1 -- 172.21.15.28:0/1224642810 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11ec005020 con 0x55f6fda84630 2024-05-09T08:37:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.093+0000 7f11d97fa640 1 -- 172.21.15.28:0/1224642810 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f11ec028070 con 0x55f6fda84630 2024-05-09T08:37:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.093+0000 7f11fa9ca740 10 client.26450 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:37:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.093+0000 7f11fa9ca740 10 client.26450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:37:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.093+0000 7f11fa9ca740 20 client.26450 populate_metadata read hostname 'smithi028' 2024-05-09T08:37:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.093+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f6fda94210 0x55f6fda96600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.093+0000 7f11fa9ca740 4 client.26450 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.093+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1224642810 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6fd9eba70 con 0x55f6fda94210 2024-05-09T08:37:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.094+0000 7f11f5978640 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f6fda94210 0x55f6fda96600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.101+0000 7f11f5978640 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f6fda94210 0x55f6fda96600 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84663c78968177af server_cookie=14d9e9a6098dd306 in_seq=0 out_seq=0 2024-05-09T08:37:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.101+0000 7f11d97fa640 10 client.26450 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.102+0000 7f11d97fa640 1 -- 172.21.15.28:0/1224642810 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6fd9eba70 con 0x55f6fda94210 2024-05-09T08:37:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.102+0000 7f11d97fa640 10 client.26450 handle_command_reply: tid=0 2024-05-09T08:37:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.102+0000 7f11fa9ca740 1 client.26450 shutdown 2024-05-09T08:37:34.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:33 smithi165 ceph-mon[34540]: pgmap v358: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 992 KiB/s wr, 230 op/s 2024-05-09T08:37:34.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:33 smithi184 ceph-mon[34509]: pgmap v358: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.7 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 992 KiB/s wr, 230 op/s 2024-05-09T08:37:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.115+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1224642810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda84630 msgr2=0x55f6fda82040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:34.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.115+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda84630 0x55f6fda82040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11ec005940 tx=0x7f11ec0046c0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.119+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1224642810 shutdown_connections 2024-05-09T08:37:34.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.119+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f6fda94210 0x55f6fda96600 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.119+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda84630 0x55f6fda82040 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.119+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda79d70 0x55f6fda81b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.119+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1224642810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79450 0x55f6fda815c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.119+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1224642810 >> 172.21.15.28:0/1224642810 conn(0x55f6fda75960 msgr2=0x55f6fda7b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:34.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.120+0000 7f11d97fa640 0 client.26450 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.128+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1224642810 shutdown_connections 2024-05-09T08:37:34.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.128+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1224642810 wait complete. 2024-05-09T08:37:34.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.131+0000 7f11fa9ca740 20 client.26450 trim_cache size 0 max 16384 2024-05-09T08:37:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.170+0000 7f11fa9ca740 1 Processor -- start 2024-05-09T08:37:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.170+0000 7f11fa9ca740 1 -- start start 2024-05-09T08:37:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.171+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79450 0x55f6fda83cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.171+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda79d70 0x55f6fda7be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.171+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda84630 0x55f6fda81e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.171+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11f01b5560 con 0x55f6fda84630 2024-05-09T08:37:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.171+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f11f01b72e0 con 0x55f6fda79450 2024-05-09T08:37:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.171+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f11f01b5cf0 con 0x55f6fda79d70 2024-05-09T08:37:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.171+0000 7f11f5177640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda84630 0x55f6fda81e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.171+0000 7f11f5177640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda84630 0x55f6fda81e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35936/0 (socket says 172.21.15.28:35936) 2024-05-09T08:37:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.171+0000 7f11f5177640 1 -- 172.21.15.28:0/1052801562 learned_addr learned my addr 172.21.15.28:0/1052801562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.173+0000 7f11f5177640 1 -- 172.21.15.28:0/1052801562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda79d70 msgr2=0x55f6fda7be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.173+0000 7f11f5177640 1 --2- 172.21.15.28:0/1052801562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda79d70 0x55f6fda7be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.173+0000 7f11f5177640 1 -- 172.21.15.28:0/1052801562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79450 msgr2=0x55f6fda83cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.173+0000 7f11f5177640 1 --2- 172.21.15.28:0/1052801562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79450 0x55f6fda83cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.173+0000 7f11f5177640 1 -- 172.21.15.28:0/1052801562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6fda73f00 con 0x55f6fda84630 2024-05-09T08:37:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.174+0000 7f11f5177640 1 --2- 172.21.15.28:0/1052801562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda84630 0x55f6fda81e60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f11ec005560 tx=0x7f11ec0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc681789ba2b61f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.175+0000 7f11db7fe640 1 -- 172.21.15.28:0/1052801562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec018050 con 0x55f6fda84630 2024-05-09T08:37:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.175+0000 7f11db7fe640 1 -- 172.21.15.28:0/1052801562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11ec005600 con 0x55f6fda84630 2024-05-09T08:37:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.176+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1052801562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda84630 msgr2=0x55f6fda81e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.176+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1052801562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda84630 0x55f6fda81e60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f11ec005560 tx=0x7f11ec0096c0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.181+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1052801562 shutdown_connections 2024-05-09T08:37:34.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.181+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1052801562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda84630 0x55f6fda81e60 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.181+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1052801562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda79d70 0x55f6fda7be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.181+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/1052801562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79450 0x55f6fda83cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.181+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1052801562 >> 172.21.15.28:0/1052801562 conn(0x55f6fda75960 msgr2=0x55f6fda74a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:34.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.185+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1052801562 shutdown_connections 2024-05-09T08:37:34.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.185+0000 7f11fa9ca740 1 -- 172.21.15.28:0/1052801562 wait complete. 2024-05-09T08:37:34.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.186+0000 7f11fa9ca740 1 -- start start 2024-05-09T08:37:34.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.186+0000 7f11fa9ca740 1 Processor -- start 2024-05-09T08:37:34.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.186+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79450 0x55f6fda7e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.186+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79d70 0x55f6fda7eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.186+0000 7f11fa9ca740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 0x55f6fdaf9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.186+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11f01b6c10 con 0x55f6fda79d70 2024-05-09T08:37:34.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.186+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f11f01b6480 con 0x55f6fda79450 2024-05-09T08:37:34.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.186+0000 7f11fa9ca740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f11f01b3560 con 0x55f6fda84630 2024-05-09T08:37:34.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.186+0000 7f11fa9ca740 10 client.0 _ll_register_callbacks cb 0x7f11ec009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.187+0000 7f11f5978640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79d70 0x55f6fda7eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.187+0000 7f11f5978640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79d70 0x55f6fda7eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35942/0 (socket says 172.21.15.28:35942) 2024-05-09T08:37:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.187+0000 7f11f5978640 1 -- 172.21.15.28:0/3011382436 learned_addr learned my addr 172.21.15.28:0/3011382436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.187+0000 7f11f5177640 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 0x55f6fdaf9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.187+0000 7f11f5978640 1 -- 172.21.15.28:0/3011382436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 msgr2=0x55f6fdaf9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.187+0000 7f11f5978640 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 0x55f6fdaf9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.187+0000 7f11f5978640 1 -- 172.21.15.28:0/3011382436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79450 msgr2=0x55f6fda7e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.187+0000 7f11f5978640 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79450 0x55f6fda7e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.187+0000 7f11f5978640 1 -- 172.21.15.28:0/3011382436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6fdaf9a70 con 0x55f6fda79d70 2024-05-09T08:37:34.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.188+0000 7f11f5978640 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79d70 0x55f6fda7eef0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f11e000a930 tx=0x7f11e000aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97464d2e33d381a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:34.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.188+0000 7f11d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:37:34.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.188+0000 7f11d97fa640 1 -- 172.21.15.28:0/3011382436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0015c20 con 0x55f6fda79d70 2024-05-09T08:37:34.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.189+0000 7f11d97fa640 1 -- 172.21.15.28:0/3011382436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7f11e0015dc0 con 0x55f6fda79d70 2024-05-09T08:37:34.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.189+0000 7f11fa9ca740 1 -- 172.21.15.28:0/3011382436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11ec005600 con 0x55f6fda79d70 2024-05-09T08:37:34.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.189+0000 7f11d97fa640 1 -- 172.21.15.28:0/3011382436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f11e0018150 con 0x55f6fda79d70 2024-05-09T08:37:34.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.195+0000 7f11fa9ca740 10 client.16950 fetch_fsmap learned FSMap version 65 2024-05-09T08:37:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.195+0000 7f11fa9ca740 1 -- 172.21.15.28:0/3011382436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55f6fdafadc0 con 0x55f6fda79d70 2024-05-09T08:37:34.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.200+0000 7f11d97fa640 1 -- 172.21.15.28:0/3011382436 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11ec005600 con 0x55f6fda79d70 2024-05-09T08:37:34.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.200+0000 7f11d97fa640 1 -- 172.21.15.28:0/3011382436 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f11e0021bd0 con 0x55f6fda79d70 2024-05-09T08:37:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.203+0000 7f11fa9ca740 10 client.16950 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:37:34.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.203+0000 7f11fa9ca740 10 client.16950 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:37:34.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.203+0000 7f11fa9ca740 20 client.16950 populate_metadata read hostname 'smithi028' 2024-05-09T08:37:34.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.203+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f6fdafb4d0 0x55f6fdafd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:34.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.203+0000 7f11fa9ca740 4 client.16950 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:37:34.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.203+0000 7f11fa9ca740 1 -- 172.21.15.28:0/3011382436 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f6fd9e5dc0 con 0x55f6fdafb4d0 2024-05-09T08:37:34.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.205+0000 7f11f5177640 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f6fdafb4d0 0x55f6fdafd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:34.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.211+0000 7f11f5177640 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f6fdafb4d0 0x55f6fdafd8f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b06fb2da28de525 server_cookie=5aa9097ab4b81226 in_seq=0 out_seq=0 2024-05-09T08:37:34.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.212+0000 7f11d97fa640 10 client.16950 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11d97fa640 1 -- 172.21.15.28:0/3011382436 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f6fd9e5dc0 con 0x55f6fdafb4d0 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11d97fa640 10 client.16950 handle_command_reply: tid=0 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 client.16950 shutdown 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 -- 172.21.15.28:0/3011382436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79d70 msgr2=0x55f6fda7eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79d70 0x55f6fda7eef0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f11e000a930 tx=0x7f11e000aef0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 -- 172.21.15.28:0/3011382436 shutdown_connections 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f6fdafb4d0 0x55f6fdafd8f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f6fda84630 0x55f6fdaf9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6fda79d70 0x55f6fda7eef0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11d97fa640 0 client.16950 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 --2- 172.21.15.28:0/3011382436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f6fda79450 0x55f6fda7e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 -- 172.21.15.28:0/3011382436 >> 172.21.15.28:0/3011382436 conn(0x55f6fda75960 msgr2=0x55f6fda8f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.309+0000 7f11fa9ca740 1 -- 172.21.15.28:0/3011382436 shutdown_connections 2024-05-09T08:37:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.310+0000 7f11fa9ca740 1 -- 172.21.15.28:0/3011382436 wait complete. 2024-05-09T08:37:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.310+0000 7f11fa9ca740 20 client.16950 trim_cache size 0 max 16384 2024-05-09T08:37:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:37:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:37:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:37:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:37:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f11dc0a2fd0 msgr2=0x7f11dc0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f11dc0a2fd0 0x7f11dc0a5490 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f11e80fdb30 tx=0x7f11e8003f90 comp rx=0 tx=0).stop 2024-05-09T08:37:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f0112820 msgr2=0x7f11f0070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f0112820 0x7f11f0070dc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f11e8040ac0 tx=0x7f11e8047600 comp rx=0 tx=0).stop 2024-05-09T08:37:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 shutdown_connections 2024-05-09T08:37:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f0112820 0x7f11f0070dc0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f11dc0a2fd0 0x7f11dc0a5490 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11f00744f0 0x7f11f00706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 --2- 172.21.15.28:0/1999410632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11f0073bd0 0x7f11f0073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 >> 172.21.15.28:0/1999410632 conn(0x7f11f00fd2e0 msgr2=0x7f11f00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 shutdown_connections 2024-05-09T08:37:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:34.311+0000 7f11f7bdc640 1 -- 172.21.15.28:0/1999410632 wait complete. 2024-05-09T08:37:35.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:35 smithi028 ceph-mon[30119]: pgmap v359: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.7 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 245 op/s 2024-05-09T08:37:36.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:35 smithi165 ceph-mon[34540]: pgmap v359: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.7 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 245 op/s 2024-05-09T08:37:36.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:35 smithi184 ceph-mon[34509]: pgmap v359: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.7 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 245 op/s 2024-05-09T08:37:37.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:37 smithi028 ceph-mon[30119]: pgmap v360: 129 pgs: 55 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 57 active+clean; 2.7 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 270 op/s 2024-05-09T08:37:38.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:37 smithi165 ceph-mon[34540]: pgmap v360: 129 pgs: 55 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 57 active+clean; 2.7 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 270 op/s 2024-05-09T08:37:38.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:37 smithi184 ceph-mon[34509]: pgmap v360: 129 pgs: 55 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 57 active+clean; 2.7 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 270 op/s 2024-05-09T08:37:40.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:39 smithi165 ceph-mon[34540]: pgmap v361: 129 pgs: 55 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 57 active+clean; 2.7 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 1.0 MiB/s wr, 222 op/s 2024-05-09T08:37:40.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:39 smithi184 ceph-mon[34509]: pgmap v361: 129 pgs: 55 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 57 active+clean; 2.7 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 1.0 MiB/s wr, 222 op/s 2024-05-09T08:37:40.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:39 smithi028 ceph-mon[30119]: pgmap v361: 129 pgs: 55 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 57 active+clean; 2.7 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 1.0 MiB/s wr, 222 op/s 2024-05-09T08:37:42.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:41 smithi165 ceph-mon[34540]: pgmap v362: 129 pgs: 41 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 74 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 253 op/s 2024-05-09T08:37:42.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:41 smithi184 ceph-mon[34509]: pgmap v362: 129 pgs: 41 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 74 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 253 op/s 2024-05-09T08:37:42.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:41 smithi028 ceph-mon[30119]: pgmap v362: 129 pgs: 41 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 74 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 253 op/s 2024-05-09T08:37:44.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:43 smithi165 ceph-mon[34540]: pgmap v363: 129 pgs: 40 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 75 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 240 op/s 2024-05-09T08:37:44.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:43 smithi184 ceph-mon[34509]: pgmap v363: 129 pgs: 40 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 75 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 240 op/s 2024-05-09T08:37:44.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:43 smithi028 ceph-mon[30119]: pgmap v363: 129 pgs: 40 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 75 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 240 op/s 2024-05-09T08:37:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:45 smithi165 ceph-mon[34540]: pgmap v364: 129 pgs: 34 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 82 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 251 op/s 2024-05-09T08:37:46.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:45 smithi184 ceph-mon[34509]: pgmap v364: 129 pgs: 34 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 82 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 251 op/s 2024-05-09T08:37:46.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:45 smithi028 ceph-mon[30119]: pgmap v364: 129 pgs: 34 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 82 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 251 op/s 2024-05-09T08:37:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.469+0000 7fcfa78f5640 1 -- 172.21.15.28:0/606000029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01051c0 msgr2=0x7fcfa010cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.469+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/606000029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01051c0 0x7fcfa010cd90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009be0 tx=0x7fcf9802a710 comp rx=0 tx=0).stop 2024-05-09T08:37:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.472+0000 7fcfa78f5640 1 -- 172.21.15.28:0/606000029 shutdown_connections 2024-05-09T08:37:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.472+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/606000029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01051c0 0x7fcfa010cd90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.472+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/606000029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0100aa0 0x7fcfa0104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.472+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/606000029 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa01000f0 0x7fcfa01004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.472+0000 7fcfa78f5640 1 -- 172.21.15.28:0/606000029 >> 172.21.15.28:0/606000029 conn(0x7fcfa0076cc0 msgr2=0x7fcfa00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.474+0000 7fcfa78f5640 1 -- 172.21.15.28:0/606000029 shutdown_connections 2024-05-09T08:37:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.476+0000 7fcfa78f5640 1 -- 172.21.15.28:0/606000029 wait complete. 2024-05-09T08:37:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.477+0000 7fcfa78f5640 1 Processor -- start 2024-05-09T08:37:47.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa78f5640 1 -- start start 2024-05-09T08:37:47.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa78f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01000f0 0x7fcfa0108130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa78f5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa01051c0 0x7fcfa0108670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa78f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0108bb0 0x7fcfa01a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa78f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa006d360 con 0x7fcfa0108bb0 2024-05-09T08:37:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa78f5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa006d1e0 con 0x7fcfa01051c0 2024-05-09T08:37:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa78f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa006d4e0 con 0x7fcfa01000f0 2024-05-09T08:37:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa5691640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01000f0 0x7fcfa0108130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa5691640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01000f0 0x7fcfa0108130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50036/0 (socket says 172.21.15.28:50036) 2024-05-09T08:37:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.479+0000 7fcfa5691640 1 -- 172.21.15.28:0/1399158258 learned_addr learned my addr 172.21.15.28:0/1399158258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.481+0000 7fcfa5691640 1 -- 172.21.15.28:0/1399158258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa01051c0 msgr2=0x7fcfa0108670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:37:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.481+0000 7fcfa5691640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa01051c0 0x7fcfa0108670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.481+0000 7fcfa5691640 1 -- 172.21.15.28:0/1399158258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0108bb0 msgr2=0x7fcfa01a1e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:37:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.481+0000 7fcfa5691640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0108bb0 0x7fcfa01a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.481+0000 7fcfa5691640 1 -- 172.21.15.28:0/1399158258 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa010da10 con 0x7fcfa01000f0 2024-05-09T08:37:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.483+0000 7fcfa5691640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01000f0 0x7fcfa0108130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c01ea20 tx=0x7fcf9c01eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.487+0000 7fcf967fc640 1 -- 172.21.15.28:0/1399158258 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c028070 con 0x7fcfa01000f0 2024-05-09T08:37:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.487+0000 7fcf967fc640 1 -- 172.21.15.28:0/1399158258 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf9c01abc0 con 0x7fcfa01000f0 2024-05-09T08:37:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.487+0000 7fcf967fc640 1 -- 172.21.15.28:0/1399158258 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c020630 con 0x7fcfa01000f0 2024-05-09T08:37:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.487+0000 7fcfa78f5640 1 -- 172.21.15.28:0/1399158258 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa010dd00 con 0x7fcfa01000f0 2024-05-09T08:37:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.487+0000 7fcfa78f5640 1 -- 172.21.15.28:0/1399158258 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa010e200 con 0x7fcfa01000f0 2024-05-09T08:37:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.493+0000 7fcfaa603740 1 Processor -- start 2024-05-09T08:37:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.494+0000 7fcf967fc640 1 -- 172.21.15.28:0/1399158258 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcf9c0207d0 con 0x7fcfa01000f0 2024-05-09T08:37:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.495+0000 7fcf967fc640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcf8c0a2f80 0x7fcf8c0a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.495+0000 7fcf967fc640 1 -- 172.21.15.28:0/1399158258 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7fcf9c0aa590 con 0x7fcfa01000f0 2024-05-09T08:37:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.498+0000 7fcfaa603740 1 -- start start 2024-05-09T08:37:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.498+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 0x55ebac82f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.498+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac83a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.498+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac83ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.498+0000 7fcfaa603740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ebac794340 con 0x55ebac82fd70 2024-05-09T08:37:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.498+0000 7fcfaa603740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ebac73c290 con 0x55ebac83a630 2024-05-09T08:37:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.498+0000 7fcfaa603740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ebac790300 con 0x55ebac82f450 2024-05-09T08:37:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.498+0000 7fcfa5e92640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcf8c0a2f80 0x7fcf8c0a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.500+0000 7fcfa5e92640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac83ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.500+0000 7fcfa5e92640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac83ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:38104/0 (socket says 172.21.15.28:38104) 2024-05-09T08:37:47.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.500+0000 7fcfa5e92640 1 -- 172.21.15.28:0/157196816 learned_addr learned my addr 172.21.15.28:0/157196816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:47.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.501+0000 7fcfa5e92640 1 -- 172.21.15.28:0/157196816 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 msgr2=0x55ebac82f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.501+0000 7fcfa5e92640 1 --2- 172.21.15.28:0/157196816 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 0x55ebac82f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.501+0000 7fcfa5e92640 1 -- 172.21.15.28:0/157196816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 msgr2=0x55ebac83a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.501+0000 7fcfa5e92640 1 --2- 172.21.15.28:0/157196816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac83a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.501+0000 7fcfa5e92640 1 -- 172.21.15.28:0/157196816 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebac7608e0 con 0x55ebac83a630 2024-05-09T08:37:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.501+0000 7fcfa5e92640 1 --2- 172.21.15.28:0/157196816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac83ca20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcf980304a0 tx=0x7fcf98041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39eedec37b0b2dbc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.502+0000 7fcf8b7fe640 1 -- 172.21.15.28:0/157196816 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9804f070 con 0x55ebac83a630 2024-05-09T08:37:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.502+0000 7fcf8b7fe640 1 -- 172.21.15.28:0/157196816 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf98031bc0 con 0x55ebac83a630 2024-05-09T08:37:47.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.502+0000 7fcf8b7fe640 1 -- 172.21.15.28:0/157196816 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98036040 con 0x55ebac83a630 2024-05-09T08:37:47.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.503+0000 7fcfaa603740 1 -- 172.21.15.28:0/157196816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 msgr2=0x55ebac83ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.503+0000 7fcfaa603740 1 --2- 172.21.15.28:0/157196816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac83ca20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcf980304a0 tx=0x7fcf98041000 comp rx=0 tx=0).stop 2024-05-09T08:37:47.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.506+0000 7fcfa5e92640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcf8c0a2f80 0x7fcf8c0a5440 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcf98030400 tx=0x7fcf98030430 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.508+0000 7fcfaa603740 1 -- 172.21.15.28:0/157196816 shutdown_connections 2024-05-09T08:37:47.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.508+0000 7fcfaa603740 1 --2- 172.21.15.28:0/157196816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac83ca20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.508+0000 7fcfaa603740 1 --2- 172.21.15.28:0/157196816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac83a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.508+0000 7fcfaa603740 1 --2- 172.21.15.28:0/157196816 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 0x55ebac82f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.508+0000 7fcfaa603740 1 -- 172.21.15.28:0/157196816 >> 172.21.15.28:0/157196816 conn(0x55ebac82b960 msgr2=0x55ebac82bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.513+0000 7fcfaa603740 1 -- 172.21.15.28:0/157196816 shutdown_connections 2024-05-09T08:37:47.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.515+0000 7fcfaa603740 1 -- 172.21.15.28:0/157196816 wait complete. 2024-05-09T08:37:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.516+0000 7fcfaa603740 1 -- start start 2024-05-09T08:37:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.516+0000 7fcfaa603740 1 Processor -- start 2024-05-09T08:37:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.516+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac837590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.516+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac837ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.516+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac83a630 0x55ebac838010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.516+0000 7fcfaa603740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ebac83d200 con 0x55ebac82fd70 2024-05-09T08:37:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.516+0000 7fcfaa603740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ebac74f790 con 0x55ebac82f450 2024-05-09T08:37:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.516+0000 7fcfaa603740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ebac83d380 con 0x55ebac83a630 2024-05-09T08:37:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.517+0000 7fcfaa603740 10 client.0 _ll_register_callbacks cb 0x55ebac78aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.518+0000 7fcfa5e92640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac83a630 0x55ebac838010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.518+0000 7fcfa5e92640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac83a630 0x55ebac838010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50054/0 (socket says 172.21.15.28:50054) 2024-05-09T08:37:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.518+0000 7fcfa5e92640 1 -- 172.21.15.28:0/4249535493 learned_addr learned my addr 172.21.15.28:0/4249535493 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.519+0000 7fcfa5691640 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac837590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.519+0000 7fcfa4e90640 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac837ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.519+0000 7fcfa5691640 1 -- 172.21.15.28:0/4249535493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac83a630 msgr2=0x55ebac838010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.519+0000 7fcfa5691640 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac83a630 0x55ebac838010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.519+0000 7fcfa5691640 1 -- 172.21.15.28:0/4249535493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 msgr2=0x55ebac837ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.519+0000 7fcfa5691640 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac837ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.519+0000 7fcfa5691640 1 -- 172.21.15.28:0/4249535493 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebac4a8210 con 0x55ebac82f450 2024-05-09T08:37:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.520+0000 7fcfa5691640 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac837590 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0dd720 tx=0x7fcf9c0a6ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9909362e7365dd76 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.520+0000 7fcf88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:37:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.520+0000 7fcf88ff9640 1 -- 172.21.15.28:0/4249535493 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c0a8f20 con 0x55ebac82f450 2024-05-09T08:37:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.521+0000 7fcfaa603740 1 -- 172.21.15.28:0/4249535493 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf98031bc0 con 0x55ebac82f450 2024-05-09T08:37:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.521+0000 7fcf88ff9640 1 -- 172.21.15.28:0/4249535493 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7fcf9c0b50a0 con 0x55ebac82f450 2024-05-09T08:37:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.521+0000 7fcf88ff9640 1 -- 172.21.15.28:0/4249535493 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fcf9c0a90c0 con 0x55ebac82f450 2024-05-09T08:37:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.522+0000 7fcfaa603740 10 client.26459 fetch_fsmap learned FSMap version 65 2024-05-09T08:37:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.522+0000 7fcfaa603740 1 -- 172.21.15.28:0/4249535493 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55ebac574620 con 0x55ebac82f450 2024-05-09T08:37:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.522+0000 7fcf88ff9640 1 -- 172.21.15.28:0/4249535493 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf98031bc0 con 0x55ebac82f450 2024-05-09T08:37:47.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.523+0000 7fcf88ff9640 1 -- 172.21.15.28:0/4249535493 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7fcf9c01f040 con 0x55ebac82f450 2024-05-09T08:37:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.523+0000 7fcfaa603740 10 client.26459 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:37:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.523+0000 7fcfaa603740 10 client.26459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:37:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.523+0000 7fcfaa603740 20 client.26459 populate_metadata read hostname 'smithi028' 2024-05-09T08:37:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.523+0000 7fcfaa603740 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ebac86c8a0 0x55ebac86ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.523+0000 7fcfaa603740 4 client.26459 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.523+0000 7fcfaa603740 1 -- 172.21.15.28:0/4249535493 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebac7a1a70 con 0x55ebac86c8a0 2024-05-09T08:37:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.535+0000 7fcfa4e90640 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ebac86c8a0 0x55ebac86ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.536+0000 7fcfa4e90640 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ebac86c8a0 0x55ebac86ecc0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6633981671bf2b15 server_cookie=d884e6251832e447 in_seq=0 out_seq=0 2024-05-09T08:37:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.536+0000 7fcf88ff9640 10 client.26459 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.545+0000 7fcf88ff9640 1 -- 172.21.15.28:0/4249535493 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebac7a1a70 con 0x55ebac86c8a0 2024-05-09T08:37:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.545+0000 7fcf88ff9640 10 client.26459 handle_command_reply: tid=0 2024-05-09T08:37:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.546+0000 7fcfaa603740 1 client.26459 shutdown 2024-05-09T08:37:47.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.547+0000 7fcfaa603740 1 -- 172.21.15.28:0/4249535493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 msgr2=0x55ebac837590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.547+0000 7fcfaa603740 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac837590 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0dd720 tx=0x7fcf9c0a6ff0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.552+0000 7fcfaa603740 1 -- 172.21.15.28:0/4249535493 shutdown_connections 2024-05-09T08:37:47.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.552+0000 7fcfaa603740 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ebac86c8a0 0x55ebac86ecc0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.552+0000 7fcfaa603740 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac83a630 0x55ebac838010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.552+0000 7fcfaa603740 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac837ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.552+0000 7fcfaa603740 1 --2- 172.21.15.28:0/4249535493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac837590 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.552+0000 7fcfaa603740 1 -- 172.21.15.28:0/4249535493 >> 172.21.15.28:0/4249535493 conn(0x55ebac82b960 msgr2=0x55ebac8316e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.555+0000 7fcf88ff9640 0 client.26459 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.558+0000 7fcfaa603740 1 -- 172.21.15.28:0/4249535493 shutdown_connections 2024-05-09T08:37:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.560+0000 7fcfaa603740 1 -- 172.21.15.28:0/4249535493 wait complete. 2024-05-09T08:37:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.563+0000 7fcfaa603740 20 client.26459 trim_cache size 0 max 16384 2024-05-09T08:37:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.595+0000 7fcfaa603740 1 Processor -- start 2024-05-09T08:37:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfaa603740 1 -- start start 2024-05-09T08:37:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac846120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82fd70 0x55ebac837e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac83a630 0x55ebac82a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfa5691640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac846120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfa5691640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac846120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:38122/0 (socket says 172.21.15.28:38122) 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfa5e92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac83a630 0x55ebac82a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfa5691640 1 -- 172.21.15.28:0/2315539530 learned_addr learned my addr 172.21.15.28:0/2315539530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfaa603740 1 -- 172.21.15.28:0/2315539530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf84010190 con 0x55ebac83a630 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfaa603740 1 -- 172.21.15.28:0/2315539530 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcf84011f10 con 0x55ebac82f450 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.596+0000 7fcfaa603740 1 -- 172.21.15.28:0/2315539530 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf84010920 con 0x55ebac82fd70 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.597+0000 7fcfa5e92640 1 -- 172.21.15.28:0/2315539530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82fd70 msgr2=0x55ebac837e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.597+0000 7fcfa5e92640 1 --2- 172.21.15.28:0/2315539530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82fd70 0x55ebac837e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.597+0000 7fcfa5e92640 1 -- 172.21.15.28:0/2315539530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 msgr2=0x55ebac846120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.597+0000 7fcfa5e92640 1 --2- 172.21.15.28:0/2315539530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac846120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.597+0000 7fcfa5e92640 1 -- 172.21.15.28:0/2315539530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebac82fba0 con 0x55ebac83a630 2024-05-09T08:37:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.597+0000 7fcfa5691640 1 --2- 172.21.15.28:0/2315539530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac846120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.597+0000 7fcfa5e92640 1 --2- 172.21.15.28:0/2315539530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac83a630 0x55ebac82a880 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009b30 tx=0x7fcf980315a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634a6f75ca8f068e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.599+0000 7fcf8affd640 1 -- 172.21.15.28:0/2315539530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98059030 con 0x55ebac83a630 2024-05-09T08:37:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.599+0000 7fcf8affd640 1 -- 172.21.15.28:0/2315539530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf98003ef0 con 0x55ebac83a630 2024-05-09T08:37:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.600+0000 7fcfaa603740 1 -- 172.21.15.28:0/2315539530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac83a630 msgr2=0x55ebac82a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.600+0000 7fcfaa603740 1 --2- 172.21.15.28:0/2315539530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac83a630 0x55ebac82a880 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009b30 tx=0x7fcf980315a0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.606+0000 7fcfaa603740 1 -- 172.21.15.28:0/2315539530 shutdown_connections 2024-05-09T08:37:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.606+0000 7fcfaa603740 1 --2- 172.21.15.28:0/2315539530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac83a630 0x55ebac82a880 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.606+0000 7fcfaa603740 1 --2- 172.21.15.28:0/2315539530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82fd70 0x55ebac837e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.606+0000 7fcfaa603740 1 --2- 172.21.15.28:0/2315539530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac82f450 0x55ebac846120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.606+0000 7fcfaa603740 1 -- 172.21.15.28:0/2315539530 >> 172.21.15.28:0/2315539530 conn(0x55ebac82b960 msgr2=0x55ebac82a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.610+0000 7fcfaa603740 1 -- 172.21.15.28:0/2315539530 shutdown_connections 2024-05-09T08:37:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.611+0000 7fcfaa603740 1 -- 172.21.15.28:0/2315539530 wait complete. 2024-05-09T08:37:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.611+0000 7fcfaa603740 1 -- start start 2024-05-09T08:37:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.611+0000 7fcfaa603740 1 Processor -- start 2024-05-09T08:37:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 0x55ebac842890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac833e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfaa603740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac834360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfaa603740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf84011840 con 0x55ebac82fd70 2024-05-09T08:37:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfaa603740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcf840110b0 con 0x55ebac83a630 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfaa603740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8400e190 con 0x55ebac82f450 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfaa603740 10 client.0 _ll_register_callbacks cb 0x7fcf9c02f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfa5e92640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac834360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfa5e92640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac834360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:38126/0 (socket says 172.21.15.28:38126) 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfa5e92640 1 -- 172.21.15.28:0/616125895 learned_addr learned my addr 172.21.15.28:0/616125895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.615+0000 7fcfa5691640 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 0x55ebac842890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.616+0000 7fcfa5691640 1 -- 172.21.15.28:0/616125895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 msgr2=0x55ebac834360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.616+0000 7fcfa5691640 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac834360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.616+0000 7fcfa5691640 1 -- 172.21.15.28:0/616125895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 msgr2=0x55ebac833e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.616+0000 7fcfa5691640 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac833e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.616+0000 7fcfa5691640 1 -- 172.21.15.28:0/616125895 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebac8349d0 con 0x55ebac82f450 2024-05-09T08:37:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.617+0000 7fcfa5691640 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 0x55ebac842890 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0dd7d0 tx=0x7fcf9c01a880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c74d7bd44446ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.619+0000 7fcf88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:37:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.619+0000 7fcf88ff9640 1 -- 172.21.15.28:0/616125895 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c0b0070 con 0x55ebac82f450 2024-05-09T08:37:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.619+0000 7fcf88ff9640 1 -- 172.21.15.28:0/616125895 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7fcf9c0a8020 con 0x55ebac82f450 2024-05-09T08:37:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.622+0000 7fcfaa603740 1 -- 172.21.15.28:0/616125895 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf98003ef0 con 0x55ebac82f450 2024-05-09T08:37:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.631+0000 7fcf88ff9640 1 -- 172.21.15.28:0/616125895 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fcf9c031040 con 0x55ebac82f450 2024-05-09T08:37:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.634+0000 7fcfaa603740 10 client.26320 fetch_fsmap learned FSMap version 65 2024-05-09T08:37:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.634+0000 7fcfaa603740 1 -- 172.21.15.28:0/616125895 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55ebac8d2ca0 con 0x55ebac82f450 2024-05-09T08:37:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.637+0000 7fcf88ff9640 1 -- 172.21.15.28:0/616125895 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf98003ef0 con 0x55ebac82f450 2024-05-09T08:37:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.638+0000 7fcf88ff9640 1 -- 172.21.15.28:0/616125895 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7fcf9c0b1a30 con 0x55ebac82f450 2024-05-09T08:37:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.640+0000 7fcfaa603740 10 client.26320 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:37:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.640+0000 7fcfaa603740 10 client.26320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:37:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.640+0000 7fcfaa603740 20 client.26320 populate_metadata read hostname 'smithi028' 2024-05-09T08:37:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.640+0000 7fcfaa603740 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ebac8d33f0 0x55ebac8d5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.640+0000 7fcfaa603740 4 client.26320 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:37:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.640+0000 7fcfaa603740 1 -- 172.21.15.28:0/616125895 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ebac79bdc0 con 0x55ebac8d33f0 2024-05-09T08:37:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.641+0000 7fcfa4e90640 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ebac8d33f0 0x55ebac8d5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.649+0000 7fcfa4e90640 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ebac8d33f0 0x55ebac8d5810 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3051302d835d348 server_cookie=7eebb099cb81a4f3 in_seq=0 out_seq=0 2024-05-09T08:37:47.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.650+0000 7fcf88ff9640 10 client.26320 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcf88ff9640 1 -- 172.21.15.28:0/616125895 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ebac79bdc0 con 0x55ebac8d33f0 2024-05-09T08:37:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcf88ff9640 10 client.26320 handle_command_reply: tid=0 2024-05-09T08:37:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcfaa603740 1 client.26320 shutdown 2024-05-09T08:37:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcfaa603740 1 -- 172.21.15.28:0/616125895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 msgr2=0x55ebac842890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcfaa603740 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 0x55ebac842890 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0dd7d0 tx=0x7fcf9c01a880 comp rx=0 tx=0).stop 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcfaa603740 1 -- 172.21.15.28:0/616125895 shutdown_connections 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcfaa603740 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ebac8d33f0 0x55ebac8d5810 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcfaa603740 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebac83a630 0x55ebac834360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcfaa603740 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebac82fd70 0x55ebac833e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcf88ff9640 0 client.26320 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcfaa603740 1 --2- 172.21.15.28:0/616125895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ebac82f450 0x55ebac842890 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.757+0000 7fcfaa603740 1 -- 172.21.15.28:0/616125895 >> 172.21.15.28:0/616125895 conn(0x55ebac82b960 msgr2=0x55ebac82aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.758+0000 7fcfaa603740 1 -- 172.21.15.28:0/616125895 shutdown_connections 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.758+0000 7fcfaa603740 1 -- 172.21.15.28:0/616125895 wait complete. 2024-05-09T08:37:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.758+0000 7fcfaa603740 20 client.26320 trim_cache size 0 max 16384 2024-05-09T08:37:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:37:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:37:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:37:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:37:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 -- 172.21.15.28:0/1399158258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcf8c0a2f80 msgr2=0x7fcf8c0a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcf8c0a2f80 0x7fcf8c0a5440 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcf98030400 tx=0x7fcf98030430 comp rx=0 tx=0).stop 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 -- 172.21.15.28:0/1399158258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01000f0 msgr2=0x7fcfa0108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01000f0 0x7fcfa0108130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c01ea20 tx=0x7fcf9c01eee0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 -- 172.21.15.28:0/1399158258 shutdown_connections 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa0108bb0 0x7fcfa01a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfa01051c0 0x7fcfa0108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fcf8c0a2f80 0x7fcf8c0a5440 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 --2- 172.21.15.28:0/1399158258 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfa01000f0 0x7fcfa0108130 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 -- 172.21.15.28:0/1399158258 >> 172.21.15.28:0/1399158258 conn(0x7fcfa0076cc0 msgr2=0x7fcfa0103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 -- 172.21.15.28:0/1399158258 shutdown_connections 2024-05-09T08:37:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:37:47.759+0000 7fcfa78f5640 1 -- 172.21.15.28:0/1399158258 wait complete. 2024-05-09T08:37:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:47 smithi165 ceph-mon[34540]: pgmap v365: 129 pgs: 34 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 83 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 278 op/s 2024-05-09T08:37:48.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:47 smithi184 ceph-mon[34509]: pgmap v365: 129 pgs: 34 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 83 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 278 op/s 2024-05-09T08:37:48.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:47 smithi028 ceph-mon[30119]: pgmap v365: 129 pgs: 34 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 83 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 278 op/s 2024-05-09T08:37:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:49 smithi165 ceph-mon[34540]: pgmap v366: 129 pgs: 27 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 91 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 238 op/s 2024-05-09T08:37:50.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:49 smithi184 ceph-mon[34509]: pgmap v366: 129 pgs: 27 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 91 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 238 op/s 2024-05-09T08:37:50.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:49 smithi028 ceph-mon[30119]: pgmap v366: 129 pgs: 27 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 91 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 238 op/s 2024-05-09T08:37:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:51 smithi165 ceph-mon[34540]: pgmap v367: 129 pgs: 27 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 92 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 274 op/s 2024-05-09T08:37:52.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:51 smithi184 ceph-mon[34509]: pgmap v367: 129 pgs: 27 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 92 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 274 op/s 2024-05-09T08:37:52.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:51 smithi028 ceph-mon[30119]: pgmap v367: 129 pgs: 27 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 92 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 274 op/s 2024-05-09T08:37:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:53 smithi165 ceph-mon[34540]: pgmap v368: 129 pgs: 27 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 92 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 272 op/s 2024-05-09T08:37:54.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:53 smithi184 ceph-mon[34509]: pgmap v368: 129 pgs: 27 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 92 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 272 op/s 2024-05-09T08:37:54.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:53 smithi028 ceph-mon[30119]: pgmap v368: 129 pgs: 27 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 92 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 272 op/s 2024-05-09T08:37:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:55 smithi165 ceph-mon[34540]: pgmap v369: 129 pgs: 19 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 102 active+clean; 2.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 998 KiB/s wr, 285 op/s 2024-05-09T08:37:56.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:55 smithi184 ceph-mon[34509]: pgmap v369: 129 pgs: 19 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 102 active+clean; 2.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 998 KiB/s wr, 285 op/s 2024-05-09T08:37:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:55 smithi028 ceph-mon[30119]: pgmap v369: 129 pgs: 19 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 102 active+clean; 2.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 998 KiB/s wr, 285 op/s 2024-05-09T08:37:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:57 smithi165 ceph-mon[34540]: pgmap v370: 129 pgs: 13 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 110 active+clean; 2.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 353 op/s 2024-05-09T08:37:58.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:57 smithi184 ceph-mon[34509]: pgmap v370: 129 pgs: 13 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 110 active+clean; 2.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 353 op/s 2024-05-09T08:37:58.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:57 smithi028 ceph-mon[30119]: pgmap v370: 129 pgs: 13 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 110 active+clean; 2.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 353 op/s 2024-05-09T08:38:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:37:59 smithi165 ceph-mon[34540]: pgmap v371: 129 pgs: 13 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 111 active+clean; 2.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 324 op/s 2024-05-09T08:38:00.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:37:59 smithi184 ceph-mon[34509]: pgmap v371: 129 pgs: 13 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 111 active+clean; 2.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 324 op/s 2024-05-09T08:38:00.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:37:59 smithi028 ceph-mon[30119]: pgmap v371: 129 pgs: 13 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 111 active+clean; 2.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 324 op/s 2024-05-09T08:38:00.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.915+0000 7f16c359e640 1 -- 172.21.15.28:0/1071057228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16bc0957f0 msgr2=0x7f16bc0a1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:00.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.915+0000 7f16c359e640 1 --2- 172.21.15.28:0/1071057228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16bc0957f0 0x7f16bc0a1ac0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f16c4066fa0 tx=0x7f16c4096260 comp rx=0 tx=0).stop 2024-05-09T08:38:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.915+0000 7f16c359e640 1 -- 172.21.15.28:0/1071057228 shutdown_connections 2024-05-09T08:38:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.915+0000 7f16c359e640 1 --2- 172.21.15.28:0/1071057228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16bc0957f0 0x7f16bc0a1ac0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.915+0000 7f16c359e640 1 --2- 172.21.15.28:0/1071057228 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16bc094e50 0x7f16bc0952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.915+0000 7f16c359e640 1 --2- 172.21.15.28:0/1071057228 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16bc099bc0 0x7f16bc099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.915+0000 7f16c359e640 1 -- 172.21.15.28:0/1071057228 >> 172.21.15.28:0/1071057228 conn(0x7f16bc00a740 msgr2=0x7f16bc00ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.915+0000 7f16c359e640 1 -- 172.21.15.28:0/1071057228 shutdown_connections 2024-05-09T08:38:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.916+0000 7f16c359e640 1 -- 172.21.15.28:0/1071057228 wait complete. 2024-05-09T08:38:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.917+0000 7f16c359e640 1 Processor -- start 2024-05-09T08:38:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.917+0000 7f16c359e640 1 -- start start 2024-05-09T08:38:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.917+0000 7f16c359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16bc094e50 0x7f16bc12b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.917+0000 7f16c359e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16bc0957f0 0x7f16bc12b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.917+0000 7f16c359e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16bc099bc0 0x7f16bc12fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.918+0000 7f16c359e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc0a4f00 con 0x7f16bc094e50 2024-05-09T08:38:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.918+0000 7f16c359e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc0a4d80 con 0x7f16bc0957f0 2024-05-09T08:38:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.918+0000 7f16c359e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc0a5080 con 0x7f16bc099bc0 2024-05-09T08:38:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.918+0000 7f16c1d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16bc0957f0 0x7f16bc12b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.918+0000 7f16c1d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16bc0957f0 0x7f16bc12b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:40856/0 (socket says 172.21.15.28:40856) 2024-05-09T08:38:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.918+0000 7f16c1d9b640 1 -- 172.21.15.28:0/3404713342 learned_addr learned my addr 172.21.15.28:0/3404713342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.918+0000 7f16c2d9d640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16bc099bc0 0x7f16bc12fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.919+0000 7f16c1d9b640 1 -- 172.21.15.28:0/3404713342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16bc099bc0 msgr2=0x7f16bc12fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.919+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16bc099bc0 0x7f16bc12fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.919+0000 7f16c1d9b640 1 -- 172.21.15.28:0/3404713342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16bc094e50 msgr2=0x7f16bc12b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:38:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.919+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16bc094e50 0x7f16bc12b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.919+0000 7f16c1d9b640 1 -- 172.21.15.28:0/3404713342 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16bc130390 con 0x7f16bc0957f0 2024-05-09T08:38:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.920+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16bc0957f0 0x7f16bc12b8e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f16b800e9f0 tx=0x7f16b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.921+0000 7f16b37fe640 1 -- 172.21.15.28:0/3404713342 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b800cd80 con 0x7f16bc0957f0 2024-05-09T08:38:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.921+0000 7f16b37fe640 1 -- 172.21.15.28:0/3404713342 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b8010910 con 0x7f16bc0957f0 2024-05-09T08:38:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.922+0000 7f16b37fe640 1 -- 172.21.15.28:0/3404713342 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b800cd80 con 0x7f16bc0957f0 2024-05-09T08:38:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.923+0000 7f16c359e640 1 -- 172.21.15.28:0/3404713342 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16bc130680 con 0x7f16bc0957f0 2024-05-09T08:38:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.923+0000 7f16c359e640 1 -- 172.21.15.28:0/3404713342 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16bc137f10 con 0x7f16bc0957f0 2024-05-09T08:38:00.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.927+0000 7f16cbc1e740 1 Processor -- start 2024-05-09T08:38:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.928+0000 7f16cbc1e740 1 -- start start 2024-05-09T08:38:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.928+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956bf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.928+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956b6810 0x555a956b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.928+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956ca050 0x555a956cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.928+0000 7f16cbc1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555a9561b340 con 0x555a956ca050 2024-05-09T08:38:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.928+0000 7f16cbc1e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x555a955c3290 con 0x555a956b6810 2024-05-09T08:38:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.928+0000 7f16cbc1e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555a95617300 con 0x555a956b6430 2024-05-09T08:38:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.930+0000 7f16c2d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956bf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.930+0000 7f16c2d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956bf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:43496/0 (socket says 172.21.15.28:43496) 2024-05-09T08:38:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.930+0000 7f16c2d9d640 1 -- 172.21.15.28:0/3131064480 learned_addr learned my addr 172.21.15.28:0/3131064480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.930+0000 7f16b37fe640 1 -- 172.21.15.28:0/3404713342 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f16b801f020 con 0x7f16bc0957f0 2024-05-09T08:38:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.932+0000 7f16b37fe640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f16ac0a3140 0x7f16ac0a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.932+0000 7f16c2d9d640 1 -- 172.21.15.28:0/3131064480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956b6810 msgr2=0x555a956b6c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:38:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.932+0000 7f16c2d9d640 1 --2- 172.21.15.28:0/3131064480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956b6810 0x555a956b6c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.932+0000 7f16c2d9d640 1 -- 172.21.15.28:0/3131064480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956ca050 msgr2=0x555a956cc440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:38:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.933+0000 7f16c2d9d640 1 --2- 172.21.15.28:0/3131064480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956ca050 0x555a956cc440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.933+0000 7f16c2d9d640 1 -- 172.21.15.28:0/3131064480 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a955e78e0 con 0x555a956b6430 2024-05-09T08:38:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.933+0000 7f16c2d9d640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f16ac0a3140 0x7f16ac0a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.933+0000 7f16b37fe640 1 -- 172.21.15.28:0/3404713342 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7f16b800cb80 con 0x7f16bc0957f0 2024-05-09T08:38:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.936+0000 7f16c2d9d640 1 --2- 172.21.15.28:0/3131064480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956bf260 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f16c4067540 tx=0x7f16c4067b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9d3515819b93d02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.940+0000 7f16b0ff9640 1 -- 172.21.15.28:0/3131064480 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c40b6430 con 0x555a956b6430 2024-05-09T08:38:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.940+0000 7f16b0ff9640 1 -- 172.21.15.28:0/3131064480 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c40b6ab0 con 0x555a956b6430 2024-05-09T08:38:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.941+0000 7f16cbc1e740 1 -- 172.21.15.28:0/3131064480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 msgr2=0x555a956bf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.941+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/3131064480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956bf260 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f16c4067540 tx=0x7f16c4067b00 comp rx=0 tx=0).stop 2024-05-09T08:38:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.941+0000 7f16c2d9d640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f16ac0a3140 0x7f16ac0a5600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x555a956b7320 tx=0x7f16c40b3000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.943+0000 7f16cbc1e740 1 -- 172.21.15.28:0/3131064480 shutdown_connections 2024-05-09T08:38:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.943+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/3131064480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956ca050 0x555a956cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.943+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/3131064480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956b6810 0x555a956b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.943+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/3131064480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956bf260 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.943+0000 7f16cbc1e740 1 -- 172.21.15.28:0/3131064480 >> 172.21.15.28:0/3131064480 conn(0x555a956b2ae0 msgr2=0x555a956b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.947+0000 7f16cbc1e740 1 -- 172.21.15.28:0/3131064480 shutdown_connections 2024-05-09T08:38:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.949+0000 7f16cbc1e740 1 -- 172.21.15.28:0/3131064480 wait complete. 2024-05-09T08:38:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.949+0000 7f16cbc1e740 1 -- start start 2024-05-09T08:38:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.949+0000 7f16cbc1e740 1 Processor -- start 2024-05-09T08:38:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.950+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.950+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6810 0x555a956c3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.950+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 0x555a956c9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.950+0000 7f16c259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.950+0000 7f16c259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956c3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:43500/0 (socket says 172.21.15.28:43500) 2024-05-09T08:38:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.950+0000 7f16c259c640 1 -- 172.21.15.28:0/1530325899 learned_addr learned my addr 172.21.15.28:0/1530325899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.950+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555a956bfb10 con 0x555a956b6810 2024-05-09T08:38:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.950+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x555a955d6790 con 0x555a956ca050 2024-05-09T08:38:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.950+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555a956bfc90 con 0x555a956b6430 2024-05-09T08:38:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.951+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6810 0x555a956c3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.951+0000 7f16cbc1e740 10 client.0 _ll_register_callbacks cb 0x555a95611a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.951+0000 7f16c2d9d640 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 0x555a956c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.951+0000 7f16c1d9b640 1 -- 172.21.15.28:0/1530325899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 msgr2=0x555a956c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.951+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.951+0000 7f16c1d9b640 1 -- 172.21.15.28:0/1530325899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 msgr2=0x555a956c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.951+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 0x555a956c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.951+0000 7f16c1d9b640 1 -- 172.21.15.28:0/1530325899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a9532f210 con 0x555a956b6810 2024-05-09T08:38:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.952+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6810 0x555a956c3fb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f16b80cc1d0 tx=0x7f16b80a3710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23fc04695d1ba98f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.953+0000 7f16a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:38:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.953+0000 7f16a9ffb640 1 -- 172.21.15.28:0/1530325899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8097a60 con 0x555a956b6810 2024-05-09T08:38:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.953+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c40b6ab0 con 0x555a956b6810 2024-05-09T08:38:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.954+0000 7f16a9ffb640 1 -- 172.21.15.28:0/1530325899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7f16b8097100 con 0x555a956b6810 2024-05-09T08:38:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.954+0000 7f16a9ffb640 1 -- 172.21.15.28:0/1530325899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f16b8097c00 con 0x555a956b6810 2024-05-09T08:38:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.956+0000 7f16cbc1e740 10 client.16965 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.956+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x555a953fb620 con 0x555a956b6810 2024-05-09T08:38:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.959+0000 7f16a9ffb640 1 -- 172.21.15.28:0/1530325899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c40b6ab0 con 0x555a956b6810 2024-05-09T08:38:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.959+0000 7f16a9ffb640 1 -- 172.21.15.28:0/1530325899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f16b80cc450 con 0x555a956b6810 2024-05-09T08:38:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.959+0000 7f16cbc1e740 10 client.16965 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.959+0000 7f16cbc1e740 10 client.16965 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.960+0000 7f16cbc1e740 20 client.16965 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.960+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555a956e29f0 0x555a956e4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.960+0000 7f16cbc1e740 4 client.16965 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.960+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a95628a70 con 0x555a956e29f0 2024-05-09T08:38:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.962+0000 7f16c259c640 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555a956e29f0 0x555a956e4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.964+0000 7f16c259c640 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555a956e29f0 0x555a956e4e50 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1829c32a5af18439 server_cookie=80383f74d8bd443d in_seq=0 out_seq=0 2024-05-09T08:38:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.964+0000 7f16a9ffb640 10 client.16965 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.970+0000 7f16a9ffb640 1 -- 172.21.15.28:0/1530325899 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a95628a70 con 0x555a956e29f0 2024-05-09T08:38:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.971+0000 7f16a9ffb640 10 client.16965 handle_command_reply: tid=0 2024-05-09T08:38:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.973+0000 7f16cbc1e740 1 client.16965 shutdown 2024-05-09T08:38:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.974+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6810 msgr2=0x555a956c3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.974+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6810 0x555a956c3fb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f16b80cc1d0 tx=0x7f16b80a3710 comp rx=0 tx=0).stop 2024-05-09T08:38:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.976+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 shutdown_connections 2024-05-09T08:38:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.976+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555a956e29f0 0x555a956e4e50 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.976+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 0x555a956c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.976+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6810 0x555a956c3fb0 secure :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f16b80cc1d0 tx=0x7f16b80a3710 comp rx=0 tx=0).stop 2024-05-09T08:38:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.976+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/1530325899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.976+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 >> 172.21.15.28:0/1530325899 conn(0x555a956b2ae0 msgr2=0x555a956b8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.977+0000 7f16a9ffb640 0 client.16965 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.978+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 shutdown_connections 2024-05-09T08:38:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.979+0000 7f16cbc1e740 1 -- 172.21.15.28:0/1530325899 wait complete. 2024-05-09T08:38:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:00.980+0000 7f16cbc1e740 20 client.16965 trim_cache size 0 max 16384 2024-05-09T08:38:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.010+0000 7f16cbc1e740 1 Processor -- start 2024-05-09T08:38:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.012+0000 7f16cbc1e740 1 -- start start 2024-05-09T08:38:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.012+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.012+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956ca050 0x555a956ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.012+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956b2060 0x555a956eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.012+0000 7f16cbc1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1411a0 con 0x555a956ca050 2024-05-09T08:38:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.012+0000 7f16cbc1e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc142f20 con 0x555a956b2060 2024-05-09T08:38:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.012+0000 7f16cbc1e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc141930 con 0x555a956b6430 2024-05-09T08:38:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:43510/0 (socket says 172.21.15.28:43510) 2024-05-09T08:38:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c259c640 1 -- 172.21.15.28:0/2907522945 learned_addr learned my addr 172.21.15.28:0/2907522945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/2907522945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956ca050 0x555a956ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c2d9d640 1 --2- 172.21.15.28:0/2907522945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956b2060 0x555a956eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c259c640 1 -- 172.21.15.28:0/2907522945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956b2060 msgr2=0x555a956eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c259c640 1 --2- 172.21.15.28:0/2907522945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956b2060 0x555a956eaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c259c640 1 -- 172.21.15.28:0/2907522945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956ca050 msgr2=0x555a956ea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c259c640 1 --2- 172.21.15.28:0/2907522945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956ca050 0x555a956ea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c259c640 1 -- 172.21.15.28:0/2907522945 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a956bd610 con 0x555a956b6430 2024-05-09T08:38:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.013+0000 7f16c259c640 1 --2- 172.21.15.28:0/2907522945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956b1b20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f16b4000c00 tx=0x7f16b4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=892a73605f5213a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.014+0000 7f16abfff640 1 -- 172.21.15.28:0/2907522945 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b401b5b0 con 0x555a956b6430 2024-05-09T08:38:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.014+0000 7f16abfff640 1 -- 172.21.15.28:0/2907522945 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b4013da0 con 0x555a956b6430 2024-05-09T08:38:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.015+0000 7f16abfff640 1 -- 172.21.15.28:0/2907522945 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b40047a0 con 0x555a956b6430 2024-05-09T08:38:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.015+0000 7f16cbc1e740 1 -- 172.21.15.28:0/2907522945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 msgr2=0x555a956b1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.015+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/2907522945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956b1b20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f16b4000c00 tx=0x7f16b4005f90 comp rx=0 tx=0).stop 2024-05-09T08:38:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.015+0000 7f16cbc1e740 1 -- 172.21.15.28:0/2907522945 shutdown_connections 2024-05-09T08:38:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.016+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/2907522945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956b2060 0x555a956eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.016+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/2907522945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956ca050 0x555a956ea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.016+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/2907522945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b6430 0x555a956b1b20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.016+0000 7f16cbc1e740 1 -- 172.21.15.28:0/2907522945 >> 172.21.15.28:0/2907522945 conn(0x555a956b2ae0 msgr2=0x555a956b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:01.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.018+0000 7f16cbc1e740 1 -- 172.21.15.28:0/2907522945 shutdown_connections 2024-05-09T08:38:01.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.018+0000 7f16cbc1e740 1 -- 172.21.15.28:0/2907522945 wait complete. 2024-05-09T08:38:01.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.019+0000 7f16cbc1e740 1 -- start start 2024-05-09T08:38:01.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.019+0000 7f16cbc1e740 1 Processor -- start 2024-05-09T08:38:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.019+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b2060 0x555a956f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6430 0x555a956f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16cbc1e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 0x555a956f4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16cbc1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc142850 con 0x555a956b6430 2024-05-09T08:38:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16c1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6430 0x555a956f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16c1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6430 0x555a956f60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60770/0 (socket says 172.21.15.28:60770) 2024-05-09T08:38:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16cbc1e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1420c0 con 0x555a956ca050 2024-05-09T08:38:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16cbc1e740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc13f1a0 con 0x555a956b2060 2024-05-09T08:38:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16cbc1e740 10 client.0 _ll_register_callbacks cb 0x7f16b4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16c1d9b640 1 -- 172.21.15.28:0/145389718 learned_addr learned my addr 172.21.15.28:0/145389718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.020+0000 7f16c2d9d640 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 0x555a956f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.021+0000 7f16c1d9b640 1 -- 172.21.15.28:0/145389718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b2060 msgr2=0x555a956f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.021+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b2060 0x555a956f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.021+0000 7f16c1d9b640 1 -- 172.21.15.28:0/145389718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 msgr2=0x555a956f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.021+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 0x555a956f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.021+0000 7f16c1d9b640 1 -- 172.21.15.28:0/145389718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a956f4810 con 0x555a956b6430 2024-05-09T08:38:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.021+0000 7f16c1d9b640 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6430 0x555a956f60f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f16b8084640 tx=0x7f16b80bb050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428abbefba472c95 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.021+0000 7f16a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:38:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.021+0000 7f16a9ffb640 1 -- 172.21.15.28:0/145389718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8097da0 con 0x555a956b6430 2024-05-09T08:38:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.021+0000 7f16a9ffb640 1 -- 172.21.15.28:0/145389718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9671+0+0 (secure 0 0 0) 0x7f16b800cb80 con 0x555a956b6430 2024-05-09T08:38:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.022+0000 7f16cbc1e740 1 -- 172.21.15.28:0/145389718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16b4013da0 con 0x555a956b6430 2024-05-09T08:38:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.022+0000 7f16a9ffb640 1 -- 172.21.15.28:0/145389718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f16b8097f70 con 0x555a956b6430 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.022+0000 7f16cbc1e740 10 client.16971 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.022+0000 7f16cbc1e740 1 -- 172.21.15.28:0/145389718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x555a95749650 con 0x555a956b6430 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.023+0000 7f16a9ffb640 1 -- 172.21.15.28:0/145389718 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b4013da0 con 0x555a956b6430 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.023+0000 7f16a9ffb640 1 -- 172.21.15.28:0/145389718 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f16b809a510 con 0x555a956b6430 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.023+0000 7f16cbc1e740 10 client.16971 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.023+0000 7f16cbc1e740 10 client.16971 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.023+0000 7f16cbc1e740 20 client.16971 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.023+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555a95749c30 0x555a9574c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.023+0000 7f16cbc1e740 4 client.16971 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.023+0000 7f16cbc1e740 1 -- 172.21.15.28:0/145389718 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a95622dc0 con 0x555a95749c30 2024-05-09T08:38:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.024+0000 7f16c259c640 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555a95749c30 0x555a9574c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:01.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.025+0000 7f16c259c640 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555a95749c30 0x555a9574c090 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ea3aa393b6b129d server_cookie=79fc99df05c35cd5 in_seq=0 out_seq=0 2024-05-09T08:38:01.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.025+0000 7f16a9ffb640 10 client.16971 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:01.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.077+0000 7f16a9ffb640 1 -- 172.21.15.28:0/145389718 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a95622dc0 con 0x555a95749c30 2024-05-09T08:38:01.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.077+0000 7f16a9ffb640 10 client.16971 handle_command_reply: tid=0 2024-05-09T08:38:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.077+0000 7f16cbc1e740 1 client.16971 shutdown 2024-05-09T08:38:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.081+0000 7f16cbc1e740 1 -- 172.21.15.28:0/145389718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6430 msgr2=0x555a956f60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.081+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6430 0x555a956f60f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f16b8084640 tx=0x7f16b80bb050 comp rx=0 tx=0).stop 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.083+0000 7f16cbc1e740 1 -- 172.21.15.28:0/145389718 shutdown_connections 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.083+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555a95749c30 0x555a9574c090 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.083+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555a956ca050 0x555a956f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.083+0000 7f16a9ffb640 0 client.16971 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.083+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a956b6430 0x555a956f60f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.083+0000 7f16cbc1e740 1 --2- 172.21.15.28:0/145389718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555a956b2060 0x555a956f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.083+0000 7f16cbc1e740 1 -- 172.21.15.28:0/145389718 >> 172.21.15.28:0/145389718 conn(0x555a956b2ae0 msgr2=0x555a956b80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.083+0000 7f16cbc1e740 1 -- 172.21.15.28:0/145389718 shutdown_connections 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.083+0000 7f16cbc1e740 1 -- 172.21.15.28:0/145389718 wait complete. 2024-05-09T08:38:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.084+0000 7f16cbc1e740 20 client.16971 trim_cache size 0 max 16384 2024-05-09T08:38:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:38:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:38:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:38:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:38:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 -- 172.21.15.28:0/3404713342 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f16ac0a3140 msgr2=0x7f16ac0a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f16ac0a3140 0x7f16ac0a5600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x555a956b7320 tx=0x7f16c40b3000 comp rx=0 tx=0).stop 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 -- 172.21.15.28:0/3404713342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16bc0957f0 msgr2=0x7f16bc12b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16bc0957f0 0x7f16bc12b8e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f16b800e9f0 tx=0x7f16b800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 -- 172.21.15.28:0/3404713342 shutdown_connections 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f16ac0a3140 0x7f16ac0a5600 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16bc099bc0 0x7f16bc12fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16bc0957f0 0x7f16bc12b8e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 --2- 172.21.15.28:0/3404713342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16bc094e50 0x7f16bc12b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.085+0000 7f16c359e640 1 -- 172.21.15.28:0/3404713342 >> 172.21.15.28:0/3404713342 conn(0x7f16bc00a740 msgr2=0x7f16bc092ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.086+0000 7f16c359e640 1 -- 172.21.15.28:0/3404713342 shutdown_connections 2024-05-09T08:38:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:01.086+0000 7f16c359e640 1 -- 172.21.15.28:0/3404713342 wait complete. 2024-05-09T08:38:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:01 smithi165 ceph-mon[34540]: pgmap v372: 129 pgs: 13 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 112 active+clean; 2.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 387 op/s 2024-05-09T08:38:02.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:01 smithi184 ceph-mon[34509]: pgmap v372: 129 pgs: 13 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 112 active+clean; 2.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 387 op/s 2024-05-09T08:38:02.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:01 smithi028 ceph-mon[30119]: pgmap v372: 129 pgs: 13 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 112 active+clean; 2.7 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 387 op/s 2024-05-09T08:38:04.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:03 smithi165 ceph-mon[34540]: pgmap v373: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 410 op/s 2024-05-09T08:38:04.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:03 smithi184 ceph-mon[34509]: pgmap v373: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 410 op/s 2024-05-09T08:38:04.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:03 smithi028 ceph-mon[30119]: pgmap v373: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 410 op/s 2024-05-09T08:38:05.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:04 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:38:05.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:04 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:38:05.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:04 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:38:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:05 smithi165 ceph-mon[34540]: pgmap v374: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 443 op/s 2024-05-09T08:38:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:05 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:05 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:05 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:05 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:05 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:05 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:05 smithi165 ceph-mon[34540]: osdmap e113: 12 total, 12 up, 12 in 2024-05-09T08:38:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:05 smithi184 ceph-mon[34509]: pgmap v374: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 443 op/s 2024-05-09T08:38:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:05 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:05 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:05 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:05 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:05 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:05 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:05 smithi184 ceph-mon[34509]: osdmap e113: 12 total, 12 up, 12 in 2024-05-09T08:38:06.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:05 smithi028 ceph-mon[30119]: pgmap v374: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 443 op/s 2024-05-09T08:38:06.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:05 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:05 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:05 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:05 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:05 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:05 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:06.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:05 smithi028 ceph-mon[30119]: osdmap e113: 12 total, 12 up, 12 in 2024-05-09T08:38:07.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:06 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:38:07.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:06 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:38:07.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:06 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:07.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:06 smithi165 ceph-mon[34540]: osdmap e114: 12 total, 12 up, 12 in 2024-05-09T08:38:07.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:06 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:38:07.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:06 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:38:07.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:06 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:07.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:06 smithi184 ceph-mon[34509]: osdmap e114: 12 total, 12 up, 12 in 2024-05-09T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:06 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:06 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:06 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:38:07.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:06 smithi028 ceph-mon[30119]: osdmap e114: 12 total, 12 up, 12 in 2024-05-09T08:38:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:07 smithi165 ceph-mon[34540]: pgmap v376: 129 pgs: 129 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 510 op/s 2024-05-09T08:38:08.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:07 smithi184 ceph-mon[34509]: pgmap v376: 129 pgs: 129 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 510 op/s 2024-05-09T08:38:08.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:07 smithi028 ceph-mon[30119]: pgmap v376: 129 pgs: 129 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 510 op/s 2024-05-09T08:38:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:09 smithi165 ceph-mon[34540]: pgmap v378: 129 pgs: 15 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 110 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 496 op/s 2024-05-09T08:38:10.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:09 smithi184 ceph-mon[34509]: pgmap v378: 129 pgs: 15 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 110 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 496 op/s 2024-05-09T08:38:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:09 smithi028 ceph-mon[30119]: pgmap v378: 129 pgs: 15 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 110 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 496 op/s 2024-05-09T08:38:11.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:11 smithi028 ceph-mon[30119]: pgmap v379: 129 pgs: 46 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 67 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 447 op/s 2024-05-09T08:38:11.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:11 smithi028 ceph-mon[30119]: osdmap e115: 12 total, 12 up, 12 in 2024-05-09T08:38:12.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:11 smithi165 ceph-mon[34540]: pgmap v379: 129 pgs: 46 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 67 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 447 op/s 2024-05-09T08:38:12.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:11 smithi165 ceph-mon[34540]: osdmap e115: 12 total, 12 up, 12 in 2024-05-09T08:38:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:11 smithi184 ceph-mon[34509]: pgmap v379: 129 pgs: 46 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 67 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 447 op/s 2024-05-09T08:38:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:11 smithi184 ceph-mon[34509]: osdmap e115: 12 total, 12 up, 12 in 2024-05-09T08:38:13.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:13 smithi028 ceph-mon[30119]: pgmap v381: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 277 op/s 2024-05-09T08:38:14.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:13 smithi165 ceph-mon[34540]: pgmap v381: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 277 op/s 2024-05-09T08:38:14.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:13 smithi184 ceph-mon[34509]: pgmap v381: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 277 op/s 2024-05-09T08:38:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.237+0000 7f191f7fe640 1 -- 172.21.15.28:0/2688063527 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19180319e0 con 0x7f1920104220 2024-05-09T08:38:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.237+0000 7f1926ce1640 1 -- 172.21.15.28:0/2688063527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1920104220 msgr2=0x7f19201046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.237+0000 7f1926ce1640 1 --2- 172.21.15.28:0/2688063527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1920104220 0x7f19201046a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1918009900 tx=0x7f191802a440 comp rx=0 tx=0).stop 2024-05-09T08:38:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.237+0000 7f1926ce1640 1 -- 172.21.15.28:0/2688063527 shutdown_connections 2024-05-09T08:38:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.237+0000 7f1926ce1640 1 --2- 172.21.15.28:0/2688063527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1920104220 0x7f19201046a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.237+0000 7f1926ce1640 1 --2- 172.21.15.28:0/2688063527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f192006a5a0 0x7f192006aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.237+0000 7f1926ce1640 1 --2- 172.21.15.28:0/2688063527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1920106880 0x7f192006a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.237+0000 7f1926ce1640 1 -- 172.21.15.28:0/2688063527 >> 172.21.15.28:0/2688063527 conn(0x7f1920076530 msgr2=0x7f1920076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.237+0000 7f1926ce1640 1 -- 172.21.15.28:0/2688063527 shutdown_connections 2024-05-09T08:38:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.238+0000 7f1926ce1640 1 -- 172.21.15.28:0/2688063527 wait complete. 2024-05-09T08:38:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.238+0000 7f1926ce1640 1 Processor -- start 2024-05-09T08:38:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.238+0000 7f1926ce1640 1 -- start start 2024-05-09T08:38:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.239+0000 7f1926ce1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f192006a5a0 0x7f192019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.239+0000 7f1926ce1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1920104220 0x7f192019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.239+0000 7f1926ce1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1920106880 0x7f19201a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.239+0000 7f1926ce1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19201182b0 con 0x7f1920106880 2024-05-09T08:38:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.239+0000 7f1926ce1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1920118130 con 0x7f192006a5a0 2024-05-09T08:38:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.239+0000 7f1926ce1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1920118430 con 0x7f1920104220 2024-05-09T08:38:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.239+0000 7f192527e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1920106880 0x7f19201a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.239+0000 7f192527e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1920106880 0x7f19201a2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46924/0 (socket says 172.21.15.28:46924) 2024-05-09T08:38:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.239+0000 7f192527e640 1 -- 172.21.15.28:0/294490881 learned_addr learned my addr 172.21.15.28:0/294490881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.240+0000 7f192527e640 1 -- 172.21.15.28:0/294490881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1920104220 msgr2=0x7f192019ebd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:38:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.240+0000 7f192527e640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1920104220 0x7f192019ebd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.240+0000 7f192527e640 1 -- 172.21.15.28:0/294490881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f192006a5a0 msgr2=0x7f192019e690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:38:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.240+0000 7f192527e640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f192006a5a0 0x7f192019e690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.240+0000 7f192527e640 1 -- 172.21.15.28:0/294490881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19201a3650 con 0x7f1920106880 2024-05-09T08:38:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.240+0000 7f192527e640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1920106880 0x7f19201a2f50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1918009a20 tx=0x7f1918032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.241+0000 7f191dffb640 1 -- 172.21.15.28:0/294490881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191803f030 con 0x7f1920106880 2024-05-09T08:38:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.241+0000 7f1926ce1640 1 -- 172.21.15.28:0/294490881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19201a38e0 con 0x7f1920106880 2024-05-09T08:38:14.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.241+0000 7f1926ce1640 1 -- 172.21.15.28:0/294490881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19201070a0 con 0x7f1920106880 2024-05-09T08:38:14.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.241+0000 7f191dffb640 1 -- 172.21.15.28:0/294490881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1918003ce0 con 0x7f1920106880 2024-05-09T08:38:14.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.241+0000 7f191dffb640 1 -- 172.21.15.28:0/294490881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918035070 con 0x7f1920106880 2024-05-09T08:38:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.245+0000 7f19299f2740 1 Processor -- start 2024-05-09T08:38:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.246+0000 7f191dffb640 1 -- 172.21.15.28:0/294490881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1918042030 con 0x7f1920106880 2024-05-09T08:38:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.247+0000 7f191dffb640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f190c0a2ea0 0x7f190c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.248+0000 7f191dffb640 1 -- 172.21.15.28:0/294490881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7f19180f01a0 con 0x7f1920106880 2024-05-09T08:38:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.248+0000 7f19299f2740 1 -- start start 2024-05-09T08:38:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.248+0000 7f19299f2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4400 0x55bad9cd47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.248+0000 7f19299f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 0x55bad9cdf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.248+0000 7f19299f2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.248+0000 7f19299f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bad9c39340 con 0x55bad9cd4d20 2024-05-09T08:38:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.248+0000 7f19299f2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55bad9be1290 con 0x55bad9cdf5e0 2024-05-09T08:38:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.248+0000 7f19299f2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bad9c35300 con 0x55bad9cd4400 2024-05-09T08:38:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.249+0000 7f192527e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.249+0000 7f192527e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60188/0 (socket says 172.21.15.28:60188) 2024-05-09T08:38:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.249+0000 7f192527e640 1 -- 172.21.15.28:0/1817850703 learned_addr learned my addr 172.21.15.28:0/1817850703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.249+0000 7f192527e640 1 -- 172.21.15.28:0/1817850703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4400 msgr2=0x55bad9cd47e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:38:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.249+0000 7f192527e640 1 --2- 172.21.15.28:0/1817850703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4400 0x55bad9cd47e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.249+0000 7f192527e640 1 -- 172.21.15.28:0/1817850703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 msgr2=0x55bad9cdf0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:38:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.249+0000 7f192527e640 1 --2- 172.21.15.28:0/1817850703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 0x55bad9cdf0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.249+0000 7f192527e640 1 -- 172.21.15.28:0/1817850703 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bad9c058e0 con 0x55bad9cdf5e0 2024-05-09T08:38:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.250+0000 7f192527e640 1 --2- 172.21.15.28:0/1817850703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce19d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f19180c4930 tx=0x7f19180bbb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64a1739134d04a61 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.251+0000 7f191ffff640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f190c0a2ea0 0x7f190c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.251+0000 7f1916ffd640 1 -- 172.21.15.28:0/1817850703 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918049040 con 0x55bad9cdf5e0 2024-05-09T08:38:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.252+0000 7f191ffff640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f190c0a2ea0 0x7f190c0a5360 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1908006fd0 tx=0x7f1908006bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.252+0000 7f1916ffd640 1 -- 172.21.15.28:0/1817850703 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19180c05a0 con 0x55bad9cdf5e0 2024-05-09T08:38:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.252+0000 7f1916ffd640 1 -- 172.21.15.28:0/1817850703 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19180ed6a0 con 0x55bad9cdf5e0 2024-05-09T08:38:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.254+0000 7f19299f2740 1 -- 172.21.15.28:0/1817850703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 msgr2=0x55bad9ce19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.254+0000 7f19299f2740 1 --2- 172.21.15.28:0/1817850703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce19d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f19180c4930 tx=0x7f19180bbb50 comp rx=0 tx=0).stop 2024-05-09T08:38:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.256+0000 7f19299f2740 1 -- 172.21.15.28:0/1817850703 shutdown_connections 2024-05-09T08:38:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.256+0000 7f19299f2740 1 --2- 172.21.15.28:0/1817850703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce19d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.256+0000 7f19299f2740 1 --2- 172.21.15.28:0/1817850703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 0x55bad9cdf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.256+0000 7f19299f2740 1 --2- 172.21.15.28:0/1817850703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4400 0x55bad9cd47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.256+0000 7f19299f2740 1 -- 172.21.15.28:0/1817850703 >> 172.21.15.28:0/1817850703 conn(0x55bad9cd0910 msgr2=0x55bad9cd0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.257+0000 7f19299f2740 1 -- 172.21.15.28:0/1817850703 shutdown_connections 2024-05-09T08:38:14.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.257+0000 7f19299f2740 1 -- 172.21.15.28:0/1817850703 wait complete. 2024-05-09T08:38:14.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.258+0000 7f19299f2740 1 -- start start 2024-05-09T08:38:14.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.258+0000 7f19299f2740 1 Processor -- start 2024-05-09T08:38:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.262+0000 7f19299f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4400 0x55bad9cdc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.262+0000 7f19299f2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9cdca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.262+0000 7f19299f2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9cdcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.262+0000 7f19299f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bad9ce1fd0 con 0x55bad9cd4400 2024-05-09T08:38:14.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.262+0000 7f19299f2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55bad9bf4790 con 0x55bad9cdf5e0 2024-05-09T08:38:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.262+0000 7f19299f2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bad9ce2150 con 0x55bad9cd4d20 2024-05-09T08:38:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.262+0000 7f19299f2740 10 client.0 _ll_register_callbacks cb 0x55bad9c2fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f191ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9cdca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f191ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9cdca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:45826/0 (socket says 172.21.15.28:45826) 2024-05-09T08:38:14.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f191ffff640 1 -- 172.21.15.28:0/1143132357 learned_addr learned my addr 172.21.15.28:0/1143132357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:14.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f192527e640 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9cdcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f1924a7d640 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4400 0x55bad9cdc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f191ffff640 1 -- 172.21.15.28:0/1143132357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 msgr2=0x55bad9cdcf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:14.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f191ffff640 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9cdcf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f191ffff640 1 -- 172.21.15.28:0/1143132357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4400 msgr2=0x55bad9cdc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:14.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f191ffff640 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4400 0x55bad9cdc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.263+0000 7f191ffff640 1 -- 172.21.15.28:0/1143132357 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bad994d210 con 0x55bad9cd4d20 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.264+0000 7f191ffff640 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9cdca00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1908006d60 tx=0x7f19080047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1607911d3d726347 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.264+0000 7f18fffff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.264+0000 7f18fffff640 1 -- 172.21.15.28:0/1143132357 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908010770 con 0x55bad9cd4d20 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.264+0000 7f19299f2740 1 -- 172.21.15.28:0/1143132357 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19180c05a0 con 0x55bad9cd4d20 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.264+0000 7f18fffff640 1 -- 172.21.15.28:0/1143132357 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7f1908010910 con 0x55bad9cd4d20 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.264+0000 7f18fffff640 1 -- 172.21.15.28:0/1143132357 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f1908010b30 con 0x55bad9cd4d20 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.264+0000 7f19299f2740 10 client.26341 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.264+0000 7f19299f2740 1 -- 172.21.15.28:0/1143132357 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55bad9a19620 con 0x55bad9cd4d20 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.265+0000 7f18fffff640 1 -- 172.21.15.28:0/1143132357 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19180c05a0 con 0x55bad9cd4d20 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.265+0000 7f18fffff640 1 -- 172.21.15.28:0/1143132357 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f1908012070 con 0x55bad9cd4d20 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.265+0000 7f19299f2740 10 client.26341 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.265+0000 7f19299f2740 10 client.26341 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.265+0000 7f19299f2740 20 client.26341 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.265+0000 7f19299f2740 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bad9d000f0 0x55bad9d02550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.265+0000 7f19299f2740 4 client.26341 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.265+0000 7f19299f2740 1 -- 172.21.15.28:0/1143132357 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bad9c46a70 con 0x55bad9d000f0 2024-05-09T08:38:14.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.266+0000 7f1924a7d640 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bad9d000f0 0x55bad9d02550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.266+0000 7f1924a7d640 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bad9d000f0 0x55bad9d02550 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a41d192a547689 server_cookie=8ae0f1694090dd59 in_seq=0 out_seq=0 2024-05-09T08:38:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.266+0000 7f18fffff640 10 client.26341 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f18fffff640 1 -- 172.21.15.28:0/1143132357 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bad9c46a70 con 0x55bad9d000f0 2024-05-09T08:38:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f18fffff640 10 client.26341 handle_command_reply: tid=0 2024-05-09T08:38:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 client.26341 shutdown 2024-05-09T08:38:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 -- 172.21.15.28:0/1143132357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 msgr2=0x55bad9cdca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9cdca00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1908006d60 tx=0x7f19080047d0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 -- 172.21.15.28:0/1143132357 shutdown_connections 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bad9d000f0 0x55bad9d02550 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cdf5e0 0x55bad9cdcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9cdca00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 --2- 172.21.15.28:0/1143132357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4400 0x55bad9cdc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f18fffff640 0 client.26341 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 -- 172.21.15.28:0/1143132357 >> 172.21.15.28:0/1143132357 conn(0x55bad9cd0910 msgr2=0x55bad9cd6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 -- 172.21.15.28:0/1143132357 shutdown_connections 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 1 -- 172.21.15.28:0/1143132357 wait complete. 2024-05-09T08:38:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.267+0000 7f19299f2740 20 client.26341 trim_cache size 0 max 16384 2024-05-09T08:38:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.283+0000 7f19299f2740 1 Processor -- start 2024-05-09T08:38:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.283+0000 7f19299f2740 1 -- start start 2024-05-09T08:38:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f19299f2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9cdebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f19299f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 0x55bad9ceac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f19299f2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cdf5e0 0x55bad9cdcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f19299f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1920109ec0 con 0x55bad9cd4d20 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f19299f2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f192010bc40 con 0x55bad9cd4400 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f19299f2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f192010a650 con 0x55bad9cdf5e0 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f191ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 0x55bad9ceac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f191ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 0x55bad9ceac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46950/0 (socket says 172.21.15.28:46950) 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f191ffff640 1 -- 172.21.15.28:0/262218164 learned_addr learned my addr 172.21.15.28:0/262218164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f1924a7d640 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9cdebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f192527e640 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cdf5e0 0x55bad9cdcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f191ffff640 1 -- 172.21.15.28:0/262218164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cdf5e0 msgr2=0x55bad9cdcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f191ffff640 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cdf5e0 0x55bad9cdcd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f191ffff640 1 -- 172.21.15.28:0/262218164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 msgr2=0x55bad9cdebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f191ffff640 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9cdebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f191ffff640 1 -- 172.21.15.28:0/262218164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bad9d02bd0 con 0x55bad9cd4d20 2024-05-09T08:38:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.284+0000 7f192527e640 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cdf5e0 0x55bad9cdcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.285+0000 7f191ffff640 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 0x55bad9ceac50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f190801df80 tx=0x7f1908004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b03c12bbaf1c545 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.285+0000 7f19167fc640 1 -- 172.21.15.28:0/262218164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908035070 con 0x55bad9cd4d20 2024-05-09T08:38:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.285+0000 7f19167fc640 1 -- 172.21.15.28:0/262218164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908022070 con 0x55bad9cd4d20 2024-05-09T08:38:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.285+0000 7f19167fc640 1 -- 172.21.15.28:0/262218164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908005de0 con 0x55bad9cd4d20 2024-05-09T08:38:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 -- 172.21.15.28:0/262218164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 msgr2=0x55bad9ceac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 0x55bad9ceac50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f190801df80 tx=0x7f1908004640 comp rx=0 tx=0).stop 2024-05-09T08:38:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 -- 172.21.15.28:0/262218164 shutdown_connections 2024-05-09T08:38:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cdf5e0 0x55bad9cdcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cd4d20 0x55bad9ceac50 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 --2- 172.21.15.28:0/262218164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9cdebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 -- 172.21.15.28:0/262218164 >> 172.21.15.28:0/262218164 conn(0x55bad9cd0910 msgr2=0x55bad9ccb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 -- 172.21.15.28:0/262218164 shutdown_connections 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 -- 172.21.15.28:0/262218164 wait complete. 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.286+0000 7f19299f2740 1 -- start start 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.287+0000 7f19299f2740 1 Processor -- start 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.287+0000 7f19299f2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9ce7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f19299f2740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9ce2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f19299f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f1924a7d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9ce7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f1924a7d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9ce7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60212/0 (socket says 172.21.15.28:60212) 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f19299f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f192010b570 con 0x55bad9cdf5e0 2024-05-09T08:38:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f19299f2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f192010ade0 con 0x55bad9cd4400 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f19299f2740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f192004f5f0 con 0x55bad9cd4d20 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f1924a7d640 1 -- 172.21.15.28:0/1682337447 learned_addr learned my addr 172.21.15.28:0/1682337447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f19299f2740 10 client.0 _ll_register_callbacks cb 0x7f1908013c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.288+0000 7f192527e640 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.289+0000 7f191ffff640 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9ce2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.289+0000 7f1924a7d640 1 -- 172.21.15.28:0/1682337447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 msgr2=0x55bad9ce2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.289+0000 7f1924a7d640 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9ce2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.289+0000 7f1924a7d640 1 -- 172.21.15.28:0/1682337447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cdf5e0 msgr2=0x55bad9ce2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.289+0000 7f1924a7d640 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.289+0000 7f1924a7d640 1 -- 172.21.15.28:0/1682337447 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bad9ce3280 con 0x55bad9cd4400 2024-05-09T08:38:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.289+0000 7f1924a7d640 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9ce7390 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x55bad9d02da0 tx=0x7f191000bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0436fd3f5bb97de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.289+0000 7f18fffff640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:38:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.289+0000 7f18fffff640 1 -- 172.21.15.28:0/1682337447 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910019900 con 0x55bad9cd4400 2024-05-09T08:38:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.290+0000 7f19299f2740 1 -- 172.21.15.28:0/1682337447 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1908022070 con 0x55bad9cd4400 2024-05-09T08:38:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.290+0000 7f18fffff640 1 -- 172.21.15.28:0/1682337447 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7f19100167e0 con 0x55bad9cd4400 2024-05-09T08:38:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.290+0000 7f18fffff640 1 -- 172.21.15.28:0/1682337447 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f191002a050 con 0x55bad9cd4400 2024-05-09T08:38:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.293+0000 7f19299f2740 10 client.26492 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.293+0000 7f19299f2740 1 -- 172.21.15.28:0/1682337447 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55bad9d66590 con 0x55bad9cd4400 2024-05-09T08:38:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.294+0000 7f18fffff640 1 -- 172.21.15.28:0/1682337447 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908022070 con 0x55bad9cd4400 2024-05-09T08:38:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.294+0000 7f18fffff640 1 -- 172.21.15.28:0/1682337447 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f1910022cc0 con 0x55bad9cd4400 2024-05-09T08:38:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.294+0000 7f19299f2740 10 client.26492 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.294+0000 7f19299f2740 10 client.26492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.294+0000 7f19299f2740 20 client.26492 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.294+0000 7f19299f2740 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bad9d66dd0 0x55bad9d691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.294+0000 7f19299f2740 4 client.26492 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.294+0000 7f19299f2740 1 -- 172.21.15.28:0/1682337447 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bad9c40dc0 con 0x55bad9d66dd0 2024-05-09T08:38:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.295+0000 7f191ffff640 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bad9d66dd0 0x55bad9d691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.296+0000 7f18fffff640 10 client.26492 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.296+0000 7f191ffff640 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bad9d66dd0 0x55bad9d691f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a106cd7d0730f5c server_cookie=8861092913bef849 in_seq=0 out_seq=0 2024-05-09T08:38:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f18fffff640 1 -- 172.21.15.28:0/1682337447 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bad9c40dc0 con 0x55bad9d66dd0 2024-05-09T08:38:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f18fffff640 10 client.26492 handle_command_reply: tid=0 2024-05-09T08:38:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 client.26492 shutdown 2024-05-09T08:38:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 -- 172.21.15.28:0/1682337447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 msgr2=0x55bad9ce7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9ce7390 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x55bad9d02da0 tx=0x7f191000bce0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 -- 172.21.15.28:0/1682337447 shutdown_connections 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bad9d66dd0 0x55bad9d691f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bad9cdf5e0 0x55bad9ce2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bad9cd4d20 0x55bad9ce2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 --2- 172.21.15.28:0/1682337447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bad9cd4400 0x55bad9ce7390 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 -- 172.21.15.28:0/1682337447 >> 172.21.15.28:0/1682337447 conn(0x55bad9cd0910 msgr2=0x55bad9ccf120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f18fffff640 0 client.26492 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 -- 172.21.15.28:0/1682337447 shutdown_connections 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.413+0000 7f19299f2740 1 -- 172.21.15.28:0/1682337447 wait complete. 2024-05-09T08:38:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f19299f2740 20 client.26492 trim_cache size 0 max 16384 2024-05-09T08:38:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:38:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:38:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:38:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 -- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f190c0a2ea0 msgr2=0x7f190c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f190c0a2ea0 0x7f190c0a5360 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1908006fd0 tx=0x7f1908006bc0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 -- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1920106880 msgr2=0x7f19201a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1920106880 0x7f19201a2f50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1918009a20 tx=0x7f1918032b90 comp rx=0 tx=0).stop 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 -- 172.21.15.28:0/294490881 shutdown_connections 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1920106880 0x7f19201a2f50 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f190c0a2ea0 0x7f190c0a5360 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1920104220 0x7f192019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 --2- 172.21.15.28:0/294490881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f192006a5a0 0x7f192019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.414+0000 7f1926ce1640 1 -- 172.21.15.28:0/294490881 >> 172.21.15.28:0/294490881 conn(0x7f1920076530 msgr2=0x7f1920102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.415+0000 7f1926ce1640 1 -- 172.21.15.28:0/294490881 shutdown_connections 2024-05-09T08:38:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:14.415+0000 7f1926ce1640 1 -- 172.21.15.28:0/294490881 wait complete. 2024-05-09T08:38:15.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:15 smithi028 ceph-mon[30119]: pgmap v382: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-05-09T08:38:16.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:15 smithi165 ceph-mon[34540]: pgmap v382: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-05-09T08:38:16.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:15 smithi184 ceph-mon[34509]: pgmap v382: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-05-09T08:38:17.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:17 smithi028 ceph-mon[30119]: pgmap v383: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 220 op/s 2024-05-09T08:38:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:17 smithi165 ceph-mon[34540]: pgmap v383: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 220 op/s 2024-05-09T08:38:18.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:17 smithi184 ceph-mon[34509]: pgmap v383: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 220 op/s 2024-05-09T08:38:18.931 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:38:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:38:18.531+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:38:19.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:19 smithi028 ceph-mon[30119]: pgmap v384: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 210 op/s 2024-05-09T08:38:20.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:19 smithi165 ceph-mon[34540]: pgmap v384: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 210 op/s 2024-05-09T08:38:20.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:19 smithi184 ceph-mon[34509]: pgmap v384: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 210 op/s 2024-05-09T08:38:21.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:21 smithi028 ceph-mon[30119]: pgmap v385: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 207 op/s 2024-05-09T08:38:22.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:21 smithi165 ceph-mon[34540]: pgmap v385: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 207 op/s 2024-05-09T08:38:22.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:21 smithi184 ceph-mon[34509]: pgmap v385: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 207 op/s 2024-05-09T08:38:23.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:23 smithi028 ceph-mon[30119]: pgmap v386: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 219 op/s 2024-05-09T08:38:24.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:23 smithi165 ceph-mon[34540]: pgmap v386: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 219 op/s 2024-05-09T08:38:24.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:23 smithi184 ceph-mon[34509]: pgmap v386: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 219 op/s 2024-05-09T08:38:25.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:25 smithi028 ceph-mon[30119]: pgmap v387: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 906 KiB/s wr, 238 op/s 2024-05-09T08:38:26.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:25 smithi165 ceph-mon[34540]: pgmap v387: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 906 KiB/s wr, 238 op/s 2024-05-09T08:38:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:25 smithi184 ceph-mon[34509]: pgmap v387: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 906 KiB/s wr, 238 op/s 2024-05-09T08:38:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.561+0000 7f79667fc640 1 -- 172.21.15.28:0/1518087346 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79580319b0 con 0x7f7968106720 2024-05-09T08:38:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.561+0000 7f796dfb7640 1 -- 172.21.15.28:0/1518087346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7968106720 msgr2=0x7f7968106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.561+0000 7f796dfb7640 1 --2- 172.21.15.28:0/1518087346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7968106720 0x7f7968106b00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7958009930 tx=0x7f795802a440 comp rx=0 tx=0).stop 2024-05-09T08:38:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.561+0000 7f796dfb7640 1 -- 172.21.15.28:0/1518087346 shutdown_connections 2024-05-09T08:38:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.561+0000 7f796dfb7640 1 --2- 172.21.15.28:0/1518087346 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79681010d0 0x7f796810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.561+0000 7f796dfb7640 1 --2- 172.21.15.28:0/1518087346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7968100710 0x7f7968100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.561+0000 7f796dfb7640 1 --2- 172.21.15.28:0/1518087346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7968106720 0x7f7968106b00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.561+0000 7f796dfb7640 1 -- 172.21.15.28:0/1518087346 >> 172.21.15.28:0/1518087346 conn(0x7f79680fc200 msgr2=0x7f79680fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.562+0000 7f796dfb7640 1 -- 172.21.15.28:0/1518087346 shutdown_connections 2024-05-09T08:38:27.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.562+0000 7f796dfb7640 1 -- 172.21.15.28:0/1518087346 wait complete. 2024-05-09T08:38:27.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.563+0000 7f796dfb7640 1 Processor -- start 2024-05-09T08:38:27.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.563+0000 7f796dfb7640 1 -- start start 2024-05-09T08:38:27.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.563+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7968100710 0x7f7968109b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.563+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79681010d0 0x7f796810a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.563+0000 7f796dfb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796810a610 0x7f79680732a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.563+0000 7f796dfb7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f796810fb20 con 0x7f796810a610 2024-05-09T08:38:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.563+0000 7f796dfb7640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f796810f9a0 con 0x7f79681010d0 2024-05-09T08:38:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.563+0000 7f796dfb7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f796810fca0 con 0x7f7968100710 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.564+0000 7f7967fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796810a610 0x7f79680732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.564+0000 7f7967fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796810a610 0x7f79680732a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38956/0 (socket says 172.21.15.28:38956) 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.564+0000 7f7967fff640 1 -- 172.21.15.28:0/3470458949 learned_addr learned my addr 172.21.15.28:0/3470458949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.564+0000 7f7966ffd640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79681010d0 0x7f796810a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.564+0000 7f79677fe640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7968100710 0x7f7968109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f7966ffd640 1 -- 172.21.15.28:0/3470458949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7968100710 msgr2=0x7f7968109b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f7966ffd640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7968100710 0x7f7968109b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f7966ffd640 1 -- 172.21.15.28:0/3470458949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796810a610 msgr2=0x7f79680732a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f7966ffd640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796810a610 0x7f79680732a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f7966ffd640 1 -- 172.21.15.28:0/3470458949 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79681b3870 con 0x7f79681010d0 2024-05-09T08:38:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f7966ffd640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79681010d0 0x7f796810a0d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f795c009fd0 tx=0x7f795c00b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f7964ff9640 1 -- 172.21.15.28:0/3470458949 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c015020 con 0x7f79681010d0 2024-05-09T08:38:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f796dfb7640 1 -- 172.21.15.28:0/3470458949 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79681b3b60 con 0x7f79681010d0 2024-05-09T08:38:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f7964ff9640 1 -- 172.21.15.28:0/3470458949 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f795c011d10 con 0x7f79681010d0 2024-05-09T08:38:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f7964ff9640 1 -- 172.21.15.28:0/3470458949 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c00f2e0 con 0x7f79681010d0 2024-05-09T08:38:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.565+0000 7f796dfb7640 1 -- 172.21.15.28:0/3470458949 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79681b40b0 con 0x7f79681010d0 2024-05-09T08:38:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.569+0000 7f7964ff9640 1 -- 172.21.15.28:0/3470458949 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f795c018020 con 0x7f79681010d0 2024-05-09T08:38:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.569+0000 7f7970da1740 1 Processor -- start 2024-05-09T08:38:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.569+0000 7f7970da1740 1 -- start start 2024-05-09T08:38:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.569+0000 7f7970da1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35eb450 0x55dda35eb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.569+0000 7f7970da1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35ebd70 0x55dda35f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7970da1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35f8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7970da1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dda3550340 con 0x55dda35eb450 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7970da1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55dda34f8290 con 0x55dda35f6630 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7970da1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dda354c300 con 0x55dda35ebd70 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7967fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35eb450 0x55dda35eb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7967fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35eb450 0x55dda35eb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38966/0 (socket says 172.21.15.28:38966) 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7967fff640 1 -- 172.21.15.28:0/1576663465 learned_addr learned my addr 172.21.15.28:0/1576663465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f79677fe640 1 --2- 172.21.15.28:0/1576663465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35ebd70 0x55dda35f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7967fff640 1 -- 172.21.15.28:0/1576663465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35ebd70 msgr2=0x55dda35f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7967fff640 1 --2- 172.21.15.28:0/1576663465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35ebd70 0x55dda35f60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7967fff640 1 -- 172.21.15.28:0/1576663465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 msgr2=0x55dda35f8a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:38:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7967fff640 1 --2- 172.21.15.28:0/1576663465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35f8a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7967fff640 1 -- 172.21.15.28:0/1576663465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dda351c8e0 con 0x55dda35eb450 2024-05-09T08:38:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7964ff9640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79500a2ea0 0x7f79500a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.570+0000 7f7967fff640 1 --2- 172.21.15.28:0/1576663465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35eb450 0x55dda35eb830 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f796000e1c0 tx=0x7f7960009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af08fa4fb0b1ca36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.571+0000 7f7964ff9640 1 -- 172.21.15.28:0/3470458949 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7f795c0cf050 con 0x7f79681010d0 2024-05-09T08:38:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.571+0000 7f794dffb640 1 -- 172.21.15.28:0/1576663465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960018020 con 0x55dda35eb450 2024-05-09T08:38:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.571+0000 7f79677fe640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79500a2ea0 0x7f79500a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.572+0000 7f794dffb640 1 -- 172.21.15.28:0/1576663465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7960014b60 con 0x55dda35eb450 2024-05-09T08:38:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.572+0000 7f794dffb640 1 -- 172.21.15.28:0/1576663465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960013c40 con 0x55dda35eb450 2024-05-09T08:38:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.573+0000 7f7970da1740 1 -- 172.21.15.28:0/1576663465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35eb450 msgr2=0x55dda35eb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.573+0000 7f7970da1740 1 --2- 172.21.15.28:0/1576663465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35eb450 0x55dda35eb830 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f796000e1c0 tx=0x7f7960009710 comp rx=0 tx=0).stop 2024-05-09T08:38:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.573+0000 7f79677fe640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79500a2ea0 0x7f79500a5360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7958009230 tx=0x7f7958005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.573+0000 7f7970da1740 1 -- 172.21.15.28:0/1576663465 shutdown_connections 2024-05-09T08:38:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.573+0000 7f7970da1740 1 --2- 172.21.15.28:0/1576663465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35f8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.573+0000 7f7970da1740 1 --2- 172.21.15.28:0/1576663465 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35ebd70 0x55dda35f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.573+0000 7f7970da1740 1 --2- 172.21.15.28:0/1576663465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35eb450 0x55dda35eb830 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.573+0000 7f7970da1740 1 -- 172.21.15.28:0/1576663465 >> 172.21.15.28:0/1576663465 conn(0x55dda35e7960 msgr2=0x55dda35e7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.574+0000 7f7970da1740 1 -- 172.21.15.28:0/1576663465 shutdown_connections 2024-05-09T08:38:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.574+0000 7f7970da1740 1 -- 172.21.15.28:0/1576663465 wait complete. 2024-05-09T08:38:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.574+0000 7f7970da1740 1 -- start start 2024-05-09T08:38:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.575+0000 7f7970da1740 1 Processor -- start 2024-05-09T08:38:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.575+0000 7f7970da1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.575+0000 7f7970da1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.575+0000 7f7970da1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35f4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.575+0000 7f7970da1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dda35f9260 con 0x55dda35ebd70 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.575+0000 7f7970da1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55dda350b790 con 0x55dda35f6630 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.575+0000 7f7970da1740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dda35f93e0 con 0x55dda35eb450 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.575+0000 7f7970da1740 10 client.0 _ll_register_callbacks cb 0x55dda3546a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f79677fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f79677fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35f35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:45158/0 (socket says 172.21.15.28:45158) 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f79677fe640 1 -- 172.21.15.28:0/4050128354 learned_addr learned my addr 172.21.15.28:0/4050128354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f7967fff640 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f7966ffd640 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f7966ffd640 1 -- 172.21.15.28:0/4050128354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 msgr2=0x55dda35f35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f7966ffd640 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35f35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f7966ffd640 1 -- 172.21.15.28:0/4050128354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 msgr2=0x55dda35f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f7966ffd640 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.576+0000 7f7966ffd640 1 -- 172.21.15.28:0/4050128354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dda3264210 con 0x55dda35ebd70 2024-05-09T08:38:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.577+0000 7f7966ffd640 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3af0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f795c0cc3e0 tx=0x7f795c097510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3738b4437aa361f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.577+0000 7f7946ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:38:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.577+0000 7f7946ffd640 1 -- 172.21.15.28:0/4050128354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c09b8b0 con 0x55dda35ebd70 2024-05-09T08:38:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.577+0000 7f7946ffd640 1 -- 172.21.15.28:0/4050128354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7f795c097820 con 0x55dda35ebd70 2024-05-09T08:38:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.578+0000 7f7970da1740 1 -- 172.21.15.28:0/4050128354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7960014b60 con 0x55dda35ebd70 2024-05-09T08:38:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.578+0000 7f7946ffd640 1 -- 172.21.15.28:0/4050128354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f795c09b1b0 con 0x55dda35ebd70 2024-05-09T08:38:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.578+0000 7f7970da1740 10 client.16992 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.578+0000 7f7970da1740 1 -- 172.21.15.28:0/4050128354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55dda3330620 con 0x55dda35ebd70 2024-05-09T08:38:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.579+0000 7f7946ffd640 1 -- 172.21.15.28:0/4050128354 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7960014b60 con 0x55dda35ebd70 2024-05-09T08:38:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.579+0000 7f7946ffd640 1 -- 172.21.15.28:0/4050128354 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f795c09f170 con 0x55dda35ebd70 2024-05-09T08:38:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.579+0000 7f7970da1740 10 client.16992 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.579+0000 7f7970da1740 10 client.16992 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.579+0000 7f7970da1740 20 client.16992 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.579+0000 7f7970da1740 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dda3606410 0x55dda3608800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.579+0000 7f7970da1740 4 client.16992 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.579+0000 7f7970da1740 1 -- 172.21.15.28:0/4050128354 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dda355da70 con 0x55dda3606410 2024-05-09T08:38:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.580+0000 7f79677fe640 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dda3606410 0x55dda3608800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.580+0000 7f79677fe640 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dda3606410 0x55dda3608800 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e47df6f881ce49d8 server_cookie=b246a67a2f9f5152 in_seq=0 out_seq=0 2024-05-09T08:38:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.580+0000 7f7946ffd640 10 client.16992 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:27.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.585+0000 7f7946ffd640 1 -- 172.21.15.28:0/4050128354 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dda355da70 con 0x55dda3606410 2024-05-09T08:38:27.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.585+0000 7f7946ffd640 10 client.16992 handle_command_reply: tid=0 2024-05-09T08:38:27.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.585+0000 7f7970da1740 1 client.16992 shutdown 2024-05-09T08:38:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.592+0000 7f7970da1740 1 -- 172.21.15.28:0/4050128354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 msgr2=0x55dda35f3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.592+0000 7f7970da1740 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3af0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f795c0cc3e0 tx=0x7f795c097510 comp rx=0 tx=0).stop 2024-05-09T08:38:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.595+0000 7f7970da1740 1 -- 172.21.15.28:0/4050128354 shutdown_connections 2024-05-09T08:38:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.595+0000 7f7970da1740 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dda3606410 0x55dda3608800 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.595+0000 7f7946ffd640 0 client.16992 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.595+0000 7f7970da1740 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.595+0000 7f7970da1740 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3af0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.595+0000 7f7970da1740 1 --2- 172.21.15.28:0/4050128354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35f35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.595+0000 7f7970da1740 1 -- 172.21.15.28:0/4050128354 >> 172.21.15.28:0/4050128354 conn(0x55dda35e7960 msgr2=0x55dda35ed690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.596+0000 7f7970da1740 1 -- 172.21.15.28:0/4050128354 shutdown_connections 2024-05-09T08:38:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.597+0000 7f7970da1740 1 -- 172.21.15.28:0/4050128354 wait complete. 2024-05-09T08:38:27.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.597+0000 7f7970da1740 20 client.16992 trim_cache size 0 max 16384 2024-05-09T08:38:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.620+0000 7f7970da1740 1 Processor -- start 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.621+0000 7f7970da1740 1 -- start start 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.622+0000 7f7970da1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35eb450 0x55dda3602120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.622+0000 7f7970da1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.622+0000 7f7970da1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35f6630 0x55dda360e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.622+0000 7f7966ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.622+0000 7f7966ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38986/0 (socket says 172.21.15.28:38986) 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.622+0000 7f7966ffd640 1 -- 172.21.15.28:0/739161624 learned_addr learned my addr 172.21.15.28:0/739161624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.622+0000 7f7970da1740 1 -- 172.21.15.28:0/739161624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b5da0 con 0x55dda35ebd70 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.622+0000 7f7970da1740 1 -- 172.21.15.28:0/739161624 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b7b20 con 0x55dda35eb450 2024-05-09T08:38:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.622+0000 7f7967fff640 1 --2- 172.21.15.28:0/739161624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35f6630 0x55dda360e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.623+0000 7f7970da1740 1 -- 172.21.15.28:0/739161624 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b6530 con 0x55dda35f6630 2024-05-09T08:38:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.623+0000 7f7967fff640 1 -- 172.21.15.28:0/739161624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35eb450 msgr2=0x55dda3602120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:38:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.623+0000 7f7967fff640 1 --2- 172.21.15.28:0/739161624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35eb450 0x55dda3602120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.623+0000 7f7967fff640 1 -- 172.21.15.28:0/739161624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 msgr2=0x55dda35f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.623+0000 7f7967fff640 1 --2- 172.21.15.28:0/739161624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.623+0000 7f7967fff640 1 -- 172.21.15.28:0/739161624 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dda35e5f00 con 0x55dda35f6630 2024-05-09T08:38:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.624+0000 7f7967fff640 1 --2- 172.21.15.28:0/739161624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35f6630 0x55dda360e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7960004b90 tx=0x7f796000efd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7702966cba0f64 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.624+0000 7f794d7fa640 1 -- 172.21.15.28:0/739161624 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796002b070 con 0x55dda35f6630 2024-05-09T08:38:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.624+0000 7f794d7fa640 1 -- 172.21.15.28:0/739161624 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7960013cf0 con 0x55dda35f6630 2024-05-09T08:38:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.624+0000 7f794d7fa640 1 -- 172.21.15.28:0/739161624 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960018040 con 0x55dda35f6630 2024-05-09T08:38:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.625+0000 7f7970da1740 1 -- 172.21.15.28:0/739161624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35f6630 msgr2=0x55dda360e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.625+0000 7f7970da1740 1 --2- 172.21.15.28:0/739161624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35f6630 0x55dda360e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7960004b90 tx=0x7f796000efd0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.625+0000 7f7970da1740 1 -- 172.21.15.28:0/739161624 shutdown_connections 2024-05-09T08:38:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.626+0000 7f7970da1740 1 --2- 172.21.15.28:0/739161624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35f6630 0x55dda360e090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.626+0000 7f7970da1740 1 --2- 172.21.15.28:0/739161624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.626+0000 7f7970da1740 1 --2- 172.21.15.28:0/739161624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35eb450 0x55dda3602120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.626+0000 7f7970da1740 1 -- 172.21.15.28:0/739161624 >> 172.21.15.28:0/739161624 conn(0x55dda35e7960 msgr2=0x55dda35f3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.632+0000 7f7970da1740 1 -- 172.21.15.28:0/739161624 shutdown_connections 2024-05-09T08:38:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.634+0000 7f7970da1740 1 -- 172.21.15.28:0/739161624 wait complete. 2024-05-09T08:38:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.635+0000 7f7970da1740 1 -- start start 2024-05-09T08:38:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.635+0000 7f7970da1740 1 Processor -- start 2024-05-09T08:38:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.636+0000 7f7970da1740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35fe630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.636+0000 7f7970da1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35feb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.636+0000 7f79677fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35fe630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.636+0000 7f79677fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35fe630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:45170/0 (socket says 172.21.15.28:45170) 2024-05-09T08:38:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.636+0000 7f79677fe640 1 -- 172.21.15.28:0/1027032982 learned_addr learned my addr 172.21.15.28:0/1027032982 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.636+0000 7f7970da1740 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35fb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.636+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b7450 con 0x55dda35ebd70 2024-05-09T08:38:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.637+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b6cc0 con 0x55dda35f6630 2024-05-09T08:38:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.637+0000 7f7967fff640 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35fb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.637+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f796811d660 con 0x55dda35eb450 2024-05-09T08:38:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.637+0000 7f7970da1740 10 client.0 _ll_register_callbacks cb 0x7f795c010510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.637+0000 7f79677fe640 1 -- 172.21.15.28:0/1027032982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 msgr2=0x55dda35fb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.637+0000 7f79677fe640 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35fb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.637+0000 7f79677fe640 1 -- 172.21.15.28:0/1027032982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 msgr2=0x55dda35feb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.637+0000 7f79677fe640 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35feb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.637+0000 7f79677fe640 1 -- 172.21.15.28:0/1027032982 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dda35fbbf0 con 0x55dda35eb450 2024-05-09T08:38:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.638+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7960013cf0 con 0x55dda35eb450 2024-05-09T08:38:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.638+0000 7f79677fe640 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35fe630 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7958009900 tx=0x7f795803d070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab90ceacb1a07ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.639+0000 7f7946ffd640 10 client.26365 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:38:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.642+0000 7f7946ffd640 1 -- 172.21.15.28:0/1027032982 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958035050 con 0x55dda35eb450 2024-05-09T08:38:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.642+0000 7f7946ffd640 1 -- 172.21.15.28:0/1027032982 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7f7958062d50 con 0x55dda35eb450 2024-05-09T08:38:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.642+0000 7f7946ffd640 1 -- 172.21.15.28:0/1027032982 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f7958062380 con 0x55dda35eb450 2024-05-09T08:38:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.644+0000 7f7970da1740 10 client.26365 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.644+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=65}) -- 0x55dda366d110 con 0x55dda35eb450 2024-05-09T08:38:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.646+0000 7f7946ffd640 1 -- 172.21.15.28:0/1027032982 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f7958062520 con 0x55dda35eb450 2024-05-09T08:38:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.646+0000 7f7970da1740 10 client.26365 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.646+0000 7f7970da1740 10 client.26365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.646+0000 7f7970da1740 20 client.26365 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.646+0000 7f7970da1740 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dda3606410 0x55dda366f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.646+0000 7f7970da1740 4 client.26365 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.646+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dda3557dc0 con 0x55dda3606410 2024-05-09T08:38:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.649+0000 7f7966ffd640 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dda3606410 0x55dda366f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.655+0000 7f7966ffd640 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dda3606410 0x55dda366f8c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b18186f91785323e server_cookie=bdfd650b95f7298b in_seq=0 out_seq=0 2024-05-09T08:38:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:27.656+0000 7f7946ffd640 10 client.26365 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:27.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:27 smithi028 ceph-mon[30119]: pgmap v388: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 860 KiB/s wr, 304 op/s 2024-05-09T08:38:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:27 smithi165 ceph-mon[34540]: pgmap v388: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 860 KiB/s wr, 304 op/s 2024-05-09T08:38:28.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:27 smithi184 ceph-mon[34509]: pgmap v388: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 860 KiB/s wr, 304 op/s 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.695+0000 7f7946ffd640 1 -- 172.21.15.28:0/1027032982 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dda3557dc0 con 0x55dda3606410 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.695+0000 7f7946ffd640 10 client.26365 handle_command_reply: tid=0 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.695+0000 7f7970da1740 1 client.26365 shutdown 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.695+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 msgr2=0x55dda35fe630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.695+0000 7f7970da1740 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35fe630 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7958009900 tx=0x7f795803d070 comp rx=0 tx=0).stop 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.695+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 shutdown_connections 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.695+0000 7f7970da1740 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55dda3606410 0x55dda366f8c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.696+0000 7f7970da1740 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55dda35f6630 0x55dda35fb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.696+0000 7f7946ffd640 0 client.26365 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.696+0000 7f7970da1740 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dda35ebd70 0x55dda35feb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.696+0000 7f7970da1740 1 --2- 172.21.15.28:0/1027032982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dda35eb450 0x55dda35fe630 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.696+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 >> 172.21.15.28:0/1027032982 conn(0x55dda35e7960 msgr2=0x55dda35f3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.696+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 shutdown_connections 2024-05-09T08:38:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.696+0000 7f7970da1740 1 -- 172.21.15.28:0/1027032982 wait complete. 2024-05-09T08:38:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.696+0000 7f7970da1740 20 client.26365 trim_cache size 0 max 16384 2024-05-09T08:38:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:38:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:38:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:38:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:38:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 -- 172.21.15.28:0/3470458949 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79500a2ea0 msgr2=0x7f79500a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79500a2ea0 0x7f79500a5360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7958009230 tx=0x7f7958005fd0 comp rx=0 tx=0).stop 2024-05-09T08:38:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 -- 172.21.15.28:0/3470458949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79681010d0 msgr2=0x7f796810a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79681010d0 0x7f796810a0d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f795c009fd0 tx=0x7f795c00b870 comp rx=0 tx=0).stop 2024-05-09T08:38:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 -- 172.21.15.28:0/3470458949 shutdown_connections 2024-05-09T08:38:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796810a610 0x7f79680732a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79681010d0 0x7f796810a0d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79500a2ea0 0x7f79500a5360 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 --2- 172.21.15.28:0/3470458949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7968100710 0x7f7968109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 -- 172.21.15.28:0/3470458949 >> 172.21.15.28:0/3470458949 conn(0x7f79680fc200 msgr2=0x7f79680fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.698+0000 7f796dfb7640 1 -- 172.21.15.28:0/3470458949 shutdown_connections 2024-05-09T08:38:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:28.699+0000 7f796dfb7640 1 -- 172.21.15.28:0/3470458949 wait complete. 2024-05-09T08:38:29.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:29 smithi028 ceph-mon[30119]: pgmap v389: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 314 op/s 2024-05-09T08:38:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:29 smithi165 ceph-mon[34540]: pgmap v389: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 314 op/s 2024-05-09T08:38:30.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:29 smithi184 ceph-mon[34509]: pgmap v389: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 314 op/s 2024-05-09T08:38:31.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:31 smithi028 ceph-mon[30119]: pgmap v390: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 887 KiB/s wr, 333 op/s 2024-05-09T08:38:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:31 smithi165 ceph-mon[34540]: pgmap v390: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 887 KiB/s wr, 333 op/s 2024-05-09T08:38:32.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:31 smithi184 ceph-mon[34509]: pgmap v390: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 887 KiB/s wr, 333 op/s 2024-05-09T08:38:34.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:33 smithi165 ceph-mon[34540]: pgmap v391: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1003 KiB/s wr, 360 op/s 2024-05-09T08:38:34.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:33 smithi184 ceph-mon[34509]: pgmap v391: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1003 KiB/s wr, 360 op/s 2024-05-09T08:38:34.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:33 smithi028 ceph-mon[30119]: pgmap v391: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1003 KiB/s wr, 360 op/s 2024-05-09T08:38:36.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:35 smithi165 ceph-mon[34540]: pgmap v392: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 968 KiB/s wr, 349 op/s 2024-05-09T08:38:36.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:35 smithi184 ceph-mon[34509]: pgmap v392: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 968 KiB/s wr, 349 op/s 2024-05-09T08:38:36.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:35 smithi028 ceph-mon[30119]: pgmap v392: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 968 KiB/s wr, 349 op/s 2024-05-09T08:38:38.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:37 smithi165 ceph-mon[34540]: pgmap v393: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 358 op/s 2024-05-09T08:38:38.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:37 smithi184 ceph-mon[34509]: pgmap v393: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 358 op/s 2024-05-09T08:38:38.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:37 smithi028 ceph-mon[30119]: pgmap v393: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 358 op/s 2024-05-09T08:38:40.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:39 smithi165 ceph-mon[34540]: pgmap v394: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 310 op/s 2024-05-09T08:38:40.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:39 smithi184 ceph-mon[34509]: pgmap v394: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 310 op/s 2024-05-09T08:38:40.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:39 smithi028 ceph-mon[30119]: pgmap v394: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 310 op/s 2024-05-09T08:38:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.860+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3812375529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 msgr2=0x7fc5b4073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.860+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3812375529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 0x7fc5b4073fb0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009ca0 tx=0x7fc5ac02a980 comp rx=0 tx=0).stop 2024-05-09T08:38:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.868+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3812375529 shutdown_connections 2024-05-09T08:38:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.868+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3812375529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b4112850 0x7fc5b4114c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.868+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3812375529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5b40744f0 0x7fc5b4112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.868+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3812375529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 0x7fc5b4073fb0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.868+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3812375529 >> 172.21.15.28:0/3812375529 conn(0x7fc5b40fd2f0 msgr2=0x7fc5b40ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.869+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3812375529 shutdown_connections 2024-05-09T08:38:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.870+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3812375529 wait complete. 2024-05-09T08:38:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.871+0000 7fc5bae5d640 1 Processor -- start 2024-05-09T08:38:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.874+0000 7fc5bae5d640 1 -- start start 2024-05-09T08:38:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.874+0000 7fc5bae5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 0x7fc5b40734b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.874+0000 7fc5bae5d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5b40744f0 0x7fc5b4070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.874+0000 7fc5bae5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b4112850 0x7fc5b4070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.874+0000 7fc5bae5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b4106530 con 0x7fc5b4073bd0 2024-05-09T08:38:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.874+0000 7fc5bae5d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b41063b0 con 0x7fc5b4112850 2024-05-09T08:38:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.874+0000 7fc5bae5d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b41066b0 con 0x7fc5b40744f0 2024-05-09T08:38:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.875+0000 7fc5b8bf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 0x7fc5b40734b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.875+0000 7fc5b8bf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 0x7fc5b40734b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45008/0 (socket says 172.21.15.28:45008) 2024-05-09T08:38:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.875+0000 7fc5b8bf9640 1 -- 172.21.15.28:0/3374939481 learned_addr learned my addr 172.21.15.28:0/3374939481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.875+0000 7fc5b3fff640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5b40744f0 0x7fc5b4070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.876+0000 7fc5b93fa640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b4112850 0x7fc5b4070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.878+0000 7fc5b8bf9640 1 -- 172.21.15.28:0/3374939481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5b40744f0 msgr2=0x7fc5b4070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.878+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5b40744f0 0x7fc5b4070550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.878+0000 7fc5b8bf9640 1 -- 172.21.15.28:0/3374939481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b4112850 msgr2=0x7fc5b4070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.878+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b4112850 0x7fc5b4070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.878+0000 7fc5b8bf9640 1 -- 172.21.15.28:0/3374939481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b410fd50 con 0x7fc5b4073bd0 2024-05-09T08:38:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.878+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 0x7fc5b40734b0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac006010 tx=0x7fc5ac0308b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.879+0000 7fc5b1ffb640 1 -- 172.21.15.28:0/3374939481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac034050 con 0x7fc5b4073bd0 2024-05-09T08:38:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.879+0000 7fc5b1ffb640 1 -- 172.21.15.28:0/3374939481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5ac004040 con 0x7fc5b4073bd0 2024-05-09T08:38:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.881+0000 7fc5b1ffb640 1 -- 172.21.15.28:0/3374939481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac039710 con 0x7fc5b4073bd0 2024-05-09T08:38:41.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.884+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3374939481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b410ffe0 con 0x7fc5b4073bd0 2024-05-09T08:38:41.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.885+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3374939481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b4110450 con 0x7fc5b4073bd0 2024-05-09T08:38:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.893+0000 7fc5bdb69740 1 Processor -- start 2024-05-09T08:38:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5bdb69740 1 -- start start 2024-05-09T08:38:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67dedc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5b1ffb640 1 -- 172.21.15.28:0/3374939481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc5ac037020 con 0x7fc5b4073bd0 2024-05-09T08:38:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dee7630 0x55e67dee9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e67de41340 con 0x55e67dedcd70 2024-05-09T08:38:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e67dde9290 con 0x55e67dee7630 2024-05-09T08:38:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e67de3d300 con 0x55e67dedc450 2024-05-09T08:38:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5b93fa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67dedc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5b93fa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67dedc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48716/0 (socket says 172.21.15.28:48716) 2024-05-09T08:38:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.894+0000 7fc5b93fa640 1 -- 172.21.15.28:0/259408926 learned_addr learned my addr 172.21.15.28:0/259408926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.895+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/259408926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.901+0000 7fc5b1ffb640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc59c0a2ea0 0x7fc59c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.905+0000 7fc5b8bf9640 1 -- 172.21.15.28:0/259408926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 msgr2=0x55e67dedc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.905+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/259408926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67dedc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.905+0000 7fc5b8bf9640 1 -- 172.21.15.28:0/259408926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dee7630 msgr2=0x55e67dee9a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:38:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.905+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/259408926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dee7630 0x55e67dee9a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.905+0000 7fc5b8bf9640 1 -- 172.21.15.28:0/259408926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e67de0d8e0 con 0x55e67dedcd70 2024-05-09T08:38:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.905+0000 7fc5b93fa640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc59c0a2ea0 0x7fc59c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.906+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/259408926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee70f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0ede50 tx=0x7fc5ac0f9240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55da9f69b0fb01a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.907+0000 7fc59affd640 1 -- 172.21.15.28:0/259408926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0b8ec0 con 0x55e67dedcd70 2024-05-09T08:38:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.908+0000 7fc59affd640 1 -- 172.21.15.28:0/259408926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5ac0bd320 con 0x55e67dedcd70 2024-05-09T08:38:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.909+0000 7fc5bdb69740 1 -- 172.21.15.28:0/259408926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 msgr2=0x55e67dee70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.909+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/259408926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee70f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0ede50 tx=0x7fc5ac0f9240 comp rx=0 tx=0).stop 2024-05-09T08:38:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.910+0000 7fc5b93fa640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc59c0a2ea0 0x7fc59c0a5360 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8005fd0 tx=0x7fc5a8004730 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.910+0000 7fc5bdb69740 1 -- 172.21.15.28:0/259408926 shutdown_connections 2024-05-09T08:38:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.910+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/259408926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dee7630 0x55e67dee9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.910+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/259408926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee70f0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.910+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/259408926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67dedc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.910+0000 7fc5bdb69740 1 -- 172.21.15.28:0/259408926 >> 172.21.15.28:0/259408926 conn(0x55e67ded8960 msgr2=0x55e67ded8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.910+0000 7fc5b1ffb640 1 -- 172.21.15.28:0/3374939481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7fc5ac0f0bd0 con 0x7fc5b4073bd0 2024-05-09T08:38:41.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.917+0000 7fc5bdb69740 1 -- 172.21.15.28:0/259408926 shutdown_connections 2024-05-09T08:38:41.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.919+0000 7fc5bdb69740 1 -- 172.21.15.28:0/259408926 wait complete. 2024-05-09T08:38:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.919+0000 7fc5bdb69740 1 -- start start 2024-05-09T08:38:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.919+0000 7fc5bdb69740 1 Processor -- start 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67dee45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dedcd70 0x55e67dee4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dee7630 0x55e67dee5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e67de41340 con 0x55e67dee7630 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e67ddfc790 con 0x55e67dedcd70 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e67deea020 con 0x55e67dedc450 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5bdb69740 10 client.0 _ll_register_callbacks cb 0x55e67de37a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5b93fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dee7630 0x55e67dee5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5b93fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dee7630 0x55e67dee5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45024/0 (socket says 172.21.15.28:45024) 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.920+0000 7fc5b93fa640 1 -- 172.21.15.28:0/3301553093 learned_addr learned my addr 172.21.15.28:0/3301553093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.921+0000 7fc5b93fa640 1 -- 172.21.15.28:0/3301553093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 msgr2=0x55e67dee45e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.921+0000 7fc5b93fa640 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67dee45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.921+0000 7fc5b93fa640 1 -- 172.21.15.28:0/3301553093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dedcd70 msgr2=0x55e67dee4b20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:38:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.921+0000 7fc5b93fa640 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dedcd70 0x55e67dee4b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.921+0000 7fc5b93fa640 1 -- 172.21.15.28:0/3301553093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e67db55210 con 0x55e67dee7630 2024-05-09T08:38:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.922+0000 7fc5b93fa640 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dee7630 0x55e67dee5060 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800fde0 tx=0x7fc5a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c815e24809219e8a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.922+0000 7fc593fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:38:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.922+0000 7fc593fff640 1 -- 172.21.15.28:0/3301553093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8018070 con 0x55e67dee7630 2024-05-09T08:38:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.923+0000 7fc5bdb69740 1 -- 172.21.15.28:0/3301553093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e67de4ea70 con 0x55e67dee7630 2024-05-09T08:38:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.923+0000 7fc593fff640 1 -- 172.21.15.28:0/3301553093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7fc5a8010c10 con 0x55e67dee7630 2024-05-09T08:38:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.923+0000 7fc593fff640 1 -- 172.21.15.28:0/3301553093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fc5a8010e10 con 0x55e67dee7630 2024-05-09T08:38:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.924+0000 7fc5bdb69740 10 client.17007 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.924+0000 7fc5bdb69740 1 -- 172.21.15.28:0/3301553093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55e67dc21620 con 0x55e67dee7630 2024-05-09T08:38:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.925+0000 7fc593fff640 1 -- 172.21.15.28:0/3301553093 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e67de4ea70 con 0x55e67dee7630 2024-05-09T08:38:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.925+0000 7fc593fff640 1 -- 172.21.15.28:0/3301553093 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7fc5a801e620 con 0x55e67dee7630 2024-05-09T08:38:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.925+0000 7fc5bdb69740 10 client.17007 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:41.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.925+0000 7fc5bdb69740 10 client.17007 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.925+0000 7fc5bdb69740 20 client.17007 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.925+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e67def7220 0x55e67def96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.925+0000 7fc5bdb69740 4 client.17007 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.925+0000 7fc5bdb69740 1 -- 172.21.15.28:0/3301553093 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e67de48dc0 con 0x55e67def7220 2024-05-09T08:38:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.927+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e67def7220 0x55e67def96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.928+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e67def7220 0x55e67def96e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e9afbb4679ae331 server_cookie=59b0172f88c564d3 in_seq=0 out_seq=0 2024-05-09T08:38:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.928+0000 7fc593fff640 10 client.17007 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.932+0000 7fc593fff640 1 -- 172.21.15.28:0/3301553093 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e67de48dc0 con 0x55e67def7220 2024-05-09T08:38:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.932+0000 7fc593fff640 10 client.17007 handle_command_reply: tid=0 2024-05-09T08:38:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.932+0000 7fc5bdb69740 1 client.17007 shutdown 2024-05-09T08:38:41.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.935+0000 7fc5bdb69740 1 -- 172.21.15.28:0/3301553093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dee7630 msgr2=0x55e67dee5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:41.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.935+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dee7630 0x55e67dee5060 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800fde0 tx=0x7fc5a80046c0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.937+0000 7fc5bdb69740 1 -- 172.21.15.28:0/3301553093 shutdown_connections 2024-05-09T08:38:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.937+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e67def7220 0x55e67def96e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.937+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dee7630 0x55e67dee5060 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.937+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dedcd70 0x55e67dee4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.937+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/3301553093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67dee45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.937+0000 7fc5bdb69740 1 -- 172.21.15.28:0/3301553093 >> 172.21.15.28:0/3301553093 conn(0x55e67ded8960 msgr2=0x55e67dede710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.937+0000 7fc593fff640 0 client.17007 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.938+0000 7fc5bdb69740 1 -- 172.21.15.28:0/3301553093 shutdown_connections 2024-05-09T08:38:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.938+0000 7fc5bdb69740 1 -- 172.21.15.28:0/3301553093 wait complete. 2024-05-09T08:38:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.939+0000 7fc5bdb69740 20 client.17007 trim_cache size 0 max 16384 2024-05-09T08:38:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.967+0000 7fc5bdb69740 1 Processor -- start 2024-05-09T08:38:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.967+0000 7fc5bdb69740 1 -- start start 2024-05-09T08:38:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.967+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67def2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.968+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.968+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dee7630 0x55e67dedee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.968+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b41c6360 con 0x55e67dedcd70 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.968+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b41c80e0 con 0x55e67dee7630 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.968+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b41c6af0 con 0x55e67dedc450 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.968+0000 7fc5b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.968+0000 7fc5b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45040/0 (socket says 172.21.15.28:45040) 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.968+0000 7fc5b3fff640 1 -- 172.21.15.28:0/4142304612 learned_addr learned my addr 172.21.15.28:0/4142304612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.969+0000 7fc5b3fff640 1 -- 172.21.15.28:0/4142304612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 msgr2=0x55e67def2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.969+0000 7fc5b3fff640 1 --2- 172.21.15.28:0/4142304612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67def2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.969+0000 7fc5b3fff640 1 -- 172.21.15.28:0/4142304612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dee7630 msgr2=0x55e67dedee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.969+0000 7fc5b3fff640 1 --2- 172.21.15.28:0/4142304612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dee7630 0x55e67dedee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.969+0000 7fc5b3fff640 1 -- 172.21.15.28:0/4142304612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e67dee3590 con 0x55e67dedcd70 2024-05-09T08:38:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.969+0000 7fc5b3fff640 1 --2- 172.21.15.28:0/4142304612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee2eb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4071da0 tx=0x7fc5a400aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5f32798a59746f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.970+0000 7fc59a7fc640 1 -- 172.21.15.28:0/4142304612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a400fe20 con 0x55e67dedcd70 2024-05-09T08:38:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.970+0000 7fc59a7fc640 1 -- 172.21.15.28:0/4142304612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5a4005500 con 0x55e67dedcd70 2024-05-09T08:38:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.970+0000 7fc59a7fc640 1 -- 172.21.15.28:0/4142304612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40186a0 con 0x55e67dedcd70 2024-05-09T08:38:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.974+0000 7fc5bdb69740 1 -- 172.21.15.28:0/4142304612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 msgr2=0x55e67dee2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.974+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/4142304612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee2eb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4071da0 tx=0x7fc5a400aeb0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.974+0000 7fc5bdb69740 1 -- 172.21.15.28:0/4142304612 shutdown_connections 2024-05-09T08:38:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.974+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/4142304612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dee7630 0x55e67dedee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.974+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/4142304612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedcd70 0x55e67dee2eb0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.974+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/4142304612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dedc450 0x55e67def2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.974+0000 7fc5bdb69740 1 -- 172.21.15.28:0/4142304612 >> 172.21.15.28:0/4142304612 conn(0x55e67ded8960 msgr2=0x55e67ded7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:41.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.976+0000 7fc5bdb69740 1 -- 172.21.15.28:0/4142304612 shutdown_connections 2024-05-09T08:38:41.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.976+0000 7fc5bdb69740 1 -- 172.21.15.28:0/4142304612 wait complete. 2024-05-09T08:38:41.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.976+0000 7fc5bdb69740 1 -- start start 2024-05-09T08:38:41.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.976+0000 7fc5bdb69740 1 Processor -- start 2024-05-09T08:38:41.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.977+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedc450 0x55e67deef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.977+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dedcd70 0x55e67deef7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.977+0000 7fc5bdb69740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dee7630 0x55e67dee0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.977+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b41c7a10 con 0x55e67dedc450 2024-05-09T08:38:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.977+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b41c7280 con 0x55e67dedcd70 2024-05-09T08:38:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.977+0000 7fc5bdb69740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b41c4360 con 0x55e67dee7630 2024-05-09T08:38:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.977+0000 7fc5bdb69740 10 client.0 _ll_register_callbacks cb 0x7fc5a800cda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b93fa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dee7630 0x55e67dee0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b93fa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dee7630 0x55e67dee0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48732/0 (socket says 172.21.15.28:48732) 2024-05-09T08:38:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b93fa640 1 -- 172.21.15.28:0/1435750781 learned_addr learned my addr 172.21.15.28:0/1435750781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedc450 0x55e67deef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b3fff640 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dedcd70 0x55e67deef7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b93fa640 1 -- 172.21.15.28:0/1435750781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dedcd70 msgr2=0x55e67deef7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b93fa640 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dedcd70 0x55e67deef7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b93fa640 1 -- 172.21.15.28:0/1435750781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedc450 msgr2=0x55e67deef270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b93fa640 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedc450 0x55e67deef270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.978+0000 7fc5b93fa640 1 -- 172.21.15.28:0/1435750781 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e67dee14f0 con 0x55e67dee7630 2024-05-09T08:38:41.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.979+0000 7fc5b93fa640 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dee7630 0x55e67dee0cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800ce70 tx=0x7fc5a801eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d96bdea70c2140c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:41.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.979+0000 7fc593fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:38:41.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.979+0000 7fc593fff640 1 -- 172.21.15.28:0/1435750781 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8018040 con 0x55e67dee7630 2024-05-09T08:38:41.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.980+0000 7fc593fff640 1 -- 172.21.15.28:0/1435750781 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7fc5a8010c10 con 0x55e67dee7630 2024-05-09T08:38:41.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.980+0000 7fc5bdb69740 1 -- 172.21.15.28:0/1435750781 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5a4005500 con 0x55e67dee7630 2024-05-09T08:38:41.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.982+0000 7fc593fff640 1 -- 172.21.15.28:0/1435750781 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fc5a801e420 con 0x55e67dee7630 2024-05-09T08:38:41.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.982+0000 7fc5bdb69740 10 client.26374 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:41.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.982+0000 7fc5bdb69740 1 -- 172.21.15.28:0/1435750781 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x55e67df5d3b0 con 0x55e67dee7630 2024-05-09T08:38:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.983+0000 7fc593fff640 1 -- 172.21.15.28:0/1435750781 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5a4005500 con 0x55e67dee7630 2024-05-09T08:38:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.983+0000 7fc593fff640 1 -- 172.21.15.28:0/1435750781 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7fc5a80035f0 con 0x55e67dee7630 2024-05-09T08:38:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.983+0000 7fc5bdb69740 10 client.26374 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.983+0000 7fc5bdb69740 10 client.26374 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.983+0000 7fc5bdb69740 20 client.26374 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.983+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e67df5dba0 0x55e67df5ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.983+0000 7fc5bdb69740 4 client.26374 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.983+0000 7fc5bdb69740 1 -- 172.21.15.28:0/1435750781 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e67ddc68e0 con 0x55e67df5dba0 2024-05-09T08:38:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.984+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e67df5dba0 0x55e67df5ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.985+0000 7fc5b8bf9640 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e67df5dba0 0x55e67df5ffc0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b163761c2c333b3d server_cookie=be002f338437cf78 in_seq=0 out_seq=0 2024-05-09T08:38:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:41.985+0000 7fc593fff640 10 client.26374 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:42.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:41 smithi165 ceph-mon[34540]: pgmap v395: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 850 KiB/s wr, 275 op/s 2024-05-09T08:38:42.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:41 smithi184 ceph-mon[34509]: pgmap v395: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 850 KiB/s wr, 275 op/s 2024-05-09T08:38:42.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:41 smithi028 ceph-mon[30119]: pgmap v395: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 850 KiB/s wr, 275 op/s 2024-05-09T08:38:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.294+0000 7fc593fff640 1 -- 172.21.15.28:0/1435750781 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e67ddc68e0 con 0x55e67df5dba0 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc593fff640 10 client.26374 handle_command_reply: tid=0 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc5bdb69740 1 client.26374 shutdown 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc5bdb69740 1 -- 172.21.15.28:0/1435750781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dee7630 msgr2=0x55e67dee0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dee7630 0x55e67dee0cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800ce70 tx=0x7fc5a801eeb0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc5bdb69740 1 -- 172.21.15.28:0/1435750781 shutdown_connections 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e67df5dba0 0x55e67df5ffc0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e67dee7630 0x55e67dee0cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e67dedcd70 0x55e67deef7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc5bdb69740 1 --2- 172.21.15.28:0/1435750781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e67dedc450 0x55e67deef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc5bdb69740 1 -- 172.21.15.28:0/1435750781 >> 172.21.15.28:0/1435750781 conn(0x55e67ded8960 msgr2=0x55e67ded7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.295+0000 7fc593fff640 0 client.26374 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.296+0000 7fc5bdb69740 1 -- 172.21.15.28:0/1435750781 shutdown_connections 2024-05-09T08:38:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.296+0000 7fc5bdb69740 1 -- 172.21.15.28:0/1435750781 wait complete. 2024-05-09T08:38:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.296+0000 7fc5bdb69740 20 client.26374 trim_cache size 0 max 16384 2024-05-09T08:38:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:38:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:38:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:38:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:38:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc59c0a2ea0 msgr2=0x7fc59c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc59c0a2ea0 0x7fc59c0a5360 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8005fd0 tx=0x7fc5a8004730 comp rx=0 tx=0).stop 2024-05-09T08:38:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 msgr2=0x7fc5b40734b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 0x7fc5b40734b0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac006010 tx=0x7fc5ac0308b0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3374939481 shutdown_connections 2024-05-09T08:38:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b4112850 0x7fc5b4070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc59c0a2ea0 0x7fc59c0a5360 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5b40744f0 0x7fc5b4070550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 --2- 172.21.15.28:0/3374939481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5b4073bd0 0x7fc5b40734b0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3374939481 >> 172.21.15.28:0/3374939481 conn(0x7fc5b40fd2f0 msgr2=0x7fc5b40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3374939481 shutdown_connections 2024-05-09T08:38:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:42.297+0000 7fc5bae5d640 1 -- 172.21.15.28:0/3374939481 wait complete. 2024-05-09T08:38:44.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:43 smithi165 ceph-mon[34540]: pgmap v396: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-05-09T08:38:44.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:43 smithi184 ceph-mon[34509]: pgmap v396: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-05-09T08:38:44.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:43 smithi028 ceph-mon[30119]: pgmap v396: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 271 op/s 2024-05-09T08:38:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:45 smithi165 ceph-mon[34540]: pgmap v397: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 220 op/s 2024-05-09T08:38:46.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:45 smithi184 ceph-mon[34509]: pgmap v397: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 220 op/s 2024-05-09T08:38:46.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:45 smithi028 ceph-mon[30119]: pgmap v397: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 220 op/s 2024-05-09T08:38:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:47 smithi165 ceph-mon[34540]: pgmap v398: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 238 op/s 2024-05-09T08:38:48.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:47 smithi184 ceph-mon[34509]: pgmap v398: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 238 op/s 2024-05-09T08:38:48.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:47 smithi028 ceph-mon[30119]: pgmap v398: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 238 op/s 2024-05-09T08:38:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:49 smithi165 ceph-mon[34540]: pgmap v399: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 208 op/s 2024-05-09T08:38:50.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:49 smithi184 ceph-mon[34509]: pgmap v399: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 208 op/s 2024-05-09T08:38:50.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:49 smithi028 ceph-mon[30119]: pgmap v399: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 208 op/s 2024-05-09T08:38:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:51 smithi165 ceph-mon[34540]: pgmap v400: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 206 op/s 2024-05-09T08:38:52.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:51 smithi184 ceph-mon[34509]: pgmap v400: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 206 op/s 2024-05-09T08:38:52.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:51 smithi028 ceph-mon[30119]: pgmap v400: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 206 op/s 2024-05-09T08:38:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:53 smithi165 ceph-mon[34540]: pgmap v401: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 223 op/s 2024-05-09T08:38:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:53 smithi184 ceph-mon[34509]: pgmap v401: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 223 op/s 2024-05-09T08:38:54.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:53 smithi028 ceph-mon[30119]: pgmap v401: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 223 op/s 2024-05-09T08:38:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.537+0000 7f00452e7640 1 -- 172.21.15.28:0/3185551078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 msgr2=0x7f00380b9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.537+0000 7f00452e7640 1 --2- 172.21.15.28:0/3185551078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 0x7f00380b9e90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0040066fa0 tx=0x7f004008d8b0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.538+0000 7f00452e7640 1 -- 172.21.15.28:0/3185551078 shutdown_connections 2024-05-09T08:38:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.538+0000 7f00452e7640 1 --2- 172.21.15.28:0/3185551078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 0x7f00380b9e90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.538+0000 7f00452e7640 1 --2- 172.21.15.28:0/3185551078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00380a5730 0x7f00380b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.538+0000 7f00452e7640 1 --2- 172.21.15.28:0/3185551078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00380a4e10 0x7f00380a51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.538+0000 7f00452e7640 1 -- 172.21.15.28:0/3185551078 >> 172.21.15.28:0/3185551078 conn(0x7f003801a670 msgr2=0x7f003801aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.539+0000 7f00452e7640 1 -- 172.21.15.28:0/3185551078 shutdown_connections 2024-05-09T08:38:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.542+0000 7f00452e7640 1 -- 172.21.15.28:0/3185551078 wait complete. 2024-05-09T08:38:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.542+0000 7f00452e7640 1 Processor -- start 2024-05-09T08:38:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.543+0000 7f00452e7640 1 -- start start 2024-05-09T08:38:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.544+0000 7f00452e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00380a4e10 0x7f00381496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.544+0000 7f00452e7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00380a5730 0x7f0038149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.544+0000 7f00452e7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 0x7f0038143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.544+0000 7f00452e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00380bd650 con 0x7f00380a4e10 2024-05-09T08:38:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.544+0000 7f00452e7640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f00380bd4d0 con 0x7f00380a5730 2024-05-09T08:38:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.544+0000 7f00452e7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f00380bd7d0 con 0x7f00380b7aa0 2024-05-09T08:38:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.545+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 0x7f0038143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.545+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 0x7f0038143870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:47098/0 (socket says 172.21.15.28:47098) 2024-05-09T08:38:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.545+0000 7f003f7fe640 1 -- 172.21.15.28:0/2993827263 learned_addr learned my addr 172.21.15.28:0/2993827263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.545+0000 7f003e7fc640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00380a5730 0x7f0038149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.547+0000 7f003f7fe640 1 -- 172.21.15.28:0/2993827263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00380a5730 msgr2=0x7f0038149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.547+0000 7f003f7fe640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00380a5730 0x7f0038149c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.547+0000 7f003f7fe640 1 -- 172.21.15.28:0/2993827263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00380a4e10 msgr2=0x7f00381496c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:38:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.547+0000 7f003f7fe640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00380a4e10 0x7f00381496c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.547+0000 7f003f7fe640 1 -- 172.21.15.28:0/2993827263 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0038143ff0 con 0x7f00380b7aa0 2024-05-09T08:38:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.548+0000 7f003f7fe640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 0x7f0038143870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0040067540 tx=0x7f004006c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:55.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.549+0000 7f0027fff640 1 -- 172.21.15.28:0/2993827263 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400a5070 con 0x7f00380b7aa0 2024-05-09T08:38:55.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.549+0000 7f0027fff640 1 -- 172.21.15.28:0/2993827263 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0040095950 con 0x7f00380b7aa0 2024-05-09T08:38:55.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.549+0000 7f0027fff640 1 -- 172.21.15.28:0/2993827263 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400a07f0 con 0x7f00380b7aa0 2024-05-09T08:38:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.550+0000 7f00452e7640 1 -- 172.21.15.28:0/2993827263 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0038144280 con 0x7f00380b7aa0 2024-05-09T08:38:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.550+0000 7f00452e7640 1 -- 172.21.15.28:0/2993827263 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0038150580 con 0x7f00380b7aa0 2024-05-09T08:38:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.555+0000 7f0027fff640 1 -- 172.21.15.28:0/2993827263 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0040094bb0 con 0x7f00380b7aa0 2024-05-09T08:38:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.556+0000 7f0027fff640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f00280a2e80 0x7f00280a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.557+0000 7f0027fff640 1 -- 172.21.15.28:0/2993827263 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7f0040152390 con 0x7f00380b7aa0 2024-05-09T08:38:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.564+0000 7f003effd640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f00280a2e80 0x7f00280a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.566+0000 7f003effd640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f00280a2e80 0x7f00280a5340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f00300058c0 tx=0x7f0030005850 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.582+0000 7f0047ff0740 1 Processor -- start 2024-05-09T08:38:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.583+0000 7f0047ff0740 1 -- start start 2024-05-09T08:38:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.584+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 0x5601ca77e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.584+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77ef20 0x5601ca7892a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.584+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca78bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.584+0000 7f0047ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5601ca6e3340 con 0x5601ca77e600 2024-05-09T08:38:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.584+0000 7f0047ff0740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5601ca68b290 con 0x5601ca7897e0 2024-05-09T08:38:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.584+0000 7f0047ff0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5601ca6df300 con 0x5601ca77ef20 2024-05-09T08:38:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.584+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca78bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.584+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca78bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:43938/0 (socket says 172.21.15.28:43938) 2024-05-09T08:38:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.584+0000 7f003f7fe640 1 -- 172.21.15.28:0/1264147430 learned_addr learned my addr 172.21.15.28:0/1264147430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.585+0000 7f003f7fe640 1 -- 172.21.15.28:0/1264147430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77ef20 msgr2=0x5601ca7892a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:38:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.585+0000 7f003f7fe640 1 --2- 172.21.15.28:0/1264147430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77ef20 0x5601ca7892a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.585+0000 7f003f7fe640 1 -- 172.21.15.28:0/1264147430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 msgr2=0x5601ca77e9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:38:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.585+0000 7f003f7fe640 1 --2- 172.21.15.28:0/1264147430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 0x5601ca77e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.585+0000 7f003f7fe640 1 -- 172.21.15.28:0/1264147430 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601ca6af8e0 con 0x5601ca7897e0 2024-05-09T08:38:55.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.586+0000 7f003f7fe640 1 --2- 172.21.15.28:0/1264147430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca78bbd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0040000b60 tx=0x7f0040094f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea860a00e1e129fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:55.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.586+0000 7f00257fa640 1 -- 172.21.15.28:0/1264147430 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400943e0 con 0x5601ca7897e0 2024-05-09T08:38:55.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.587+0000 7f00257fa640 1 -- 172.21.15.28:0/1264147430 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0040152180 con 0x5601ca7897e0 2024-05-09T08:38:55.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.588+0000 7f0047ff0740 1 -- 172.21.15.28:0/1264147430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 msgr2=0x5601ca78bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:55.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.588+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1264147430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca78bbd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0040000b60 tx=0x7f0040094f50 comp rx=0 tx=0).stop 2024-05-09T08:38:55.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.588+0000 7f0047ff0740 1 -- 172.21.15.28:0/1264147430 shutdown_connections 2024-05-09T08:38:55.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.588+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1264147430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca78bbd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.588+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1264147430 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77ef20 0x5601ca7892a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.588+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1264147430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 0x5601ca77e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.588+0000 7f0047ff0740 1 -- 172.21.15.28:0/1264147430 >> 172.21.15.28:0/1264147430 conn(0x5601ca77ab10 msgr2=0x5601ca77aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.594+0000 7f0047ff0740 1 -- 172.21.15.28:0/1264147430 shutdown_connections 2024-05-09T08:38:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.595+0000 7f0047ff0740 1 -- 172.21.15.28:0/1264147430 wait complete. 2024-05-09T08:38:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.596+0000 7f0047ff0740 1 -- start start 2024-05-09T08:38:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.596+0000 7f0047ff0740 1 Processor -- start 2024-05-09T08:38:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.597+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 0x5601ca7866a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.597+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77ef20 0x5601ca786be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.597+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 0x5601ca787120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.597+0000 7f0047ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5601ca68b290 con 0x5601ca77e600 2024-05-09T08:38:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.597+0000 7f0047ff0740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5601ca69e790 con 0x5601ca77ef20 2024-05-09T08:38:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.597+0000 7f0047ff0740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5601ca79d1e0 con 0x5601ca7897e0 2024-05-09T08:38:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.597+0000 7f0047ff0740 10 client.0 _ll_register_callbacks cb 0x5601ca6d9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.598+0000 7f003effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 0x5601ca7866a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.598+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 0x5601ca787120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.598+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 0x5601ca787120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:47112/0 (socket says 172.21.15.28:47112) 2024-05-09T08:38:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.598+0000 7f003f7fe640 1 -- 172.21.15.28:0/1163388787 learned_addr learned my addr 172.21.15.28:0/1163388787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.599+0000 7f003effd640 1 -- 172.21.15.28:0/1163388787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 msgr2=0x5601ca787120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.599+0000 7f003effd640 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 0x5601ca787120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.599+0000 7f003effd640 1 -- 172.21.15.28:0/1163388787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77ef20 msgr2=0x5601ca786be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:38:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.599+0000 7f003effd640 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77ef20 0x5601ca786be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.599+0000 7f003effd640 1 -- 172.21.15.28:0/1163388787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601ca3f7210 con 0x5601ca77e600 2024-05-09T08:38:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.599+0000 7f003effd640 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 0x5601ca7866a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f00381449b0 tx=0x7f0030004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baeabb29a5a36510 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.599+0000 7f0022ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:38:55.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.600+0000 7f0022ff5640 1 -- 172.21.15.28:0/1163388787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030021020 con 0x5601ca77e600 2024-05-09T08:38:55.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.600+0000 7f0047ff0740 1 -- 172.21.15.28:0/1163388787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0040152180 con 0x5601ca77e600 2024-05-09T08:38:55.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.600+0000 7f0022ff5640 1 -- 172.21.15.28:0/1163388787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7f003001d640 con 0x5601ca77e600 2024-05-09T08:38:55.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.601+0000 7f0022ff5640 1 -- 172.21.15.28:0/1163388787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f003002c050 con 0x5601ca77e600 2024-05-09T08:38:55.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.601+0000 7f0047ff0740 10 client.17019 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:55.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.601+0000 7f0047ff0740 1 -- 172.21.15.28:0/1163388787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5601ca4c3620 con 0x5601ca77e600 2024-05-09T08:38:55.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.602+0000 7f0022ff5640 1 -- 172.21.15.28:0/1163388787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0040152180 con 0x5601ca77e600 2024-05-09T08:38:55.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.602+0000 7f0022ff5640 1 -- 172.21.15.28:0/1163388787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f003000f860 con 0x5601ca77e600 2024-05-09T08:38:55.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.602+0000 7f0047ff0740 10 client.17019 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:55.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.602+0000 7f0047ff0740 10 client.17019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:55.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.602+0000 7f0047ff0740 20 client.17019 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:55.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.602+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5601ca7aa700 0x5601ca7acb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.602+0000 7f0047ff0740 4 client.17019 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:55.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.602+0000 7f0047ff0740 1 -- 172.21.15.28:0/1163388787 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601ca6f0a70 con 0x5601ca7aa700 2024-05-09T08:38:55.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.603+0000 7f003e7fc640 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5601ca7aa700 0x5601ca7acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.605+0000 7f003e7fc640 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5601ca7aa700 0x5601ca7acb80 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25f81a96ef539ee5 server_cookie=ce1e9845bcc6f881 in_seq=0 out_seq=0 2024-05-09T08:38:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.605+0000 7f0022ff5640 10 client.17019 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.606+0000 7f0022ff5640 1 -- 172.21.15.28:0/1163388787 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5601ca6f0a70 con 0x5601ca7aa700 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.606+0000 7f0022ff5640 10 client.17019 handle_command_reply: tid=0 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.606+0000 7f0047ff0740 1 client.17019 shutdown 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.606+0000 7f0047ff0740 1 -- 172.21.15.28:0/1163388787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 msgr2=0x5601ca7866a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.606+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 0x5601ca7866a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f00381449b0 tx=0x7f0030004730 comp rx=0 tx=0).stop 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.607+0000 7f0047ff0740 1 -- 172.21.15.28:0/1163388787 shutdown_connections 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.607+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5601ca7aa700 0x5601ca7acb80 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.607+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 0x5601ca787120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.607+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77ef20 0x5601ca786be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.607+0000 7f0047ff0740 1 --2- 172.21.15.28:0/1163388787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77e600 0x5601ca7866a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.607+0000 7f0047ff0740 1 -- 172.21.15.28:0/1163388787 >> 172.21.15.28:0/1163388787 conn(0x5601ca77ab10 msgr2=0x5601ca780890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.607+0000 7f0022ff5640 0 client.17019 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.607+0000 7f0047ff0740 1 -- 172.21.15.28:0/1163388787 shutdown_connections 2024-05-09T08:38:55.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.607+0000 7f0047ff0740 1 -- 172.21.15.28:0/1163388787 wait complete. 2024-05-09T08:38:55.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.609+0000 7f0047ff0740 20 client.17019 trim_cache size 0 max 16384 2024-05-09T08:38:55.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.638+0000 7f0047ff0740 1 Processor -- start 2024-05-09T08:38:55.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.638+0000 7f0047ff0740 1 -- start start 2024-05-09T08:38:55.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.639+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77e600 0x5601ca788dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.639+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 0x5601ca780fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.639+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca786f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.641+0000 7f003effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77e600 0x5601ca788dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.641+0000 7f003effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77e600 0x5601ca788dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:47126/0 (socket says 172.21.15.28:47126) 2024-05-09T08:38:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.641+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca786f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.641+0000 7f003f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca786f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:43964/0 (socket says 172.21.15.28:43964) 2024-05-09T08:38:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.641+0000 7f003effd640 1 -- 172.21.15.28:0/3089747908 learned_addr learned my addr 172.21.15.28:0/3089747908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.641+0000 7f0047ff0740 1 -- 172.21.15.28:0/3089747908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00381521b0 con 0x5601ca77ef20 2024-05-09T08:38:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.641+0000 7f0047ff0740 1 -- 172.21.15.28:0/3089747908 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0038153f30 con 0x5601ca7897e0 2024-05-09T08:38:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.641+0000 7f003e7fc640 1 --2- 172.21.15.28:0/3089747908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 0x5601ca780fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.641+0000 7f0047ff0740 1 -- 172.21.15.28:0/3089747908 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0038152940 con 0x5601ca77e600 2024-05-09T08:38:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.642+0000 7f003f7fe640 1 -- 172.21.15.28:0/3089747908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77e600 msgr2=0x5601ca788dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.642+0000 7f003f7fe640 1 --2- 172.21.15.28:0/3089747908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77e600 0x5601ca788dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.642+0000 7f003f7fe640 1 -- 172.21.15.28:0/3089747908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 msgr2=0x5601ca780fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.642+0000 7f003f7fe640 1 --2- 172.21.15.28:0/3089747908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 0x5601ca780fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.642+0000 7f003f7fe640 1 -- 172.21.15.28:0/3089747908 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601ca784bf0 con 0x5601ca7897e0 2024-05-09T08:38:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.642+0000 7f003f7fe640 1 --2- 172.21.15.28:0/3089747908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca786f40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f004011ae30 tx=0x7f004004fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf66f0f9a114b9da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:55.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.644+0000 7f0024ff9640 1 -- 172.21.15.28:0/3089747908 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004011ebf0 con 0x5601ca7897e0 2024-05-09T08:38:55.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.644+0000 7f0024ff9640 1 -- 172.21.15.28:0/3089747908 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f004011e4f0 con 0x5601ca7897e0 2024-05-09T08:38:55.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.644+0000 7f0024ff9640 1 -- 172.21.15.28:0/3089747908 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004011d940 con 0x5601ca7897e0 2024-05-09T08:38:55.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.647+0000 7f0047ff0740 1 -- 172.21.15.28:0/3089747908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 msgr2=0x5601ca786f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:55.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.647+0000 7f0047ff0740 1 --2- 172.21.15.28:0/3089747908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca786f40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f004011ae30 tx=0x7f004004fc00 comp rx=0 tx=0).stop 2024-05-09T08:38:55.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.655+0000 7f0047ff0740 1 -- 172.21.15.28:0/3089747908 shutdown_connections 2024-05-09T08:38:55.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.655+0000 7f0047ff0740 1 --2- 172.21.15.28:0/3089747908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca7897e0 0x5601ca786f40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.655+0000 7f0047ff0740 1 --2- 172.21.15.28:0/3089747908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 0x5601ca780fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.655+0000 7f0047ff0740 1 --2- 172.21.15.28:0/3089747908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca77e600 0x5601ca788dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.655+0000 7f0047ff0740 1 -- 172.21.15.28:0/3089747908 >> 172.21.15.28:0/3089747908 conn(0x5601ca77ab10 msgr2=0x5601ca779320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:55.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.657+0000 7f0047ff0740 1 -- 172.21.15.28:0/3089747908 shutdown_connections 2024-05-09T08:38:55.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.657+0000 7f0047ff0740 1 -- 172.21.15.28:0/3089747908 wait complete. 2024-05-09T08:38:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.658+0000 7f0047ff0740 1 -- start start 2024-05-09T08:38:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.658+0000 7f0047ff0740 1 Processor -- start 2024-05-09T08:38:55.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.658+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77e600 0x5601ca7a2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.659+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 0x5601ca7a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.659+0000 7f0047ff0740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 0x5601ca79d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.659+0000 7f0047ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0038153860 con 0x5601ca77ef20 2024-05-09T08:38:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.659+0000 7f003effd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77e600 0x5601ca7a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.659+0000 7f003effd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77e600 0x5601ca7a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:43980/0 (socket says 172.21.15.28:43980) 2024-05-09T08:38:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.659+0000 7f003effd640 1 -- 172.21.15.28:0/2973727064 learned_addr learned my addr 172.21.15.28:0/2973727064 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:38:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.659+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f00381530d0 con 0x5601ca77e600 2024-05-09T08:38:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.659+0000 7f003f7fe640 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 0x5601ca79d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.660+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0038005170 con 0x5601ca7897e0 2024-05-09T08:38:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.660+0000 7f003e7fc640 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 0x5601ca7a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.660+0000 7f0047ff0740 10 client.0 _ll_register_callbacks cb 0x7f003000b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:55.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.660+0000 7f003effd640 1 -- 172.21.15.28:0/2973727064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 msgr2=0x5601ca79d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:55.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.660+0000 7f003effd640 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 0x5601ca79d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.660+0000 7f003effd640 1 -- 172.21.15.28:0/2973727064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 msgr2=0x5601ca7a28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.660+0000 7f003effd640 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 0x5601ca7a28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.660+0000 7f003effd640 1 -- 172.21.15.28:0/2973727064 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601ca79e0b0 con 0x5601ca77e600 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.660+0000 7f003effd640 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77e600 0x5601ca7a2370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0030004700 tx=0x7f003000f5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=636f436e2dac7609 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.661+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f004011e4f0 con 0x5601ca77e600 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.661+0000 7f0022ff5640 10 client.26519 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.661+0000 7f0022ff5640 1 -- 172.21.15.28:0/2973727064 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030021070 con 0x5601ca77e600 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.661+0000 7f0022ff5640 1 -- 172.21.15.28:0/2973727064 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9655+0+0 (secure 0 0 0) 0x7f003000f960 con 0x5601ca77e600 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.661+0000 7f0022ff5640 1 -- 172.21.15.28:0/2973727064 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f003000da20 con 0x5601ca77e600 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.662+0000 7f0047ff0740 10 client.26519 fetch_fsmap learned FSMap version 65 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.662+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({fsmap=65}) -- 0x5601ca8107a0 con 0x5601ca77e600 2024-05-09T08:38:55.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.662+0000 7f0022ff5640 1 -- 172.21.15.28:0/2973727064 <== mon.2 v2:172.21.15.165:3300/0 4 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f003003a470 con 0x5601ca77e600 2024-05-09T08:38:55.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.664+0000 7f0047ff0740 10 client.26519 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:38:55.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.664+0000 7f0047ff0740 10 client.26519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:38:55.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.664+0000 7f0047ff0740 20 client.26519 populate_metadata read hostname 'smithi028' 2024-05-09T08:38:55.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.664+0000 7f0047ff0740 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5601ca810e80 0x5601ca813320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:55.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.664+0000 7f0047ff0740 4 client.26519 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:55.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.664+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f004011e4f0 con 0x5601ca810e80 2024-05-09T08:38:55.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.665+0000 7f003e7fc640 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 0x5601ca7a28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:55.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.667+0000 7f003e7fc640 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5601ca810e80 0x5601ca813320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.675+0000 7f0022ff5640 10 client.26519 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:55.675+0000 7f003e7fc640 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5601ca810e80 0x5601ca813320 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee52b6b3376a2f56 server_cookie=2f415d729bb74e5e in_seq=0 out_seq=0 2024-05-09T08:38:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.084+0000 7f0022ff5640 1 -- 172.21.15.28:0/2973727064 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f004011e4f0 con 0x5601ca810e80 2024-05-09T08:38:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.084+0000 7f0022ff5640 10 client.26519 handle_command_reply: tid=0 2024-05-09T08:38:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.084+0000 7f0047ff0740 1 client.26519 shutdown 2024-05-09T08:38:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77e600 msgr2=0x5601ca7a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77e600 0x5601ca7a2370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0030004700 tx=0x7f003000f5f0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 shutdown_connections 2024-05-09T08:38:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5601ca810e80 0x5601ca813320 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5601ca7897e0 0x5601ca79d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601ca77ef20 0x5601ca7a28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 --2- 172.21.15.28:0/2973727064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5601ca77e600 0x5601ca7a2370 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 >> 172.21.15.28:0/2973727064 conn(0x5601ca77ab10 msgr2=0x5601ca779bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0022ff5640 0 client.26519 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:38:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 shutdown_connections 2024-05-09T08:38:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 1 -- 172.21.15.28:0/2973727064 wait complete. 2024-05-09T08:38:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.085+0000 7f0047ff0740 20 client.26519 trim_cache size 0 max 16384 2024-05-09T08:38:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:38:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:38:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:38:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:38:56.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 -- 172.21.15.28:0/2993827263 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f00280a2e80 msgr2=0x7f00280a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:56.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f00280a2e80 0x7f00280a5340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f00300058c0 tx=0x7f0030005850 comp rx=0 tx=0).stop 2024-05-09T08:38:56.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 -- 172.21.15.28:0/2993827263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 msgr2=0x7f0038143870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:56.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 0x7f0038143870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0040067540 tx=0x7f004006c9b0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 -- 172.21.15.28:0/2993827263 shutdown_connections 2024-05-09T08:38:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f00280a2e80 0x7f00280a5340 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00380b7aa0 0x7f0038143870 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00380a5730 0x7f0038149c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 --2- 172.21.15.28:0/2993827263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00380a4e10 0x7f00381496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:56.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.087+0000 7f00452e7640 1 -- 172.21.15.28:0/2993827263 >> 172.21.15.28:0/2993827263 conn(0x7f003801a670 msgr2=0x7f00380b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:56.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.089+0000 7f00452e7640 1 -- 172.21.15.28:0/2993827263 shutdown_connections 2024-05-09T08:38:56.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:38:56.089+0000 7f00452e7640 1 -- 172.21.15.28:0/2993827263 wait complete. 2024-05-09T08:38:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:55 smithi165 ceph-mon[34540]: pgmap v402: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 726 KiB/s wr, 215 op/s 2024-05-09T08:38:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:55 smithi184 ceph-mon[34509]: pgmap v402: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 726 KiB/s wr, 215 op/s 2024-05-09T08:38:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:55 smithi028 ceph-mon[30119]: pgmap v402: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 726 KiB/s wr, 215 op/s 2024-05-09T08:38:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:57 smithi165 ceph-mon[34540]: pgmap v403: 129 pgs: 57 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 53 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 284 op/s 2024-05-09T08:38:58.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:57 smithi184 ceph-mon[34509]: pgmap v403: 129 pgs: 57 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 53 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 284 op/s 2024-05-09T08:38:58.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:57 smithi028 ceph-mon[30119]: pgmap v403: 129 pgs: 57 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 53 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 284 op/s 2024-05-09T08:39:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:38:59 smithi165 ceph-mon[34540]: pgmap v404: 129 pgs: 57 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 53 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 890 KiB/s wr, 275 op/s 2024-05-09T08:39:00.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:38:59 smithi184 ceph-mon[34509]: pgmap v404: 129 pgs: 57 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 53 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 890 KiB/s wr, 275 op/s 2024-05-09T08:39:00.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:38:59 smithi028 ceph-mon[30119]: pgmap v404: 129 pgs: 57 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 53 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 890 KiB/s wr, 275 op/s 2024-05-09T08:39:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:01 smithi165 ceph-mon[34540]: pgmap v405: 129 pgs: 57 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 53 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 889 KiB/s wr, 293 op/s 2024-05-09T08:39:02.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:01 smithi184 ceph-mon[34509]: pgmap v405: 129 pgs: 57 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 53 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 889 KiB/s wr, 293 op/s 2024-05-09T08:39:02.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:01 smithi028 ceph-mon[30119]: pgmap v405: 129 pgs: 57 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 53 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 889 KiB/s wr, 293 op/s 2024-05-09T08:39:04.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:03 smithi165 ceph-mon[34540]: pgmap v406: 129 pgs: 55 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 57 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 1020 KiB/s wr, 347 op/s 2024-05-09T08:39:04.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:03 smithi184 ceph-mon[34509]: pgmap v406: 129 pgs: 55 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 57 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 1020 KiB/s wr, 347 op/s 2024-05-09T08:39:04.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:03 smithi028 ceph-mon[30119]: pgmap v406: 129 pgs: 55 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 57 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 1020 KiB/s wr, 347 op/s 2024-05-09T08:39:05.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:04 smithi165 ceph-mon[34540]: osdmap e116: 12 total, 12 up, 12 in 2024-05-09T08:39:05.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:04 smithi184 ceph-mon[34509]: osdmap e116: 12 total, 12 up, 12 in 2024-05-09T08:39:05.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:04 smithi028 ceph-mon[30119]: osdmap e116: 12 total, 12 up, 12 in 2024-05-09T08:39:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:05 smithi165 ceph-mon[34540]: pgmap v408: 129 pgs: 48 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 66 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 374 op/s 2024-05-09T08:39:06.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:05 smithi184 ceph-mon[34509]: pgmap v408: 129 pgs: 48 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 66 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 374 op/s 2024-05-09T08:39:06.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:05 smithi028 ceph-mon[30119]: pgmap v408: 129 pgs: 48 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 66 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 374 op/s 2024-05-09T08:39:07.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:06 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:39:07.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:06 smithi165 ceph-mon[34540]: pgmap v409: 129 pgs: 65 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 42 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 387 op/s 2024-05-09T08:39:07.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:06 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:39:07.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:06 smithi184 ceph-mon[34509]: pgmap v409: 129 pgs: 65 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 42 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 387 op/s 2024-05-09T08:39:07.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:06 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:39:07.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:06 smithi028 ceph-mon[30119]: pgmap v409: 129 pgs: 65 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 42 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 387 op/s 2024-05-09T08:39:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:07 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:39:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:07 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:39:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:07 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:39:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:07 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:39:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:07 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:39:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:07 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:39:08.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:07 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:39:08.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:07 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:39:08.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:07 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:39:09.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:08 smithi165 ceph-mon[34540]: pgmap v410: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 372 op/s 2024-05-09T08:39:09.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:08 smithi184 ceph-mon[34509]: pgmap v410: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 372 op/s 2024-05-09T08:39:09.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:08 smithi028 ceph-mon[30119]: pgmap v410: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 372 op/s 2024-05-09T08:39:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.257+0000 7f32cebb0640 1 -- 172.21.15.28:0/2100136883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c8100760 msgr2=0x7f32c8100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.257+0000 7f32cebb0640 1 --2- 172.21.15.28:0/2100136883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c8100760 0x7f32c8100be0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f32c001ace0 tx=0x7f32c003b970 comp rx=0 tx=0).stop 2024-05-09T08:39:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.260+0000 7f32cebb0640 1 -- 172.21.15.28:0/2100136883 shutdown_connections 2024-05-09T08:39:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.260+0000 7f32cebb0640 1 --2- 172.21.15.28:0/2100136883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8101120 0x7f32c810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.260+0000 7f32cebb0640 1 --2- 172.21.15.28:0/2100136883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c8100760 0x7f32c8100be0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.260+0000 7f32cebb0640 1 --2- 172.21.15.28:0/2100136883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32c8106720 0x7f32c8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.260+0000 7f32cebb0640 1 -- 172.21.15.28:0/2100136883 >> 172.21.15.28:0/2100136883 conn(0x7f32c80778b0 msgr2=0x7f32c8077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:09.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.266+0000 7f32cebb0640 1 -- 172.21.15.28:0/2100136883 shutdown_connections 2024-05-09T08:39:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.269+0000 7f32cebb0640 1 -- 172.21.15.28:0/2100136883 wait complete. 2024-05-09T08:39:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.270+0000 7f32cebb0640 1 Processor -- start 2024-05-09T08:39:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.271+0000 7f32cebb0640 1 -- start start 2024-05-09T08:39:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.271+0000 7f32cebb0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8100760 0x7f32c8073190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.271+0000 7f32cebb0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32c8101120 0x7f32c806d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.271+0000 7f32cebb0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c8106720 0x7f32c806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.271+0000 7f32cebb0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32c806bc60 con 0x7f32c8106720 2024-05-09T08:39:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.271+0000 7f32cebb0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f32c806bae0 con 0x7f32c8100760 2024-05-09T08:39:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.271+0000 7f32cebb0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f32c806bde0 con 0x7f32c8101120 2024-05-09T08:39:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.271+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32c8101120 0x7f32c806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.272+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32c8101120 0x7f32c806d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50036/0 (socket says 172.21.15.28:50036) 2024-05-09T08:39:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.272+0000 7f32c7fff640 1 -- 172.21.15.28:0/3862910704 learned_addr learned my addr 172.21.15.28:0/3862910704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.272+0000 7f32cd14d640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c8106720 0x7f32c806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.273+0000 7f32c7fff640 1 -- 172.21.15.28:0/3862910704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8100760 msgr2=0x7f32c8073190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:39:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.273+0000 7f32c7fff640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8100760 0x7f32c8073190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.273+0000 7f32c7fff640 1 -- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c8106720 msgr2=0x7f32c806d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.273+0000 7f32c7fff640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c8106720 0x7f32c806d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.273+0000 7f32c7fff640 1 -- 172.21.15.28:0/3862910704 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c806dec0 con 0x7f32c8101120 2024-05-09T08:39:09.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.273+0000 7f32c7fff640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32c8101120 0x7f32c806d250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f32c0005c40 tx=0x7f32c00405b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.275+0000 7f32c5ffb640 1 -- 172.21.15.28:0/3862910704 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c0045070 con 0x7f32c8101120 2024-05-09T08:39:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.275+0000 7f32c5ffb640 1 -- 172.21.15.28:0/3862910704 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32c004f070 con 0x7f32c8101120 2024-05-09T08:39:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.275+0000 7f32c5ffb640 1 -- 172.21.15.28:0/3862910704 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c00043e0 con 0x7f32c8101120 2024-05-09T08:39:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.275+0000 7f32cebb0640 1 -- 172.21.15.28:0/3862910704 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c81aadb0 con 0x7f32c8101120 2024-05-09T08:39:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.276+0000 7f32cebb0640 1 -- 172.21.15.28:0/3862910704 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c81ab160 con 0x7f32c8101120 2024-05-09T08:39:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.282+0000 7f32c5ffb640 1 -- 172.21.15.28:0/3862910704 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f32c0048020 con 0x7f32c8101120 2024-05-09T08:39:09.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.288+0000 7f32d199f740 1 Processor -- start 2024-05-09T08:39:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.290+0000 7f32d199f740 1 -- start start 2024-05-09T08:39:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.290+0000 7f32d199f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190450 0x563ccf190830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.290+0000 7f32d199f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190d70 0x563ccf19b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.290+0000 7f32d199f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 0x563ccf19da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.290+0000 7f32d199f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ccf0f5340 con 0x563ccf190450 2024-05-09T08:39:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.290+0000 7f32d199f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x563ccf09d290 con 0x563ccf19b630 2024-05-09T08:39:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.290+0000 7f32d199f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563ccf0f1300 con 0x563ccf190d70 2024-05-09T08:39:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32cd14d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190450 0x563ccf190830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32cd14d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190450 0x563ccf190830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46112/0 (socket says 172.21.15.28:46112) 2024-05-09T08:39:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32cd14d640 1 -- 172.21.15.28:0/3381291027 learned_addr learned my addr 172.21.15.28:0/3381291027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32c7fff640 1 --2- 172.21.15.28:0/3381291027 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 0x563ccf19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32c7fff640 1 -- 172.21.15.28:0/3381291027 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190d70 msgr2=0x563ccf19b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:39:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32c7fff640 1 --2- 172.21.15.28:0/3381291027 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190d70 0x563ccf19b0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32c7fff640 1 -- 172.21.15.28:0/3381291027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190450 msgr2=0x563ccf190830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32c7fff640 1 --2- 172.21.15.28:0/3381291027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190450 0x563ccf190830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32c7fff640 1 -- 172.21.15.28:0/3381291027 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ccf0c18e0 con 0x563ccf19b630 2024-05-09T08:39:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.294+0000 7f32c5ffb640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f32b40a2fd0 0x7f32b40a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.295+0000 7f32c7fff640 1 --2- 172.21.15.28:0/3381291027 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 0x563ccf19da20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f32c00ff730 tx=0x7f32c0045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ac11ca9ed7e9a6b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.295+0000 7f32c5ffb640 1 -- 172.21.15.28:0/3862910704 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9703+0+0 (secure 0 0 0) 0x7f32c01023d0 con 0x7f32c8101120 2024-05-09T08:39:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.295+0000 7f32beffd640 1 -- 172.21.15.28:0/3381291027 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c00ca7a0 con 0x563ccf19b630 2024-05-09T08:39:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.295+0000 7f32beffd640 1 -- 172.21.15.28:0/3381291027 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32c010f030 con 0x563ccf19b630 2024-05-09T08:39:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.296+0000 7f32d199f740 1 -- 172.21.15.28:0/3381291027 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 msgr2=0x563ccf19da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.296+0000 7f32d199f740 1 --2- 172.21.15.28:0/3381291027 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 0x563ccf19da20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f32c00ff730 tx=0x7f32c0045000 comp rx=0 tx=0).stop 2024-05-09T08:39:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.299+0000 7f32cd14d640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f32b40a2fd0 0x7f32b40a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.300+0000 7f32d199f740 1 -- 172.21.15.28:0/3381291027 shutdown_connections 2024-05-09T08:39:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.300+0000 7f32d199f740 1 --2- 172.21.15.28:0/3381291027 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 0x563ccf19da20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.300+0000 7f32d199f740 1 --2- 172.21.15.28:0/3381291027 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190d70 0x563ccf19b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.300+0000 7f32d199f740 1 --2- 172.21.15.28:0/3381291027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190450 0x563ccf190830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.300+0000 7f32d199f740 1 -- 172.21.15.28:0/3381291027 >> 172.21.15.28:0/3381291027 conn(0x563ccf18c960 msgr2=0x563ccf18cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.305+0000 7f32d199f740 1 -- 172.21.15.28:0/3381291027 shutdown_connections 2024-05-09T08:39:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.307+0000 7f32d199f740 1 -- 172.21.15.28:0/3381291027 wait complete. 2024-05-09T08:39:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.307+0000 7f32d199f740 1 -- start start 2024-05-09T08:39:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.307+0000 7f32d199f740 1 Processor -- start 2024-05-09T08:39:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32d199f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf198520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32d199f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf190d70 0x563ccf198a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32d199f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 0x563ccf198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32d199f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ccf09d290 con 0x563ccf19b630 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32d199f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x563ccf0b0790 con 0x563ccf190d70 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32d199f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563ccf19e020 con 0x563ccf190450 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32d199f740 10 client.0 _ll_register_callbacks cb 0x563ccf0eba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32cd14d640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f32b40a2fd0 0x7f32b40a5490 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f32b0005fd0 tx=0x7f32b0005ee0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf190d70 0x563ccf198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf190d70 0x563ccf198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:43396/0 (socket says 172.21.15.28:43396) 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32cc94c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.308+0000 7f32c7fff640 1 -- 172.21.15.28:0/1770054655 learned_addr learned my addr 172.21.15.28:0/1770054655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.309+0000 7f32cd14d640 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 0x563ccf198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.309+0000 7f32cd14d640 1 -- 172.21.15.28:0/1770054655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 msgr2=0x563ccf198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.309+0000 7f32cd14d640 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.309+0000 7f32cd14d640 1 -- 172.21.15.28:0/1770054655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf190d70 msgr2=0x563ccf198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.309+0000 7f32cd14d640 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf190d70 0x563ccf198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.309+0000 7f32cd14d640 1 -- 172.21.15.28:0/1770054655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ccee09210 con 0x563ccf19b630 2024-05-09T08:39:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.309+0000 7f32cd14d640 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 0x563ccf198fa0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f32b0005d10 tx=0x7f32b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed46f85be79ea41e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.310+0000 7f32affff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:39:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.310+0000 7f32affff640 1 -- 172.21.15.28:0/1770054655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b000ede0 con 0x563ccf19b630 2024-05-09T08:39:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.310+0000 7f32d199f740 1 -- 172.21.15.28:0/1770054655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32c010f030 con 0x563ccf19b630 2024-05-09T08:39:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.310+0000 7f32affff640 1 -- 172.21.15.28:0/1770054655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9703+0+0 (secure 0 0 0) 0x7f32b001e420 con 0x563ccf19b630 2024-05-09T08:39:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.315+0000 7f32affff640 1 -- 172.21.15.28:0/1770054655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f32b001e620 con 0x563ccf19b630 2024-05-09T08:39:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.317+0000 7f32d199f740 10 client.17034 fetch_fsmap learned FSMap version 65 2024-05-09T08:39:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.317+0000 7f32d199f740 1 -- 172.21.15.28:0/1770054655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x563cceed5620 con 0x563ccf19b630 2024-05-09T08:39:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.323+0000 7f32affff640 1 -- 172.21.15.28:0/1770054655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32c010f030 con 0x563ccf19b630 2024-05-09T08:39:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.323+0000 7f32affff640 1 -- 172.21.15.28:0/1770054655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f32b00309c0 con 0x563ccf19b630 2024-05-09T08:39:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.325+0000 7f32d199f740 10 client.17034 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:39:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.325+0000 7f32d199f740 10 client.17034 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:39:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.325+0000 7f32d199f740 20 client.17034 populate_metadata read hostname 'smithi028' 2024-05-09T08:39:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.325+0000 7f32d199f740 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563ccf1ab000 0x563ccf1ad490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.325+0000 7f32d199f740 4 client.17034 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.325+0000 7f32d199f740 1 -- 172.21.15.28:0/1770054655 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ccf102a70 con 0x563ccf1ab000 2024-05-09T08:39:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.328+0000 7f32cc94c640 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563ccf1ab000 0x563ccf1ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.336+0000 7f32cc94c640 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563ccf1ab000 0x563ccf1ad490 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85d7a2dc88ae3e12 server_cookie=fa77263a5f04cf76 in_seq=0 out_seq=0 2024-05-09T08:39:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.337+0000 7f32affff640 10 client.17034 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.343+0000 7f32affff640 1 -- 172.21.15.28:0/1770054655 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ccf102a70 con 0x563ccf1ab000 2024-05-09T08:39:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.343+0000 7f32affff640 10 client.17034 handle_command_reply: tid=0 2024-05-09T08:39:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.346+0000 7f32d199f740 1 client.17034 shutdown 2024-05-09T08:39:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.347+0000 7f32d199f740 1 -- 172.21.15.28:0/1770054655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 msgr2=0x563ccf198fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.347+0000 7f32d199f740 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 0x563ccf198fa0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f32b0005d10 tx=0x7f32b00046c0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.349+0000 7f32d199f740 1 -- 172.21.15.28:0/1770054655 shutdown_connections 2024-05-09T08:39:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.349+0000 7f32d199f740 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563ccf1ab000 0x563ccf1ad490 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.349+0000 7f32d199f740 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 0x563ccf198fa0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.349+0000 7f32d199f740 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf190d70 0x563ccf198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.349+0000 7f32d199f740 1 --2- 172.21.15.28:0/1770054655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.349+0000 7f32d199f740 1 -- 172.21.15.28:0/1770054655 >> 172.21.15.28:0/1770054655 conn(0x563ccf18c960 msgr2=0x563ccf1926e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.349+0000 7f32affff640 0 client.17034 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.350+0000 7f32d199f740 1 -- 172.21.15.28:0/1770054655 shutdown_connections 2024-05-09T08:39:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.353+0000 7f32d199f740 1 -- 172.21.15.28:0/1770054655 wait complete. 2024-05-09T08:39:09.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.354+0000 7f32d199f740 20 client.17034 trim_cache size 0 max 16384 2024-05-09T08:39:09.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.387+0000 7f32d199f740 1 Processor -- start 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.390+0000 7f32d199f740 1 -- start start 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.391+0000 7f32d199f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.391+0000 7f32d199f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190d70 0x563ccf1a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.391+0000 7f32d199f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 0x563ccf1b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.391+0000 7f32d199f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32a8010330 con 0x563ccf190d70 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.391+0000 7f32d199f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f32a80120b0 con 0x563ccf19b630 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.391+0000 7f32d199f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f32a8010ac0 con 0x563ccf190450 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.392+0000 7f32cd14d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 0x563ccf1b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.392+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190d70 0x563ccf1a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.393+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190d70 0x563ccf1a6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46140/0 (socket says 172.21.15.28:46140) 2024-05-09T08:39:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.393+0000 7f32c7fff640 1 -- 172.21.15.28:0/822234431 learned_addr learned my addr 172.21.15.28:0/822234431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.394+0000 7f32cc94c640 1 --2- 172.21.15.28:0/822234431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.395+0000 7f32c7fff640 1 -- 172.21.15.28:0/822234431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 msgr2=0x563ccf19ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.395+0000 7f32c7fff640 1 --2- 172.21.15.28:0/822234431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.395+0000 7f32c7fff640 1 -- 172.21.15.28:0/822234431 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 msgr2=0x563ccf1b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.395+0000 7f32c7fff640 1 --2- 172.21.15.28:0/822234431 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 0x563ccf1b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.395+0000 7f32c7fff640 1 -- 172.21.15.28:0/822234431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ccf196a70 con 0x563ccf190d70 2024-05-09T08:39:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.395+0000 7f32c7fff640 1 --2- 172.21.15.28:0/822234431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190d70 0x563ccf1a6d60 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f32c00caed0 tx=0x7f32c0003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65af1d7580b35503 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.395+0000 7f32be7fc640 1 -- 172.21.15.28:0/822234431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c00ce040 con 0x563ccf190d70 2024-05-09T08:39:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.396+0000 7f32be7fc640 1 -- 172.21.15.28:0/822234431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32c010f040 con 0x563ccf190d70 2024-05-09T08:39:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.396+0000 7f32d199f740 1 -- 172.21.15.28:0/822234431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190d70 msgr2=0x563ccf1a6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.397+0000 7f32d199f740 1 --2- 172.21.15.28:0/822234431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190d70 0x563ccf1a6d60 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f32c00caed0 tx=0x7f32c0003dd0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.398+0000 7f32d199f740 1 -- 172.21.15.28:0/822234431 shutdown_connections 2024-05-09T08:39:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.398+0000 7f32d199f740 1 --2- 172.21.15.28:0/822234431 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19b630 0x563ccf1b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.398+0000 7f32d199f740 1 --2- 172.21.15.28:0/822234431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf190d70 0x563ccf1a6d60 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.398+0000 7f32d199f740 1 --2- 172.21.15.28:0/822234431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.398+0000 7f32d199f740 1 -- 172.21.15.28:0/822234431 >> 172.21.15.28:0/822234431 conn(0x563ccf18c960 msgr2=0x563ccf18ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.399+0000 7f32d199f740 1 -- 172.21.15.28:0/822234431 shutdown_connections 2024-05-09T08:39:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.399+0000 7f32d199f740 1 -- 172.21.15.28:0/822234431 wait complete. 2024-05-09T08:39:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.399+0000 7f32d199f740 1 -- start start 2024-05-09T08:39:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.399+0000 7f32d199f740 1 Processor -- start 2024-05-09T08:39:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.401+0000 7f32d199f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.401+0000 7f32d199f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 0x563ccf19f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.401+0000 7f32d199f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19f9a0 0x563ccf210610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.401+0000 7f32d199f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32a80119e0 con 0x563ccf19b630 2024-05-09T08:39:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.401+0000 7f32d199f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f32a8011250 con 0x563ccf19f9a0 2024-05-09T08:39:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.401+0000 7f32d199f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f32a800e330 con 0x563ccf190450 2024-05-09T08:39:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.401+0000 7f32d199f740 10 client.0 _ll_register_callbacks cb 0x7f32b000c880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.402+0000 7f32cc94c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.402+0000 7f32cc94c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50072/0 (socket says 172.21.15.28:50072) 2024-05-09T08:39:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.402+0000 7f32cc94c640 1 -- 172.21.15.28:0/3973479443 learned_addr learned my addr 172.21.15.28:0/3973479443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.402+0000 7f32cc94c640 1 -- 172.21.15.28:0/3973479443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19f9a0 msgr2=0x563ccf210610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.402+0000 7f32cc94c640 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19f9a0 0x563ccf210610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.402+0000 7f32cc94c640 1 -- 172.21.15.28:0/3973479443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 msgr2=0x563ccf19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.402+0000 7f32cc94c640 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 0x563ccf19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.403+0000 7f32cc94c640 1 -- 172.21.15.28:0/3973479443 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ccf210cd0 con 0x563ccf190450 2024-05-09T08:39:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.404+0000 7f32cc94c640 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ef20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f32b8005940 tx=0x7f32b8010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3e1ead2da7b43d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.404+0000 7f32affff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:39:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.405+0000 7f32affff640 1 -- 172.21.15.28:0/3973479443 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b801c070 con 0x563ccf190450 2024-05-09T08:39:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.405+0000 7f32d199f740 1 -- 172.21.15.28:0/3973479443 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32c010f040 con 0x563ccf190450 2024-05-09T08:39:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.405+0000 7f32affff640 1 -- 172.21.15.28:0/3973479443 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9703+0+0 (secure 0 0 0) 0x7f32b8018dc0 con 0x563ccf190450 2024-05-09T08:39:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.408+0000 7f32affff640 1 -- 172.21.15.28:0/3973479443 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f32b8005020 con 0x563ccf190450 2024-05-09T08:39:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.409+0000 7f32d199f740 10 client.26401 fetch_fsmap learned FSMap version 65 2024-05-09T08:39:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.409+0000 7f32d199f740 1 -- 172.21.15.28:0/3973479443 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x563ccf212020 con 0x563ccf190450 2024-05-09T08:39:09.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.411+0000 7f32affff640 1 -- 172.21.15.28:0/3973479443 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32c010f040 con 0x563ccf190450 2024-05-09T08:39:09.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.412+0000 7f32affff640 1 -- 172.21.15.28:0/3973479443 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 65) ==== 8442+0+0 (secure 0 0 0) 0x7f32b80047a0 con 0x563ccf190450 2024-05-09T08:39:09.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.412+0000 7f32d199f740 10 client.26401 fetch_fsmap finished waiting for FSMap version 65 2024-05-09T08:39:09.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.412+0000 7f32d199f740 10 client.26401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:39:09.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.412+0000 7f32d199f740 20 client.26401 populate_metadata read hostname 'smithi028' 2024-05-09T08:39:09.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.412+0000 7f32d199f740 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563ccf212810 0x563ccf214c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:09.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.412+0000 7f32d199f740 4 client.26401 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:39:09.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.412+0000 7f32d199f740 1 -- 172.21.15.28:0/3973479443 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563ccf0fcdc0 con 0x563ccf212810 2024-05-09T08:39:09.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.415+0000 7f32c7fff640 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563ccf212810 0x563ccf214c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.421+0000 7f32c7fff640 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563ccf212810 0x563ccf214c90 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d9da7ad5c7dab6 server_cookie=b1b3317d90224a3e in_seq=0 out_seq=0 2024-05-09T08:39:09.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:09.422+0000 7f32affff640 10 client.26401 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32affff640 1 -- 172.21.15.28:0/3973479443 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563ccf0fcdc0 con 0x563ccf212810 2024-05-09T08:39:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32affff640 10 client.26401 handle_command_reply: tid=0 2024-05-09T08:39:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 client.26401 shutdown 2024-05-09T08:39:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 -- 172.21.15.28:0/3973479443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 msgr2=0x563ccf19ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ef20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f32b8005940 tx=0x7f32b8010800 comp rx=0 tx=0).stop 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 -- 172.21.15.28:0/3973479443 shutdown_connections 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563ccf212810 0x563ccf214c90 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563ccf19f9a0 0x563ccf210610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ccf19b630 0x563ccf19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 --2- 172.21.15.28:0/3973479443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563ccf190450 0x563ccf19ef20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 -- 172.21.15.28:0/3973479443 >> 172.21.15.28:0/3973479443 conn(0x563ccf18c960 msgr2=0x563ccf191e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32affff640 0 client.26401 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 -- 172.21.15.28:0/3973479443 shutdown_connections 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.571+0000 7f32d199f740 1 -- 172.21.15.28:0/3973479443 wait complete. 2024-05-09T08:39:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.572+0000 7f32d199f740 20 client.26401 trim_cache size 0 max 16384 2024-05-09T08:39:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:39:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:39:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:39:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:39:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 -- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f32b40a2fd0 msgr2=0x7f32b40a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f32b40a2fd0 0x7f32b40a5490 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f32b0005fd0 tx=0x7f32b0005ee0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 -- 172.21.15.28:0/3862910704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32c8101120 msgr2=0x7f32c806d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32c8101120 0x7f32c806d250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f32c0005c40 tx=0x7f32c00405b0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 -- 172.21.15.28:0/3862910704 shutdown_connections 2024-05-09T08:39:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c8106720 0x7f32c806d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f32b40a2fd0 0x7f32b40a5490 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32c8101120 0x7f32c806d250 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 --2- 172.21.15.28:0/3862910704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8100760 0x7f32c8073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.573+0000 7f32cebb0640 1 -- 172.21.15.28:0/3862910704 >> 172.21.15.28:0/3862910704 conn(0x7f32c80778b0 msgr2=0x7f32c8103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.574+0000 7f32cebb0640 1 -- 172.21.15.28:0/3862910704 shutdown_connections 2024-05-09T08:39:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:10.574+0000 7f32cebb0640 1 -- 172.21.15.28:0/3862910704 wait complete. 2024-05-09T08:39:10.932 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:39:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[91682]: ignoring --setuser ceph since I am not root 2024-05-09T08:39:10.932 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:39:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[91682]: ignoring --setgroup ceph since I am not root 2024-05-09T08:39:10.932 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:39:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[91682]: starting mds.d at 2024-05-09T08:39:11.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:11 smithi028 ceph-mon[30119]: pgmap v411: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 355 op/s 2024-05-09T08:39:11.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:11 smithi028 ceph-mon[30119]: daemon mds.d restarted 2024-05-09T08:39:11.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:11 smithi165 ceph-mon[34540]: pgmap v411: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 355 op/s 2024-05-09T08:39:11.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:11 smithi165 ceph-mon[34540]: daemon mds.d restarted 2024-05-09T08:39:11.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:11 smithi184 ceph-mon[34509]: pgmap v411: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 355 op/s 2024-05-09T08:39:11.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:11 smithi184 ceph-mon[34509]: daemon mds.d restarted 2024-05-09T08:39:12.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:12 smithi028 ceph-mon[30119]: osdmap e117: 12 total, 12 up, 12 in 2024-05-09T08:39:12.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.28:6836/1569049047,v1:172.21.15.28:6837/1569049047] up:boot 2024-05-09T08:39:12.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:12 smithi028 ceph-mon[30119]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:39:12.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:39:12.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:12 smithi028 ceph-mon[30119]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:39:12.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:12 smithi165 ceph-mon[34540]: osdmap e117: 12 total, 12 up, 12 in 2024-05-09T08:39:12.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.28:6836/1569049047,v1:172.21.15.28:6837/1569049047] up:boot 2024-05-09T08:39:12.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:12 smithi165 ceph-mon[34540]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:39:12.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:39:12.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:12 smithi165 ceph-mon[34540]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:39:12.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:12 smithi184 ceph-mon[34509]: osdmap e117: 12 total, 12 up, 12 in 2024-05-09T08:39:12.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.28:6836/1569049047,v1:172.21.15.28:6837/1569049047] up:boot 2024-05-09T08:39:12.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:12 smithi184 ceph-mon[34509]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-09T08:39:12.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:39:12.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:12 smithi184 ceph-mon[34509]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-09T08:39:13.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:13 smithi028 ceph-mon[30119]: pgmap v413: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 342 op/s 2024-05-09T08:39:13.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:13 smithi165 ceph-mon[34540]: pgmap v413: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 342 op/s 2024-05-09T08:39:13.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:13 smithi184 ceph-mon[34509]: pgmap v413: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 342 op/s 2024-05-09T08:39:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:15 smithi028 ceph-mon[30119]: pgmap v414: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 338 op/s 2024-05-09T08:39:15.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:15 smithi165 ceph-mon[34540]: pgmap v414: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 338 op/s 2024-05-09T08:39:15.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:15 smithi184 ceph-mon[34509]: pgmap v414: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 338 op/s 2024-05-09T08:39:17.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:17 smithi028 ceph-mon[30119]: pgmap v415: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 842 KiB/s wr, 296 op/s 2024-05-09T08:39:17.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:17 smithi165 ceph-mon[34540]: pgmap v415: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 842 KiB/s wr, 296 op/s 2024-05-09T08:39:17.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:17 smithi184 ceph-mon[34509]: pgmap v415: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 842 KiB/s wr, 296 op/s 2024-05-09T08:39:18.932 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:39:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:39:18.564+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:39:18.932 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:39:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:39:18.564+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:39:19.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:19 smithi028 ceph-mon[30119]: pgmap v416: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 909 KiB/s wr, 272 op/s 2024-05-09T08:39:19.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:19 smithi165 ceph-mon[34540]: pgmap v416: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 909 KiB/s wr, 272 op/s 2024-05-09T08:39:19.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:19 smithi184 ceph-mon[34509]: pgmap v416: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 909 KiB/s wr, 272 op/s 2024-05-09T08:39:21.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:21 smithi028 ceph-mon[30119]: pgmap v417: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.0 MiB/s wr, 288 op/s 2024-05-09T08:39:21.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:21 smithi165 ceph-mon[34540]: pgmap v417: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.0 MiB/s wr, 288 op/s 2024-05-09T08:39:21.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:21 smithi184 ceph-mon[34509]: pgmap v417: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.0 MiB/s wr, 288 op/s 2024-05-09T08:39:23.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.739+0000 7fad4df9e640 1 -- 172.21.15.28:0/4270184894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4806db20 msgr2=0x7fad48108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.739+0000 7fad4df9e640 1 --2- 172.21.15.28:0/4270184894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4806db20 0x7fad48108830 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fad40009ca0 tx=0x7fad4002a8f0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.741+0000 7fad4df9e640 1 -- 172.21.15.28:0/4270184894 shutdown_connections 2024-05-09T08:39:23.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.741+0000 7fad4df9e640 1 --2- 172.21.15.28:0/4270184894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad48114570 0x7fad48116960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.741+0000 7fad4df9e640 1 --2- 172.21.15.28:0/4270184894 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad48108e00 0x7fad48109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.741+0000 7fad4df9e640 1 --2- 172.21.15.28:0/4270184894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4806db20 0x7fad48108830 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.741+0000 7fad4df9e640 1 -- 172.21.15.28:0/4270184894 >> 172.21.15.28:0/4270184894 conn(0x7fad4806c770 msgr2=0x7fad4806cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.746+0000 7fad4df9e640 1 -- 172.21.15.28:0/4270184894 shutdown_connections 2024-05-09T08:39:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.746+0000 7fad4df9e640 1 -- 172.21.15.28:0/4270184894 wait complete. 2024-05-09T08:39:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.747+0000 7fad4df9e640 1 Processor -- start 2024-05-09T08:39:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.750+0000 7fad4df9e640 1 -- start start 2024-05-09T08:39:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.750+0000 7fad4df9e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad4806db20 0x7fad48113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.750+0000 7fad4df9e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad48108e00 0x7fad48113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.750+0000 7fad4df9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad48114570 0x7fad4810a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.750+0000 7fad4df9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad4811a340 con 0x7fad48114570 2024-05-09T08:39:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.750+0000 7fad4df9e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad4811a1c0 con 0x7fad48108e00 2024-05-09T08:39:23.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.750+0000 7fad4df9e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad4811a4c0 con 0x7fad4806db20 2024-05-09T08:39:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.753+0000 7fad4cf9c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad4806db20 0x7fad48113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.753+0000 7fad4cf9c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad4806db20 0x7fad48113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:55716/0 (socket says 172.21.15.28:55716) 2024-05-09T08:39:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.753+0000 7fad4cf9c640 1 -- 172.21.15.28:0/1231686360 learned_addr learned my addr 172.21.15.28:0/1231686360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.754+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad48114570 0x7fad4810a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.754+0000 7fad4cf9c640 1 -- 172.21.15.28:0/1231686360 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad48108e00 msgr2=0x7fad48113e50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:39:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.754+0000 7fad4cf9c640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad48108e00 0x7fad48113e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.754+0000 7fad4cf9c640 1 -- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad48114570 msgr2=0x7fad4810a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.754+0000 7fad4cf9c640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad48114570 0x7fad4810a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.754+0000 7fad4cf9c640 1 -- 172.21.15.28:0/1231686360 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4810b2a0 con 0x7fad4806db20 2024-05-09T08:39:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.761+0000 7fad4cf9c640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad4806db20 0x7fad48113910 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fad40006010 tx=0x7fad40031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.766+0000 7fad45ffb640 1 -- 172.21.15.28:0/1231686360 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad400307f0 con 0x7fad4806db20 2024-05-09T08:39:23.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.766+0000 7fad45ffb640 1 -- 172.21.15.28:0/1231686360 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad4002d070 con 0x7fad4806db20 2024-05-09T08:39:23.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.766+0000 7fad45ffb640 1 -- 172.21.15.28:0/1231686360 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4003c410 con 0x7fad4806db20 2024-05-09T08:39:23.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.768+0000 7fad4df9e640 1 -- 172.21.15.28:0/1231686360 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad4810b530 con 0x7fad4806db20 2024-05-09T08:39:23.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.768+0000 7fad4df9e640 1 -- 172.21.15.28:0/1231686360 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad4810f530 con 0x7fad4806db20 2024-05-09T08:39:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.773+0000 7fad51fef740 1 Processor -- start 2024-05-09T08:39:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.779+0000 7fad45ffb640 1 -- 172.21.15.28:0/1231686360 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fad40004040 con 0x7fad4806db20 2024-05-09T08:39:23.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.781+0000 7fad45ffb640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad300a2ec0 0x7fad300a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.781+0000 7fad45ffb640 1 -- 172.21.15.28:0/1231686360 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9615+0+0 (secure 0 0 0) 0x7fad400f0e20 con 0x7fad4806db20 2024-05-09T08:39:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.781+0000 7fad51fef740 1 -- start start 2024-05-09T08:39:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.782+0000 7fad51fef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfada5d0 0x557acfada9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.782+0000 7fad51fef740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfadaef0 0x557acfae5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.782+0000 7fad51fef740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfae57b0 0x557acfae7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.782+0000 7fad51fef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557acfa3f340 con 0x557acfada5d0 2024-05-09T08:39:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.782+0000 7fad51fef740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557acf9e7290 con 0x557acfae57b0 2024-05-09T08:39:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.782+0000 7fad51fef740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557acfa3b300 con 0x557acfadaef0 2024-05-09T08:39:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.782+0000 7fad4d79d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfae57b0 0x557acfae7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.783+0000 7fad4d79d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfae57b0 0x557acfae7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:56714/0 (socket says 172.21.15.28:56714) 2024-05-09T08:39:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.783+0000 7fad4d79d640 1 -- 172.21.15.28:0/3680724160 learned_addr learned my addr 172.21.15.28:0/3680724160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.783+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad300a2ec0 0x7fad300a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.783+0000 7fad4d79d640 1 -- 172.21.15.28:0/3680724160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfadaef0 msgr2=0x557acfae5270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:39:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.783+0000 7fad4d79d640 1 --2- 172.21.15.28:0/3680724160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfadaef0 0x557acfae5270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.783+0000 7fad4d79d640 1 -- 172.21.15.28:0/3680724160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfada5d0 msgr2=0x557acfada9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:39:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.783+0000 7fad4d79d640 1 --2- 172.21.15.28:0/3680724160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfada5d0 0x557acfada9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.783+0000 7fad4d79d640 1 -- 172.21.15.28:0/3680724160 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557acfa0b8e0 con 0x557acfae57b0 2024-05-09T08:39:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.784+0000 7fad4d79d640 1 --2- 172.21.15.28:0/3680724160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfae57b0 0x557acfae7ba0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fad38017950 tx=0x7fad38017e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b39d368f5e9cc1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.785+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad300a2ec0 0x7fad300a5380 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fad38008000 tx=0x7fad3801b000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.785+0000 7fad2effd640 1 -- 172.21.15.28:0/3680724160 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad38009750 con 0x557acfae57b0 2024-05-09T08:39:23.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.785+0000 7fad2effd640 1 -- 172.21.15.28:0/3680724160 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad3801c070 con 0x557acfae57b0 2024-05-09T08:39:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.786+0000 7fad51fef740 1 -- 172.21.15.28:0/3680724160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfae57b0 msgr2=0x557acfae7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.786+0000 7fad51fef740 1 --2- 172.21.15.28:0/3680724160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfae57b0 0x557acfae7ba0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fad38017950 tx=0x7fad38017e10 comp rx=0 tx=0).stop 2024-05-09T08:39:23.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.794+0000 7fad51fef740 1 -- 172.21.15.28:0/3680724160 shutdown_connections 2024-05-09T08:39:23.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.794+0000 7fad51fef740 1 --2- 172.21.15.28:0/3680724160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfae57b0 0x557acfae7ba0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.794+0000 7fad51fef740 1 --2- 172.21.15.28:0/3680724160 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfadaef0 0x557acfae5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.794+0000 7fad51fef740 1 --2- 172.21.15.28:0/3680724160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfada5d0 0x557acfada9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.794+0000 7fad51fef740 1 -- 172.21.15.28:0/3680724160 >> 172.21.15.28:0/3680724160 conn(0x557acfad6ae0 msgr2=0x557acfad6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.799+0000 7fad51fef740 1 -- 172.21.15.28:0/3680724160 shutdown_connections 2024-05-09T08:39:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.799+0000 7fad51fef740 1 -- 172.21.15.28:0/3680724160 wait complete. 2024-05-09T08:39:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.800+0000 7fad51fef740 1 -- start start 2024-05-09T08:39:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.800+0000 7fad51fef740 1 Processor -- start 2024-05-09T08:39:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.808+0000 7fad51fef740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfada5d0 0x557acfae26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.808+0000 7fad51fef740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.808+0000 7fad51fef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.808+0000 7fad51fef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557acfae8320 con 0x557acfae57b0 2024-05-09T08:39:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.808+0000 7fad51fef740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557acf9fa790 con 0x557acfadaef0 2024-05-09T08:39:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.808+0000 7fad51fef740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557acfae84a0 con 0x557acfada5d0 2024-05-09T08:39:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.808+0000 7fad51fef740 10 client.0 _ll_register_callbacks cb 0x557acfa35a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.810+0000 7fad47fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.810+0000 7fad47fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:56722/0 (socket says 172.21.15.28:56722) 2024-05-09T08:39:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.810+0000 7fad47fff640 1 -- 172.21.15.28:0/1204796461 learned_addr learned my addr 172.21.15.28:0/1204796461 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.812+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.812+0000 7fad4cf9c640 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfada5d0 0x557acfae26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.816+0000 7fad4d79d640 1 -- 172.21.15.28:0/1204796461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfada5d0 msgr2=0x557acfae26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.816+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfada5d0 0x557acfae26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.816+0000 7fad4d79d640 1 -- 172.21.15.28:0/1204796461 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 msgr2=0x557acfae2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.816+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.816+0000 7fad4d79d640 1 -- 172.21.15.28:0/1204796461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557acf753210 con 0x557acfae57b0 2024-05-09T08:39:23.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.819+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae3170 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fad38004810 tx=0x7fad38009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cbd4185273ca039 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.820+0000 7fad27fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:39:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.821+0000 7fad27fff640 1 -- 172.21.15.28:0/1204796461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad38029440 con 0x557acfae57b0 2024-05-09T08:39:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.821+0000 7fad51fef740 1 -- 172.21.15.28:0/1204796461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad3801c070 con 0x557acfae57b0 2024-05-09T08:39:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.821+0000 7fad27fff640 1 -- 172.21.15.28:0/1204796461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9615+0+0 (secure 0 0 0) 0x7fad380295e0 con 0x557acfae57b0 2024-05-09T08:39:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.824+0000 7fad27fff640 1 -- 172.21.15.28:0/1204796461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fad380059c0 con 0x557acfae57b0 2024-05-09T08:39:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.827+0000 7fad51fef740 10 client.17052 fetch_fsmap learned FSMap version 67 2024-05-09T08:39:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.827+0000 7fad51fef740 1 -- 172.21.15.28:0/1204796461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x557acf81f620 con 0x557acfae57b0 2024-05-09T08:39:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.829+0000 7fad27fff640 1 -- 172.21.15.28:0/1204796461 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad3801c070 con 0x557acfae57b0 2024-05-09T08:39:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.829+0000 7fad27fff640 1 -- 172.21.15.28:0/1204796461 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fad38006020 con 0x557acfae57b0 2024-05-09T08:39:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.829+0000 7fad51fef740 10 client.17052 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:39:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.829+0000 7fad51fef740 10 client.17052 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:39:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.829+0000 7fad51fef740 20 client.17052 populate_metadata read hostname 'smithi028' 2024-05-09T08:39:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.829+0000 7fad51fef740 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557acfaf59f0 0x557acfaf7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.829+0000 7fad51fef740 4 client.17052 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.829+0000 7fad51fef740 1 -- 172.21.15.28:0/1204796461 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557acfa4ca70 con 0x557acfaf59f0 2024-05-09T08:39:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.830+0000 7fad4cf9c640 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557acfaf59f0 0x557acfaf7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.842+0000 7fad4cf9c640 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557acfaf59f0 0x557acfaf7e50 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d28dd352f025f775 server_cookie=9d8b37dfc7d833e5 in_seq=0 out_seq=0 2024-05-09T08:39:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.843+0000 7fad27fff640 10 client.17052 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:23.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:23 smithi165 ceph-mon[34540]: pgmap v418: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.0 MiB/s wr, 257 op/s 2024-05-09T08:39:23.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:23 smithi184 ceph-mon[34509]: pgmap v418: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.0 MiB/s wr, 257 op/s 2024-05-09T08:39:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.868+0000 7fad27fff640 1 -- 172.21.15.28:0/1204796461 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557acfa4ca70 con 0x557acfaf59f0 2024-05-09T08:39:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.868+0000 7fad27fff640 10 client.17052 handle_command_reply: tid=0 2024-05-09T08:39:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.868+0000 7fad51fef740 1 client.17052 shutdown 2024-05-09T08:39:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.870+0000 7fad51fef740 1 -- 172.21.15.28:0/1204796461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 msgr2=0x557acfae3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.870+0000 7fad51fef740 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae3170 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fad38004810 tx=0x7fad38009f20 comp rx=0 tx=0).stop 2024-05-09T08:39:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.870+0000 7fad51fef740 1 -- 172.21.15.28:0/1204796461 shutdown_connections 2024-05-09T08:39:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.870+0000 7fad51fef740 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557acfaf59f0 0x557acfaf7e50 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.870+0000 7fad51fef740 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae3170 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.870+0000 7fad51fef740 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.870+0000 7fad51fef740 1 --2- 172.21.15.28:0/1204796461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfada5d0 0x557acfae26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.870+0000 7fad51fef740 1 -- 172.21.15.28:0/1204796461 >> 172.21.15.28:0/1204796461 conn(0x557acfad6ae0 msgr2=0x557acfadc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.870+0000 7fad27fff640 0 client.17052 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.873+0000 7fad51fef740 1 -- 172.21.15.28:0/1204796461 shutdown_connections 2024-05-09T08:39:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.874+0000 7fad51fef740 1 -- 172.21.15.28:0/1204796461 wait complete. 2024-05-09T08:39:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.874+0000 7fad51fef740 20 client.17052 trim_cache size 0 max 16384 2024-05-09T08:39:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.903+0000 7fad51fef740 1 Processor -- start 2024-05-09T08:39:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.903+0000 7fad51fef740 1 -- start start 2024-05-09T08:39:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.904+0000 7fad51fef740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfada5d0 0x557acfad6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.904+0000 7fad51fef740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.904+0000 7fad51fef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.904+0000 7fad47fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.904+0000 7fad47fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:56736/0 (socket says 172.21.15.28:56736) 2024-05-09T08:39:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.904+0000 7fad47fff640 1 -- 172.21.15.28:0/1000064829 learned_addr learned my addr 172.21.15.28:0/1000064829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.904+0000 7fad51fef740 1 -- 172.21.15.28:0/1000064829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad481c6500 con 0x557acfae57b0 2024-05-09T08:39:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.905+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1000064829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.905+0000 7fad51fef740 1 -- 172.21.15.28:0/1000064829 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad481c8280 con 0x557acfadaef0 2024-05-09T08:39:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.905+0000 7fad51fef740 1 -- 172.21.15.28:0/1000064829 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad481c6c90 con 0x557acfada5d0 2024-05-09T08:39:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.906+0000 7fad4d79d640 1 -- 172.21.15.28:0/1000064829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfada5d0 msgr2=0x557acfad6420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:39:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.906+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1000064829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfada5d0 0x557acfad6420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.906+0000 7fad4d79d640 1 -- 172.21.15.28:0/1000064829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 msgr2=0x557acfae2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.906+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1000064829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.906+0000 7fad4d79d640 1 -- 172.21.15.28:0/1000064829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557acfad7d10 con 0x557acfae57b0 2024-05-09T08:39:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.906+0000 7fad4d79d640 1 --2- 172.21.15.28:0/1000064829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae44b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fad3800b040 tx=0x7fad38020ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55163d00f9a6832a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.908+0000 7fad2e7fc640 1 -- 172.21.15.28:0/1000064829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad38006630 con 0x557acfae57b0 2024-05-09T08:39:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.908+0000 7fad2e7fc640 1 -- 172.21.15.28:0/1000064829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad3801c040 con 0x557acfae57b0 2024-05-09T08:39:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.909+0000 7fad51fef740 1 -- 172.21.15.28:0/1000064829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 msgr2=0x557acfae44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.909+0000 7fad51fef740 1 --2- 172.21.15.28:0/1000064829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae44b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fad3800b040 tx=0x7fad38020ce0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.910+0000 7fad51fef740 1 -- 172.21.15.28:0/1000064829 shutdown_connections 2024-05-09T08:39:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.910+0000 7fad51fef740 1 --2- 172.21.15.28:0/1000064829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae57b0 0x557acfae44b0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.910+0000 7fad51fef740 1 --2- 172.21.15.28:0/1000064829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfadaef0 0x557acfae2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.910+0000 7fad51fef740 1 --2- 172.21.15.28:0/1000064829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfada5d0 0x557acfad6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.910+0000 7fad51fef740 1 -- 172.21.15.28:0/1000064829 >> 172.21.15.28:0/1000064829 conn(0x557acfad6ae0 msgr2=0x557acfae3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.913+0000 7fad51fef740 1 -- 172.21.15.28:0/1000064829 shutdown_connections 2024-05-09T08:39:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.913+0000 7fad51fef740 1 -- 172.21.15.28:0/1000064829 wait complete. 2024-05-09T08:39:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.913+0000 7fad51fef740 1 -- start start 2024-05-09T08:39:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.914+0000 7fad51fef740 1 Processor -- start 2024-05-09T08:39:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.914+0000 7fad51fef740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfada5d0 0x557acfae84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.914+0000 7fad51fef740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfadaef0 0x557acfae8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.914+0000 7fad51fef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae8f70 0x557acfb59f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.914+0000 7fad51fef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad481c7bb0 con 0x557acfae8f70 2024-05-09T08:39:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.914+0000 7fad51fef740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad481c7420 con 0x557acfada5d0 2024-05-09T08:39:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.914+0000 7fad51fef740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad481c44e0 con 0x557acfadaef0 2024-05-09T08:39:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.914+0000 7fad51fef740 10 client.0 _ll_register_callbacks cb 0x7fad38017720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.915+0000 7fad4d79d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae8f70 0x557acfb59f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.915+0000 7fad4d79d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae8f70 0x557acfb59f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49900/0 (socket says 172.21.15.28:49900) 2024-05-09T08:39:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.915+0000 7fad4d79d640 1 -- 172.21.15.28:0/2288465617 learned_addr learned my addr 172.21.15.28:0/2288465617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.915+0000 7fad4d79d640 1 -- 172.21.15.28:0/2288465617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfadaef0 msgr2=0x557acfae8a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:39:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.915+0000 7fad4d79d640 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfadaef0 0x557acfae8a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.915+0000 7fad4d79d640 1 -- 172.21.15.28:0/2288465617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfada5d0 msgr2=0x557acfae84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.915+0000 7fad4d79d640 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfada5d0 0x557acfae84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.915+0000 7fad4d79d640 1 -- 172.21.15.28:0/2288465617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557acfb5a4c0 con 0x557acfae8f70 2024-05-09T08:39:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.916+0000 7fad4d79d640 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae8f70 0x557acfb59f80 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fad38020d70 tx=0x7fad38015a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e07a428d73731c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.916+0000 7fad27fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:39:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.916+0000 7fad27fff640 1 -- 172.21.15.28:0/2288465617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad38006630 con 0x557acfae8f70 2024-05-09T08:39:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.917+0000 7fad51fef740 1 -- 172.21.15.28:0/2288465617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557acfa46dc0 con 0x557acfae8f70 2024-05-09T08:39:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.917+0000 7fad27fff640 1 -- 172.21.15.28:0/2288465617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9615+0+0 (secure 0 0 0) 0x7fad380067d0 con 0x557acfae8f70 2024-05-09T08:39:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.921+0000 7fad27fff640 1 -- 172.21.15.28:0/2288465617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fad38005b60 con 0x557acfae8f70 2024-05-09T08:39:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.922+0000 7fad51fef740 10 client.17058 fetch_fsmap learned FSMap version 67 2024-05-09T08:39:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.922+0000 7fad51fef740 1 -- 172.21.15.28:0/2288465617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x557acfb5b820 con 0x557acfae8f70 2024-05-09T08:39:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.928+0000 7fad27fff640 1 -- 172.21.15.28:0/2288465617 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557acfa46dc0 con 0x557acfae8f70 2024-05-09T08:39:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.928+0000 7fad27fff640 1 -- 172.21.15.28:0/2288465617 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fad38029440 con 0x557acfae8f70 2024-05-09T08:39:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.928+0000 7fad51fef740 10 client.17058 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:39:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.928+0000 7fad51fef740 10 client.17058 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:39:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.928+0000 7fad51fef740 20 client.17058 populate_metadata read hostname 'smithi028' 2024-05-09T08:39:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.928+0000 7fad51fef740 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557acfb5bd50 0x557acfb5e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.928+0000 7fad51fef740 4 client.17058 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:39:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.928+0000 7fad51fef740 1 -- 172.21.15.28:0/2288465617 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557acf9c48e0 con 0x557acfb5bd50 2024-05-09T08:39:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.932+0000 7fad4cf9c640 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557acfb5bd50 0x557acfb5e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:23 smithi028 ceph-mon[30119]: pgmap v418: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.0 MiB/s wr, 257 op/s 2024-05-09T08:39:23.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.942+0000 7fad4cf9c640 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557acfb5bd50 0x557acfb5e170 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b737c72d810ea4c7 server_cookie=8f3a3586b039aad6 in_seq=0 out_seq=0 2024-05-09T08:39:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:23.942+0000 7fad27fff640 10 client.17058 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.465+0000 7fad27fff640 1 -- 172.21.15.28:0/2288465617 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557acf9c48e0 con 0x557acfb5bd50 2024-05-09T08:39:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.465+0000 7fad27fff640 10 client.17058 handle_command_reply: tid=0 2024-05-09T08:39:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.465+0000 7fad51fef740 1 client.17058 shutdown 2024-05-09T08:39:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 -- 172.21.15.28:0/2288465617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae8f70 msgr2=0x557acfb59f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae8f70 0x557acfb59f80 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fad38020d70 tx=0x7fad38015a60 comp rx=0 tx=0).stop 2024-05-09T08:39:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 -- 172.21.15.28:0/2288465617 shutdown_connections 2024-05-09T08:39:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557acfb5bd50 0x557acfb5e170 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557acfae8f70 0x557acfb59f80 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557acfadaef0 0x557acfae8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 --2- 172.21.15.28:0/2288465617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557acfada5d0 0x557acfae84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 -- 172.21.15.28:0/2288465617 >> 172.21.15.28:0/2288465617 conn(0x557acfad6ae0 msgr2=0x557acfae3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad27fff640 0 client.17058 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 -- 172.21.15.28:0/2288465617 shutdown_connections 2024-05-09T08:39:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 1 -- 172.21.15.28:0/2288465617 wait complete. 2024-05-09T08:39:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.466+0000 7fad51fef740 20 client.17058 trim_cache size 0 max 16384 2024-05-09T08:39:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:39:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:39:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:39:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:39:25.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 -- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad300a2ec0 msgr2=0x7fad300a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad300a2ec0 0x7fad300a5380 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fad38008000 tx=0x7fad3801b000 comp rx=0 tx=0).stop 2024-05-09T08:39:25.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 -- 172.21.15.28:0/1231686360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad4806db20 msgr2=0x7fad48113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad4806db20 0x7fad48113910 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fad40006010 tx=0x7fad40031a20 comp rx=0 tx=0).stop 2024-05-09T08:39:25.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 -- 172.21.15.28:0/1231686360 shutdown_connections 2024-05-09T08:39:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad48114570 0x7fad4810a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad48108e00 0x7fad48113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fad300a2ec0 0x7fad300a5380 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 --2- 172.21.15.28:0/1231686360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad4806db20 0x7fad48113910 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.467+0000 7fad4df9e640 1 -- 172.21.15.28:0/1231686360 >> 172.21.15.28:0/1231686360 conn(0x7fad4806c770 msgr2=0x7fad48070120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.468+0000 7fad4df9e640 1 -- 172.21.15.28:0/1231686360 shutdown_connections 2024-05-09T08:39:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:25.468+0000 7fad4df9e640 1 -- 172.21.15.28:0/1231686360 wait complete. 2024-05-09T08:39:25.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:25 smithi165 ceph-mon[34540]: pgmap v419: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 245 op/s 2024-05-09T08:39:25.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:25 smithi184 ceph-mon[34509]: pgmap v419: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 245 op/s 2024-05-09T08:39:25.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:25 smithi028 ceph-mon[30119]: pgmap v419: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 245 op/s 2024-05-09T08:39:27.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:27 smithi165 ceph-mon[34540]: pgmap v420: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 958 KiB/s wr, 255 op/s 2024-05-09T08:39:27.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:27 smithi184 ceph-mon[34509]: pgmap v420: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 958 KiB/s wr, 255 op/s 2024-05-09T08:39:27.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:27 smithi028 ceph-mon[30119]: pgmap v420: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 958 KiB/s wr, 255 op/s 2024-05-09T08:39:29.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:29 smithi165 ceph-mon[34540]: pgmap v421: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 938 KiB/s wr, 227 op/s 2024-05-09T08:39:29.855 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:29 smithi184 ceph-mon[34509]: pgmap v421: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 938 KiB/s wr, 227 op/s 2024-05-09T08:39:29.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:29 smithi028 ceph-mon[30119]: pgmap v421: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 938 KiB/s wr, 227 op/s 2024-05-09T08:39:31.733 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:31 smithi028 ceph-mon[30119]: pgmap v422: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 266 op/s 2024-05-09T08:39:31.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:31 smithi165 ceph-mon[34540]: pgmap v422: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 266 op/s 2024-05-09T08:39:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:31 smithi184 ceph-mon[34509]: pgmap v422: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 266 op/s 2024-05-09T08:39:33.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:33 smithi165 ceph-mon[34540]: pgmap v423: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 1.0 MiB/s wr, 283 op/s 2024-05-09T08:39:33.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:33 smithi184 ceph-mon[34509]: pgmap v423: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 1.0 MiB/s wr, 283 op/s 2024-05-09T08:39:33.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:33 smithi028 ceph-mon[30119]: pgmap v423: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 1.0 MiB/s wr, 283 op/s 2024-05-09T08:39:35.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:35 smithi165 ceph-mon[34540]: pgmap v424: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 968 KiB/s wr, 264 op/s 2024-05-09T08:39:35.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:35 smithi184 ceph-mon[34509]: pgmap v424: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 968 KiB/s wr, 264 op/s 2024-05-09T08:39:35.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:35 smithi028 ceph-mon[30119]: pgmap v424: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 968 KiB/s wr, 264 op/s 2024-05-09T08:39:37.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:37 smithi165 ceph-mon[34540]: pgmap v425: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 842 KiB/s wr, 304 op/s 2024-05-09T08:39:37.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:37 smithi184 ceph-mon[34509]: pgmap v425: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 842 KiB/s wr, 304 op/s 2024-05-09T08:39:37.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:37 smithi028 ceph-mon[30119]: pgmap v425: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 842 KiB/s wr, 304 op/s 2024-05-09T08:39:38.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.620+0000 7f016267d640 1 -- 172.21.15.28:0/2415312480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101120 msgr2=0x7f015c10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.620+0000 7f016267d640 1 --2- 172.21.15.28:0/2415312480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101120 0x7f015c10d5e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f0154009cd0 tx=0x7f015402a910 comp rx=0 tx=0).stop 2024-05-09T08:39:38.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.621+0000 7f016267d640 1 -- 172.21.15.28:0/2415312480 shutdown_connections 2024-05-09T08:39:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.621+0000 7f016267d640 1 --2- 172.21.15.28:0/2415312480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c101120 0x7f015c10d5e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.621+0000 7f016267d640 1 --2- 172.21.15.28:0/2415312480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c100760 0x7f015c100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.621+0000 7f016267d640 1 --2- 172.21.15.28:0/2415312480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c106720 0x7f015c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.621+0000 7f016267d640 1 -- 172.21.15.28:0/2415312480 >> 172.21.15.28:0/2415312480 conn(0x7f015c0778b0 msgr2=0x7f015c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.622+0000 7f016267d640 1 -- 172.21.15.28:0/2415312480 shutdown_connections 2024-05-09T08:39:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.626+0000 7f016267d640 1 -- 172.21.15.28:0/2415312480 wait complete. 2024-05-09T08:39:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.626+0000 7f016267d640 1 Processor -- start 2024-05-09T08:39:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.627+0000 7f016267d640 1 -- start start 2024-05-09T08:39:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.627+0000 7f016267d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c100760 0x7f015c072f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.627+0000 7f016267d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c101120 0x7f015c073490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.627+0000 7f016267d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c106720 0x7f015c06c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.627+0000 7f016267d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f015c10fa40 con 0x7f015c106720 2024-05-09T08:39:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.627+0000 7f016267d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f015c10f8c0 con 0x7f015c100760 2024-05-09T08:39:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.627+0000 7f016267d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f015c10fbc0 con 0x7f015c101120 2024-05-09T08:39:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.628+0000 7f0160c1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c106720 0x7f015c06c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.628+0000 7f0160c1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c106720 0x7f015c06c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42962/0 (socket says 172.21.15.28:42962) 2024-05-09T08:39:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.628+0000 7f0160c1a640 1 -- 172.21.15.28:0/827358413 learned_addr learned my addr 172.21.15.28:0/827358413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.629+0000 7f0160c1a640 1 -- 172.21.15.28:0/827358413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c101120 msgr2=0x7f015c073490 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:39:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.629+0000 7f0160c1a640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c101120 0x7f015c073490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.629+0000 7f0160c1a640 1 -- 172.21.15.28:0/827358413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c100760 msgr2=0x7f015c072f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:39:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.629+0000 7f0160c1a640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c100760 0x7f015c072f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.629+0000 7f0160c1a640 1 -- 172.21.15.28:0/827358413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015c06ca30 con 0x7f015c106720 2024-05-09T08:39:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.630+0000 7f0160c1a640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c106720 0x7f015c06c200 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0154009bf0 tx=0x7f0154003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.630+0000 7f01597fa640 1 -- 172.21.15.28:0/827358413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154031660 con 0x7f015c106720 2024-05-09T08:39:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.630+0000 7f016267d640 1 -- 172.21.15.28:0/827358413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f015c06ccc0 con 0x7f015c106720 2024-05-09T08:39:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.630+0000 7f016267d640 1 -- 172.21.15.28:0/827358413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f015c070c50 con 0x7f015c106720 2024-05-09T08:39:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.631+0000 7f01597fa640 1 -- 172.21.15.28:0/827358413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0154031800 con 0x7f015c106720 2024-05-09T08:39:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.631+0000 7f01597fa640 1 -- 172.21.15.28:0/827358413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015402faa0 con 0x7f015c106720 2024-05-09T08:39:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.636+0000 7f016538d740 1 Processor -- start 2024-05-09T08:39:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.637+0000 7f01597fa640 1 -- 172.21.15.28:0/827358413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f015402fc40 con 0x7f015c106720 2024-05-09T08:39:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.638+0000 7f016538d740 1 -- start start 2024-05-09T08:39:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.638+0000 7f016538d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b04450 0x55e261b04830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.638+0000 7f016538d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261b0f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.638+0000 7f016538d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 0x55e261b11a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.638+0000 7f016538d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e261a69340 con 0x55e261b04450 2024-05-09T08:39:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.638+0000 7f016538d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e261a11290 con 0x55e261b0f630 2024-05-09T08:39:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.638+0000 7f016538d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e261a65300 con 0x55e261b04d70 2024-05-09T08:39:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.638+0000 7f015b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b04450 0x55e261b04830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.639+0000 7f015b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b04450 0x55e261b04830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42968/0 (socket says 172.21.15.28:42968) 2024-05-09T08:39:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.639+0000 7f015b7fe640 1 -- 172.21.15.28:0/2575749774 learned_addr learned my addr 172.21.15.28:0/2575749774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.639+0000 7f01597fa640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f01440a2fd0 0x7f01440a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.639+0000 7f0160c1a640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f01440a2fd0 0x7f01440a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.640+0000 7f01597fa640 1 -- 172.21.15.28:0/827358413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9615+0+0 (secure 0 0 0) 0x7f01540f0d70 con 0x7f015c106720 2024-05-09T08:39:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.640+0000 7f0160c1a640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f01440a2fd0 0x7f01440a5490 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f01540b9310 tx=0x7f01540b9280 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.642+0000 7f015b7fe640 1 --2- 172.21.15.28:0/2575749774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 0x55e261b11a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.642+0000 7f015b7fe640 1 -- 172.21.15.28:0/2575749774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 msgr2=0x55e261b0f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:39:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.642+0000 7f015b7fe640 1 --2- 172.21.15.28:0/2575749774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261b0f0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.642+0000 7f015b7fe640 1 -- 172.21.15.28:0/2575749774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 msgr2=0x55e261b11a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.642+0000 7f015b7fe640 1 --2- 172.21.15.28:0/2575749774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 0x55e261b11a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.642+0000 7f015b7fe640 1 -- 172.21.15.28:0/2575749774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e261a358e0 con 0x55e261b04450 2024-05-09T08:39:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.642+0000 7f015b7fe640 1 --2- 172.21.15.28:0/2575749774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b04450 0x55e261b04830 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f014c00bfd0 tx=0x7f014c00b080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff14c0f95ac3cfc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.645+0000 7f01427fc640 1 -- 172.21.15.28:0/2575749774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c011040 con 0x55e261b04450 2024-05-09T08:39:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.645+0000 7f01427fc640 1 -- 172.21.15.28:0/2575749774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f014c004500 con 0x55e261b04450 2024-05-09T08:39:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.645+0000 7f01427fc640 1 -- 172.21.15.28:0/2575749774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c004a60 con 0x55e261b04450 2024-05-09T08:39:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.646+0000 7f016538d740 1 -- 172.21.15.28:0/2575749774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b04450 msgr2=0x55e261b04830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.646+0000 7f016538d740 1 --2- 172.21.15.28:0/2575749774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b04450 0x55e261b04830 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f014c00bfd0 tx=0x7f014c00b080 comp rx=0 tx=0).stop 2024-05-09T08:39:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.646+0000 7f016538d740 1 -- 172.21.15.28:0/2575749774 shutdown_connections 2024-05-09T08:39:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.646+0000 7f016538d740 1 --2- 172.21.15.28:0/2575749774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 0x55e261b11a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.646+0000 7f016538d740 1 --2- 172.21.15.28:0/2575749774 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261b0f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.646+0000 7f016538d740 1 --2- 172.21.15.28:0/2575749774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b04450 0x55e261b04830 secure :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f014c00bfd0 tx=0x7f014c00b080 comp rx=0 tx=0).stop 2024-05-09T08:39:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.646+0000 7f016538d740 1 -- 172.21.15.28:0/2575749774 >> 172.21.15.28:0/2575749774 conn(0x55e261b00960 msgr2=0x55e261b00d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.652+0000 7f016538d740 1 -- 172.21.15.28:0/2575749774 shutdown_connections 2024-05-09T08:39:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.673+0000 7f016538d740 1 -- 172.21.15.28:0/2575749774 wait complete. 2024-05-09T08:39:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.673+0000 7f016538d740 1 -- start start 2024-05-09T08:39:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.673+0000 7f016538d740 1 Processor -- start 2024-05-09T08:39:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.674+0000 7f016538d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261b0c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.674+0000 7f016538d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 0x55e261b0cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.674+0000 7f016538d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0d110 0x55e261b1b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.674+0000 7f016538d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e261b12260 con 0x55e261b0d110 2024-05-09T08:39:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.674+0000 7f016538d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e261a24790 con 0x55e261b0f630 2024-05-09T08:39:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.674+0000 7f016538d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e261b123e0 con 0x55e261b04d70 2024-05-09T08:39:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.674+0000 7f016538d740 10 client.0 _ll_register_callbacks cb 0x55e261a5fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.677+0000 7f015bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 0x55e261b0cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.678+0000 7f015bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 0x55e261b0cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60252/0 (socket says 172.21.15.28:60252) 2024-05-09T08:39:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.678+0000 7f015bfff640 1 -- 172.21.15.28:0/2636960170 learned_addr learned my addr 172.21.15.28:0/2636960170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.679+0000 7f015b7fe640 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0d110 0x55e261b1b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.679+0000 7f015b7fe640 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261b0c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.680+0000 7f015b7fe640 1 -- 172.21.15.28:0/2636960170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 msgr2=0x55e261b0c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.680+0000 7f015b7fe640 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261b0c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.680+0000 7f015b7fe640 1 -- 172.21.15.28:0/2636960170 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 msgr2=0x55e261b0cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.680+0000 7f015b7fe640 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 0x55e261b0cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.680+0000 7f015b7fe640 1 -- 172.21.15.28:0/2636960170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e26177d210 con 0x55e261b0d110 2024-05-09T08:39:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.682+0000 7f015b7fe640 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0d110 0x55e261b1b730 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f014c00a000 tx=0x7f014c00b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17224143374cc165 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.682+0000 7f013b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:39:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.683+0000 7f013b7fe640 1 -- 172.21.15.28:0/2636960170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c02a060 con 0x55e261b0d110 2024-05-09T08:39:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.683+0000 7f016538d740 1 -- 172.21.15.28:0/2636960170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f014c004500 con 0x55e261b0d110 2024-05-09T08:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.683+0000 7f013b7fe640 1 -- 172.21.15.28:0/2636960170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9615+0+0 (secure 0 0 0) 0x7f014c0060f0 con 0x55e261b0d110 2024-05-09T08:39:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.684+0000 7f013b7fe640 1 -- 172.21.15.28:0/2636960170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f014c0062f0 con 0x55e261b0d110 2024-05-09T08:39:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.684+0000 7f016538d740 10 client.17070 fetch_fsmap learned FSMap version 67 2024-05-09T08:39:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.685+0000 7f016538d740 1 -- 172.21.15.28:0/2636960170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55e261849620 con 0x55e261b0d110 2024-05-09T08:39:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.685+0000 7f013b7fe640 1 -- 172.21.15.28:0/2636960170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f014c004500 con 0x55e261b0d110 2024-05-09T08:39:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.686+0000 7f013b7fe640 1 -- 172.21.15.28:0/2636960170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f014c020070 con 0x55e261b0d110 2024-05-09T08:39:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.686+0000 7f016538d740 10 client.17070 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:39:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.686+0000 7f016538d740 10 client.17070 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:39:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.686+0000 7f016538d740 20 client.17070 populate_metadata read hostname 'smithi028' 2024-05-09T08:39:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.686+0000 7f016538d740 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e261b1fcd0 0x55e261b220f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.686+0000 7f016538d740 4 client.17070 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.686+0000 7f016538d740 1 -- 172.21.15.28:0/2636960170 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e261a76a70 con 0x55e261b1fcd0 2024-05-09T08:39:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.686+0000 7f015bfff640 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e261b1fcd0 0x55e261b220f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.692+0000 7f015bfff640 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e261b1fcd0 0x55e261b220f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba58e22d597175fd server_cookie=8b8d6420b11ded95 in_seq=0 out_seq=0 2024-05-09T08:39:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.692+0000 7f013b7fe640 10 client.17070 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.700+0000 7f013b7fe640 1 -- 172.21.15.28:0/2636960170 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e261a76a70 con 0x55e261b1fcd0 2024-05-09T08:39:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.700+0000 7f013b7fe640 10 client.17070 handle_command_reply: tid=0 2024-05-09T08:39:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.700+0000 7f016538d740 1 client.17070 shutdown 2024-05-09T08:39:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.704+0000 7f016538d740 1 -- 172.21.15.28:0/2636960170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0d110 msgr2=0x55e261b1b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.704+0000 7f016538d740 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0d110 0x55e261b1b730 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f014c00a000 tx=0x7f014c00b6f0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.707+0000 7f016538d740 1 -- 172.21.15.28:0/2636960170 shutdown_connections 2024-05-09T08:39:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.707+0000 7f016538d740 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e261b1fcd0 0x55e261b220f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.707+0000 7f016538d740 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0d110 0x55e261b1b730 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.707+0000 7f016538d740 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0f630 0x55e261b0cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.707+0000 7f016538d740 1 --2- 172.21.15.28:0/2636960170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261b0c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.707+0000 7f016538d740 1 -- 172.21.15.28:0/2636960170 >> 172.21.15.28:0/2636960170 conn(0x55e261b00960 msgr2=0x55e261b066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.707+0000 7f013b7fe640 0 client.17070 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.709+0000 7f016538d740 1 -- 172.21.15.28:0/2636960170 shutdown_connections 2024-05-09T08:39:38.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.709+0000 7f016538d740 1 -- 172.21.15.28:0/2636960170 wait complete. 2024-05-09T08:39:38.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.711+0000 7f016538d740 20 client.17070 trim_cache size 0 max 16384 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.744+0000 7f016538d740 1 Processor -- start 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.744+0000 7f016538d740 1 -- start start 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.744+0000 7f016538d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261afb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.744+0000 7f016538d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 0x55e261aff880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.744+0000 7f016538d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 0x55e261b279c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.744+0000 7f016538d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0148010180 con 0x55e261b0f630 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.744+0000 7f016538d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0148011f00 con 0x55e261b0d110 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.744+0000 7f016538d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0148010910 con 0x55e261b04d70 2024-05-09T08:39:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.748+0000 7f015b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261afb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.748+0000 7f015b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261afb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:47180/0 (socket says 172.21.15.28:47180) 2024-05-09T08:39:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.748+0000 7f015b7fe640 1 -- 172.21.15.28:0/718736284 learned_addr learned my addr 172.21.15.28:0/718736284 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.749+0000 7f015bfff640 1 --2- 172.21.15.28:0/718736284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 0x55e261aff880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.750+0000 7f015b7fe640 1 -- 172.21.15.28:0/718736284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 msgr2=0x55e261aff880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.750+0000 7f015b7fe640 1 --2- 172.21.15.28:0/718736284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 0x55e261aff880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.750+0000 7f015b7fe640 1 -- 172.21.15.28:0/718736284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 msgr2=0x55e261b279c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.750+0000 7f015b7fe640 1 --2- 172.21.15.28:0/718736284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 0x55e261b279c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.750+0000 7f015b7fe640 1 -- 172.21.15.28:0/718736284 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e261b0b640 con 0x55e261b04d70 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.751+0000 7f015b7fe640 1 --2- 172.21.15.28:0/718736284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261afb8a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f014c026e60 tx=0x7f014c00b080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7affa088dd0c4903 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.753+0000 7f0141ffb640 1 -- 172.21.15.28:0/718736284 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c02a070 con 0x55e261b04d70 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.753+0000 7f0141ffb640 1 -- 172.21.15.28:0/718736284 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f014c0060f0 con 0x55e261b04d70 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.754+0000 7f016538d740 1 -- 172.21.15.28:0/718736284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 msgr2=0x55e261afb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.754+0000 7f016538d740 1 --2- 172.21.15.28:0/718736284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261afb8a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f014c026e60 tx=0x7f014c00b080 comp rx=0 tx=0).stop 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.754+0000 7f016538d740 1 -- 172.21.15.28:0/718736284 shutdown_connections 2024-05-09T08:39:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.754+0000 7f016538d740 1 --2- 172.21.15.28:0/718736284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 0x55e261b279c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.754+0000 7f016538d740 1 --2- 172.21.15.28:0/718736284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 0x55e261aff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.754+0000 7f016538d740 1 --2- 172.21.15.28:0/718736284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b04d70 0x55e261afb8a0 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f014c026e60 tx=0x7f014c00b080 comp rx=0 tx=0).stop 2024-05-09T08:39:38.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.754+0000 7f016538d740 1 -- 172.21.15.28:0/718736284 >> 172.21.15.28:0/718736284 conn(0x55e261b00960 msgr2=0x55e261aff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.756+0000 7f016538d740 1 -- 172.21.15.28:0/718736284 shutdown_connections 2024-05-09T08:39:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.756+0000 7f016538d740 1 -- 172.21.15.28:0/718736284 wait complete. 2024-05-09T08:39:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.756+0000 7f016538d740 1 -- start start 2024-05-09T08:39:38.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.757+0000 7f016538d740 1 Processor -- start 2024-05-09T08:39:38.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.761+0000 7f016538d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 0x55e261b7e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.761+0000 7f016538d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 0x55e261b7eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.761+0000 7f016538d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b12830 0x55e261b12cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.761+0000 7f016538d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0148011830 con 0x55e261b0f630 2024-05-09T08:39:38.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.761+0000 7f016538d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f01480110a0 con 0x55e261b0d110 2024-05-09T08:39:38.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.761+0000 7f016538d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f014800e160 con 0x55e261b12830 2024-05-09T08:39:38.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.761+0000 7f016538d740 10 client.0 _ll_register_callbacks cb 0x7f014c00ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:38.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.764+0000 7f015b7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 0x55e261b7e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.764+0000 7f015b7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b12830 0x55e261b12cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.764+0000 7f015bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 0x55e261b7eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.764+0000 7f015bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 0x55e261b7eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42992/0 (socket says 172.21.15.28:42992) 2024-05-09T08:39:38.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.764+0000 7f015bfff640 1 -- 172.21.15.28:0/1909172413 learned_addr learned my addr 172.21.15.28:0/1909172413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.767+0000 7f015bfff640 1 -- 172.21.15.28:0/1909172413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b12830 msgr2=0x55e261b12cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.767+0000 7f015bfff640 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b12830 0x55e261b12cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.767+0000 7f015bfff640 1 -- 172.21.15.28:0/1909172413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 msgr2=0x55e261b7e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.767+0000 7f015bfff640 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 0x55e261b7e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.767+0000 7f015bfff640 1 -- 172.21.15.28:0/1909172413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e261b132c0 con 0x55e261b0f630 2024-05-09T08:39:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.769+0000 7f015bfff640 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 0x55e261b7eae0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x55e261b22960 tx=0x7f015000bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ecd4226f05c560 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.771+0000 7f013b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:39:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.774+0000 7f013b7fe640 1 -- 172.21.15.28:0/1909172413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015001a740 con 0x55e261b0f630 2024-05-09T08:39:38.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.774+0000 7f013b7fe640 1 -- 172.21.15.28:0/1909172413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9615+0+0 (secure 0 0 0) 0x7f015001a8e0 con 0x55e261b0f630 2024-05-09T08:39:38.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.775+0000 7f016538d740 1 -- 172.21.15.28:0/1909172413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f014c0060f0 con 0x55e261b0f630 2024-05-09T08:39:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.777+0000 7f013b7fe640 1 -- 172.21.15.28:0/1909172413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f0150027050 con 0x55e261b0f630 2024-05-09T08:39:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.777+0000 7f016538d740 10 client.17073 fetch_fsmap learned FSMap version 67 2024-05-09T08:39:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.777+0000 7f016538d740 1 -- 172.21.15.28:0/1909172413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55e261b861a0 con 0x55e261b0f630 2024-05-09T08:39:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.778+0000 7f013b7fe640 1 -- 172.21.15.28:0/1909172413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f014c0060f0 con 0x55e261b0f630 2024-05-09T08:39:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.778+0000 7f013b7fe640 1 -- 172.21.15.28:0/1909172413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f01500035f0 con 0x55e261b0f630 2024-05-09T08:39:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.778+0000 7f016538d740 10 client.17073 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:39:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.778+0000 7f016538d740 10 client.17073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:39:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.778+0000 7f016538d740 20 client.17073 populate_metadata read hostname 'smithi028' 2024-05-09T08:39:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.778+0000 7f016538d740 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e261b86a40 0x55e261b88e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.778+0000 7f016538d740 4 client.17073 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:39:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.778+0000 7f016538d740 1 -- 172.21.15.28:0/1909172413 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e261a70dc0 con 0x55e261b86a40 2024-05-09T08:39:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.779+0000 7f015b7fe640 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e261b86a40 0x55e261b88e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.790+0000 7f015b7fe640 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e261b86a40 0x55e261b88e60 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3358831322e3cc5 server_cookie=c9505e315171ac26 in_seq=0 out_seq=0 2024-05-09T08:39:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:38.790+0000 7f013b7fe640 10 client.17073 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f013b7fe640 1 -- 172.21.15.28:0/1909172413 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e261a70dc0 con 0x55e261b86a40 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f013b7fe640 10 client.17073 handle_command_reply: tid=0 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f016538d740 1 client.17073 shutdown 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f016538d740 1 -- 172.21.15.28:0/1909172413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 msgr2=0x55e261b7eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f016538d740 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 0x55e261b7eae0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x55e261b22960 tx=0x7f015000bc50 comp rx=0 tx=0).stop 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f016538d740 1 -- 172.21.15.28:0/1909172413 shutdown_connections 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f016538d740 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e261b86a40 0x55e261b88e60 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f016538d740 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e261b12830 0x55e261b12cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f016538d740 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e261b0f630 0x55e261b7eae0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f016538d740 1 --2- 172.21.15.28:0/1909172413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e261b0d110 0x55e261b7e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.712+0000 7f016538d740 1 -- 172.21.15.28:0/1909172413 >> 172.21.15.28:0/1909172413 conn(0x55e261b00960 msgr2=0x55e261affa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:39.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.713+0000 7f013b7fe640 0 client.17073 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:39.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.713+0000 7f016538d740 1 -- 172.21.15.28:0/1909172413 shutdown_connections 2024-05-09T08:39:39.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.713+0000 7f016538d740 1 -- 172.21.15.28:0/1909172413 wait complete. 2024-05-09T08:39:39.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.713+0000 7f016538d740 20 client.17073 trim_cache size 0 max 16384 2024-05-09T08:39:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:39:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:39:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:39:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:39:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 -- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f01440a2fd0 msgr2=0x7f01440a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f01440a2fd0 0x7f01440a5490 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f01540b9310 tx=0x7f01540b9280 comp rx=0 tx=0).stop 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 -- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c106720 msgr2=0x7f015c06c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c106720 0x7f015c06c200 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0154009bf0 tx=0x7f0154003f10 comp rx=0 tx=0).stop 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 -- 172.21.15.28:0/827358413 shutdown_connections 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015c106720 0x7f015c06c200 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f01440a2fd0 0x7f01440a5490 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f015c101120 0x7f015c073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 --2- 172.21.15.28:0/827358413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015c100760 0x7f015c072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 -- 172.21.15.28:0/827358413 >> 172.21.15.28:0/827358413 conn(0x7f015c0778b0 msgr2=0x7f015c1031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 -- 172.21.15.28:0/827358413 shutdown_connections 2024-05-09T08:39:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:39.714+0000 7f016267d640 1 -- 172.21.15.28:0/827358413 wait complete. 2024-05-09T08:39:39.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:39 smithi165 ceph-mon[34540]: pgmap v426: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.0 MiB/s wr, 290 op/s 2024-05-09T08:39:39.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:39 smithi184 ceph-mon[34509]: pgmap v426: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.0 MiB/s wr, 290 op/s 2024-05-09T08:39:39.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:39 smithi028 ceph-mon[30119]: pgmap v426: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.0 MiB/s wr, 290 op/s 2024-05-09T08:39:41.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:41 smithi165 ceph-mon[34540]: pgmap v427: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.0 MiB/s wr, 299 op/s 2024-05-09T08:39:41.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:41 smithi184 ceph-mon[34509]: pgmap v427: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.0 MiB/s wr, 299 op/s 2024-05-09T08:39:41.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:41 smithi028 ceph-mon[30119]: pgmap v427: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.0 MiB/s wr, 299 op/s 2024-05-09T08:39:43.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:43 smithi165 ceph-mon[34540]: pgmap v428: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 319 op/s 2024-05-09T08:39:43.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:43 smithi184 ceph-mon[34509]: pgmap v428: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 319 op/s 2024-05-09T08:39:43.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:43 smithi028 ceph-mon[30119]: pgmap v428: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 319 op/s 2024-05-09T08:39:45.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:45 smithi165 ceph-mon[34540]: pgmap v429: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 1019 KiB/s wr, 285 op/s 2024-05-09T08:39:45.857 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:45 smithi184 ceph-mon[34509]: pgmap v429: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 1019 KiB/s wr, 285 op/s 2024-05-09T08:39:45.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:45 smithi028 ceph-mon[30119]: pgmap v429: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 1019 KiB/s wr, 285 op/s 2024-05-09T08:39:46.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:46 smithi028 ceph-mon[30119]: osdmap e118: 12 total, 12 up, 12 in 2024-05-09T08:39:47.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:46 smithi165 ceph-mon[34540]: osdmap e118: 12 total, 12 up, 12 in 2024-05-09T08:39:47.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:46 smithi184 ceph-mon[34509]: osdmap e118: 12 total, 12 up, 12 in 2024-05-09T08:39:47.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:47 smithi028 ceph-mon[30119]: pgmap v431: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 983 KiB/s wr, 273 op/s 2024-05-09T08:39:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:47 smithi165 ceph-mon[34540]: pgmap v431: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 983 KiB/s wr, 273 op/s 2024-05-09T08:39:48.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:47 smithi184 ceph-mon[34509]: pgmap v431: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 983 KiB/s wr, 273 op/s 2024-05-09T08:39:49.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:49 smithi028 ceph-mon[30119]: pgmap v432: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 873 KiB/s wr, 270 op/s 2024-05-09T08:39:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:49 smithi165 ceph-mon[34540]: pgmap v432: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 873 KiB/s wr, 270 op/s 2024-05-09T08:39:50.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:49 smithi184 ceph-mon[34509]: pgmap v432: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 873 KiB/s wr, 270 op/s 2024-05-09T08:39:51.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:51 smithi028 ceph-mon[30119]: pgmap v433: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 265 op/s 2024-05-09T08:39:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:51 smithi165 ceph-mon[34540]: pgmap v433: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 265 op/s 2024-05-09T08:39:52.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:51 smithi184 ceph-mon[34509]: pgmap v433: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 265 op/s 2024-05-09T08:39:52.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.872+0000 7f09ae49a640 1 -- 172.21.15.28:0/2920412271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09a811b010 msgr2=0x7f09a811d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:52.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.872+0000 7f09ae49a640 1 --2- 172.21.15.28:0/2920412271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09a811b010 0x7f09a811d400 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f099c009930 tx=0x7f099c02a4f0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.879+0000 7f09ae49a640 1 -- 172.21.15.28:0/2920412271 shutdown_connections 2024-05-09T08:39:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.879+0000 7f09ae49a640 1 --2- 172.21.15.28:0/2920412271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09a811b010 0x7f09a811d400 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.879+0000 7f09ae49a640 1 --2- 172.21.15.28:0/2920412271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09a8110400 0x7f09a8108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.879+0000 7f09ae49a640 1 --2- 172.21.15.28:0/2920412271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09a8110020 0x7f09a8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.879+0000 7f09ae49a640 1 -- 172.21.15.28:0/2920412271 >> 172.21.15.28:0/2920412271 conn(0x7f09a806c900 msgr2=0x7f09a806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:52.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.882+0000 7f09ae49a640 1 -- 172.21.15.28:0/2920412271 shutdown_connections 2024-05-09T08:39:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.885+0000 7f09ae49a640 1 -- 172.21.15.28:0/2920412271 wait complete. 2024-05-09T08:39:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.886+0000 7f09ae49a640 1 Processor -- start 2024-05-09T08:39:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.886+0000 7f09ae49a640 1 -- start start 2024-05-09T08:39:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.887+0000 7f09ae49a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09a8110020 0x7f09a81a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.887+0000 7f09ae49a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09a811b010 0x7f09a81a74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.887+0000 7f09ae49a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09a81ab830 0x7f09a81abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.887+0000 7f09ae49a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8120ab0 con 0x7f09a811b010 2024-05-09T08:39:52.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.887+0000 7f09ae49a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8120930 con 0x7f09a81ab830 2024-05-09T08:39:52.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.887+0000 7f09ae49a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8120c30 con 0x7f09a8110020 2024-05-09T08:39:52.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.887+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09a8110020 0x7f09a81a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.887+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09a8110020 0x7f09a81a6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:36620/0 (socket says 172.21.15.28:36620) 2024-05-09T08:39:52.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.887+0000 7f09a7fff640 1 -- 172.21.15.28:0/3564017868 learned_addr learned my addr 172.21.15.28:0/3564017868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:52.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.888+0000 7f09aca37640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09a81ab830 0x7f09a81abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.888+0000 7f09a7fff640 1 -- 172.21.15.28:0/3564017868 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09a81ab830 msgr2=0x7f09a81abce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.888+0000 7f09a7fff640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09a81ab830 0x7f09a81abce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.888+0000 7f09a7fff640 1 -- 172.21.15.28:0/3564017868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09a811b010 msgr2=0x7f09a81a74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.888+0000 7f09a7fff640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09a811b010 0x7f09a81a74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.888+0000 7f09a7fff640 1 -- 172.21.15.28:0/3564017868 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a81ac320 con 0x7f09a8110020 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.888+0000 7f09a7fff640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09a8110020 0x7f09a81a6f70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f099800d770 tx=0x7f099800dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.889+0000 7f09a57fa640 1 -- 172.21.15.28:0/3564017868 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998017070 con 0x7f09a8110020 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.889+0000 7f09ae49a640 1 -- 172.21.15.28:0/3564017868 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a81ac610 con 0x7f09a8110020 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.889+0000 7f09ae49a640 1 -- 172.21.15.28:0/3564017868 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a81b3e00 con 0x7f09a8110020 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.890+0000 7f09a57fa640 1 -- 172.21.15.28:0/3564017868 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0998005500 con 0x7f09a8110020 2024-05-09T08:39:52.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.890+0000 7f09a57fa640 1 -- 172.21.15.28:0/3564017868 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998004e00 con 0x7f09a8110020 2024-05-09T08:39:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.893+0000 7f09a57fa640 1 -- 172.21.15.28:0/3564017868 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0998010460 con 0x7f09a8110020 2024-05-09T08:39:52.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.896+0000 7f09a57fa640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f09900a2ef0 0x7f09900a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.896+0000 7f09a57fa640 1 -- 172.21.15.28:0/3564017868 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9643+0+0 (secure 0 0 0) 0x7f09980d03a0 con 0x7f09a8110020 2024-05-09T08:39:52.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.897+0000 7f09b11a6740 1 Processor -- start 2024-05-09T08:39:52.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.898+0000 7f09a77fe640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f09900a2ef0 0x7f09900a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.899+0000 7f09b11a6740 1 -- start start 2024-05-09T08:39:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.900+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755450 0x558afe755830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.900+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755d70 0x558afe7600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.900+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755450 0x558afe755830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.900+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755450 0x558afe755830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42498/0 (socket says 172.21.15.28:42498) 2024-05-09T08:39:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.900+0000 7f09a7fff640 1 -- 172.21.15.28:0/2101225498 learned_addr learned my addr 172.21.15.28:0/2101225498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.900+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 0x558afe762a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.900+0000 7f09b11a6740 1 -- 172.21.15.28:0/2101225498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558afe6ba340 con 0x558afe755450 2024-05-09T08:39:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.901+0000 7f09b11a6740 1 -- 172.21.15.28:0/2101225498 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558afe662290 con 0x558afe760630 2024-05-09T08:39:52.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.901+0000 7f09b11a6740 1 -- 172.21.15.28:0/2101225498 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558afe6b6300 con 0x558afe755d70 2024-05-09T08:39:52.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.901+0000 7f09aca37640 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 0x558afe762a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.901+0000 7f09a7fff640 1 -- 172.21.15.28:0/2101225498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755d70 msgr2=0x558afe7600f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:39:52.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.901+0000 7f09a7fff640 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755d70 0x558afe7600f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.901+0000 7f09a7fff640 1 -- 172.21.15.28:0/2101225498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 msgr2=0x558afe762a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:52.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.901+0000 7f09a7fff640 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 0x558afe762a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.901+0000 7f09a7fff640 1 -- 172.21.15.28:0/2101225498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558afe6868e0 con 0x558afe755450 2024-05-09T08:39:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.901+0000 7f09aca37640 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 0x558afe762a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.902+0000 7f09a7fff640 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755450 0x558afe755830 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f0998004a50 tx=0x7f0998098040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a1b475b8189afa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.902+0000 7f098e7fc640 1 -- 172.21.15.28:0/2101225498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998020070 con 0x558afe755450 2024-05-09T08:39:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.903+0000 7f098e7fc640 1 -- 172.21.15.28:0/2101225498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099809cb00 con 0x558afe755450 2024-05-09T08:39:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.903+0000 7f098e7fc640 1 -- 172.21.15.28:0/2101225498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998098c10 con 0x558afe755450 2024-05-09T08:39:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.904+0000 7f09b11a6740 1 -- 172.21.15.28:0/2101225498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755450 msgr2=0x558afe755830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.904+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755450 0x558afe755830 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f0998004a50 tx=0x7f0998098040 comp rx=0 tx=0).stop 2024-05-09T08:39:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.904+0000 7f09b11a6740 1 -- 172.21.15.28:0/2101225498 shutdown_connections 2024-05-09T08:39:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.904+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 0x558afe762a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.904+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755d70 0x558afe7600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.904+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2101225498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755450 0x558afe755830 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.904+0000 7f09b11a6740 1 -- 172.21.15.28:0/2101225498 >> 172.21.15.28:0/2101225498 conn(0x558afe751960 msgr2=0x558afe751d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.905+0000 7f09b11a6740 1 -- 172.21.15.28:0/2101225498 shutdown_connections 2024-05-09T08:39:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.907+0000 7f09b11a6740 1 -- 172.21.15.28:0/2101225498 wait complete. 2024-05-09T08:39:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.907+0000 7f09b11a6740 1 -- start start 2024-05-09T08:39:52.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.908+0000 7f09b11a6740 1 Processor -- start 2024-05-09T08:39:52.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.908+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755450 0x558afe75d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.908+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755d70 0x558afe75da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.908+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.909+0000 7f09b11a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558afe7631a0 con 0x558afe760630 2024-05-09T08:39:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.909+0000 7f09aca37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.909+0000 7f09aca37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42500/0 (socket says 172.21.15.28:42500) 2024-05-09T08:39:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.909+0000 7f09aca37640 1 -- 172.21.15.28:0/2429817390 learned_addr learned my addr 172.21.15.28:0/2429817390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.909+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558afe675790 con 0x558afe755450 2024-05-09T08:39:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.909+0000 7f09a7fff640 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755450 0x558afe75d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.909+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558afe763320 con 0x558afe755d70 2024-05-09T08:39:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.909+0000 7f09b11a6740 10 client.0 _ll_register_callbacks cb 0x558afe6b0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.910+0000 7f09aca37640 1 -- 172.21.15.28:0/2429817390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755d70 msgr2=0x558afe75da40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:39:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.910+0000 7f09aca37640 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755d70 0x558afe75da40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.910+0000 7f09aca37640 1 -- 172.21.15.28:0/2429817390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755450 msgr2=0x558afe75d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.910+0000 7f09aca37640 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755450 0x558afe75d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.910+0000 7f09aca37640 1 -- 172.21.15.28:0/2429817390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558afe3ce210 con 0x558afe760630 2024-05-09T08:39:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.911+0000 7f09aca37640 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75df80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f099c009930 tx=0x7f099c005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6cea80617bb7308 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.911+0000 7f09877fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:39:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.911+0000 7f09877fe640 1 -- 172.21.15.28:0/2429817390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c035030 con 0x558afe760630 2024-05-09T08:39:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.911+0000 7f09877fe640 1 -- 172.21.15.28:0/2429817390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9643+0+0 (secure 0 0 0) 0x7f099c0046c0 con 0x558afe760630 2024-05-09T08:39:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.911+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f099809cb00 con 0x558afe760630 2024-05-09T08:39:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09877fe640 1 -- 172.21.15.28:0/2429817390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f099c0048c0 con 0x558afe760630 2024-05-09T08:39:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09b11a6740 10 client.17082 fetch_fsmap learned FSMap version 67 2024-05-09T08:39:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x558afe49a620 con 0x558afe760630 2024-05-09T08:39:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09877fe640 1 -- 172.21.15.28:0/2429817390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099809cb00 con 0x558afe760630 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09877fe640 1 -- 172.21.15.28:0/2429817390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f099c0315b0 con 0x558afe760630 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09b11a6740 10 client.17082 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09b11a6740 10 client.17082 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09b11a6740 20 client.17082 populate_metadata read hostname 'smithi028' 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558afe7701b0 0x558afe7725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09b11a6740 4 client.17082 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.912+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558afe6c7a70 con 0x558afe7701b0 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.913+0000 7f09a7fff640 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558afe7701b0 0x558afe7725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.913+0000 7f09a7fff640 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558afe7701b0 0x558afe7725d0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d3c5924b511ce77 server_cookie=39bf54d919ba91e8 in_seq=0 out_seq=0 2024-05-09T08:39:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.913+0000 7f09877fe640 10 client.17082 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:52.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.916+0000 7f09a77fe640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f09900a2ef0 0x7f09900a53b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x558afe7569f0 tx=0x7f09a0009250 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.918+0000 7f09877fe640 1 -- 172.21.15.28:0/2429817390 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558afe6c7a70 con 0x558afe7701b0 2024-05-09T08:39:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.918+0000 7f09877fe640 10 client.17082 handle_command_reply: tid=0 2024-05-09T08:39:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.918+0000 7f09b11a6740 1 client.17082 shutdown 2024-05-09T08:39:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.921+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 msgr2=0x558afe75df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.921+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75df80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f099c009930 tx=0x7f099c005250 comp rx=0 tx=0).stop 2024-05-09T08:39:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.923+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 shutdown_connections 2024-05-09T08:39:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.923+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558afe7701b0 0x558afe7725d0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.923+0000 7f09877fe640 0 client.17082 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.923+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75df80 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.923+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755d70 0x558afe75da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.923+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2429817390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755450 0x558afe75d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.923+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 >> 172.21.15.28:0/2429817390 conn(0x558afe751960 msgr2=0x558afe7576e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.923+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 shutdown_connections 2024-05-09T08:39:52.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.924+0000 7f09b11a6740 1 -- 172.21.15.28:0/2429817390 wait complete. 2024-05-09T08:39:52.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.926+0000 7f09b11a6740 20 client.17082 trim_cache size 0 max 16384 2024-05-09T08:39:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.958+0000 7f09b11a6740 1 Processor -- start 2024-05-09T08:39:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.962+0000 7f09b11a6740 1 -- start start 2024-05-09T08:39:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.962+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 0x558afe75fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.962+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755d70 0x558afe76c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.962+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 0x558afe75c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.962+0000 7f09b11a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0994010130 con 0x558afe755d70 2024-05-09T08:39:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.962+0000 7f09b11a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0994011eb0 con 0x558afe760630 2024-05-09T08:39:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.962+0000 7f09b11a6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09940108c0 con 0x558afe755450 2024-05-09T08:39:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.963+0000 7f09a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755d70 0x558afe76c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.963+0000 7f09a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755d70 0x558afe76c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42508/0 (socket says 172.21.15.28:42508) 2024-05-09T08:39:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.963+0000 7f09a77fe640 1 -- 172.21.15.28:0/51989447 learned_addr learned my addr 172.21.15.28:0/51989447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.963+0000 7f09aca37640 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 0x558afe75c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.963+0000 7f09a7fff640 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 0x558afe75fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.969+0000 7f09a77fe640 1 -- 172.21.15.28:0/51989447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 msgr2=0x558afe75fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.969+0000 7f09a77fe640 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 0x558afe75fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.969+0000 7f09a77fe640 1 -- 172.21.15.28:0/51989447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 msgr2=0x558afe75c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.969+0000 7f09a77fe640 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 0x558afe75c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.969+0000 7f09a77fe640 1 -- 172.21.15.28:0/51989447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558afe755ba0 con 0x558afe755d70 2024-05-09T08:39:52.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.970+0000 7f09a7fff640 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 0x558afe75fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:52.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.973+0000 7f09a77fe640 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755d70 0x558afe76c0b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f09a000ae40 tx=0x7f09a000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315c9f4eaa4bee1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:52.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.976+0000 7f098dffb640 1 -- 172.21.15.28:0/51989447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0009b20 con 0x558afe755d70 2024-05-09T08:39:52.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.976+0000 7f098dffb640 1 -- 172.21.15.28:0/51989447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a001d070 con 0x558afe755d70 2024-05-09T08:39:52.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.976+0000 7f098dffb640 1 -- 172.21.15.28:0/51989447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0018730 con 0x558afe755d70 2024-05-09T08:39:52.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.977+0000 7f09b11a6740 1 -- 172.21.15.28:0/51989447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755d70 msgr2=0x558afe76c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:52.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.977+0000 7f09b11a6740 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755d70 0x558afe76c0b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f09a000ae40 tx=0x7f09a000f000 comp rx=0 tx=0).stop 2024-05-09T08:39:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.980+0000 7f09b11a6740 1 -- 172.21.15.28:0/51989447 shutdown_connections 2024-05-09T08:39:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.980+0000 7f09b11a6740 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe760630 0x558afe75c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.980+0000 7f09b11a6740 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe755d70 0x558afe76c0b0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.980+0000 7f09b11a6740 1 --2- 172.21.15.28:0/51989447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 0x558afe75fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.980+0000 7f09b11a6740 1 -- 172.21.15.28:0/51989447 >> 172.21.15.28:0/51989447 conn(0x558afe751960 msgr2=0x558afe750170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.986+0000 7f09b11a6740 1 -- 172.21.15.28:0/51989447 shutdown_connections 2024-05-09T08:39:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.988+0000 7f09b11a6740 1 -- 172.21.15.28:0/51989447 wait complete. 2024-05-09T08:39:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.989+0000 7f09b11a6740 1 -- start start 2024-05-09T08:39:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.989+0000 7f09b11a6740 1 Processor -- start 2024-05-09T08:39:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.990+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 0x558afe768820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.990+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755d70 0x558afe759d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.990+0000 7f09b11a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.990+0000 7f09b11a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09940117e0 con 0x558afe760630 2024-05-09T08:39:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.990+0000 7f09b11a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0994011050 con 0x558afe755d70 2024-05-09T08:39:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.990+0000 7f09b11a6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f099400e110 con 0x558afe755450 2024-05-09T08:39:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.990+0000 7f09b11a6740 10 client.0 _ll_register_callbacks cb 0x7f099c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.991+0000 7f09aca37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.991+0000 7f09aca37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42512/0 (socket says 172.21.15.28:42512) 2024-05-09T08:39:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.991+0000 7f09aca37640 1 -- 172.21.15.28:0/2304572062 learned_addr learned my addr 172.21.15.28:0/2304572062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:39:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.992+0000 7f09a77fe640 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755d70 0x558afe759d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.995+0000 7f09aca37640 1 -- 172.21.15.28:0/2304572062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 msgr2=0x558afe768820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:39:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.995+0000 7f09aca37640 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 0x558afe768820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.995+0000 7f09aca37640 1 -- 172.21.15.28:0/2304572062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755d70 msgr2=0x558afe759d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.995+0000 7f09aca37640 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755d70 0x558afe759d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:52.995+0000 7f09aca37640 1 -- 172.21.15.28:0/2304572062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558afe75a8d0 con 0x558afe760630 2024-05-09T08:39:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.003+0000 7f09aca37640 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75a290 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f099c02a530 tx=0x7f099c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29835f27b9ab65cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:53.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.004+0000 7f09877fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:39:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.008+0000 7f09877fe640 1 -- 172.21.15.28:0/2304572062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c035070 con 0x558afe760630 2024-05-09T08:39:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.008+0000 7f09877fe640 1 -- 172.21.15.28:0/2304572062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9643+0+0 (secure 0 0 0) 0x7f099c004aa0 con 0x558afe760630 2024-05-09T08:39:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.012+0000 7f09b11a6740 1 -- 172.21.15.28:0/2304572062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a001d070 con 0x558afe760630 2024-05-09T08:39:53.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.013+0000 7f09877fe640 1 -- 172.21.15.28:0/2304572062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f099c0317c0 con 0x558afe760630 2024-05-09T08:39:53.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.018+0000 7f09b11a6740 10 client.17088 fetch_fsmap learned FSMap version 67 2024-05-09T08:39:53.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.018+0000 7f09b11a6740 1 -- 172.21.15.28:0/2304572062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x558afe7e75a0 con 0x558afe760630 2024-05-09T08:39:53.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.024+0000 7f09877fe640 1 -- 172.21.15.28:0/2304572062 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a001d070 con 0x558afe760630 2024-05-09T08:39:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.024+0000 7f09877fe640 1 -- 172.21.15.28:0/2304572062 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f099c044470 con 0x558afe760630 2024-05-09T08:39:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.024+0000 7f09b11a6740 10 client.17088 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:39:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.024+0000 7f09b11a6740 10 client.17088 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:39:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.024+0000 7f09b11a6740 20 client.17088 populate_metadata read hostname 'smithi028' 2024-05-09T08:39:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.024+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558afe7701b0 0x558afe7e9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.024+0000 7f09b11a6740 4 client.17088 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:39:53.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.024+0000 7f09b11a6740 1 -- 172.21.15.28:0/2304572062 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558afe6c1dc0 con 0x558afe7701b0 2024-05-09T08:39:53.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.027+0000 7f09a7fff640 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558afe7701b0 0x558afe7e9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.032+0000 7f09a7fff640 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558afe7701b0 0x558afe7e9d90 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9847c93ec0bdd0f5 server_cookie=4405985e8697b330 in_seq=0 out_seq=0 2024-05-09T08:39:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:53.032+0000 7f09877fe640 10 client.17088 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:53.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:53 smithi028 ceph-mon[30119]: pgmap v434: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 629 KiB/s wr, 232 op/s 2024-05-09T08:39:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:53 smithi165 ceph-mon[34540]: pgmap v434: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 629 KiB/s wr, 232 op/s 2024-05-09T08:39:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.102+0000 7f09877fe640 1 -- 172.21.15.28:0/2304572062 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558afe6c1dc0 con 0x558afe7701b0 2024-05-09T08:39:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.102+0000 7f09877fe640 10 client.17088 handle_command_reply: tid=0 2024-05-09T08:39:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.102+0000 7f09b11a6740 1 client.17088 shutdown 2024-05-09T08:39:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 -- 172.21.15.28:0/2304572062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 msgr2=0x558afe75a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75a290 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f099c02a530 tx=0x7f099c005780 comp rx=0 tx=0).stop 2024-05-09T08:39:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 -- 172.21.15.28:0/2304572062 shutdown_connections 2024-05-09T08:39:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558afe7701b0 0x558afe7e9d90 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558afe760630 0x558afe75a290 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558afe755d70 0x558afe759d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 --2- 172.21.15.28:0/2304572062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558afe755450 0x558afe768820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 -- 172.21.15.28:0/2304572062 >> 172.21.15.28:0/2304572062 conn(0x558afe751960 msgr2=0x558afe750a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09877fe640 0 client.17088 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:39:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 -- 172.21.15.28:0/2304572062 shutdown_connections 2024-05-09T08:39:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.103+0000 7f09b11a6740 1 -- 172.21.15.28:0/2304572062 wait complete. 2024-05-09T08:39:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.104+0000 7f09b11a6740 20 client.17088 trim_cache size 0 max 16384 2024-05-09T08:39:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:39:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:39:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:39:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:39:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 -- 172.21.15.28:0/3564017868 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f09900a2ef0 msgr2=0x7f09900a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f09900a2ef0 0x7f09900a53b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x558afe7569f0 tx=0x7f09a0009250 comp rx=0 tx=0).stop 2024-05-09T08:39:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 -- 172.21.15.28:0/3564017868 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09a8110020 msgr2=0x7f09a81a6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09a8110020 0x7f09a81a6f70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f099800d770 tx=0x7f099800dc30 comp rx=0 tx=0).stop 2024-05-09T08:39:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 -- 172.21.15.28:0/3564017868 shutdown_connections 2024-05-09T08:39:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09a81ab830 0x7f09a81abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09a811b010 0x7f09a81a74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f09900a2ef0 0x7f09900a53b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 --2- 172.21.15.28:0/3564017868 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09a8110020 0x7f09a81a6f70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 -- 172.21.15.28:0/3564017868 >> 172.21.15.28:0/3564017868 conn(0x7f09a806c900 msgr2=0x7f09a810dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 -- 172.21.15.28:0/3564017868 shutdown_connections 2024-05-09T08:39:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:39:54.105+0000 7f09ae49a640 1 -- 172.21.15.28:0/3564017868 wait complete. 2024-05-09T08:39:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:53 smithi184 ceph-mon[34509]: pgmap v434: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 629 KiB/s wr, 232 op/s 2024-05-09T08:39:55.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:55 smithi028 ceph-mon[30119]: pgmap v435: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 622 KiB/s wr, 232 op/s 2024-05-09T08:39:56.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:55 smithi165 ceph-mon[34540]: pgmap v435: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 622 KiB/s wr, 232 op/s 2024-05-09T08:39:56.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:55 smithi184 ceph-mon[34509]: pgmap v435: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 622 KiB/s wr, 232 op/s 2024-05-09T08:39:57.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:57 smithi028 ceph-mon[30119]: pgmap v436: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 952 KiB/s wr, 251 op/s 2024-05-09T08:39:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:57 smithi165 ceph-mon[34540]: pgmap v436: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 952 KiB/s wr, 251 op/s 2024-05-09T08:39:58.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:57 smithi184 ceph-mon[34509]: pgmap v436: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 952 KiB/s wr, 251 op/s 2024-05-09T08:39:59.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:39:59 smithi028 ceph-mon[30119]: pgmap v437: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 852 KiB/s wr, 246 op/s 2024-05-09T08:40:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:39:59 smithi165 ceph-mon[34540]: pgmap v437: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 852 KiB/s wr, 246 op/s 2024-05-09T08:40:00.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:39:59 smithi184 ceph-mon[34509]: pgmap v437: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 852 KiB/s wr, 246 op/s 2024-05-09T08:40:00.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:39:59.999+0000 7f8912d3a700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 4 MDSs report damaged metadata 2024-05-09T08:40:01.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:00 smithi165 ceph-mon[34540]: overall HEALTH_ERR 4 MDSs report damaged metadata 2024-05-09T08:40:01.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:00 smithi184 ceph-mon[34509]: overall HEALTH_ERR 4 MDSs report damaged metadata 2024-05-09T08:40:01.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:00 smithi028 ceph-mon[30119]: overall HEALTH_ERR 4 MDSs report damaged metadata 2024-05-09T08:40:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:01 smithi165 ceph-mon[34540]: pgmap v438: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 712 KiB/s wr, 255 op/s 2024-05-09T08:40:02.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:01 smithi184 ceph-mon[34509]: pgmap v438: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 712 KiB/s wr, 255 op/s 2024-05-09T08:40:02.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:01 smithi028 ceph-mon[30119]: pgmap v438: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 712 KiB/s wr, 255 op/s 2024-05-09T08:40:04.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:03 smithi165 ceph-mon[34540]: pgmap v439: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 886 KiB/s wr, 271 op/s 2024-05-09T08:40:04.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:03 smithi184 ceph-mon[34509]: pgmap v439: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 886 KiB/s wr, 271 op/s 2024-05-09T08:40:04.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:03 smithi028 ceph-mon[30119]: pgmap v439: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 886 KiB/s wr, 271 op/s 2024-05-09T08:40:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:05 smithi165 ceph-mon[34540]: pgmap v440: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 855 KiB/s wr, 251 op/s 2024-05-09T08:40:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:05 smithi184 ceph-mon[34509]: pgmap v440: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 855 KiB/s wr, 251 op/s 2024-05-09T08:40:06.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:05 smithi028 ceph-mon[30119]: pgmap v440: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 855 KiB/s wr, 251 op/s 2024-05-09T08:40:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.263+0000 7f0044c4c640 1 -- 172.21.15.28:0/1350090700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0040103a30 msgr2=0x7f0040103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.263+0000 7f0044c4c640 1 --2- 172.21.15.28:0/1350090700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0040103a30 0x7f0040103eb0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0028009890 tx=0x7f002802a360 comp rx=0 tx=0).stop 2024-05-09T08:40:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.269+0000 7f0044c4c640 1 -- 172.21.15.28:0/1350090700 shutdown_connections 2024-05-09T08:40:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.269+0000 7f0044c4c640 1 --2- 172.21.15.28:0/1350090700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00401043f0 0x7f0040110900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.269+0000 7f0044c4c640 1 --2- 172.21.15.28:0/1350090700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0040103a30 0x7f0040103eb0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.269+0000 7f0044c4c640 1 --2- 172.21.15.28:0/1350090700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0040109a40 0x7f0040109e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.269+0000 7f0044c4c640 1 -- 172.21.15.28:0/1350090700 >> 172.21.15.28:0/1350090700 conn(0x7f00400ff560 msgr2=0x7f0040101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:07.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.273+0000 7f0044c4c640 1 -- 172.21.15.28:0/1350090700 shutdown_connections 2024-05-09T08:40:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.276+0000 7f0044c4c640 1 -- 172.21.15.28:0/1350090700 wait complete. 2024-05-09T08:40:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.277+0000 7f0044c4c640 1 Processor -- start 2024-05-09T08:40:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.280+0000 7f0044c4c640 1 -- start start 2024-05-09T08:40:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.280+0000 7f0044c4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0040103a30 0x7f00401107c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.280+0000 7f0044c4c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00401043f0 0x7f004010d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.280+0000 7f0044c4c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0040109a40 0x7f004010ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.280+0000 7f0044c4c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0040113ea0 con 0x7f0040103a30 2024-05-09T08:40:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.280+0000 7f0044c4c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0040113d20 con 0x7f00401043f0 2024-05-09T08:40:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.280+0000 7f0044c4c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0040114020 con 0x7f0040109a40 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.283+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0040109a40 0x7f004010ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.283+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0040109a40 0x7f004010ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38258/0 (socket says 172.21.15.28:38258) 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.283+0000 7f003ed9d640 1 -- 172.21.15.28:0/3886869657 learned_addr learned my addr 172.21.15.28:0/3886869657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.283+0000 7f0037fff640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00401043f0 0x7f004010d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.283+0000 7f003ed9d640 1 -- 172.21.15.28:0/3886869657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00401043f0 msgr2=0x7f004010d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.283+0000 7f003ed9d640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00401043f0 0x7f004010d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.284+0000 7f003ed9d640 1 -- 172.21.15.28:0/3886869657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0040103a30 msgr2=0x7f00401107c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.284+0000 7f003ed9d640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0040103a30 0x7f00401107c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.284+0000 7f003ed9d640 1 -- 172.21.15.28:0/3886869657 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004010e680 con 0x7f0040109a40 2024-05-09T08:40:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.284+0000 7f003ed9d640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0040109a40 0x7f004010ddc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0038009fd0 tx=0x7f003800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.287+0000 7f00377fe640 1 -- 172.21.15.28:0/3886869657 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038004b20 con 0x7f0040109a40 2024-05-09T08:40:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.287+0000 7f0044c4c640 1 -- 172.21.15.28:0/3886869657 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f004006b370 con 0x7f0040109a40 2024-05-09T08:40:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.287+0000 7f0044c4c640 1 -- 172.21.15.28:0/3886869657 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f004006b7d0 con 0x7f0040109a40 2024-05-09T08:40:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.288+0000 7f00377fe640 1 -- 172.21.15.28:0/3886869657 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0038003e50 con 0x7f0040109a40 2024-05-09T08:40:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.288+0000 7f00377fe640 1 -- 172.21.15.28:0/3886869657 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003800b630 con 0x7f0040109a40 2024-05-09T08:40:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.292+0000 7f00377fe640 1 -- 172.21.15.28:0/3886869657 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f003801d020 con 0x7f0040109a40 2024-05-09T08:40:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.293+0000 7f00377fe640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f002c0a2fd0 0x7f002c0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.293+0000 7f00377fe640 1 -- 172.21.15.28:0/3886869657 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9643+0+0 (secure 0 0 0) 0x7f00380cf720 con 0x7f0040109a40 2024-05-09T08:40:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.296+0000 7f0047a31740 1 Processor -- start 2024-05-09T08:40:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.296+0000 7f0047a31740 1 -- start start 2024-05-09T08:40:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.297+0000 7f0047a31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fc1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.297+0000 7f0047a31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 0x55fe37fcc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.297+0000 7f0047a31740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fcea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.297+0000 7f0047a31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe37f26340 con 0x55fe37fc1450 2024-05-09T08:40:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.297+0000 7f0047a31740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fe37ece290 con 0x55fe37fcc630 2024-05-09T08:40:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.297+0000 7f0047a31740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fe37f22300 con 0x55fe37fc1d70 2024-05-09T08:40:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.299+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fcea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.299+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fcea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:54240/0 (socket says 172.21.15.28:54240) 2024-05-09T08:40:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.299+0000 7f003ed9d640 1 -- 172.21.15.28:0/164964151 learned_addr learned my addr 172.21.15.28:0/164964151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.299+0000 7f003e59c640 1 --2- 172.21.15.28:0/164964151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fc1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.299+0000 7f003e59c640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f002c0a2fd0 0x7f002c0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.300+0000 7f003ed9d640 1 -- 172.21.15.28:0/164964151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 msgr2=0x55fe37fcc0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:40:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.300+0000 7f003ed9d640 1 --2- 172.21.15.28:0/164964151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 0x55fe37fcc0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.300+0000 7f003ed9d640 1 -- 172.21.15.28:0/164964151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 msgr2=0x55fe37fc1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.300+0000 7f003ed9d640 1 --2- 172.21.15.28:0/164964151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fc1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.300+0000 7f003ed9d640 1 -- 172.21.15.28:0/164964151 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe37ef28e0 con 0x55fe37fcc630 2024-05-09T08:40:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.300+0000 7f003ed9d640 1 --2- 172.21.15.28:0/164964151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fcea20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0038098150 tx=0x7f003809be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f764cf601945e1a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.300+0000 7f0034ff9640 1 -- 172.21.15.28:0/164964151 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380b4070 con 0x55fe37fcc630 2024-05-09T08:40:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.300+0000 7f0034ff9640 1 -- 172.21.15.28:0/164964151 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00380a43f0 con 0x55fe37fcc630 2024-05-09T08:40:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.300+0000 7f0034ff9640 1 -- 172.21.15.28:0/164964151 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380cf070 con 0x55fe37fcc630 2024-05-09T08:40:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.301+0000 7f0047a31740 1 -- 172.21.15.28:0/164964151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 msgr2=0x55fe37fcea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:07.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.301+0000 7f0047a31740 1 --2- 172.21.15.28:0/164964151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fcea20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0038098150 tx=0x7f003809be80 comp rx=0 tx=0).stop 2024-05-09T08:40:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.302+0000 7f0047a31740 1 -- 172.21.15.28:0/164964151 shutdown_connections 2024-05-09T08:40:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.302+0000 7f0047a31740 1 --2- 172.21.15.28:0/164964151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fcea20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.302+0000 7f0047a31740 1 --2- 172.21.15.28:0/164964151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 0x55fe37fcc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.302+0000 7f0047a31740 1 --2- 172.21.15.28:0/164964151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fc1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.302+0000 7f0047a31740 1 -- 172.21.15.28:0/164964151 >> 172.21.15.28:0/164964151 conn(0x55fe37fbd960 msgr2=0x55fe37fbdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:07.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.302+0000 7f003e59c640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f002c0a2fd0 0x7f002c0a5490 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0030005fd0 tx=0x7f0030005c90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:07.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.306+0000 7f0047a31740 1 -- 172.21.15.28:0/164964151 shutdown_connections 2024-05-09T08:40:07.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.308+0000 7f0047a31740 1 -- 172.21.15.28:0/164964151 wait complete. 2024-05-09T08:40:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.308+0000 7f0047a31740 1 -- start start 2024-05-09T08:40:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.308+0000 7f0047a31740 1 Processor -- start 2024-05-09T08:40:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.310+0000 7f0047a31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fc94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.310+0000 7f0047a31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 0x55fe37fc9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.310+0000 7f0047a31740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fc9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.311+0000 7f0047a31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe37fcf260 con 0x55fe37fc1450 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.311+0000 7f0047a31740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55fe37ee1790 con 0x55fe37fcc630 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.311+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fc9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.311+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fc9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:54242/0 (socket says 172.21.15.28:54242) 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.311+0000 7f003ed9d640 1 -- 172.21.15.28:0/1527676474 learned_addr learned my addr 172.21.15.28:0/1527676474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.311+0000 7f0047a31740 1 -- 172.21.15.28:0/1527676474 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fe37fcf3e0 con 0x55fe37fc1d70 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.311+0000 7f0047a31740 10 client.0 _ll_register_callbacks cb 0x55fe37f1ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.312+0000 7f003ed9d640 1 -- 172.21.15.28:0/1527676474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 msgr2=0x55fe37fc9a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.312+0000 7f003ed9d640 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 0x55fe37fc9a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.312+0000 7f003ed9d640 1 -- 172.21.15.28:0/1527676474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 msgr2=0x55fe37fc94f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:40:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.312+0000 7f003ed9d640 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fc94f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.312+0000 7f003ed9d640 1 -- 172.21.15.28:0/1527676474 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe37c3a210 con 0x55fe37fcc630 2024-05-09T08:40:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.314+0000 7f003ed9d640 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fc9f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f003809b710 tx=0x7f00380ccee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8beac33a13c577d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.315+0000 7f0025ffb640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:40:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.315+0000 7f0025ffb640 1 -- 172.21.15.28:0/1527676474 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380b4040 con 0x55fe37fcc630 2024-05-09T08:40:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.315+0000 7f0047a31740 1 -- 172.21.15.28:0/1527676474 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00380a43f0 con 0x55fe37fcc630 2024-05-09T08:40:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.315+0000 7f0025ffb640 1 -- 172.21.15.28:0/1527676474 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9643+0+0 (secure 0 0 0) 0x7f00380a4170 con 0x55fe37fcc630 2024-05-09T08:40:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.316+0000 7f0025ffb640 1 -- 172.21.15.28:0/1527676474 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f00380cf040 con 0x55fe37fcc630 2024-05-09T08:40:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.316+0000 7f0047a31740 10 client.26582 fetch_fsmap learned FSMap version 67 2024-05-09T08:40:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.316+0000 7f0047a31740 1 -- 172.21.15.28:0/1527676474 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55fe37d06620 con 0x55fe37fcc630 2024-05-09T08:40:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.317+0000 7f0025ffb640 1 -- 172.21.15.28:0/1527676474 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00380a43f0 con 0x55fe37fcc630 2024-05-09T08:40:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.317+0000 7f0025ffb640 1 -- 172.21.15.28:0/1527676474 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f0038097040 con 0x55fe37fcc630 2024-05-09T08:40:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.318+0000 7f0047a31740 10 client.26582 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:40:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.318+0000 7f0047a31740 10 client.26582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:40:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.318+0000 7f0047a31740 20 client.26582 populate_metadata read hostname 'smithi028' 2024-05-09T08:40:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.318+0000 7f0047a31740 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fe37fdc190 0x55fe37fde610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.318+0000 7f0047a31740 4 client.26582 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.318+0000 7f0047a31740 1 -- 172.21.15.28:0/1527676474 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe37f33a70 con 0x55fe37fdc190 2024-05-09T08:40:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.318+0000 7f003e59c640 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fe37fdc190 0x55fe37fde610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.341+0000 7f003e59c640 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fe37fdc190 0x55fe37fde610 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f704646da6ad2150 server_cookie=bb7a411840a92814 in_seq=0 out_seq=0 2024-05-09T08:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.341+0000 7f0025ffb640 10 client.26582 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.344+0000 7f0025ffb640 1 -- 172.21.15.28:0/1527676474 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe37f33a70 con 0x55fe37fdc190 2024-05-09T08:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.344+0000 7f0025ffb640 10 client.26582 handle_command_reply: tid=0 2024-05-09T08:40:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.344+0000 7f0047a31740 1 client.26582 shutdown 2024-05-09T08:40:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.345+0000 7f0047a31740 1 -- 172.21.15.28:0/1527676474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 msgr2=0x55fe37fc9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.345+0000 7f0047a31740 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fc9f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f003809b710 tx=0x7f00380ccee0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.349+0000 7f0047a31740 1 -- 172.21.15.28:0/1527676474 shutdown_connections 2024-05-09T08:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.349+0000 7f0047a31740 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fe37fdc190 0x55fe37fde610 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.349+0000 7f0047a31740 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fcc630 0x55fe37fc9f70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.349+0000 7f0047a31740 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 0x55fe37fc9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.349+0000 7f0047a31740 1 --2- 172.21.15.28:0/1527676474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fc94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.349+0000 7f0047a31740 1 -- 172.21.15.28:0/1527676474 >> 172.21.15.28:0/1527676474 conn(0x55fe37fbd960 msgr2=0x55fe37fc36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.349+0000 7f0025ffb640 0 client.26582 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.352+0000 7f0047a31740 1 -- 172.21.15.28:0/1527676474 shutdown_connections 2024-05-09T08:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.352+0000 7f0047a31740 1 -- 172.21.15.28:0/1527676474 wait complete. 2024-05-09T08:40:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.352+0000 7f0047a31740 20 client.26582 trim_cache size 0 max 16384 2024-05-09T08:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.383+0000 7f0047a31740 1 Processor -- start 2024-05-09T08:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.383+0000 7f0047a31740 1 -- start start 2024-05-09T08:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.383+0000 7f0047a31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fd8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.383+0000 7f0047a31740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1d70 0x55fe37fc3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.383+0000 7f0047a31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fcc630 0x55fe37fe3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.383+0000 7f0047a31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0020010180 con 0x55fe37fc1450 2024-05-09T08:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.383+0000 7f0047a31740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0020011f00 con 0x55fe37fc1d70 2024-05-09T08:40:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.383+0000 7f0047a31740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0020010910 con 0x55fe37fcc630 2024-05-09T08:40:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fcc630 0x55fe37fe3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fcc630 0x55fe37fe3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38286/0 (socket says 172.21.15.28:38286) 2024-05-09T08:40:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003ed9d640 1 -- 172.21.15.28:0/2111347727 learned_addr learned my addr 172.21.15.28:0/2111347727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003e59c640 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fd8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f0037fff640 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1d70 0x55fe37fc3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003ed9d640 1 -- 172.21.15.28:0/2111347727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1d70 msgr2=0x55fe37fc3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003ed9d640 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1d70 0x55fe37fc3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003ed9d640 1 -- 172.21.15.28:0/2111347727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 msgr2=0x55fe37fd8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003ed9d640 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fd8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003ed9d640 1 -- 172.21.15.28:0/2111347727 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe37fbbf00 con 0x55fe37fcc630 2024-05-09T08:40:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.384+0000 7f003e59c640 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fd8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.385+0000 7f003ed9d640 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fcc630 0x55fe37fe3ee0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f003809b480 tx=0x7f003809be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5bebfe40557db1c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.388+0000 7f0027fff640 1 -- 172.21.15.28:0/2111347727 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380b4040 con 0x55fe37fcc630 2024-05-09T08:40:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.388+0000 7f0027fff640 1 -- 172.21.15.28:0/2111347727 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00380ba070 con 0x55fe37fcc630 2024-05-09T08:40:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.388+0000 7f0027fff640 1 -- 172.21.15.28:0/2111347727 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003801a040 con 0x55fe37fcc630 2024-05-09T08:40:07.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.392+0000 7f0047a31740 1 -- 172.21.15.28:0/2111347727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fcc630 msgr2=0x55fe37fe3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:07.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.392+0000 7f0047a31740 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fcc630 0x55fe37fe3ee0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f003809b480 tx=0x7f003809be80 comp rx=0 tx=0).stop 2024-05-09T08:40:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.396+0000 7f0047a31740 1 -- 172.21.15.28:0/2111347727 shutdown_connections 2024-05-09T08:40:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.396+0000 7f0047a31740 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fcc630 0x55fe37fe3ee0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.396+0000 7f0047a31740 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1d70 0x55fe37fc3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.396+0000 7f0047a31740 1 --2- 172.21.15.28:0/2111347727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fc1450 0x55fe37fd8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.396+0000 7f0047a31740 1 -- 172.21.15.28:0/2111347727 >> 172.21.15.28:0/2111347727 conn(0x55fe37fbd960 msgr2=0x55fe37fbca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.400+0000 7f0047a31740 1 -- 172.21.15.28:0/2111347727 shutdown_connections 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.403+0000 7f0047a31740 1 -- 172.21.15.28:0/2111347727 wait complete. 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.403+0000 7f0047a31740 1 -- start start 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.403+0000 7f0047a31740 1 Processor -- start 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.403+0000 7f0047a31740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1450 0x55fe37fd4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.403+0000 7f0047a31740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 0x55fe37fc5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.403+0000 7f0047a31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fcc630 0x55fe37fc6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.403+0000 7f0047a31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0020011830 con 0x55fe37fcc630 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.403+0000 7f0047a31740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f00200110a0 con 0x55fe37fc1450 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.403+0000 7f0047a31740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f002000e160 con 0x55fe37fc1d70 2024-05-09T08:40:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.404+0000 7f0047a31740 10 client.0 _ll_register_callbacks cb 0x7f00380b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.406+0000 7f003e59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1450 0x55fe37fd4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.406+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fcc630 0x55fe37fc6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.406+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fcc630 0x55fe37fc6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41126/0 (socket says 172.21.15.28:41126) 2024-05-09T08:40:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.406+0000 7f003ed9d640 1 -- 172.21.15.28:0/507555196 learned_addr learned my addr 172.21.15.28:0/507555196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.406+0000 7f003ed9d640 1 -- 172.21.15.28:0/507555196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 msgr2=0x55fe37fc5d20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:40:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.406+0000 7f003ed9d640 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 0x55fe37fc5d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.406+0000 7f003ed9d640 1 -- 172.21.15.28:0/507555196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1450 msgr2=0x55fe37fd4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.406+0000 7f003ed9d640 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1450 0x55fe37fd4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.406+0000 7f003ed9d640 1 -- 172.21.15.28:0/507555196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe37fc67a0 con 0x55fe37fcc630 2024-05-09T08:40:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.407+0000 7f003ed9d640 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fcc630 0x55fe37fc6260 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0038097be0 tx=0x7f00380973e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30e4d4f13b48ab1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.407+0000 7f0025ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:40:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.407+0000 7f0025ffb640 1 -- 172.21.15.28:0/507555196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003801a070 con 0x55fe37fcc630 2024-05-09T08:40:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.407+0000 7f0047a31740 1 -- 172.21.15.28:0/507555196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00380ba070 con 0x55fe37fcc630 2024-05-09T08:40:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.407+0000 7f0025ffb640 1 -- 172.21.15.28:0/507555196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9643+0+0 (secure 0 0 0) 0x7f003809abd0 con 0x55fe37fcc630 2024-05-09T08:40:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0025ffb640 1 -- 172.21.15.28:0/507555196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f003809b550 con 0x55fe37fcc630 2024-05-09T08:40:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0047a31740 10 client.17100 fetch_fsmap learned FSMap version 67 2024-05-09T08:40:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0047a31740 1 -- 172.21.15.28:0/507555196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55fe37fc67a0 con 0x55fe37fcc630 2024-05-09T08:40:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0025ffb640 1 -- 172.21.15.28:0/507555196 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00380ba070 con 0x55fe37fcc630 2024-05-09T08:40:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0025ffb640 1 -- 172.21.15.28:0/507555196 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f0038097040 con 0x55fe37fcc630 2024-05-09T08:40:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0047a31740 10 client.17100 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:40:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0047a31740 10 client.17100 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:40:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0047a31740 20 client.17100 populate_metadata read hostname 'smithi028' 2024-05-09T08:40:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0047a31740 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fe380757b0 0x55fe38077bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0047a31740 4 client.17100 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:40:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.408+0000 7f0047a31740 1 -- 172.21.15.28:0/507555196 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fe37f2ddc0 con 0x55fe380757b0 2024-05-09T08:40:07.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.412+0000 7f003e59c640 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fe380757b0 0x55fe38077bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.425+0000 7f003e59c640 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fe380757b0 0x55fe38077bd0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af6d01127d1187fb server_cookie=44f6b422ad7f06cf in_seq=0 out_seq=0 2024-05-09T08:40:07.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:07.425+0000 7f0025ffb640 10 client.17100 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:07 smithi165 ceph-mon[34540]: pgmap v441: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 906 KiB/s wr, 283 op/s 2024-05-09T08:40:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:07 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:07 smithi184 ceph-mon[34509]: pgmap v441: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 906 KiB/s wr, 283 op/s 2024-05-09T08:40:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:07 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:08.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:07 smithi028 ceph-mon[30119]: pgmap v441: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 906 KiB/s wr, 283 op/s 2024-05-09T08:40:08.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:07 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:08.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0025ffb640 1 -- 172.21.15.28:0/507555196 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fe37f2ddc0 con 0x55fe380757b0 2024-05-09T08:40:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0025ffb640 10 client.17100 handle_command_reply: tid=0 2024-05-09T08:40:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 client.17100 shutdown 2024-05-09T08:40:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 -- 172.21.15.28:0/507555196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fcc630 msgr2=0x55fe37fc6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fcc630 0x55fe37fc6260 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0038097be0 tx=0x7f00380973e0 comp rx=0 tx=0).stop 2024-05-09T08:40:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 -- 172.21.15.28:0/507555196 shutdown_connections 2024-05-09T08:40:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55fe380757b0 0x55fe38077bd0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe37fcc630 0x55fe37fc6260 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:08.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fe37fc1d70 0x55fe37fc5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:08.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 --2- 172.21.15.28:0/507555196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55fe37fc1450 0x55fe37fd4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:08.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 -- 172.21.15.28:0/507555196 >> 172.21.15.28:0/507555196 conn(0x55fe37fbd960 msgr2=0x55fe37fc94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:08.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0025ffb640 0 client.17100 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:08.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 -- 172.21.15.28:0/507555196 shutdown_connections 2024-05-09T08:40:08.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 1 -- 172.21.15.28:0/507555196 wait complete. 2024-05-09T08:40:08.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.275+0000 7f0047a31740 20 client.17100 trim_cache size 0 max 16384 2024-05-09T08:40:08.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:40:08.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:40:08.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:40:08.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:40:08.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.276+0000 7f0044c4c640 1 -- 172.21.15.28:0/3886869657 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f002c0a2fd0 msgr2=0x7f002c0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:08.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.276+0000 7f0044c4c640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f002c0a2fd0 0x7f002c0a5490 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0030005fd0 tx=0x7f0030005c90 comp rx=0 tx=0).stop 2024-05-09T08:40:08.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.277+0000 7f0044c4c640 1 -- 172.21.15.28:0/3886869657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0040109a40 msgr2=0x7f004010ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:08.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.277+0000 7f0044c4c640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0040109a40 0x7f004010ddc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0038009fd0 tx=0x7f003800c800 comp rx=0 tx=0).stop 2024-05-09T08:40:08.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.277+0000 7f0044c4c640 1 -- 172.21.15.28:0/3886869657 shutdown_connections 2024-05-09T08:40:08.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.277+0000 7f0044c4c640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f002c0a2fd0 0x7f002c0a5490 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:08.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.277+0000 7f0044c4c640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0040109a40 0x7f004010ddc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:08.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.277+0000 7f0044c4c640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00401043f0 0x7f004010d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.277+0000 7f0044c4c640 1 --2- 172.21.15.28:0/3886869657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0040103a30 0x7f00401107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.277+0000 7f0044c4c640 1 -- 172.21.15.28:0/3886869657 >> 172.21.15.28:0/3886869657 conn(0x7f00400ff560 msgr2=0x7f00400ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.278+0000 7f0044c4c640 1 -- 172.21.15.28:0/3886869657 shutdown_connections 2024-05-09T08:40:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:08.278+0000 7f0044c4c640 1 -- 172.21.15.28:0/3886869657 wait complete. 2024-05-09T08:40:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:09 smithi165 ceph-mon[34540]: pgmap v442: 129 pgs: 48 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 63 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 830 KiB/s wr, 262 op/s 2024-05-09T08:40:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:09 smithi165 ceph-mon[34540]: osdmap e119: 12 total, 12 up, 12 in 2024-05-09T08:40:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:09 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:40:10.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:09 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:40:10.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:09 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:40:10.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:09 smithi184 ceph-mon[34509]: pgmap v442: 129 pgs: 48 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 63 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 830 KiB/s wr, 262 op/s 2024-05-09T08:40:10.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:09 smithi184 ceph-mon[34509]: osdmap e119: 12 total, 12 up, 12 in 2024-05-09T08:40:10.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:09 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:40:10.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:09 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:40:10.107 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:09 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:40:10.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:09 smithi028 ceph-mon[30119]: pgmap v442: 129 pgs: 48 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 63 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 830 KiB/s wr, 262 op/s 2024-05-09T08:40:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:09 smithi028 ceph-mon[30119]: osdmap e119: 12 total, 12 up, 12 in 2024-05-09T08:40:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:09 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:40:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:09 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:40:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:09 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:40:12.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:11 smithi165 ceph-mon[34540]: pgmap v444: 129 pgs: 50 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 59 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 293 op/s 2024-05-09T08:40:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:11 smithi184 ceph-mon[34509]: pgmap v444: 129 pgs: 50 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 59 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 293 op/s 2024-05-09T08:40:12.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:11 smithi028 ceph-mon[30119]: pgmap v444: 129 pgs: 50 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 59 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 293 op/s 2024-05-09T08:40:14.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:13 smithi165 ceph-mon[34540]: pgmap v445: 129 pgs: 65 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 41 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 841 KiB/s wr, 312 op/s 2024-05-09T08:40:14.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:13 smithi184 ceph-mon[34509]: pgmap v445: 129 pgs: 65 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 41 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 841 KiB/s wr, 312 op/s 2024-05-09T08:40:14.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:13 smithi028 ceph-mon[30119]: pgmap v445: 129 pgs: 65 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 41 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 841 KiB/s wr, 312 op/s 2024-05-09T08:40:15.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:15 smithi028 ceph-mon[30119]: pgmap v446: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 857 KiB/s wr, 323 op/s 2024-05-09T08:40:16.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:15 smithi165 ceph-mon[34540]: pgmap v446: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 857 KiB/s wr, 323 op/s 2024-05-09T08:40:16.105 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:15 smithi184 ceph-mon[34509]: pgmap v446: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 857 KiB/s wr, 323 op/s 2024-05-09T08:40:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:17 smithi165 ceph-mon[34540]: pgmap v447: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 334 op/s 2024-05-09T08:40:18.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:17 smithi184 ceph-mon[34509]: pgmap v447: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 334 op/s 2024-05-09T08:40:18.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:17 smithi028 ceph-mon[30119]: pgmap v447: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 334 op/s 2024-05-09T08:40:18.931 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:40:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:40:18.595+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:40:18.932 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:40:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:40:18.595+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:40:20.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:19 smithi165 ceph-mon[34540]: pgmap v448: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 936 KiB/s wr, 337 op/s 2024-05-09T08:40:20.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:19 smithi184 ceph-mon[34509]: pgmap v448: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 936 KiB/s wr, 337 op/s 2024-05-09T08:40:20.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:19 smithi028 ceph-mon[30119]: pgmap v448: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 936 KiB/s wr, 337 op/s 2024-05-09T08:40:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.440+0000 7f0afeaa3640 1 -- 172.21.15.28:0/1446261770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0af81091f0 msgr2=0x7f0af8115120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.440+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/1446261770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0af81091f0 0x7f0af8115120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0af000dfa0 tx=0x7f0af0032aa0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.451+0000 7f0afeaa3640 1 -- 172.21.15.28:0/1446261770 shutdown_connections 2024-05-09T08:40:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.451+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/1446261770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0af81091f0 0x7f0af8115120 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.451+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/1446261770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af8108830 0x7f0af8108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.451+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/1446261770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0af810d290 0x7f0af810d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.451+0000 7f0afeaa3640 1 -- 172.21.15.28:0/1446261770 >> 172.21.15.28:0/1446261770 conn(0x7f0af806cd90 msgr2=0x7f0af806d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.457+0000 7f0afeaa3640 1 -- 172.21.15.28:0/1446261770 shutdown_connections 2024-05-09T08:40:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.461+0000 7f0afeaa3640 1 -- 172.21.15.28:0/1446261770 wait complete. 2024-05-09T08:40:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.461+0000 7f0afeaa3640 1 Processor -- start 2024-05-09T08:40:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.461+0000 7f0afeaa3640 1 -- start start 2024-05-09T08:40:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.462+0000 7f0afeaa3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0af8108830 0x7f0af81af630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.462+0000 7f0afeaa3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0af81091f0 0x7f0af81afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.462+0000 7f0afeaa3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af810d290 0x7f0af81b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.462+0000 7f0afeaa3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0af8118240 con 0x7f0af810d290 2024-05-09T08:40:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.462+0000 7f0afeaa3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0af81180c0 con 0x7f0af8108830 2024-05-09T08:40:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.462+0000 7f0afeaa3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0af81183c0 con 0x7f0af81091f0 2024-05-09T08:40:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.463+0000 7f0afd040640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af810d290 0x7f0af81b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.463+0000 7f0afd040640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af810d290 0x7f0af81b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53524/0 (socket says 172.21.15.28:53524) 2024-05-09T08:40:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.463+0000 7f0afd040640 1 -- 172.21.15.28:0/307424780 learned_addr learned my addr 172.21.15.28:0/307424780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.464+0000 7f0af7fff640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0af81091f0 0x7f0af81afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.464+0000 7f0afc83f640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0af8108830 0x7f0af81af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.467+0000 7f0afd040640 1 -- 172.21.15.28:0/307424780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0af81091f0 msgr2=0x7f0af81afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.467+0000 7f0afd040640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0af81091f0 0x7f0af81afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.468+0000 7f0afd040640 1 -- 172.21.15.28:0/307424780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0af8108830 msgr2=0x7f0af81af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.468+0000 7f0afd040640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0af8108830 0x7f0af81af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.468+0000 7f0afd040640 1 -- 172.21.15.28:0/307424780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af81b4620 con 0x7f0af810d290 2024-05-09T08:40:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.468+0000 7f0afd040640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af810d290 0x7f0af81b3ef0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0af0037a80 tx=0x7f0af0037ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.471+0000 7f0af5ffb640 1 -- 172.21.15.28:0/307424780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af003a040 con 0x7f0af810d290 2024-05-09T08:40:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.471+0000 7f0af5ffb640 1 -- 172.21.15.28:0/307424780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0af0004040 con 0x7f0af810d290 2024-05-09T08:40:21.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.471+0000 7f0af5ffb640 1 -- 172.21.15.28:0/307424780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af000b830 con 0x7f0af810d290 2024-05-09T08:40:21.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.473+0000 7f0afeaa3640 1 -- 172.21.15.28:0/307424780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af81b48b0 con 0x7f0af810d290 2024-05-09T08:40:21.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.473+0000 7f0afeaa3640 1 -- 172.21.15.28:0/307424780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af81a3740 con 0x7f0af810d290 2024-05-09T08:40:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.478+0000 7f0b017a8740 1 Processor -- start 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.483+0000 7f0b017a8740 1 -- start start 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.483+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 0x5580a64b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.483+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2d70 0x5580a64bd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.483+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64bd630 0x5580a64bfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.483+0000 7f0b017a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5580a6417340 con 0x5580a64b2d70 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.483+0000 7f0b017a8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5580a63bf290 con 0x5580a64b2450 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.483+0000 7f0b017a8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5580a6413300 con 0x5580a64bd630 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.484+0000 7f0af7fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 0x5580a64b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.484+0000 7f0af7fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 0x5580a64b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:49890/0 (socket says 172.21.15.28:49890) 2024-05-09T08:40:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.484+0000 7f0af7fff640 1 -- 172.21.15.28:0/556064808 learned_addr learned my addr 172.21.15.28:0/556064808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.484+0000 7f0af7fff640 1 --2- 172.21.15.28:0/556064808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64bd630 0x5580a64bfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.484+0000 7f0af7fff640 1 -- 172.21.15.28:0/556064808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64bd630 msgr2=0x5580a64bfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.484+0000 7f0af7fff640 1 --2- 172.21.15.28:0/556064808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64bd630 0x5580a64bfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.484+0000 7f0af7fff640 1 -- 172.21.15.28:0/556064808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2d70 msgr2=0x5580a64bd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.484+0000 7f0af7fff640 1 --2- 172.21.15.28:0/556064808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2d70 0x5580a64bd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.484+0000 7f0af7fff640 1 -- 172.21.15.28:0/556064808 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580a63e38e0 con 0x5580a64b2450 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.486+0000 7f0af5ffb640 1 -- 172.21.15.28:0/307424780 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0af0046060 con 0x7f0af810d290 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.487+0000 7f0af5ffb640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ae00c4fe0 0x7f0ae00c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.487+0000 7f0af7fff640 1 --2- 172.21.15.28:0/556064808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 0x5580a64b2830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0aec016950 tx=0x7f0aec00ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a079bb14adee01a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.487+0000 7f0af5ffb640 1 -- 172.21.15.28:0/307424780 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9767+0+0 (secure 0 0 0) 0x7f0af00eed10 con 0x7f0af810d290 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.488+0000 7f0adeffd640 1 -- 172.21.15.28:0/556064808 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec0160d0 con 0x5580a64b2450 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.488+0000 7f0adeffd640 1 -- 172.21.15.28:0/556064808 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0aec016270 con 0x5580a64b2450 2024-05-09T08:40:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.488+0000 7f0adeffd640 1 -- 172.21.15.28:0/556064808 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec004a00 con 0x5580a64b2450 2024-05-09T08:40:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.489+0000 7f0b017a8740 1 -- 172.21.15.28:0/556064808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 msgr2=0x5580a64b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.489+0000 7f0b017a8740 1 --2- 172.21.15.28:0/556064808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 0x5580a64b2830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0aec016950 tx=0x7f0aec00ac00 comp rx=0 tx=0).stop 2024-05-09T08:40:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.490+0000 7f0afc83f640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ae00c4fe0 0x7f0ae00c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.491+0000 7f0b017a8740 1 -- 172.21.15.28:0/556064808 shutdown_connections 2024-05-09T08:40:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.491+0000 7f0b017a8740 1 --2- 172.21.15.28:0/556064808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64bd630 0x5580a64bfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.491+0000 7f0b017a8740 1 --2- 172.21.15.28:0/556064808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2d70 0x5580a64bd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.491+0000 7f0b017a8740 1 --2- 172.21.15.28:0/556064808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 0x5580a64b2830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.491+0000 7f0b017a8740 1 -- 172.21.15.28:0/556064808 >> 172.21.15.28:0/556064808 conn(0x5580a64ae960 msgr2=0x5580a64aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.496+0000 7f0b017a8740 1 -- 172.21.15.28:0/556064808 shutdown_connections 2024-05-09T08:40:21.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.499+0000 7f0afc83f640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ae00c4fe0 0x7f0ae00c74a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8006fd0 tx=0x7f0ae8008040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:21.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.502+0000 7f0b017a8740 1 -- 172.21.15.28:0/556064808 wait complete. 2024-05-09T08:40:21.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.502+0000 7f0b017a8740 1 -- start start 2024-05-09T08:40:21.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.502+0000 7f0b017a8740 1 Processor -- start 2024-05-09T08:40:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.503+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 0x5580a64ba510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.503+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64baa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.503+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64bd630 0x5580a64baf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.503+0000 7f0b017a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5580a64c0200 con 0x5580a64b2450 2024-05-09T08:40:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.503+0000 7f0b017a8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5580a63d2790 con 0x5580a64bd630 2024-05-09T08:40:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.503+0000 7f0b017a8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5580a64c0380 con 0x5580a64b2d70 2024-05-09T08:40:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.503+0000 7f0b017a8740 10 client.0 _ll_register_callbacks cb 0x5580a640da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.504+0000 7f0af7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64baa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.506+0000 7f0af7fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64baa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:53410/0 (socket says 172.21.15.28:53410) 2024-05-09T08:40:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.506+0000 7f0af7fff640 1 -- 172.21.15.28:0/2310863040 learned_addr learned my addr 172.21.15.28:0/2310863040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.507+0000 7f0af7fff640 1 -- 172.21.15.28:0/2310863040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64bd630 msgr2=0x5580a64baf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.507+0000 7f0af7fff640 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64bd630 0x5580a64baf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.507+0000 7f0af7fff640 1 -- 172.21.15.28:0/2310863040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 msgr2=0x5580a64ba510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.507+0000 7f0af7fff640 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 0x5580a64ba510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.507+0000 7f0af7fff640 1 -- 172.21.15.28:0/2310863040 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580a612b210 con 0x5580a64b2d70 2024-05-09T08:40:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.508+0000 7f0af7fff640 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64baa50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0aec005c80 tx=0x7f0aec005730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22e1fd39b9e84633 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.509+0000 7f0ad7fff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:40:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.517+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2310863040 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec0160d0 con 0x5580a64b2d70 2024-05-09T08:40:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.517+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2310863040 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9767+0+0 (secure 0 0 0) 0x7f0aec011b60 con 0x5580a64b2d70 2024-05-09T08:40:21.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.517+0000 7f0b017a8740 1 -- 172.21.15.28:0/2310863040 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0aec016270 con 0x5580a64b2d70 2024-05-09T08:40:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.525+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2310863040 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f0aec016610 con 0x5580a64b2d70 2024-05-09T08:40:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.527+0000 7f0b017a8740 10 client.26437 fetch_fsmap learned FSMap version 67 2024-05-09T08:40:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.527+0000 7f0b017a8740 1 -- 172.21.15.28:0/2310863040 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5580a61f7620 con 0x5580a64b2d70 2024-05-09T08:40:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.535+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2310863040 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0aec016270 con 0x5580a64b2d70 2024-05-09T08:40:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.535+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2310863040 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f0aec0035f0 con 0x5580a64b2d70 2024-05-09T08:40:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.538+0000 7f0b017a8740 10 client.26437 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:40:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.538+0000 7f0b017a8740 10 client.26437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:40:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.538+0000 7f0b017a8740 20 client.26437 populate_metadata read hostname 'smithi028' 2024-05-09T08:40:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.538+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5580a64cd1a0 0x5580a64cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.538+0000 7f0b017a8740 4 client.26437 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.538+0000 7f0b017a8740 1 -- 172.21.15.28:0/2310863040 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580a6424a70 con 0x5580a64cd1a0 2024-05-09T08:40:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.544+0000 7f0afc83f640 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5580a64cd1a0 0x5580a64cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.553+0000 7f0afc83f640 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5580a64cd1a0 0x5580a64cf590 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50b4ebfe6910cc73 server_cookie=b1c7eb0731114892 in_seq=0 out_seq=0 2024-05-09T08:40:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.554+0000 7f0ad7fff640 10 client.26437 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.555+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2310863040 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5580a6424a70 con 0x5580a64cd1a0 2024-05-09T08:40:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.555+0000 7f0ad7fff640 10 client.26437 handle_command_reply: tid=0 2024-05-09T08:40:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.558+0000 7f0b017a8740 1 client.26437 shutdown 2024-05-09T08:40:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.561+0000 7f0b017a8740 1 -- 172.21.15.28:0/2310863040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 msgr2=0x5580a64baa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.561+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64baa50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0aec005c80 tx=0x7f0aec005730 comp rx=0 tx=0).stop 2024-05-09T08:40:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.564+0000 7f0b017a8740 1 -- 172.21.15.28:0/2310863040 shutdown_connections 2024-05-09T08:40:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.564+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5580a64cd1a0 0x5580a64cf590 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.564+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64bd630 0x5580a64baf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.564+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64baa50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.564+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2310863040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 0x5580a64ba510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.564+0000 7f0b017a8740 1 -- 172.21.15.28:0/2310863040 >> 172.21.15.28:0/2310863040 conn(0x5580a64ae960 msgr2=0x5580a64b46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.564+0000 7f0ad7fff640 0 client.26437 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.565+0000 7f0b017a8740 1 -- 172.21.15.28:0/2310863040 shutdown_connections 2024-05-09T08:40:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.566+0000 7f0b017a8740 1 -- 172.21.15.28:0/2310863040 wait complete. 2024-05-09T08:40:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.568+0000 7f0b017a8740 20 client.26437 trim_cache size 0 max 16384 2024-05-09T08:40:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0b017a8740 1 Processor -- start 2024-05-09T08:40:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0b017a8740 1 -- start start 2024-05-09T08:40:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 0x5580a64bcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64bd630 0x5580a64ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0b017a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4010310 con 0x5580a64bd630 2024-05-09T08:40:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0b017a8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4012090 con 0x5580a64b2450 2024-05-09T08:40:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0b017a8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4010aa0 con 0x5580a64b2d70 2024-05-09T08:40:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0afd040640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64bd630 0x5580a64ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0afd040640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64bd630 0x5580a64ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53556/0 (socket says 172.21.15.28:53556) 2024-05-09T08:40:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0afd040640 1 -- 172.21.15.28:0/3481023737 learned_addr learned my addr 172.21.15.28:0/3481023737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.600+0000 7f0af7fff640 1 --2- 172.21.15.28:0/3481023737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.601+0000 7f0afd040640 1 -- 172.21.15.28:0/3481023737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 msgr2=0x5580a64c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.601+0000 7f0afd040640 1 --2- 172.21.15.28:0/3481023737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.601+0000 7f0afd040640 1 -- 172.21.15.28:0/3481023737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 msgr2=0x5580a64bcc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:40:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.601+0000 7f0afd040640 1 --2- 172.21.15.28:0/3481023737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 0x5580a64bcc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.601+0000 7f0afd040640 1 -- 172.21.15.28:0/3481023737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580a64acf00 con 0x5580a64bd630 2024-05-09T08:40:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.602+0000 7f0afd040640 1 --2- 172.21.15.28:0/3481023737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64bd630 0x5580a64ad880 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0af00b7740 tx=0x7f0af00bac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef131dfd63e32b25 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.603+0000 7f0ade7fc640 1 -- 172.21.15.28:0/3481023737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af00bb470 con 0x5580a64bd630 2024-05-09T08:40:21.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.603+0000 7f0ade7fc640 1 -- 172.21.15.28:0/3481023737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0af00bad70 con 0x5580a64bd630 2024-05-09T08:40:21.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.603+0000 7f0ade7fc640 1 -- 172.21.15.28:0/3481023737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af0004b40 con 0x5580a64bd630 2024-05-09T08:40:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.604+0000 7f0b017a8740 1 -- 172.21.15.28:0/3481023737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64bd630 msgr2=0x5580a64ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.604+0000 7f0b017a8740 1 --2- 172.21.15.28:0/3481023737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64bd630 0x5580a64ad880 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0af00b7740 tx=0x7f0af00bac60 comp rx=0 tx=0).stop 2024-05-09T08:40:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.605+0000 7f0b017a8740 1 -- 172.21.15.28:0/3481023737 shutdown_connections 2024-05-09T08:40:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.605+0000 7f0b017a8740 1 --2- 172.21.15.28:0/3481023737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64bd630 0x5580a64ad880 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.605+0000 7f0b017a8740 1 --2- 172.21.15.28:0/3481023737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.605+0000 7f0b017a8740 1 --2- 172.21.15.28:0/3481023737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64b2450 0x5580a64bcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.605+0000 7f0b017a8740 1 -- 172.21.15.28:0/3481023737 >> 172.21.15.28:0/3481023737 conn(0x5580a64ae960 msgr2=0x5580a64ad170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.608+0000 7f0b017a8740 1 -- 172.21.15.28:0/3481023737 shutdown_connections 2024-05-09T08:40:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.610+0000 7f0b017a8740 1 -- 172.21.15.28:0/3481023737 wait complete. 2024-05-09T08:40:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.610+0000 7f0b017a8740 1 -- start start 2024-05-09T08:40:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.611+0000 7f0b017a8740 1 Processor -- start 2024-05-09T08:40:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.611+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 0x5580a64c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.611+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.611+0000 7f0b017a8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64bd630 0x5580a64b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.611+0000 7f0b017a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae40119c0 con 0x5580a64b2450 2024-05-09T08:40:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.611+0000 7f0b017a8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4011230 con 0x5580a64bd630 2024-05-09T08:40:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.611+0000 7f0b017a8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae400e310 con 0x5580a64b2d70 2024-05-09T08:40:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.611+0000 7f0b017a8740 10 client.0 _ll_register_callbacks cb 0x7f0aec00a9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.612+0000 7f0afc83f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 0x5580a64c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.612+0000 7f0afc83f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 0x5580a64c59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53568/0 (socket says 172.21.15.28:53568) 2024-05-09T08:40:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.612+0000 7f0afc83f640 1 -- 172.21.15.28:0/2865749637 learned_addr learned my addr 172.21.15.28:0/2865749637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.612+0000 7f0afc83f640 1 -- 172.21.15.28:0/2865749637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 msgr2=0x5580a64b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.612+0000 7f0afc83f640 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.612+0000 7f0afc83f640 1 -- 172.21.15.28:0/2865749637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64bd630 msgr2=0x5580a64b73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.612+0000 7f0afc83f640 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64bd630 0x5580a64b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.612+0000 7f0afc83f640 1 -- 172.21.15.28:0/2865749637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580a64b7c20 con 0x5580a64b2450 2024-05-09T08:40:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.613+0000 7f0afc83f640 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 0x5580a64c59d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0ae80047b0 tx=0x7f0ae800dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffb37114bcf978 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.613+0000 7f0ad7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:40:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.613+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2865749637 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8005990 con 0x5580a64b2450 2024-05-09T08:40:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.614+0000 7f0b017a8740 1 -- 172.21.15.28:0/2865749637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0af00bad70 con 0x5580a64b2450 2024-05-09T08:40:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.614+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2865749637 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9767+0+0 (secure 0 0 0) 0x7f0ae8005b30 con 0x5580a64b2450 2024-05-09T08:40:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.614+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2865749637 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f0ae8031050 con 0x5580a64b2450 2024-05-09T08:40:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.614+0000 7f0b017a8740 10 client.17112 fetch_fsmap learned FSMap version 67 2024-05-09T08:40:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.614+0000 7f0b017a8740 1 -- 172.21.15.28:0/2865749637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5580a65335b0 con 0x5580a64b2450 2024-05-09T08:40:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.615+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2865749637 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0af00bad70 con 0x5580a64b2450 2024-05-09T08:40:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.615+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2865749637 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f0ae8013070 con 0x5580a64b2450 2024-05-09T08:40:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.615+0000 7f0b017a8740 10 client.17112 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:40:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.615+0000 7f0b017a8740 10 client.17112 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:40:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.615+0000 7f0b017a8740 20 client.17112 populate_metadata read hostname 'smithi028' 2024-05-09T08:40:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.616+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5580a6533dd0 0x5580a6536230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.616+0000 7f0b017a8740 4 client.17112 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:40:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.616+0000 7f0b017a8740 1 -- 172.21.15.28:0/2865749637 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5580a641edc0 con 0x5580a6533dd0 2024-05-09T08:40:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.616+0000 7f0af7fff640 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5580a6533dd0 0x5580a6536230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.617+0000 7f0af7fff640 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5580a6533dd0 0x5580a6536230 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6e000affd1d0ba6 server_cookie=ef93ac78d1f58ddc in_seq=0 out_seq=0 2024-05-09T08:40:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.617+0000 7f0ad7fff640 10 client.17112 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.874+0000 7f0ad7fff640 1 -- 172.21.15.28:0/2865749637 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5580a641edc0 con 0x5580a6533dd0 2024-05-09T08:40:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.874+0000 7f0ad7fff640 10 client.17112 handle_command_reply: tid=0 2024-05-09T08:40:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.874+0000 7f0b017a8740 1 client.17112 shutdown 2024-05-09T08:40:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.874+0000 7f0b017a8740 1 -- 172.21.15.28:0/2865749637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 msgr2=0x5580a64c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.874+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 0x5580a64c59d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0ae80047b0 tx=0x7f0ae800dd80 comp rx=0 tx=0).stop 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0b017a8740 1 -- 172.21.15.28:0/2865749637 shutdown_connections 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5580a6533dd0 0x5580a6536230 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5580a64bd630 0x5580a64b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5580a64b2d70 0x5580a64b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0b017a8740 1 --2- 172.21.15.28:0/2865749637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5580a64b2450 0x5580a64c59d0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0b017a8740 1 -- 172.21.15.28:0/2865749637 >> 172.21.15.28:0/2865749637 conn(0x5580a64ae960 msgr2=0x5580a64ada00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0ad7fff640 0 client.17112 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0b017a8740 1 -- 172.21.15.28:0/2865749637 shutdown_connections 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0b017a8740 1 -- 172.21.15.28:0/2865749637 wait complete. 2024-05-09T08:40:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.875+0000 7f0b017a8740 20 client.17112 trim_cache size 0 max 16384 2024-05-09T08:40:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:40:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:40:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:40:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:40:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 -- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ae00c4fe0 msgr2=0x7f0ae00c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ae00c4fe0 0x7f0ae00c74a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8006fd0 tx=0x7f0ae8008040 comp rx=0 tx=0).stop 2024-05-09T08:40:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 -- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af810d290 msgr2=0x7f0af81b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af810d290 0x7f0af81b3ef0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0af0037a80 tx=0x7f0af0037ab0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 -- 172.21.15.28:0/307424780 shutdown_connections 2024-05-09T08:40:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0af810d290 0x7f0af81b3ef0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f0ae00c4fe0 0x7f0ae00c74a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0af81091f0 0x7f0af81afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 --2- 172.21.15.28:0/307424780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0af8108830 0x7f0af81af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 -- 172.21.15.28:0/307424780 >> 172.21.15.28:0/307424780 conn(0x7f0af806cd90 msgr2=0x7f0af8070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 -- 172.21.15.28:0/307424780 shutdown_connections 2024-05-09T08:40:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:21.876+0000 7f0afeaa3640 1 -- 172.21.15.28:0/307424780 wait complete. 2024-05-09T08:40:22.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:21 smithi165 ceph-mon[34540]: pgmap v449: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 333 op/s 2024-05-09T08:40:22.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:21 smithi184 ceph-mon[34509]: pgmap v449: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 333 op/s 2024-05-09T08:40:22.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:21 smithi028 ceph-mon[30119]: pgmap v449: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 333 op/s 2024-05-09T08:40:24.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:23 smithi165 ceph-mon[34540]: pgmap v450: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 799 KiB/s wr, 323 op/s 2024-05-09T08:40:24.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:23 smithi184 ceph-mon[34509]: pgmap v450: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 799 KiB/s wr, 323 op/s 2024-05-09T08:40:24.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:23 smithi028 ceph-mon[30119]: pgmap v450: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 799 KiB/s wr, 323 op/s 2024-05-09T08:40:26.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:25 smithi165 ceph-mon[34540]: pgmap v451: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 892 KiB/s wr, 290 op/s 2024-05-09T08:40:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:25 smithi184 ceph-mon[34509]: pgmap v451: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 892 KiB/s wr, 290 op/s 2024-05-09T08:40:26.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:25 smithi028 ceph-mon[30119]: pgmap v451: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 892 KiB/s wr, 290 op/s 2024-05-09T08:40:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:27 smithi165 ceph-mon[34540]: pgmap v452: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 303 op/s 2024-05-09T08:40:28.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:27 smithi184 ceph-mon[34509]: pgmap v452: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 303 op/s 2024-05-09T08:40:28.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:27 smithi028 ceph-mon[30119]: pgmap v452: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 303 op/s 2024-05-09T08:40:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:29 smithi165 ceph-mon[34540]: pgmap v453: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 990 KiB/s wr, 258 op/s 2024-05-09T08:40:30.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:29 smithi184 ceph-mon[34509]: pgmap v453: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 990 KiB/s wr, 258 op/s 2024-05-09T08:40:30.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:29 smithi028 ceph-mon[30119]: pgmap v453: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 990 KiB/s wr, 258 op/s 2024-05-09T08:40:31.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:31 smithi028 ceph-mon[30119]: pgmap v454: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-05-09T08:40:31.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:31 smithi028 ceph-mon[30119]: osdmap e120: 12 total, 12 up, 12 in 2024-05-09T08:40:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:31 smithi165 ceph-mon[34540]: pgmap v454: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-05-09T08:40:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:31 smithi165 ceph-mon[34540]: osdmap e120: 12 total, 12 up, 12 in 2024-05-09T08:40:32.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:31 smithi184 ceph-mon[34509]: pgmap v454: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 241 op/s 2024-05-09T08:40:32.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:31 smithi184 ceph-mon[34509]: osdmap e120: 12 total, 12 up, 12 in 2024-05-09T08:40:33.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:33 smithi028 ceph-mon[30119]: pgmap v456: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 186 op/s 2024-05-09T08:40:34.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:33 smithi165 ceph-mon[34540]: pgmap v456: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 186 op/s 2024-05-09T08:40:34.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:33 smithi184 ceph-mon[34509]: pgmap v456: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 186 op/s 2024-05-09T08:40:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.039+0000 7fc6d9f63640 1 -- 172.21.15.28:0/808605927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41011c0 msgr2=0x7fc6d4105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.039+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/808605927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41011c0 0x7fc6d4105680 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009930 tx=0x7fc6c402a510 comp rx=0 tx=0).stop 2024-05-09T08:40:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.043+0000 7fc6d9f63640 1 -- 172.21.15.28:0/808605927 shutdown_connections 2024-05-09T08:40:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.043+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/808605927 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6d4105ca0 0x7fc6d4106150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.043+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/808605927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41011c0 0x7fc6d4105680 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.043+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/808605927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6d41008a0 0x7fc6d4100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.043+0000 7fc6d9f63640 1 -- 172.21.15.28:0/808605927 >> 172.21.15.28:0/808605927 conn(0x7fc6d4077880 msgr2=0x7fc6d4077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:35.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.048+0000 7fc6d9f63640 1 -- 172.21.15.28:0/808605927 shutdown_connections 2024-05-09T08:40:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.050+0000 7fc6d9f63640 1 -- 172.21.15.28:0/808605927 wait complete. 2024-05-09T08:40:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.051+0000 7fc6d9f63640 1 Processor -- start 2024-05-09T08:40:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.051+0000 7fc6d9f63640 1 -- start start 2024-05-09T08:40:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.051+0000 7fc6d9f63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41008a0 0x7fc6d4073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.051+0000 7fc6d9f63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6d41011c0 0x7fc6d406c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.051+0000 7fc6d9f63640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6d4105ca0 0x7fc6d406c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.051+0000 7fc6d9f63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d4118370 con 0x7fc6d41008a0 2024-05-09T08:40:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.051+0000 7fc6d9f63640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d41181f0 con 0x7fc6d4105ca0 2024-05-09T08:40:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.051+0000 7fc6d9f63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d41184f0 con 0x7fc6d41011c0 2024-05-09T08:40:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.052+0000 7fc6d37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41008a0 0x7fc6d4073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.052+0000 7fc6d37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41008a0 0x7fc6d4073080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55946/0 (socket says 172.21.15.28:55946) 2024-05-09T08:40:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.052+0000 7fc6d37fe640 1 -- 172.21.15.28:0/1429297880 learned_addr learned my addr 172.21.15.28:0/1429297880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.052+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6d4105ca0 0x7fc6d406c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.053+0000 7fc6d37fe640 1 -- 172.21.15.28:0/1429297880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6d41011c0 msgr2=0x7fc6d406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.053+0000 7fc6d37fe640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6d41011c0 0x7fc6d406c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.053+0000 7fc6d37fe640 1 -- 172.21.15.28:0/1429297880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6d4105ca0 msgr2=0x7fc6d406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.053+0000 7fc6d37fe640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6d4105ca0 0x7fc6d406c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.053+0000 7fc6d37fe640 1 -- 172.21.15.28:0/1429297880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d406cc80 con 0x7fc6d41008a0 2024-05-09T08:40:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.053+0000 7fc6d37fe640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41008a0 0x7fc6d4073080 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00b450 tx=0x7fc6bc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.054+0000 7fc6d0ff9640 1 -- 172.21.15.28:0/1429297880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc015020 con 0x7fc6d41008a0 2024-05-09T08:40:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.054+0000 7fc6d9f63640 1 -- 172.21.15.28:0/1429297880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d4070ba0 con 0x7fc6d41008a0 2024-05-09T08:40:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.055+0000 7fc6d9f63640 1 -- 172.21.15.28:0/1429297880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d4071140 con 0x7fc6d41008a0 2024-05-09T08:40:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.056+0000 7fc6d0ff9640 1 -- 172.21.15.28:0/1429297880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6bc011d70 con 0x7fc6d41008a0 2024-05-09T08:40:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.056+0000 7fc6d0ff9640 1 -- 172.21.15.28:0/1429297880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc00f340 con 0x7fc6d41008a0 2024-05-09T08:40:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.058+0000 7fc6dcc77740 1 Processor -- start 2024-05-09T08:40:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.063+0000 7fc6dcc77740 1 -- start start 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.066+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 0x558601c3c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.066+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c470f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.066+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c49a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.066+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558601ba1340 con 0x558601c3cd70 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.066+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558601b49290 con 0x558601c3c450 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.066+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558601b9d300 con 0x558601c47630 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 0x558601c3c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c49a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c49a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50034/0 (socket says 172.21.15.28:50034) 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 0x558601c3c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:46722/0 (socket says 172.21.15.28:46722) 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d2ffd640 1 -- 172.21.15.28:0/3585177251 learned_addr learned my addr 172.21.15.28:0/3585177251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d3fff640 1 -- 172.21.15.28:0/3585177251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 msgr2=0x558601c49a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/3585177251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c49a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d3fff640 1 -- 172.21.15.28:0/3585177251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 msgr2=0x558601c470f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:40:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/3585177251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c470f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.067+0000 7fc6d3fff640 1 -- 172.21.15.28:0/3585177251 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558601b6d8e0 con 0x558601c3c450 2024-05-09T08:40:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.069+0000 7fc6d0ff9640 1 -- 172.21.15.28:0/1429297880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc6bc018020 con 0x7fc6d41008a0 2024-05-09T08:40:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.069+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/3585177251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 0x558601c3c830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007cd0 tx=0x7fc6cc00c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7189e8b963f5eed4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.071+0000 7fc6c9ffb640 1 -- 172.21.15.28:0/3585177251 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc015020 con 0x558601c3c450 2024-05-09T08:40:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.074+0000 7fc6c9ffb640 1 -- 172.21.15.28:0/3585177251 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc00ce50 con 0x558601c3c450 2024-05-09T08:40:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.075+0000 7fc6dcc77740 1 -- 172.21.15.28:0/3585177251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 msgr2=0x558601c3c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.075+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/3585177251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 0x558601c3c830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007cd0 tx=0x7fc6cc00c540 comp rx=0 tx=0).stop 2024-05-09T08:40:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.075+0000 7fc6dcc77740 1 -- 172.21.15.28:0/3585177251 shutdown_connections 2024-05-09T08:40:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.075+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/3585177251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c49a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.075+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/3585177251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.075+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/3585177251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 0x558601c3c830 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.075+0000 7fc6dcc77740 1 -- 172.21.15.28:0/3585177251 >> 172.21.15.28:0/3585177251 conn(0x558601c38960 msgr2=0x558601c38d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.076+0000 7fc6d0ff9640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc6c00a2ef0 0x7fc6c00a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.076+0000 7fc6d0ff9640 1 -- 172.21.15.28:0/1429297880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9707+0+0 (secure 0 0 0) 0x7fc6bc0cf290 con 0x7fc6d41008a0 2024-05-09T08:40:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.077+0000 7fc6d2ffd640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc6c00a2ef0 0x7fc6c00a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.078+0000 7fc6dcc77740 1 -- 172.21.15.28:0/3585177251 shutdown_connections 2024-05-09T08:40:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.079+0000 7fc6dcc77740 1 -- 172.21.15.28:0/3585177251 wait complete. 2024-05-09T08:40:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.079+0000 7fc6dcc77740 1 -- start start 2024-05-09T08:40:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.079+0000 7fc6dcc77740 1 Processor -- start 2024-05-09T08:40:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.080+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3c450 0x558601c44530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.080+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3cd70 0x558601c44a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.080+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c44fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.080+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558601c4a260 con 0x558601c3c450 2024-05-09T08:40:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.080+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558601b5c790 con 0x558601c3cd70 2024-05-09T08:40:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.080+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558601c4a3e0 con 0x558601c47630 2024-05-09T08:40:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.080+0000 7fc6dcc77740 10 client.0 _ll_register_callbacks cb 0x558601b97a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.080+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c44fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c44fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50048/0 (socket says 172.21.15.28:50048) 2024-05-09T08:40:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d3fff640 1 -- 172.21.15.28:0/1243223935 learned_addr learned my addr 172.21.15.28:0/1243223935 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d2ffd640 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3cd70 0x558601c44a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d2ffd640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc6c00a2ef0 0x7fc6c00a53b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40096c0 tx=0x7fc6c4000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d37fe640 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3c450 0x558601c44530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d3fff640 1 -- 172.21.15.28:0/1243223935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3cd70 msgr2=0x558601c44a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3cd70 0x558601c44a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d3fff640 1 -- 172.21.15.28:0/1243223935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3c450 msgr2=0x558601c44530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3c450 0x558601c44530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.081+0000 7fc6d3fff640 1 -- 172.21.15.28:0/1243223935 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586018b5210 con 0x558601c47630 2024-05-09T08:40:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.084+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c44fb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc011ed0 tx=0x7fc6cc007a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2543587d3bc034a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.085+0000 7fc6b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:40:35.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.088+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/1243223935 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc011620 con 0x558601c47630 2024-05-09T08:40:35.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.088+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1243223935 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6cc00ce50 con 0x558601c47630 2024-05-09T08:40:35.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.088+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/1243223935 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9707+0+0 (secure 0 0 0) 0x7fc6cc0117c0 con 0x558601c47630 2024-05-09T08:40:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.089+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/1243223935 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fc6cc015030 con 0x558601c47630 2024-05-09T08:40:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.089+0000 7fc6dcc77740 10 client.26446 fetch_fsmap learned FSMap version 67 2024-05-09T08:40:35.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.089+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1243223935 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x558601981620 con 0x558601c47630 2024-05-09T08:40:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.089+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/1243223935 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc00ce50 con 0x558601c47630 2024-05-09T08:40:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.090+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/1243223935 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fc6cc023070 con 0x558601c47630 2024-05-09T08:40:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.090+0000 7fc6dcc77740 10 client.26446 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:40:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.090+0000 7fc6dcc77740 10 client.26446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:40:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.090+0000 7fc6dcc77740 20 client.26446 populate_metadata read hostname 'smithi028' 2024-05-09T08:40:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.090+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558601c797c0 0x558601c7bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.090+0000 7fc6dcc77740 4 client.26446 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.090+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1243223935 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558601baea70 con 0x558601c797c0 2024-05-09T08:40:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.091+0000 7fc6d37fe640 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558601c797c0 0x558601c7bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.100+0000 7fc6d37fe640 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558601c797c0 0x558601c7bc00 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4af68017cf0a20e server_cookie=17260760a6c51e70 in_seq=0 out_seq=0 2024-05-09T08:40:35.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.100+0000 7fc6b2ffd640 10 client.26446 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/1243223935 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558601baea70 con 0x558601c797c0 2024-05-09T08:40:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6b2ffd640 10 client.26446 handle_command_reply: tid=0 2024-05-09T08:40:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6dcc77740 1 client.26446 shutdown 2024-05-09T08:40:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1243223935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 msgr2=0x558601c44fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c44fb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc011ed0 tx=0x7fc6cc007a10 comp rx=0 tx=0).stop 2024-05-09T08:40:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1243223935 shutdown_connections 2024-05-09T08:40:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558601c797c0 0x558601c7bc00 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c44fb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3cd70 0x558601c44a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1243223935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3c450 0x558601c44530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.102+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1243223935 >> 172.21.15.28:0/1243223935 conn(0x558601c38960 msgr2=0x558601c3e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.103+0000 7fc6b2ffd640 0 client.26446 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.103+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1243223935 shutdown_connections 2024-05-09T08:40:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.103+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1243223935 wait complete. 2024-05-09T08:40:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.104+0000 7fc6dcc77740 20 client.26446 trim_cache size 0 max 16384 2024-05-09T08:40:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.137+0000 7fc6dcc77740 1 Processor -- start 2024-05-09T08:40:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6dcc77740 1 -- start start 2024-05-09T08:40:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c3c450 0x558601c53200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c3ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c47630 0x558601c44dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b40102a0 con 0x558601c3cd70 2024-05-09T08:40:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b4012020 con 0x558601c47630 2024-05-09T08:40:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b4010a30 con 0x558601c3c450 2024-05-09T08:40:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c3ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c3ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55956/0 (socket says 172.21.15.28:55956) 2024-05-09T08:40:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.138+0000 7fc6d2ffd640 1 -- 172.21.15.28:0/1440532140 learned_addr learned my addr 172.21.15.28:0/1440532140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.139+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/1440532140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c47630 0x558601c44dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.140+0000 7fc6d2ffd640 1 -- 172.21.15.28:0/1440532140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c3c450 msgr2=0x558601c53200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.140+0000 7fc6d2ffd640 1 --2- 172.21.15.28:0/1440532140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c3c450 0x558601c53200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.140+0000 7fc6d2ffd640 1 -- 172.21.15.28:0/1440532140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c47630 msgr2=0x558601c44dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.140+0000 7fc6d2ffd640 1 --2- 172.21.15.28:0/1440532140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c47630 0x558601c44dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.140+0000 7fc6d2ffd640 1 -- 172.21.15.28:0/1440532140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558601c36f00 con 0x558601c3cd70 2024-05-09T08:40:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.142+0000 7fc6d2ffd640 1 --2- 172.21.15.28:0/1440532140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c3ee60 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009230 tx=0x7fc6c403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1650a329b6ce00af server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.143+0000 7fc6c97fa640 1 -- 172.21.15.28:0/1440532140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c404e070 con 0x558601c3cd70 2024-05-09T08:40:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.143+0000 7fc6c97fa640 1 -- 172.21.15.28:0/1440532140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c4042030 con 0x558601c3cd70 2024-05-09T08:40:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.143+0000 7fc6c97fa640 1 -- 172.21.15.28:0/1440532140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4035040 con 0x558601c3cd70 2024-05-09T08:40:35.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.147+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1440532140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 msgr2=0x558601c3ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:35.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.147+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1440532140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c3ee60 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009230 tx=0x7fc6c403c000 comp rx=0 tx=0).stop 2024-05-09T08:40:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.149+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1440532140 shutdown_connections 2024-05-09T08:40:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.149+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1440532140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c47630 0x558601c44dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.149+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1440532140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c3ee60 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.149+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/1440532140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c3c450 0x558601c53200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.149+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1440532140 >> 172.21.15.28:0/1440532140 conn(0x558601c38960 msgr2=0x558601c42e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.153+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1440532140 shutdown_connections 2024-05-09T08:40:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.154+0000 7fc6dcc77740 1 -- 172.21.15.28:0/1440532140 wait complete. 2024-05-09T08:40:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.154+0000 7fc6dcc77740 1 -- start start 2024-05-09T08:40:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.155+0000 7fc6dcc77740 1 Processor -- start 2024-05-09T08:40:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.155+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 0x558601c4f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.155+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c4fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.155+0000 7fc6dcc77740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c4a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.155+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b4011950 con 0x558601c3cd70 2024-05-09T08:40:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.155+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b40111c0 con 0x558601c3c450 2024-05-09T08:40:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.155+0000 7fc6dcc77740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b400e280 con 0x558601c47630 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.155+0000 7fc6dcc77740 10 client.0 _ll_register_callbacks cb 0x7fc6cc007b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.156+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c4a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.156+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c4a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50064/0 (socket says 172.21.15.28:50064) 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.156+0000 7fc6d3fff640 1 -- 172.21.15.28:0/2670114134 learned_addr learned my addr 172.21.15.28:0/2670114134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.156+0000 7fc6d3fff640 1 -- 172.21.15.28:0/2670114134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 msgr2=0x558601c4f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.156+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 0x558601c4f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.156+0000 7fc6d3fff640 1 -- 172.21.15.28:0/2670114134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 msgr2=0x558601c4fc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.156+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c4fc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.156+0000 7fc6d3fff640 1 -- 172.21.15.28:0/2670114134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558601c4b0d0 con 0x558601c47630 2024-05-09T08:40:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.156+0000 7fc6d3fff640 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c4a8d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc004420 tx=0x7fc6cc003fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dca1e826a0aef9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.157+0000 7fc6b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:40:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.157+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/2670114134 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc039060 con 0x558601c47630 2024-05-09T08:40:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.157+0000 7fc6dcc77740 1 -- 172.21.15.28:0/2670114134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6c4042030 con 0x558601c47630 2024-05-09T08:40:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.157+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/2670114134 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9707+0+0 (secure 0 0 0) 0x7fc6cc00cc40 con 0x558601c47630 2024-05-09T08:40:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.157+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/2670114134 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fc6cc040050 con 0x558601c47630 2024-05-09T08:40:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.160+0000 7fc6dcc77740 10 client.26449 fetch_fsmap learned FSMap version 67 2024-05-09T08:40:35.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.160+0000 7fc6dcc77740 1 -- 172.21.15.28:0/2670114134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x558601d00a00 con 0x558601c47630 2024-05-09T08:40:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.164+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/2670114134 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c4042030 con 0x558601c47630 2024-05-09T08:40:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.164+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/2670114134 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fc6cc033070 con 0x558601c47630 2024-05-09T08:40:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.164+0000 7fc6dcc77740 10 client.26449 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:40:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.164+0000 7fc6dcc77740 10 client.26449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:40:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.164+0000 7fc6dcc77740 20 client.26449 populate_metadata read hostname 'smithi028' 2024-05-09T08:40:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.164+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558601d01210 0x558601d03650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.164+0000 7fc6dcc77740 4 client.26449 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:40:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.164+0000 7fc6dcc77740 1 -- 172.21.15.28:0/2670114134 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558601ba8dc0 con 0x558601d01210 2024-05-09T08:40:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.166+0000 7fc6d37fe640 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558601d01210 0x558601d03650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:35.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.180+0000 7fc6d37fe640 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558601d01210 0x558601d03650 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8778d217fc9ba97 server_cookie=d46200c90abe7f71 in_seq=0 out_seq=0 2024-05-09T08:40:35.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.181+0000 7fc6b2ffd640 10 client.26449 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.320+0000 7fc6b2ffd640 1 -- 172.21.15.28:0/2670114134 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558601ba8dc0 con 0x558601d01210 2024-05-09T08:40:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.320+0000 7fc6b2ffd640 10 client.26449 handle_command_reply: tid=0 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.320+0000 7fc6dcc77740 1 client.26449 shutdown 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 -- 172.21.15.28:0/2670114134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 msgr2=0x558601c4a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c4a8d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc004420 tx=0x7fc6cc003fc0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 -- 172.21.15.28:0/2670114134 shutdown_connections 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558601d01210 0x558601d03650 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558601c47630 0x558601c4a8d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558601c3cd70 0x558601c4fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 --2- 172.21.15.28:0/2670114134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558601c3c450 0x558601c4f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 -- 172.21.15.28:0/2670114134 >> 172.21.15.28:0/2670114134 conn(0x558601c38960 msgr2=0x558601c37a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6b2ffd640 0 client.26449 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 -- 172.21.15.28:0/2670114134 shutdown_connections 2024-05-09T08:40:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 1 -- 172.21.15.28:0/2670114134 wait complete. 2024-05-09T08:40:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.321+0000 7fc6dcc77740 20 client.26449 trim_cache size 0 max 16384 2024-05-09T08:40:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:40:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:40:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:40:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:40:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.322+0000 7fc6d9f63640 1 -- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc6c00a2ef0 msgr2=0x7fc6c00a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.322+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc6c00a2ef0 0x7fc6c00a53b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40096c0 tx=0x7fc6c4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.322+0000 7fc6d9f63640 1 -- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41008a0 msgr2=0x7fc6d4073080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.322+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41008a0 0x7fc6d4073080 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00b450 tx=0x7fc6bc00b910 comp rx=0 tx=0).stop 2024-05-09T08:40:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.323+0000 7fc6d9f63640 1 -- 172.21.15.28:0/1429297880 shutdown_connections 2024-05-09T08:40:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.323+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6d4105ca0 0x7fc6d406c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.323+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc6c00a2ef0 0x7fc6c00a53b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.323+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6d41011c0 0x7fc6d406c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.323+0000 7fc6d9f63640 1 --2- 172.21.15.28:0/1429297880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d41008a0 0x7fc6d4073080 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.323+0000 7fc6d9f63640 1 -- 172.21.15.28:0/1429297880 >> 172.21.15.28:0/1429297880 conn(0x7fc6d4077880 msgr2=0x7fc6d4103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.323+0000 7fc6d9f63640 1 -- 172.21.15.28:0/1429297880 shutdown_connections 2024-05-09T08:40:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:35.323+0000 7fc6d9f63640 1 -- 172.21.15.28:0/1429297880 wait complete. 2024-05-09T08:40:35.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:35 smithi028 ceph-mon[30119]: pgmap v457: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 170 op/s 2024-05-09T08:40:36.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:35 smithi165 ceph-mon[34540]: pgmap v457: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 170 op/s 2024-05-09T08:40:36.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:35 smithi184 ceph-mon[34509]: pgmap v457: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 170 op/s 2024-05-09T08:40:37.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:37 smithi028 ceph-mon[30119]: pgmap v458: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-05-09T08:40:38.002 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:37 smithi184 ceph-mon[34509]: pgmap v458: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-05-09T08:40:38.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:37 smithi165 ceph-mon[34540]: pgmap v458: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-05-09T08:40:39.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:39 smithi028 ceph-mon[30119]: pgmap v459: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-05-09T08:40:40.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:39 smithi165 ceph-mon[34540]: pgmap v459: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-05-09T08:40:40.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:39 smithi184 ceph-mon[34509]: pgmap v459: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-05-09T08:40:42.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:41 smithi165 ceph-mon[34540]: pgmap v460: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-05-09T08:40:42.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:41 smithi184 ceph-mon[34509]: pgmap v460: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-05-09T08:40:42.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:41 smithi028 ceph-mon[30119]: pgmap v460: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-05-09T08:40:44.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:43 smithi165 ceph-mon[34540]: pgmap v461: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-05-09T08:40:44.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:43 smithi184 ceph-mon[34509]: pgmap v461: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-05-09T08:40:44.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:43 smithi028 ceph-mon[30119]: pgmap v461: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-05-09T08:40:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:45 smithi165 ceph-mon[34540]: pgmap v462: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-05-09T08:40:46.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:45 smithi184 ceph-mon[34509]: pgmap v462: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-05-09T08:40:46.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:45 smithi028 ceph-mon[30119]: pgmap v462: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-05-09T08:40:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:47 smithi165 ceph-mon[34540]: pgmap v463: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-05-09T08:40:48.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:47 smithi184 ceph-mon[34509]: pgmap v463: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-05-09T08:40:48.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:47 smithi028 ceph-mon[30119]: pgmap v463: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-05-09T08:40:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.479+0000 7f0568fee640 1 -- 172.21.15.28:0/1382498139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f056410d290 msgr2=0x7f056410d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.479+0000 7f0568fee640 1 --2- 172.21.15.28:0/1382498139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f056410d290 0x7f056410d670 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0558013f10 tx=0x7f05580329c0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.479+0000 7f0568fee640 1 -- 172.21.15.28:0/1382498139 shutdown_connections 2024-05-09T08:40:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.479+0000 7f0568fee640 1 --2- 172.21.15.28:0/1382498139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0564108d70 0x7f05641091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.479+0000 7f0568fee640 1 --2- 172.21.15.28:0/1382498139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0564069580 0x7f0564108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.479+0000 7f0568fee640 1 --2- 172.21.15.28:0/1382498139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f056410d290 0x7f056410d670 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.479+0000 7f0568fee640 1 -- 172.21.15.28:0/1382498139 >> 172.21.15.28:0/1382498139 conn(0x7f056406cd90 msgr2=0x7f056406d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.480+0000 7f0568fee640 1 -- 172.21.15.28:0/1382498139 shutdown_connections 2024-05-09T08:40:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.480+0000 7f0568fee640 1 -- 172.21.15.28:0/1382498139 wait complete. 2024-05-09T08:40:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.481+0000 7f0568fee640 1 Processor -- start 2024-05-09T08:40:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.481+0000 7f0568fee640 1 -- start start 2024-05-09T08:40:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.482+0000 7f0568fee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0564069580 0x7f05641b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.482+0000 7f0568fee640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0564108d70 0x7f05641b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.482+0000 7f0568fee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f056410d290 0x7f05641af620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.482+0000 7f0568fee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0564129450 con 0x7f0564069580 2024-05-09T08:40:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.482+0000 7f0568fee640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f05641161c0 con 0x7f0564108d70 2024-05-09T08:40:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.482+0000 7f0568fee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f05641295d0 con 0x7f056410d290 2024-05-09T08:40:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.482+0000 7f056259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0564069580 0x7f05641b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.483+0000 7f056259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0564069580 0x7f05641b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55922/0 (socket says 172.21.15.28:55922) 2024-05-09T08:40:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.483+0000 7f056259c640 1 -- 172.21.15.28:0/4145201034 learned_addr learned my addr 172.21.15.28:0/4145201034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.483+0000 7f0561d9b640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0564108d70 0x7f05641b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.483+0000 7f056259c640 1 -- 172.21.15.28:0/4145201034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f056410d290 msgr2=0x7f05641af620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.483+0000 7f056259c640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f056410d290 0x7f05641af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.483+0000 7f056259c640 1 -- 172.21.15.28:0/4145201034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0564108d70 msgr2=0x7f05641b59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.483+0000 7f056259c640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0564108d70 0x7f05641b59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.483+0000 7f056259c640 1 -- 172.21.15.28:0/4145201034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05641afda0 con 0x7f0564069580 2024-05-09T08:40:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.484+0000 7f056259c640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0564069580 0x7f05641b5470 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0558009230 tx=0x7f0558004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.484+0000 7f05537fe640 1 -- 172.21.15.28:0/4145201034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558004020 con 0x7f0564069580 2024-05-09T08:40:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.484+0000 7f05537fe640 1 -- 172.21.15.28:0/4145201034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05580125f0 con 0x7f0564069580 2024-05-09T08:40:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.484+0000 7f0568fee640 1 -- 172.21.15.28:0/4145201034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05641b0030 con 0x7f0564069580 2024-05-09T08:40:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.485+0000 7f0568fee640 1 -- 172.21.15.28:0/4145201034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05641bc330 con 0x7f0564069580 2024-05-09T08:40:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.485+0000 7f05537fe640 1 -- 172.21.15.28:0/4145201034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558003970 con 0x7f0564069580 2024-05-09T08:40:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.491+0000 7f056bdd9740 1 Processor -- start 2024-05-09T08:40:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.491+0000 7f05537fe640 1 -- 172.21.15.28:0/4145201034 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0558047060 con 0x7f0564069580 2024-05-09T08:40:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.493+0000 7f05537fe640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f054c0a2ea0 0x7f054c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.493+0000 7f05537fe640 1 -- 172.21.15.28:0/4145201034 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9707+0+0 (secure 0 0 0) 0x7f05580ef840 con 0x7f0564069580 2024-05-09T08:40:48.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.494+0000 7f0562d9d640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f054c0a2ea0 0x7f054c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056bdd9740 1 -- start start 2024-05-09T08:40:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611450 0x5594d1611830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611d70 0x5594d161c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d161c630 0x5594d161ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056bdd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5594d1576340 con 0x5594d1611d70 2024-05-09T08:40:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056bdd9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5594d151e290 con 0x5594d1611450 2024-05-09T08:40:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056bdd9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5594d1572300 con 0x5594d161c630 2024-05-09T08:40:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611450 0x5594d1611830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611450 0x5594d1611830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:39752/0 (socket says 172.21.15.28:39752) 2024-05-09T08:40:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.495+0000 7f056259c640 1 -- 172.21.15.28:0/2344870363 learned_addr learned my addr 172.21.15.28:0/2344870363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.496+0000 7f0561d9b640 1 --2- 172.21.15.28:0/2344870363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611d70 0x5594d161c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.496+0000 7f056259c640 1 -- 172.21.15.28:0/2344870363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d161c630 msgr2=0x5594d161ea20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:40:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.496+0000 7f056259c640 1 --2- 172.21.15.28:0/2344870363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d161c630 0x5594d161ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.496+0000 7f056259c640 1 -- 172.21.15.28:0/2344870363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611d70 msgr2=0x5594d161c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.496+0000 7f056259c640 1 --2- 172.21.15.28:0/2344870363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611d70 0x5594d161c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.496+0000 7f056259c640 1 -- 172.21.15.28:0/2344870363 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594d15428e0 con 0x5594d1611450 2024-05-09T08:40:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.496+0000 7f056259c640 1 --2- 172.21.15.28:0/2344870363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611450 0x5594d1611830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f05580b7cc0 tx=0x7f05580bc020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=654da90085efcbcd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.497+0000 7f0562d9d640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f054c0a2ea0 0x7f054c0a5360 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f055c00ab10 tx=0x7f055c009230 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.498+0000 7f0550ff9640 1 -- 172.21.15.28:0/2344870363 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05580d3020 con 0x5594d1611450 2024-05-09T08:40:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.498+0000 7f0550ff9640 1 -- 172.21.15.28:0/2344870363 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05580c44d0 con 0x5594d1611450 2024-05-09T08:40:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.499+0000 7f056bdd9740 1 -- 172.21.15.28:0/2344870363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611450 msgr2=0x5594d1611830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.499+0000 7f056bdd9740 1 --2- 172.21.15.28:0/2344870363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611450 0x5594d1611830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f05580b7cc0 tx=0x7f05580bc020 comp rx=0 tx=0).stop 2024-05-09T08:40:48.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.503+0000 7f056bdd9740 1 -- 172.21.15.28:0/2344870363 shutdown_connections 2024-05-09T08:40:48.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.503+0000 7f056bdd9740 1 --2- 172.21.15.28:0/2344870363 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d161c630 0x5594d161ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.503+0000 7f056bdd9740 1 --2- 172.21.15.28:0/2344870363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611d70 0x5594d161c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.503+0000 7f056bdd9740 1 --2- 172.21.15.28:0/2344870363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611450 0x5594d1611830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.503+0000 7f056bdd9740 1 -- 172.21.15.28:0/2344870363 >> 172.21.15.28:0/2344870363 conn(0x5594d160d960 msgr2=0x5594d160dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.506+0000 7f056bdd9740 1 -- 172.21.15.28:0/2344870363 shutdown_connections 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.508+0000 7f056bdd9740 1 -- 172.21.15.28:0/2344870363 wait complete. 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.509+0000 7f056bdd9740 1 -- start start 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.509+0000 7f056bdd9740 1 Processor -- start 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.510+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611450 0x5594d1619500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.510+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611d70 0x5594d1619a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.510+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d161c630 0x5594d1619f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.510+0000 7f056bdd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5594d161f020 con 0x5594d1611450 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.510+0000 7f056bdd9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5594d1531790 con 0x5594d161c630 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.510+0000 7f056bdd9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5594d161f1a0 con 0x5594d1611d70 2024-05-09T08:40:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.510+0000 7f056bdd9740 10 client.0 _ll_register_callbacks cb 0x5594d156ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.512+0000 7f0562d9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d161c630 0x5594d1619f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.512+0000 7f0562d9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d161c630 0x5594d1619f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:39766/0 (socket says 172.21.15.28:39766) 2024-05-09T08:40:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.512+0000 7f0562d9d640 1 -- 172.21.15.28:0/3974911203 learned_addr learned my addr 172.21.15.28:0/3974911203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.513+0000 7f0561d9b640 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611d70 0x5594d1619a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.514+0000 7f056259c640 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611450 0x5594d1619500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.514+0000 7f0561d9b640 1 -- 172.21.15.28:0/3974911203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d161c630 msgr2=0x5594d1619f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.514+0000 7f0561d9b640 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d161c630 0x5594d1619f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.514+0000 7f0561d9b640 1 -- 172.21.15.28:0/3974911203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611450 msgr2=0x5594d1619500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.514+0000 7f0561d9b640 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611450 0x5594d1619500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.514+0000 7f0561d9b640 1 -- 172.21.15.28:0/3974911203 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594d128a210 con 0x5594d1611d70 2024-05-09T08:40:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.515+0000 7f0561d9b640 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611d70 0x5594d1619a40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f05641167d0 tx=0x7f0554011630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5d6e539676d24c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.516+0000 7f0549ffb640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:40:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.519+0000 7f0549ffb640 1 -- 172.21.15.28:0/3974911203 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554019070 con 0x5594d1611d70 2024-05-09T08:40:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.519+0000 7f0549ffb640 1 -- 172.21.15.28:0/3974911203 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9707+0+0 (secure 0 0 0) 0x7f0554015660 con 0x5594d1611d70 2024-05-09T08:40:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.519+0000 7f056bdd9740 1 -- 172.21.15.28:0/3974911203 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05580c44d0 con 0x5594d1611d70 2024-05-09T08:40:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.520+0000 7f0549ffb640 1 -- 172.21.15.28:0/3974911203 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f0554023050 con 0x5594d1611d70 2024-05-09T08:40:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.521+0000 7f056bdd9740 10 client.26452 fetch_fsmap learned FSMap version 67 2024-05-09T08:40:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.521+0000 7f056bdd9740 1 -- 172.21.15.28:0/3974911203 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5594d1356620 con 0x5594d1611d70 2024-05-09T08:40:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.523+0000 7f0549ffb640 1 -- 172.21.15.28:0/3974911203 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05580c44d0 con 0x5594d1611d70 2024-05-09T08:40:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.524+0000 7f0549ffb640 1 -- 172.21.15.28:0/3974911203 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f055401eb60 con 0x5594d1611d70 2024-05-09T08:40:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.525+0000 7f056bdd9740 10 client.26452 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:40:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.525+0000 7f056bdd9740 10 client.26452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:40:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.525+0000 7f056bdd9740 20 client.26452 populate_metadata read hostname 'smithi028' 2024-05-09T08:40:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.525+0000 7f056bdd9740 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5594d162c200 0x5594d162e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.525+0000 7f056bdd9740 4 client.26452 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.525+0000 7f056bdd9740 1 -- 172.21.15.28:0/3974911203 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594d1583a70 con 0x5594d162c200 2024-05-09T08:40:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.530+0000 7f056259c640 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5594d162c200 0x5594d162e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.535+0000 7f056259c640 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5594d162c200 0x5594d162e620 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90cc93420a7fb726 server_cookie=13595a21d0f3c954 in_seq=0 out_seq=0 2024-05-09T08:40:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.535+0000 7f0549ffb640 10 client.26452 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.536+0000 7f0549ffb640 1 -- 172.21.15.28:0/3974911203 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5594d1583a70 con 0x5594d162c200 2024-05-09T08:40:48.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.536+0000 7f0549ffb640 10 client.26452 handle_command_reply: tid=0 2024-05-09T08:40:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.537+0000 7f056bdd9740 1 client.26452 shutdown 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 -- 172.21.15.28:0/3974911203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611d70 msgr2=0x5594d1619a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611d70 0x5594d1619a40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f05641167d0 tx=0x7f0554011630 comp rx=0 tx=0).stop 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 -- 172.21.15.28:0/3974911203 shutdown_connections 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5594d162c200 0x5594d162e620 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d161c630 0x5594d1619f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611d70 0x5594d1619a40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 --2- 172.21.15.28:0/3974911203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d1611450 0x5594d1619500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 -- 172.21.15.28:0/3974911203 >> 172.21.15.28:0/3974911203 conn(0x5594d160d960 msgr2=0x5594d16136e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f0549ffb640 0 client.26452 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 -- 172.21.15.28:0/3974911203 shutdown_connections 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 1 -- 172.21.15.28:0/3974911203 wait complete. 2024-05-09T08:40:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.538+0000 7f056bdd9740 20 client.26452 trim_cache size 0 max 16384 2024-05-09T08:40:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.565+0000 7f056bdd9740 1 Processor -- start 2024-05-09T08:40:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.566+0000 7f056bdd9740 1 -- start start 2024-05-09T08:40:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.566+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611450 0x5594d1633ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.566+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 0x5594d1634430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.566+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d1627ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.567+0000 7f0561d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 0x5594d1634430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.567+0000 7f0561d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 0x5594d1634430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:39780/0 (socket says 172.21.15.28:39780) 2024-05-09T08:40:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.567+0000 7f0561d9b640 1 -- 172.21.15.28:0/2565425950 learned_addr learned my addr 172.21.15.28:0/2565425950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.567+0000 7f056bdd9740 1 -- 172.21.15.28:0/2565425950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0544010300 con 0x5594d161c630 2024-05-09T08:40:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.567+0000 7f056bdd9740 1 -- 172.21.15.28:0/2565425950 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0544012080 con 0x5594d1611d70 2024-05-09T08:40:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.567+0000 7f056bdd9740 1 -- 172.21.15.28:0/2565425950 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0544010a90 con 0x5594d1611450 2024-05-09T08:40:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.567+0000 7f0562d9d640 1 --2- 172.21.15.28:0/2565425950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d1627ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.568+0000 7f056259c640 1 --2- 172.21.15.28:0/2565425950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611450 0x5594d1633ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.569+0000 7f0561d9b640 1 -- 172.21.15.28:0/2565425950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611450 msgr2=0x5594d1633ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.569+0000 7f0561d9b640 1 --2- 172.21.15.28:0/2565425950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611450 0x5594d1633ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.569+0000 7f0561d9b640 1 -- 172.21.15.28:0/2565425950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 msgr2=0x5594d1627ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.569+0000 7f0561d9b640 1 --2- 172.21.15.28:0/2565425950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d1627ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.569+0000 7f0561d9b640 1 -- 172.21.15.28:0/2565425950 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594d16142c0 con 0x5594d1611d70 2024-05-09T08:40:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.574+0000 7f0561d9b640 1 --2- 172.21.15.28:0/2565425950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 0x5594d1634430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0554000c00 tx=0x7f05540099f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed6ed53a5935a0a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.576+0000 7f054bfff640 1 -- 172.21.15.28:0/2565425950 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554019040 con 0x5594d1611d70 2024-05-09T08:40:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.576+0000 7f054bfff640 1 -- 172.21.15.28:0/2565425950 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05540036b0 con 0x5594d1611d70 2024-05-09T08:40:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.576+0000 7f054bfff640 1 -- 172.21.15.28:0/2565425950 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05540148c0 con 0x5594d1611d70 2024-05-09T08:40:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.577+0000 7f056bdd9740 1 -- 172.21.15.28:0/2565425950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 msgr2=0x5594d1634430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.577+0000 7f056bdd9740 1 --2- 172.21.15.28:0/2565425950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 0x5594d1634430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0554000c00 tx=0x7f05540099f0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.578+0000 7f056bdd9740 1 -- 172.21.15.28:0/2565425950 shutdown_connections 2024-05-09T08:40:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.578+0000 7f056bdd9740 1 --2- 172.21.15.28:0/2565425950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d1627ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.578+0000 7f056bdd9740 1 --2- 172.21.15.28:0/2565425950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 0x5594d1634430 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.578+0000 7f056bdd9740 1 --2- 172.21.15.28:0/2565425950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611450 0x5594d1633ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.578+0000 7f056bdd9740 1 -- 172.21.15.28:0/2565425950 >> 172.21.15.28:0/2565425950 conn(0x5594d160d960 msgr2=0x5594d160c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.578+0000 7f056bdd9740 1 -- 172.21.15.28:0/2565425950 shutdown_connections 2024-05-09T08:40:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.578+0000 7f056bdd9740 1 -- 172.21.15.28:0/2565425950 wait complete. 2024-05-09T08:40:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.578+0000 7f056bdd9740 1 -- start start 2024-05-09T08:40:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.578+0000 7f056bdd9740 1 Processor -- start 2024-05-09T08:40:48.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611450 0x5594d1624400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 0x5594d1624940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f056bdd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d161fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f056bdd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05440119b0 con 0x5594d161c630 2024-05-09T08:40:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f056bdd9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0544011220 con 0x5594d1611d70 2024-05-09T08:40:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f056bdd9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f054400e2e0 con 0x5594d1611450 2024-05-09T08:40:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f056bdd9740 10 client.0 _ll_register_callbacks cb 0x7f0554009480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f0562d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d161fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f0562d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d161fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55958/0 (socket says 172.21.15.28:55958) 2024-05-09T08:40:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.579+0000 7f0562d9d640 1 -- 172.21.15.28:0/4259785651 learned_addr learned my addr 172.21.15.28:0/4259785651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:40:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.580+0000 7f0562d9d640 1 -- 172.21.15.28:0/4259785651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611450 msgr2=0x5594d1624400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:40:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.580+0000 7f0562d9d640 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611450 0x5594d1624400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.580+0000 7f0562d9d640 1 -- 172.21.15.28:0/4259785651 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 msgr2=0x5594d1624940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:40:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.580+0000 7f0562d9d640 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 0x5594d1624940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.580+0000 7f0562d9d640 1 -- 172.21.15.28:0/4259785651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594d16203b0 con 0x5594d161c630 2024-05-09T08:40:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.580+0000 7f0562d9d640 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d161fbb0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f055c01bbf0 tx=0x7f055c004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b3ebcacfac747f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.581+0000 7f0549ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:40:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.581+0000 7f0549ffb640 1 -- 172.21.15.28:0/4259785651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c01abf0 con 0x5594d161c630 2024-05-09T08:40:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.581+0000 7f056bdd9740 1 -- 172.21.15.28:0/4259785651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05540036b0 con 0x5594d161c630 2024-05-09T08:40:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.582+0000 7f0549ffb640 1 -- 172.21.15.28:0/4259785651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9707+0+0 (secure 0 0 0) 0x7f055c01ad90 con 0x5594d161c630 2024-05-09T08:40:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.582+0000 7f0549ffb640 1 -- 172.21.15.28:0/4259785651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f055c025420 con 0x5594d161c630 2024-05-09T08:40:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.582+0000 7f056bdd9740 10 client.17142 fetch_fsmap learned FSMap version 67 2024-05-09T08:40:48.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.582+0000 7f056bdd9740 1 -- 172.21.15.28:0/4259785651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5594d16924b0 con 0x5594d161c630 2024-05-09T08:40:48.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.584+0000 7f0549ffb640 1 -- 172.21.15.28:0/4259785651 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05540036b0 con 0x5594d161c630 2024-05-09T08:40:48.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.584+0000 7f0549ffb640 1 -- 172.21.15.28:0/4259785651 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f055c00d850 con 0x5594d161c630 2024-05-09T08:40:48.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.584+0000 7f056bdd9740 10 client.17142 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:40:48.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.584+0000 7f056bdd9740 10 client.17142 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:40:48.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.584+0000 7f056bdd9740 20 client.17142 populate_metadata read hostname 'smithi028' 2024-05-09T08:40:48.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.584+0000 7f056bdd9740 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5594d1692d50 0x5594d1695190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:48.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.584+0000 7f056bdd9740 4 client.17142 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:40:48.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.584+0000 7f056bdd9740 1 -- 172.21.15.28:0/4259785651 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5594d157ddc0 con 0x5594d1692d50 2024-05-09T08:40:48.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.596+0000 7f056259c640 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5594d1692d50 0x5594d1695190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:48.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.601+0000 7f056259c640 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5594d1692d50 0x5594d1695190 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4733e5a223be0e5 server_cookie=e7b7ead218571686 in_seq=0 out_seq=0 2024-05-09T08:40:48.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.601+0000 7f0549ffb640 10 client.17142 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.751+0000 7f0549ffb640 1 -- 172.21.15.28:0/4259785651 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5594d157ddc0 con 0x5594d1692d50 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.751+0000 7f0549ffb640 10 client.17142 handle_command_reply: tid=0 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.751+0000 7f056bdd9740 1 client.17142 shutdown 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 -- 172.21.15.28:0/4259785651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 msgr2=0x5594d161fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d161fbb0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f055c01bbf0 tx=0x7f055c004b60 comp rx=0 tx=0).stop 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 -- 172.21.15.28:0/4259785651 shutdown_connections 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5594d1692d50 0x5594d1695190 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594d161c630 0x5594d161fbb0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5594d1611d70 0x5594d1624940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 --2- 172.21.15.28:0/4259785651 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5594d1611450 0x5594d1624400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 -- 172.21.15.28:0/4259785651 >> 172.21.15.28:0/4259785651 conn(0x5594d160d960 msgr2=0x5594d161ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f0549ffb640 0 client.17142 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:40:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 -- 172.21.15.28:0/4259785651 shutdown_connections 2024-05-09T08:40:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 1 -- 172.21.15.28:0/4259785651 wait complete. 2024-05-09T08:40:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.752+0000 7f056bdd9740 20 client.17142 trim_cache size 0 max 16384 2024-05-09T08:40:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:40:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:40:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:40:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:40:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 -- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f054c0a2ea0 msgr2=0x7f054c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f054c0a2ea0 0x7f054c0a5360 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f055c00ab10 tx=0x7f055c009230 comp rx=0 tx=0).stop 2024-05-09T08:40:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 -- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0564069580 msgr2=0x7f05641b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0564069580 0x7f05641b5470 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0558009230 tx=0x7f0558004e40 comp rx=0 tx=0).stop 2024-05-09T08:40:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 -- 172.21.15.28:0/4145201034 shutdown_connections 2024-05-09T08:40:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f054c0a2ea0 0x7f054c0a5360 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f056410d290 0x7f05641af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0564108d70 0x7f05641b59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 --2- 172.21.15.28:0/4145201034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0564069580 0x7f05641b5470 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 -- 172.21.15.28:0/4145201034 >> 172.21.15.28:0/4145201034 conn(0x7f056406cd90 msgr2=0x7f0564072320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 -- 172.21.15.28:0/4145201034 shutdown_connections 2024-05-09T08:40:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:40:48.753+0000 7f0568fee640 1 -- 172.21.15.28:0/4145201034 wait complete. 2024-05-09T08:40:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:49 smithi165 ceph-mon[34540]: pgmap v464: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-05-09T08:40:50.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:49 smithi184 ceph-mon[34509]: pgmap v464: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-05-09T08:40:50.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:49 smithi028 ceph-mon[30119]: pgmap v464: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-05-09T08:40:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:51 smithi165 ceph-mon[34540]: pgmap v465: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-05-09T08:40:52.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:51 smithi184 ceph-mon[34509]: pgmap v465: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-05-09T08:40:52.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:51 smithi028 ceph-mon[30119]: pgmap v465: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-05-09T08:40:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:53 smithi165 ceph-mon[34540]: pgmap v466: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 948 KiB/s wr, 126 op/s 2024-05-09T08:40:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:53 smithi184 ceph-mon[34509]: pgmap v466: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 948 KiB/s wr, 126 op/s 2024-05-09T08:40:54.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:53 smithi028 ceph-mon[30119]: pgmap v466: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 948 KiB/s wr, 126 op/s 2024-05-09T08:40:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:55 smithi165 ceph-mon[34540]: pgmap v467: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 943 KiB/s wr, 123 op/s 2024-05-09T08:40:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:55 smithi184 ceph-mon[34509]: pgmap v467: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 943 KiB/s wr, 123 op/s 2024-05-09T08:40:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:55 smithi028 ceph-mon[30119]: pgmap v467: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 943 KiB/s wr, 123 op/s 2024-05-09T08:40:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:57 smithi165 ceph-mon[34540]: pgmap v468: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-05-09T08:40:58.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:57 smithi184 ceph-mon[34509]: pgmap v468: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-05-09T08:40:58.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:57 smithi028 ceph-mon[30119]: pgmap v468: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-05-09T08:41:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:40:59 smithi165 ceph-mon[34540]: pgmap v469: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 398 KiB/s wr, 125 op/s 2024-05-09T08:41:00.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:40:59 smithi184 ceph-mon[34509]: pgmap v469: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 398 KiB/s wr, 125 op/s 2024-05-09T08:41:00.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:40:59 smithi028 ceph-mon[30119]: pgmap v469: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 398 KiB/s wr, 125 op/s 2024-05-09T08:41:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.933+0000 7fd65e43c640 1 -- 172.21.15.28:0/547115288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6500a51d0 msgr2=0x7fd6500a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.933+0000 7fd65e43c640 1 --2- 172.21.15.28:0/547115288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6500a51d0 0x7fd6500a55b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd658066fa0 tx=0x7fd658091e40 comp rx=0 tx=0).stop 2024-05-09T08:41:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.937+0000 7fd65e43c640 1 -- 172.21.15.28:0/547115288 shutdown_connections 2024-05-09T08:41:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.937+0000 7fd65e43c640 1 --2- 172.21.15.28:0/547115288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6500aa2a0 0x7fd6500b1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.937+0000 7fd65e43c640 1 --2- 172.21.15.28:0/547115288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6500a5b80 0x7fd6500a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.937+0000 7fd65e43c640 1 --2- 172.21.15.28:0/547115288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6500a51d0 0x7fd6500a55b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.937+0000 7fd65e43c640 1 -- 172.21.15.28:0/547115288 >> 172.21.15.28:0/547115288 conn(0x7fd65001aa70 msgr2=0x7fd65001ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.944+0000 7fd65e43c640 1 -- 172.21.15.28:0/547115288 shutdown_connections 2024-05-09T08:41:01.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.945+0000 7fd65e43c640 1 -- 172.21.15.28:0/547115288 wait complete. 2024-05-09T08:41:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.945+0000 7fd65e43c640 1 Processor -- start 2024-05-09T08:41:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.945+0000 7fd65e43c640 1 -- start start 2024-05-09T08:41:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.946+0000 7fd65e43c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6500a51d0 0x7fd65013b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.946+0000 7fd65e43c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6500aa2a0 0x7fd65013b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.946+0000 7fd65e43c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65013fcf0 0x7fd6501401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.946+0000 7fd65e43c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6500b4f10 con 0x7fd65013fcf0 2024-05-09T08:41:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.946+0000 7fd65e43c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd6500b4d90 con 0x7fd6500a51d0 2024-05-09T08:41:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.946+0000 7fd65e43c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd6500b5090 con 0x7fd6500aa2a0 2024-05-09T08:41:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.946+0000 7fd65dc3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65013fcf0 0x7fd6501401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.946+0000 7fd65dc3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65013fcf0 0x7fd6501401a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49782/0 (socket says 172.21.15.28:49782) 2024-05-09T08:41:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.946+0000 7fd65dc3b640 1 -- 172.21.15.28:0/1527855323 learned_addr learned my addr 172.21.15.28:0/1527855323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.948+0000 7fd65d43a640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6500a51d0 0x7fd65013b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.949+0000 7fd65cc39640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6500aa2a0 0x7fd65013b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.953+0000 7fd65d43a640 1 -- 172.21.15.28:0/1527855323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6500aa2a0 msgr2=0x7fd65013b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.953+0000 7fd65d43a640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6500aa2a0 0x7fd65013b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.953+0000 7fd65d43a640 1 -- 172.21.15.28:0/1527855323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65013fcf0 msgr2=0x7fd6501401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.953+0000 7fd65d43a640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65013fcf0 0x7fd6501401a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.953+0000 7fd65d43a640 1 -- 172.21.15.28:0/1527855323 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6501407e0 con 0x7fd6500a51d0 2024-05-09T08:41:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.957+0000 7fd65d43a640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6500a51d0 0x7fd65013b430 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd658092430 tx=0x7fd65809a780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.961+0000 7fd64a7fc640 1 -- 172.21.15.28:0/1527855323 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65809abc0 con 0x7fd6500a51d0 2024-05-09T08:41:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.961+0000 7fd64a7fc640 1 -- 172.21.15.28:0/1527855323 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6580945b0 con 0x7fd6500a51d0 2024-05-09T08:41:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.961+0000 7fd64a7fc640 1 -- 172.21.15.28:0/1527855323 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6580a33c0 con 0x7fd6500a51d0 2024-05-09T08:41:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.961+0000 7fd65e43c640 1 -- 172.21.15.28:0/1527855323 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd650140a70 con 0x7fd6500a51d0 2024-05-09T08:41:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.961+0000 7fd65e43c640 1 -- 172.21.15.28:0/1527855323 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd65000e660 con 0x7fd6500a51d0 2024-05-09T08:41:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd6623ad740 1 Processor -- start 2024-05-09T08:41:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd6623ad740 1 -- start start 2024-05-09T08:41:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d42f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d430d7f0 0x5564d430fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd6623ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564d4256340 con 0x5564d42f1e00 2024-05-09T08:41:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd6623ad740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5564d41fe290 con 0x5564d42f14e0 2024-05-09T08:41:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd6623ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5564d4252300 con 0x5564d430d7f0 2024-05-09T08:41:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd65dc3b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d42f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd65dc3b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d42f18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33550/0 (socket says 172.21.15.28:33550) 2024-05-09T08:41:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.970+0000 7fd65dc3b640 1 -- 172.21.15.28:0/3345246735 learned_addr learned my addr 172.21.15.28:0/3345246735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.973+0000 7fd64a7fc640 1 -- 172.21.15.28:0/1527855323 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd658074020 con 0x7fd6500a51d0 2024-05-09T08:41:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.974+0000 7fd65cc39640 1 --2- 172.21.15.28:0/3345246735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d430d7f0 0x5564d430fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.974+0000 7fd65dc3b640 1 -- 172.21.15.28:0/3345246735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d430d7f0 msgr2=0x5564d430fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.974+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/3345246735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d430d7f0 0x5564d430fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.974+0000 7fd65dc3b640 1 -- 172.21.15.28:0/3345246735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 msgr2=0x5564d430d2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:41:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.974+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/3345246735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430d2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.974+0000 7fd65dc3b640 1 -- 172.21.15.28:0/3345246735 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d42228e0 con 0x5564d42f14e0 2024-05-09T08:41:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.975+0000 7fd64a7fc640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6440a2e80 0x7fd6440a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.975+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/3345246735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d42f18c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd64c02ccc0 tx=0x7fd64c0317a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c920ad795c3ab06b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.975+0000 7fd64a7fc640 1 -- 172.21.15.28:0/1527855323 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9707+0+0 (secure 0 0 0) 0x7fd65811c5c0 con 0x7fd6500a51d0 2024-05-09T08:41:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.977+0000 7fd65d43a640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6440a2e80 0x7fd6440a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.977+0000 7fd65d43a640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6440a2e80 0x7fd6440a5340 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd65802d750 tx=0x7fd65806e040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.979+0000 7fd6437fe640 1 -- 172.21.15.28:0/3345246735 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c004220 con 0x5564d42f14e0 2024-05-09T08:41:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.979+0000 7fd6437fe640 1 -- 172.21.15.28:0/3345246735 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd64c036070 con 0x5564d42f14e0 2024-05-09T08:41:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.980+0000 7fd6623ad740 1 -- 172.21.15.28:0/3345246735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 msgr2=0x5564d42f18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.980+0000 7fd6623ad740 1 --2- 172.21.15.28:0/3345246735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d42f18c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd64c02ccc0 tx=0x7fd64c0317a0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.984+0000 7fd6623ad740 1 -- 172.21.15.28:0/3345246735 shutdown_connections 2024-05-09T08:41:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.984+0000 7fd6623ad740 1 --2- 172.21.15.28:0/3345246735 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d430d7f0 0x5564d430fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.984+0000 7fd6623ad740 1 --2- 172.21.15.28:0/3345246735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.984+0000 7fd6623ad740 1 --2- 172.21.15.28:0/3345246735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d42f18c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.984+0000 7fd6623ad740 1 -- 172.21.15.28:0/3345246735 >> 172.21.15.28:0/3345246735 conn(0x5564d42edb10 msgr2=0x5564d42edef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.988+0000 7fd6623ad740 1 -- 172.21.15.28:0/3345246735 shutdown_connections 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.990+0000 7fd6623ad740 1 -- 172.21.15.28:0/3345246735 wait complete. 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.990+0000 7fd6623ad740 1 -- start start 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.990+0000 7fd6623ad740 1 Processor -- start 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d430d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f1e00 0x5564d430a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d430d7f0 0x5564d430abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd6623ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564d4310470 con 0x5564d430d7f0 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd6623ad740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5564d4211790 con 0x5564d42f14e0 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd6623ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5564d43105f0 con 0x5564d42f1e00 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd6623ad740 10 client.0 _ll_register_callbacks cb 0x5564d424ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd65dc3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d430d7f0 0x5564d430abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd65dc3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d430d7f0 0x5564d430abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49792/0 (socket says 172.21.15.28:49792) 2024-05-09T08:41:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd65dc3b640 1 -- 172.21.15.28:0/2049330916 learned_addr learned my addr 172.21.15.28:0/2049330916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.991+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d430d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.992+0000 7fd65cc39640 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f1e00 0x5564d430a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.992+0000 7fd65dc3b640 1 -- 172.21.15.28:0/2049330916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f1e00 msgr2=0x5564d430a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.992+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f1e00 0x5564d430a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.992+0000 7fd65dc3b640 1 -- 172.21.15.28:0/2049330916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d430d7f0 msgr2=0x5564d430abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.992+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d430d7f0 0x5564d430abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.992+0000 7fd65dc3b640 1 -- 172.21.15.28:0/2049330916 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d3f6a210 con 0x5564d42f14e0 2024-05-09T08:41:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.992+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d430d210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd64c031ca0 tx=0x7fd64c031fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ab15902f7d0a0e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.992+0000 7fd640ff9640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:41:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.999+0000 7fd640ff9640 1 -- 172.21.15.28:0/2049330916 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c04d060 con 0x5564d42f14e0 2024-05-09T08:41:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.999+0000 7fd640ff9640 1 -- 172.21.15.28:0/2049330916 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9707+0+0 (secure 0 0 0) 0x7fd64c0041b0 con 0x5564d42f14e0 2024-05-09T08:41:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:01.999+0000 7fd6623ad740 1 -- 172.21.15.28:0/2049330916 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd64c036070 con 0x5564d42f14e0 2024-05-09T08:41:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.000+0000 7fd640ff9640 1 -- 172.21.15.28:0/2049330916 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fd64c0043b0 con 0x5564d42f14e0 2024-05-09T08:41:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.001+0000 7fd6623ad740 10 client.26621 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.001+0000 7fd6623ad740 1 -- 172.21.15.28:0/2049330916 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5564d4036620 con 0x5564d42f14e0 2024-05-09T08:41:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.004+0000 7fd640ff9640 1 -- 172.21.15.28:0/2049330916 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd64c036070 con 0x5564d42f14e0 2024-05-09T08:41:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.004+0000 7fd640ff9640 1 -- 172.21.15.28:0/2049330916 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fd64c045040 con 0x5564d42f14e0 2024-05-09T08:41:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.005+0000 7fd6623ad740 10 client.26621 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.005+0000 7fd6623ad740 10 client.26621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.005+0000 7fd6623ad740 20 client.26621 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.005+0000 7fd6623ad740 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d431d470 0x5564d431f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.005+0000 7fd6623ad740 4 client.26621 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.005+0000 7fd6623ad740 1 -- 172.21.15.28:0/2049330916 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564d4263a70 con 0x5564d431d470 2024-05-09T08:41:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.005+0000 7fd65cc39640 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d431d470 0x5564d431f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.006+0000 7fd65cc39640 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d431d470 0x5564d431f860 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c96b04abfc56bcd0 server_cookie=d0f88348130731a7 in_seq=0 out_seq=0 2024-05-09T08:41:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.006+0000 7fd640ff9640 10 client.26621 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.008+0000 7fd640ff9640 1 -- 172.21.15.28:0/2049330916 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564d4263a70 con 0x5564d431d470 2024-05-09T08:41:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.008+0000 7fd640ff9640 10 client.26621 handle_command_reply: tid=0 2024-05-09T08:41:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.008+0000 7fd6623ad740 1 client.26621 shutdown 2024-05-09T08:41:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.009+0000 7fd6623ad740 1 -- 172.21.15.28:0/2049330916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 msgr2=0x5564d430d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.009+0000 7fd6623ad740 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d430d210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd64c031ca0 tx=0x7fd64c031fd0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.010+0000 7fd6623ad740 1 -- 172.21.15.28:0/2049330916 shutdown_connections 2024-05-09T08:41:02.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.010+0000 7fd6623ad740 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d431d470 0x5564d431f860 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.010+0000 7fd6623ad740 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d430d7f0 0x5564d430abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.010+0000 7fd6623ad740 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f1e00 0x5564d430a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.010+0000 7fd6623ad740 1 --2- 172.21.15.28:0/2049330916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d430d210 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.010+0000 7fd6623ad740 1 -- 172.21.15.28:0/2049330916 >> 172.21.15.28:0/2049330916 conn(0x5564d42edb10 msgr2=0x5564d42f3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.013+0000 7fd640ff9640 0 client.26621 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.014+0000 7fd6623ad740 1 -- 172.21.15.28:0/2049330916 shutdown_connections 2024-05-09T08:41:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.015+0000 7fd6623ad740 1 -- 172.21.15.28:0/2049330916 wait complete. 2024-05-09T08:41:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.016+0000 7fd6623ad740 20 client.26621 trim_cache size 0 max 16384 2024-05-09T08:41:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.056+0000 7fd6623ad740 1 Processor -- start 2024-05-09T08:41:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.056+0000 7fd6623ad740 1 -- start start 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.056+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f14e0 0x5564d4309580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.056+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.056+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d430d7f0 0x5564d430d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.056+0000 7fd6623ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c010300 con 0x5564d42f1e00 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.056+0000 7fd6623ad740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c012080 con 0x5564d430d7f0 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.056+0000 7fd6623ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c010a90 con 0x5564d42f14e0 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65dc3b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f14e0 0x5564d4309580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65cc39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65cc39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49802/0 (socket says 172.21.15.28:49802) 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65cc39640 1 -- 172.21.15.28:0/1148454163 learned_addr learned my addr 172.21.15.28:0/1148454163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/1148454163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d430d7f0 0x5564d430d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65cc39640 1 -- 172.21.15.28:0/1148454163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f14e0 msgr2=0x5564d4309580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65cc39640 1 --2- 172.21.15.28:0/1148454163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f14e0 0x5564d4309580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65cc39640 1 -- 172.21.15.28:0/1148454163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d430d7f0 msgr2=0x5564d430d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65cc39640 1 --2- 172.21.15.28:0/1148454163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d430d7f0 0x5564d430d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.057+0000 7fd65cc39640 1 -- 172.21.15.28:0/1148454163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d42f1c30 con 0x5564d42f1e00 2024-05-09T08:41:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.058+0000 7fd65cc39640 1 --2- 172.21.15.28:0/1148454163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430ccd0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd6500a4b20 tx=0x7fd654008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6a30f2edbfd439 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.060+0000 7fd642ffd640 1 -- 172.21.15.28:0/1148454163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654016d70 con 0x5564d42f1e00 2024-05-09T08:41:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.060+0000 7fd642ffd640 1 -- 172.21.15.28:0/1148454163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd654023070 con 0x5564d42f1e00 2024-05-09T08:41:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.060+0000 7fd642ffd640 1 -- 172.21.15.28:0/1148454163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6540156b0 con 0x5564d42f1e00 2024-05-09T08:41:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.061+0000 7fd6623ad740 1 -- 172.21.15.28:0/1148454163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 msgr2=0x5564d430ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.061+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1148454163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430ccd0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd6500a4b20 tx=0x7fd654008040 comp rx=0 tx=0).stop 2024-05-09T08:41:02.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.064+0000 7fd6623ad740 1 -- 172.21.15.28:0/1148454163 shutdown_connections 2024-05-09T08:41:02.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.064+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1148454163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d430d7f0 0x5564d430d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.064+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1148454163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430ccd0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.064+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1148454163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d42f14e0 0x5564d4309580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.064+0000 7fd6623ad740 1 -- 172.21.15.28:0/1148454163 >> 172.21.15.28:0/1148454163 conn(0x5564d42edb10 msgr2=0x5564d4308e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.065+0000 7fd6623ad740 1 -- 172.21.15.28:0/1148454163 shutdown_connections 2024-05-09T08:41:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.066+0000 7fd6623ad740 1 -- 172.21.15.28:0/1148454163 wait complete. 2024-05-09T08:41:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.066+0000 7fd6623ad740 1 -- start start 2024-05-09T08:41:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.066+0000 7fd6623ad740 1 Processor -- start 2024-05-09T08:41:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d430ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd6623ad740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d430d7f0 0x5564d430e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd6623ad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c0119b0 con 0x5564d42f1e00 2024-05-09T08:41:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd6623ad740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c011220 con 0x5564d42f14e0 2024-05-09T08:41:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd6623ad740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c00e2e0 con 0x5564d430d7f0 2024-05-09T08:41:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd6623ad740 10 client.0 _ll_register_callbacks cb 0x7fd64c02cb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd65cc39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd65cc39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49806/0 (socket says 172.21.15.28:49806) 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.067+0000 7fd65cc39640 1 -- 172.21.15.28:0/1225364814 learned_addr learned my addr 172.21.15.28:0/1225364814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.068+0000 7fd65cc39640 1 -- 172.21.15.28:0/1225364814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d430d7f0 msgr2=0x5564d430e830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.068+0000 7fd65cc39640 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d430d7f0 0x5564d430e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.068+0000 7fd65cc39640 1 -- 172.21.15.28:0/1225364814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 msgr2=0x5564d430ddb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.068+0000 7fd65cc39640 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d430ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.068+0000 7fd65cc39640 1 -- 172.21.15.28:0/1225364814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d4314c00 con 0x5564d42f1e00 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.068+0000 7fd65cc39640 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430e2f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd654000c00 tx=0x7fd65401ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632033b121c43216 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.068+0000 7fd640ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:41:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.070+0000 7fd640ff9640 1 -- 172.21.15.28:0/1225364814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65401e2d0 con 0x5564d42f1e00 2024-05-09T08:41:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.070+0000 7fd6623ad740 1 -- 172.21.15.28:0/1225364814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd654023070 con 0x5564d42f1e00 2024-05-09T08:41:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.070+0000 7fd640ff9640 1 -- 172.21.15.28:0/1225364814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9707+0+0 (secure 0 0 0) 0x7fd65401ea30 con 0x5564d42f1e00 2024-05-09T08:41:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.071+0000 7fd640ff9640 1 -- 172.21.15.28:0/1225364814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fd654039050 con 0x5564d42f1e00 2024-05-09T08:41:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.071+0000 7fd6623ad740 10 client.17154 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.071+0000 7fd6623ad740 1 -- 172.21.15.28:0/1225364814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5564d4315eb0 con 0x5564d42f1e00 2024-05-09T08:41:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.072+0000 7fd640ff9640 1 -- 172.21.15.28:0/1225364814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd654023070 con 0x5564d42f1e00 2024-05-09T08:41:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.073+0000 7fd640ff9640 1 -- 172.21.15.28:0/1225364814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fd6540273f0 con 0x5564d42f1e00 2024-05-09T08:41:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.073+0000 7fd6623ad740 10 client.17154 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.073+0000 7fd6623ad740 10 client.17154 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.073+0000 7fd6623ad740 20 client.17154 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.073+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d43838e0 0x5564d4385d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.073+0000 7fd6623ad740 4 client.17154 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.073+0000 7fd6623ad740 1 -- 172.21.15.28:0/1225364814 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5564d425ddc0 con 0x5564d43838e0 2024-05-09T08:41:02.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.076+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d43838e0 0x5564d4385d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.093+0000 7fd640ff9640 10 client.17154 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:02.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.093+0000 7fd65dc3b640 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d43838e0 0x5564d4385d00 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6ce8edd2ad6d32 server_cookie=4920e892fa87c51a in_seq=0 out_seq=0 2024-05-09T08:41:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:01 smithi165 ceph-mon[34540]: pgmap v470: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 406 KiB/s wr, 123 op/s 2024-05-09T08:41:02.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:01 smithi184 ceph-mon[34509]: pgmap v470: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 406 KiB/s wr, 123 op/s 2024-05-09T08:41:02.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:01 smithi028 ceph-mon[30119]: pgmap v470: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 406 KiB/s wr, 123 op/s 2024-05-09T08:41:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd640ff9640 1 -- 172.21.15.28:0/1225364814 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5564d425ddc0 con 0x5564d43838e0 2024-05-09T08:41:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd640ff9640 10 client.17154 handle_command_reply: tid=0 2024-05-09T08:41:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 client.17154 shutdown 2024-05-09T08:41:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 -- 172.21.15.28:0/1225364814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 msgr2=0x5564d430e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430e2f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd654000c00 tx=0x7fd65401ec60 comp rx=0 tx=0).stop 2024-05-09T08:41:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 -- 172.21.15.28:0/1225364814 shutdown_connections 2024-05-09T08:41:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564d43838e0 0x5564d4385d00 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564d430d7f0 0x5564d430e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564d42f1e00 0x5564d430e2f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd640ff9640 0 client.17154 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 --2- 172.21.15.28:0/1225364814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564d42f14e0 0x5564d430ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 -- 172.21.15.28:0/1225364814 >> 172.21.15.28:0/1225364814 conn(0x5564d42edb10 msgr2=0x5564d4309700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.205+0000 7fd6623ad740 1 -- 172.21.15.28:0/1225364814 shutdown_connections 2024-05-09T08:41:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.206+0000 7fd6623ad740 1 -- 172.21.15.28:0/1225364814 wait complete. 2024-05-09T08:41:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.206+0000 7fd6623ad740 20 client.17154 trim_cache size 0 max 16384 2024-05-09T08:41:02.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:41:02.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:41:02.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:41:02.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:41:02.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.207+0000 7fd65e43c640 1 -- 172.21.15.28:0/1527855323 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6440a2e80 msgr2=0x7fd6440a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.207+0000 7fd65e43c640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6440a2e80 0x7fd6440a5340 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd65802d750 tx=0x7fd65806e040 comp rx=0 tx=0).stop 2024-05-09T08:41:02.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.207+0000 7fd65e43c640 1 -- 172.21.15.28:0/1527855323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6500a51d0 msgr2=0x7fd65013b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.207+0000 7fd65e43c640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6500a51d0 0x7fd65013b430 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd658092430 tx=0x7fd65809a780 comp rx=0 tx=0).stop 2024-05-09T08:41:02.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.207+0000 7fd65e43c640 1 -- 172.21.15.28:0/1527855323 shutdown_connections 2024-05-09T08:41:02.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.207+0000 7fd65e43c640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65013fcf0 0x7fd6501401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.207+0000 7fd65e43c640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd6440a2e80 0x7fd6440a5340 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.207+0000 7fd65e43c640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6500aa2a0 0x7fd65013b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.207+0000 7fd65e43c640 1 --2- 172.21.15.28:0/1527855323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd6500a51d0 0x7fd65013b430 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.208+0000 7fd65e43c640 1 -- 172.21.15.28:0/1527855323 >> 172.21.15.28:0/1527855323 conn(0x7fd65001aa70 msgr2=0x7fd6500a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.211+0000 7fd65e43c640 1 -- 172.21.15.28:0/1527855323 shutdown_connections 2024-05-09T08:41:02.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:02.212+0000 7fd65e43c640 1 -- 172.21.15.28:0/1527855323 wait complete. 2024-05-09T08:41:04.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:03 smithi165 ceph-mon[34540]: pgmap v471: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 541 KiB/s wr, 118 op/s 2024-05-09T08:41:04.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:03 smithi184 ceph-mon[34509]: pgmap v471: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 541 KiB/s wr, 118 op/s 2024-05-09T08:41:04.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:03 smithi028 ceph-mon[30119]: pgmap v471: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 541 KiB/s wr, 118 op/s 2024-05-09T08:41:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:05 smithi165 ceph-mon[34540]: pgmap v472: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 828 KiB/s wr, 121 op/s 2024-05-09T08:41:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:05 smithi165 ceph-mon[34540]: osdmap e121: 12 total, 12 up, 12 in 2024-05-09T08:41:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:05 smithi184 ceph-mon[34509]: pgmap v472: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 828 KiB/s wr, 121 op/s 2024-05-09T08:41:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:05 smithi184 ceph-mon[34509]: osdmap e121: 12 total, 12 up, 12 in 2024-05-09T08:41:06.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:05 smithi028 ceph-mon[30119]: pgmap v472: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 828 KiB/s wr, 121 op/s 2024-05-09T08:41:06.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:05 smithi028 ceph-mon[30119]: osdmap e121: 12 total, 12 up, 12 in 2024-05-09T08:41:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:07 smithi165 ceph-mon[34540]: pgmap v474: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 858 KiB/s wr, 127 op/s 2024-05-09T08:41:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:07 smithi184 ceph-mon[34509]: pgmap v474: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 858 KiB/s wr, 127 op/s 2024-05-09T08:41:08.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:07 smithi028 ceph-mon[30119]: pgmap v474: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 858 KiB/s wr, 127 op/s 2024-05-09T08:41:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:09 smithi165 ceph-mon[34540]: pgmap v475: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 143 op/s 2024-05-09T08:41:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:09 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:10.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:09 smithi184 ceph-mon[34509]: pgmap v475: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 143 op/s 2024-05-09T08:41:10.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:09 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:10.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:09 smithi028 ceph-mon[30119]: pgmap v475: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 143 op/s 2024-05-09T08:41:10.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:09 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:11.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:10 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:11.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:10 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:11.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:10 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:41:11.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:10 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:11.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:10 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:11.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:10 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:41:11.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:10 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:11.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:10 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:11.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:10 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:41:12.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:11 smithi165 ceph-mon[34540]: pgmap v476: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 145 op/s 2024-05-09T08:41:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:11 smithi184 ceph-mon[34509]: pgmap v476: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 145 op/s 2024-05-09T08:41:12.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:11 smithi028 ceph-mon[30119]: pgmap v476: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 145 op/s 2024-05-09T08:41:14.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:13 smithi165 ceph-mon[34540]: pgmap v477: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-05-09T08:41:14.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:13 smithi184 ceph-mon[34509]: pgmap v477: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-05-09T08:41:14.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:13 smithi028 ceph-mon[30119]: pgmap v477: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 142 op/s 2024-05-09T08:41:15.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:14 smithi165 ceph-mon[34540]: osdmap e122: 12 total, 12 up, 12 in 2024-05-09T08:41:15.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:14 smithi184 ceph-mon[34509]: osdmap e122: 12 total, 12 up, 12 in 2024-05-09T08:41:15.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:14 smithi028 ceph-mon[30119]: osdmap e122: 12 total, 12 up, 12 in 2024-05-09T08:41:15.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.408+0000 7f2da5590640 1 -- 172.21.15.28:0/2494551095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da011b010 msgr2=0x7f2da011d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:15.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.408+0000 7f2da5590640 1 --2- 172.21.15.28:0/2494551095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da011b010 0x7f2da011d400 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009930 tx=0x7f2d9002a4f0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.409+0000 7f2da5590640 1 -- 172.21.15.28:0/2494551095 shutdown_connections 2024-05-09T08:41:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.409+0000 7f2da5590640 1 --2- 172.21.15.28:0/2494551095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da011b010 0x7f2da011d400 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.409+0000 7f2da5590640 1 --2- 172.21.15.28:0/2494551095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2da0110400 0x7f2da0108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.409+0000 7f2da5590640 1 --2- 172.21.15.28:0/2494551095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da0110020 0x7f2da0108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.409+0000 7f2da5590640 1 -- 172.21.15.28:0/2494551095 >> 172.21.15.28:0/2494551095 conn(0x7f2da006c900 msgr2=0x7f2da006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.414+0000 7f2da5590640 1 -- 172.21.15.28:0/2494551095 shutdown_connections 2024-05-09T08:41:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.416+0000 7f2da5590640 1 -- 172.21.15.28:0/2494551095 wait complete. 2024-05-09T08:41:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.417+0000 7f2da5590640 1 Processor -- start 2024-05-09T08:41:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.422+0000 7f2da5590640 1 -- start start 2024-05-09T08:41:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.422+0000 7f2da5590640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da0110020 0x7f2da01a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.422+0000 7f2da5590640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2da011b010 0x7f2da01a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.422+0000 7f2da5590640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da01ab7d0 0x7f2da01abc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.422+0000 7f2da5590640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2da0120a30 con 0x7f2da0110020 2024-05-09T08:41:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.422+0000 7f2da5590640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2da01208b0 con 0x7f2da01ab7d0 2024-05-09T08:41:15.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.422+0000 7f2da5590640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2da0120bb0 con 0x7f2da011b010 2024-05-09T08:41:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.423+0000 7f2d9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da0110020 0x7f2da01a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.423+0000 7f2d9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da0110020 0x7f2da01a6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57914/0 (socket says 172.21.15.28:57914) 2024-05-09T08:41:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.423+0000 7f2d9effd640 1 -- 172.21.15.28:0/2916792384 learned_addr learned my addr 172.21.15.28:0/2916792384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.424+0000 7f2d9effd640 1 -- 172.21.15.28:0/2916792384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2da011b010 msgr2=0x7f2da01a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.424+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2da011b010 0x7f2da01a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.424+0000 7f2d9effd640 1 -- 172.21.15.28:0/2916792384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da01ab7d0 msgr2=0x7f2da01abc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:41:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.424+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da01ab7d0 0x7f2da01abc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.424+0000 7f2d9effd640 1 -- 172.21.15.28:0/2916792384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da01ac2c0 con 0x7f2da0110020 2024-05-09T08:41:15.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.425+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da0110020 0x7f2da01a6f10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400d950 tx=0x7f2d9400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.428+0000 7f2d87fff640 1 -- 172.21.15.28:0/2916792384 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94017070 con 0x7f2da0110020 2024-05-09T08:41:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.428+0000 7f2da5590640 1 -- 172.21.15.28:0/2916792384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da01ac5b0 con 0x7f2da0110020 2024-05-09T08:41:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.428+0000 7f2da5590640 1 -- 172.21.15.28:0/2916792384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da0110b10 con 0x7f2da0110020 2024-05-09T08:41:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.428+0000 7f2d87fff640 1 -- 172.21.15.28:0/2916792384 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d94009c30 con 0x7f2da0110020 2024-05-09T08:41:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.428+0000 7f2d87fff640 1 -- 172.21.15.28:0/2916792384 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9400f6b0 con 0x7f2da0110020 2024-05-09T08:41:15.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.432+0000 7f2da837c740 1 Processor -- start 2024-05-09T08:41:15.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.433+0000 7f2da837c740 1 -- start start 2024-05-09T08:41:15.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.433+0000 7f2da837c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc205830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.433+0000 7f2da837c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc2100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.433+0000 7f2da837c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc212a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.433+0000 7f2da837c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558efc16a340 con 0x558efc205d70 2024-05-09T08:41:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.433+0000 7f2da837c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558efc112290 con 0x558efc205450 2024-05-09T08:41:15.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.433+0000 7f2da837c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558efc166300 con 0x558efc210630 2024-05-09T08:41:15.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc212a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9e7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc212a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:59964/0 (socket says 172.21.15.28:59964) 2024-05-09T08:41:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9e7fc640 1 -- 172.21.15.28:0/2351673542 learned_addr learned my addr 172.21.15.28:0/2351673542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9f7fe640 1 --2- 172.21.15.28:0/2351673542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc205830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2351673542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc2100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9e7fc640 1 -- 172.21.15.28:0/2351673542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 msgr2=0x558efc205830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2351673542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc205830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9e7fc640 1 -- 172.21.15.28:0/2351673542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 msgr2=0x558efc2100f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2351673542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc2100f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.434+0000 7f2d9e7fc640 1 -- 172.21.15.28:0/2351673542 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558efc1368e0 con 0x558efc210630 2024-05-09T08:41:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.435+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2351673542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc212a20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009d20 tx=0x7f2d9800e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d56867cc1d63d79 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.436+0000 7f2d857fa640 1 -- 172.21.15.28:0/2351673542 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98016070 con 0x558efc210630 2024-05-09T08:41:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.436+0000 7f2d857fa640 1 -- 172.21.15.28:0/2351673542 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d98005020 con 0x558efc210630 2024-05-09T08:41:15.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.436+0000 7f2d857fa640 1 -- 172.21.15.28:0/2351673542 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d980116d0 con 0x558efc210630 2024-05-09T08:41:15.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.437+0000 7f2da837c740 1 -- 172.21.15.28:0/2351673542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 msgr2=0x558efc212a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:15.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.437+0000 7f2da837c740 1 --2- 172.21.15.28:0/2351673542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc212a20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009d20 tx=0x7f2d9800e6c0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.438+0000 7f2d87fff640 1 -- 172.21.15.28:0/2916792384 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2d9400f850 con 0x7f2da0110020 2024-05-09T08:41:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.453+0000 7f2da837c740 1 -- 172.21.15.28:0/2351673542 shutdown_connections 2024-05-09T08:41:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.453+0000 7f2da837c740 1 --2- 172.21.15.28:0/2351673542 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc212a20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.453+0000 7f2da837c740 1 --2- 172.21.15.28:0/2351673542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc2100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.453+0000 7f2da837c740 1 --2- 172.21.15.28:0/2351673542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc205830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.453+0000 7f2da837c740 1 -- 172.21.15.28:0/2351673542 >> 172.21.15.28:0/2351673542 conn(0x558efc201960 msgr2=0x558efc201d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.454+0000 7f2da837c740 1 -- 172.21.15.28:0/2351673542 shutdown_connections 2024-05-09T08:41:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.454+0000 7f2d87fff640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2d880a2ee0 0x7f2d880a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.455+0000 7f2d87fff640 1 -- 172.21.15.28:0/2916792384 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f2d940cf390 con 0x7f2da0110020 2024-05-09T08:41:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.455+0000 7f2da837c740 1 -- 172.21.15.28:0/2351673542 wait complete. 2024-05-09T08:41:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.455+0000 7f2da837c740 1 -- start start 2024-05-09T08:41:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.456+0000 7f2da837c740 1 Processor -- start 2024-05-09T08:41:15.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.456+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2d880a2ee0 0x7f2d880a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.463+0000 7f2da837c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc205450 0x558efc20d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.463+0000 7f2da837c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc20da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.463+0000 7f2da837c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc210630 0x558efc20dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.463+0000 7f2da837c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558efc213260 con 0x558efc205d70 2024-05-09T08:41:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.464+0000 7f2da837c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x558efc125790 con 0x558efc210630 2024-05-09T08:41:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.464+0000 7f2da837c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x558efc2133e0 con 0x558efc205450 2024-05-09T08:41:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.464+0000 7f2da837c740 10 client.0 _ll_register_callbacks cb 0x558efc160a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.468+0000 7f2d9effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc205450 0x558efc20d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.468+0000 7f2d9effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc205450 0x558efc20d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:59968/0 (socket says 172.21.15.28:59968) 2024-05-09T08:41:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.468+0000 7f2d9effd640 1 -- 172.21.15.28:0/2338151134 learned_addr learned my addr 172.21.15.28:0/2338151134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.468+0000 7f2d9f7fe640 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc210630 0x558efc20dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.469+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc20da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.474+0000 7f2d9effd640 1 -- 172.21.15.28:0/2338151134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc210630 msgr2=0x558efc20dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:15.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.474+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc210630 0x558efc20dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.474+0000 7f2d9effd640 1 -- 172.21.15.28:0/2338151134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 msgr2=0x558efc20da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:15.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.474+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc20da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.474+0000 7f2d9effd640 1 -- 172.21.15.28:0/2338151134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558efbe7e210 con 0x558efc205450 2024-05-09T08:41:15.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.477+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2d880a2ee0 0x7f2d880a53a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2d98000f80 tx=0x7f2d980059a0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.480+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc205450 0x558efc20d520 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2d9409b040 tx=0x7f2d9409b110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eba54c15b3c37aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.481+0000 7f2d82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:41:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.482+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2338151134 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94020070 con 0x558efc205450 2024-05-09T08:41:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.482+0000 7f2da837c740 1 -- 172.21.15.28:0/2338151134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d98005020 con 0x558efc205450 2024-05-09T08:41:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.482+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2338151134 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f2d940979b0 con 0x558efc205450 2024-05-09T08:41:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.482+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2338151134 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f2d94010d70 con 0x558efc205450 2024-05-09T08:41:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.482+0000 7f2da837c740 10 client.26479 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.482+0000 7f2da837c740 1 -- 172.21.15.28:0/2338151134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x558efbf4a620 con 0x558efc205450 2024-05-09T08:41:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.486+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2338151134 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d98005020 con 0x558efc205450 2024-05-09T08:41:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.486+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2338151134 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f2d940cc7f0 con 0x558efc205450 2024-05-09T08:41:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.486+0000 7f2da837c740 10 client.26479 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.486+0000 7f2da837c740 10 client.26479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.486+0000 7f2da837c740 20 client.26479 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.486+0000 7f2da837c740 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558efc220960 0x558efc222e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.486+0000 7f2da837c740 4 client.26479 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.486+0000 7f2da837c740 1 -- 172.21.15.28:0/2338151134 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558efc177a70 con 0x558efc220960 2024-05-09T08:41:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.489+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558efc220960 0x558efc222e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.495+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558efc220960 0x558efc222e00 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c052a411e1fd4f5 server_cookie=72ede08672a97883 in_seq=0 out_seq=0 2024-05-09T08:41:15.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.495+0000 7f2d82ff5640 10 client.26479 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.499+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2338151134 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558efc177a70 con 0x558efc220960 2024-05-09T08:41:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.499+0000 7f2d82ff5640 10 client.26479 handle_command_reply: tid=0 2024-05-09T08:41:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.499+0000 7f2da837c740 1 client.26479 shutdown 2024-05-09T08:41:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.507+0000 7f2da837c740 1 -- 172.21.15.28:0/2338151134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc205450 msgr2=0x558efc20d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.507+0000 7f2da837c740 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc205450 0x558efc20d520 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2d9409b040 tx=0x7f2d9409b110 comp rx=0 tx=0).stop 2024-05-09T08:41:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.511+0000 7f2da837c740 1 -- 172.21.15.28:0/2338151134 shutdown_connections 2024-05-09T08:41:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.511+0000 7f2da837c740 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558efc220960 0x558efc222e00 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.511+0000 7f2da837c740 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc210630 0x558efc20dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.511+0000 7f2da837c740 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc20da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.511+0000 7f2da837c740 1 --2- 172.21.15.28:0/2338151134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc205450 0x558efc20d520 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.511+0000 7f2da837c740 1 -- 172.21.15.28:0/2338151134 >> 172.21.15.28:0/2338151134 conn(0x558efc201960 msgr2=0x558efc2076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.511+0000 7f2d82ff5640 0 client.26479 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.512+0000 7f2da837c740 1 -- 172.21.15.28:0/2338151134 shutdown_connections 2024-05-09T08:41:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.516+0000 7f2da837c740 1 -- 172.21.15.28:0/2338151134 wait complete. 2024-05-09T08:41:15.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.521+0000 7f2da837c740 20 client.26479 trim_cache size 0 max 16384 2024-05-09T08:41:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.559+0000 7f2da837c740 1 Processor -- start 2024-05-09T08:41:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.563+0000 7f2da837c740 1 -- start start 2024-05-09T08:41:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.563+0000 7f2da837c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205450 0x558efc20fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.563+0000 7f2da837c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205d70 0x558efc21c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.563+0000 7f2da837c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc2286d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.564+0000 7f2da837c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c010400 con 0x558efc205450 2024-05-09T08:41:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.564+0000 7f2da837c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c012180 con 0x558efc205d70 2024-05-09T08:41:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.564+0000 7f2da837c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c010b90 con 0x558efc210630 2024-05-09T08:41:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.566+0000 7f2d9e7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205d70 0x558efc21c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.566+0000 7f2d9e7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205d70 0x558efc21c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:47252/0 (socket says 172.21.15.28:47252) 2024-05-09T08:41:15.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.566+0000 7f2d9e7fc640 1 -- 172.21.15.28:0/1840537703 learned_addr learned my addr 172.21.15.28:0/1840537703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.566+0000 7f2d9f7fe640 1 --2- 172.21.15.28:0/1840537703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc2286d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.568+0000 7f2d9e7fc640 1 -- 172.21.15.28:0/1840537703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 msgr2=0x558efc2286d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.568+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/1840537703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc2286d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.568+0000 7f2d9e7fc640 1 -- 172.21.15.28:0/1840537703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205450 msgr2=0x558efc20fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.568+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/1840537703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205450 0x558efc20fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.568+0000 7f2d9e7fc640 1 -- 172.21.15.28:0/1840537703 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558efc1fff00 con 0x558efc205d70 2024-05-09T08:41:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.571+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/1840537703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205d70 0x558efc21c180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2d98005a10 tx=0x7f2d98027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a44afb4fdc490931 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.575+0000 7f2d84ff9640 1 -- 172.21.15.28:0/1840537703 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98016040 con 0x558efc205d70 2024-05-09T08:41:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.575+0000 7f2d84ff9640 1 -- 172.21.15.28:0/1840537703 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d98003e00 con 0x558efc205d70 2024-05-09T08:41:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.576+0000 7f2da837c740 1 -- 172.21.15.28:0/1840537703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205d70 msgr2=0x558efc21c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.576+0000 7f2da837c740 1 --2- 172.21.15.28:0/1840537703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205d70 0x558efc21c180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2d98005a10 tx=0x7f2d98027000 comp rx=0 tx=0).stop 2024-05-09T08:41:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.577+0000 7f2da837c740 1 -- 172.21.15.28:0/1840537703 shutdown_connections 2024-05-09T08:41:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.577+0000 7f2da837c740 1 --2- 172.21.15.28:0/1840537703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc2286d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.577+0000 7f2da837c740 1 --2- 172.21.15.28:0/1840537703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205d70 0x558efc21c180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.577+0000 7f2da837c740 1 --2- 172.21.15.28:0/1840537703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205450 0x558efc20fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.577+0000 7f2da837c740 1 -- 172.21.15.28:0/1840537703 >> 172.21.15.28:0/1840537703 conn(0x558efc201960 msgr2=0x558efc208910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:15.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.580+0000 7f2da837c740 1 -- 172.21.15.28:0/1840537703 shutdown_connections 2024-05-09T08:41:15.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.582+0000 7f2da837c740 1 -- 172.21.15.28:0/1840537703 wait complete. 2024-05-09T08:41:15.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.582+0000 7f2da837c740 1 -- start start 2024-05-09T08:41:15.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.583+0000 7f2da837c740 1 Processor -- start 2024-05-09T08:41:15.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.583+0000 7f2da837c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc218690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.583+0000 7f2da837c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc218bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.583+0000 7f2da837c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc209930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.583+0000 7f2da837c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c011ab0 con 0x558efc205d70 2024-05-09T08:41:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.583+0000 7f2da837c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c011320 con 0x558efc205450 2024-05-09T08:41:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.583+0000 7f2da837c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c00e400 con 0x558efc210630 2024-05-09T08:41:15.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.583+0000 7f2da837c740 10 client.0 _ll_register_callbacks cb 0x7f2d940975e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.584+0000 7f2d9effd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc218690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.584+0000 7f2d9effd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc218690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:47260/0 (socket says 172.21.15.28:47260) 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.584+0000 7f2d9effd640 1 -- 172.21.15.28:0/2811903853 learned_addr learned my addr 172.21.15.28:0/2811903853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.584+0000 7f2d9f7fe640 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc209930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.584+0000 7f2d9effd640 1 -- 172.21.15.28:0/2811903853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 msgr2=0x558efc209930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.584+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc209930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.584+0000 7f2d9effd640 1 -- 172.21.15.28:0/2811903853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 msgr2=0x558efc218bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.584+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc218bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.584+0000 7f2d9effd640 1 -- 172.21.15.28:0/2811903853 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558efc209ff0 con 0x558efc205450 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.585+0000 7f2d9effd640 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc218690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2d940b42c0 tx=0x7f2d940a3fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4acc752a3f39bdd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.585+0000 7f2d82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:41:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.585+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2811903853 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94010d70 con 0x558efc205450 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.585+0000 7f2da837c740 1 -- 172.21.15.28:0/2811903853 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d98003e00 con 0x558efc205450 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.585+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2811903853 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f2d94015050 con 0x558efc205450 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.585+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2811903853 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f2d940d2b90 con 0x558efc205450 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.585+0000 7f2da837c740 10 client.26639 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.585+0000 7f2da837c740 1 -- 172.21.15.28:0/2811903853 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x558efc286910 con 0x558efc205450 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.587+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2811903853 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d98003e00 con 0x558efc205450 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.587+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2811903853 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f2d940bb0b0 con 0x558efc205450 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.588+0000 7f2da837c740 10 client.26639 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.588+0000 7f2da837c740 10 client.26639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.588+0000 7f2da837c740 20 client.26639 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.588+0000 7f2da837c740 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558efc286ea0 0x558efc2892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.588+0000 7f2da837c740 4 client.26639 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:15.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.588+0000 7f2da837c740 1 -- 172.21.15.28:0/2811903853 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558efc171dc0 con 0x558efc286ea0 2024-05-09T08:41:15.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.598+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558efc286ea0 0x558efc2892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.614+0000 7f2d82ff5640 10 client.26639 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.614+0000 7f2d9e7fc640 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558efc286ea0 0x558efc2892c0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=593fa152dbf37d23 server_cookie=f745b3ca252a67d0 in_seq=0 out_seq=0 2024-05-09T08:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.740+0000 7f2d82ff5640 1 -- 172.21.15.28:0/2811903853 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558efc171dc0 con 0x558efc286ea0 2024-05-09T08:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.740+0000 7f2d82ff5640 10 client.26639 handle_command_reply: tid=0 2024-05-09T08:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.740+0000 7f2da837c740 1 client.26639 shutdown 2024-05-09T08:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 -- 172.21.15.28:0/2811903853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 msgr2=0x558efc218690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc218690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2d940b42c0 tx=0x7f2d940a3fb0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 -- 172.21.15.28:0/2811903853 shutdown_connections 2024-05-09T08:41:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x558efc286ea0 0x558efc2892c0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558efc210630 0x558efc209930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558efc205d70 0x558efc218bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2d82ff5640 0 client.26639 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 --2- 172.21.15.28:0/2811903853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558efc205450 0x558efc218690 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 -- 172.21.15.28:0/2811903853 >> 172.21.15.28:0/2811903853 conn(0x558efc201960 msgr2=0x558efc200170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 -- 172.21.15.28:0/2811903853 shutdown_connections 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 1 -- 172.21.15.28:0/2811903853 wait complete. 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.741+0000 7f2da837c740 20 client.26639 trim_cache size 0 max 16384 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:41:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:41:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 -- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2d880a2ee0 msgr2=0x7f2d880a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2d880a2ee0 0x7f2d880a53a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2d98000f80 tx=0x7f2d980059a0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 -- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da0110020 msgr2=0x7f2da01a6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da0110020 0x7f2da01a6f10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400d950 tx=0x7f2d9400de10 comp rx=0 tx=0).stop 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 -- 172.21.15.28:0/2916792384 shutdown_connections 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da01ab7d0 0x7f2da01abc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2d880a2ee0 0x7f2d880a53a0 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2d98000f80 tx=0x7f2d980059a0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2da011b010 0x7f2da01a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 --2- 172.21.15.28:0/2916792384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2da0110020 0x7f2da01a6f10 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 -- 172.21.15.28:0/2916792384 >> 172.21.15.28:0/2916792384 conn(0x7f2da006c900 msgr2=0x7f2da010dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 -- 172.21.15.28:0/2916792384 shutdown_connections 2024-05-09T08:41:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:15.745+0000 7f2da5590640 1 -- 172.21.15.28:0/2916792384 wait complete. 2024-05-09T08:41:16.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:15 smithi165 ceph-mon[34540]: pgmap v479: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-05-09T08:41:16.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:15 smithi184 ceph-mon[34509]: pgmap v479: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-05-09T08:41:16.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:15 smithi028 ceph-mon[30119]: pgmap v479: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-05-09T08:41:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:17 smithi165 ceph-mon[34540]: pgmap v480: 129 pgs: 70 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 35 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-05-09T08:41:18.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:17 smithi184 ceph-mon[34509]: pgmap v480: 129 pgs: 70 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 35 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-05-09T08:41:18.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:17 smithi028 ceph-mon[30119]: pgmap v480: 129 pgs: 70 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 35 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-05-09T08:41:18.894 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:41:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:41:18.633+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:41:18.894 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:41:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:41:18.633+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:41:19.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:18 smithi028 ceph-mon[30119]: pgmap v481: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 123 op/s 2024-05-09T08:41:19.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:18 smithi165 ceph-mon[34540]: pgmap v481: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 123 op/s 2024-05-09T08:41:19.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:18 smithi184 ceph-mon[34509]: pgmap v481: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 123 op/s 2024-05-09T08:41:21.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:21 smithi165 ceph-mon[34540]: pgmap v482: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-05-09T08:41:21.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:21 smithi184 ceph-mon[34509]: pgmap v482: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-05-09T08:41:21.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:21 smithi028 ceph-mon[30119]: pgmap v482: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-05-09T08:41:23.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:23 smithi165 ceph-mon[34540]: pgmap v483: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-05-09T08:41:23.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:23 smithi184 ceph-mon[34509]: pgmap v483: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-05-09T08:41:23.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:23 smithi028 ceph-mon[30119]: pgmap v483: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 129 op/s 2024-05-09T08:41:25.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:25 smithi165 ceph-mon[34540]: pgmap v484: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-05-09T08:41:25.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:25 smithi184 ceph-mon[34509]: pgmap v484: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-05-09T08:41:25.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:25 smithi028 ceph-mon[30119]: pgmap v484: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-05-09T08:41:27.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:27 smithi165 ceph-mon[34540]: pgmap v485: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-05-09T08:41:27.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:27 smithi184 ceph-mon[34509]: pgmap v485: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-05-09T08:41:27.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:27 smithi028 ceph-mon[30119]: pgmap v485: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-05-09T08:41:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.906+0000 7f834509e640 1 -- 172.21.15.28:0/1612417371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8340101150 msgr2=0x7f834010d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.906+0000 7f834509e640 1 --2- 172.21.15.28:0/1612417371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8340101150 0x7f834010d610 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f8330009930 tx=0x7f833002a460 comp rx=0 tx=0).stop 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.910+0000 7f834509e640 1 -- 172.21.15.28:0/1612417371 shutdown_connections 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.910+0000 7f834509e640 1 --2- 172.21.15.28:0/1612417371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8340101150 0x7f834010d610 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.910+0000 7f834509e640 1 --2- 172.21.15.28:0/1612417371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8340100790 0x7f8340100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.910+0000 7f834509e640 1 --2- 172.21.15.28:0/1612417371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8340106750 0x7f8340106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.910+0000 7f834509e640 1 -- 172.21.15.28:0/1612417371 >> 172.21.15.28:0/1612417371 conn(0x7f8340077880 msgr2=0x7f8340077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.911+0000 7f834509e640 1 -- 172.21.15.28:0/1612417371 shutdown_connections 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.911+0000 7f834509e640 1 -- 172.21.15.28:0/1612417371 wait complete. 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.912+0000 7f834509e640 1 Processor -- start 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.912+0000 7f834509e640 1 -- start start 2024-05-09T08:41:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.912+0000 7f834509e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8340100790 0x7f8340072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.912+0000 7f834509e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8340101150 0x7f8340073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.912+0000 7f834509e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8340106750 0x7f834006c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.913+0000 7f834509e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f834006bc60 con 0x7f8340100790 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.913+0000 7f834509e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f834006bae0 con 0x7f8340101150 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.913+0000 7f834509e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f834006bde0 con 0x7f8340106750 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.913+0000 7f833e59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8340101150 0x7f8340073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.913+0000 7f833e59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8340101150 0x7f8340073480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:33100/0 (socket says 172.21.15.28:33100) 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.913+0000 7f833e59c640 1 -- 172.21.15.28:0/1226793422 learned_addr learned my addr 172.21.15.28:0/1226793422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.913+0000 7f833f59e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8340106750 0x7f834006c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.913+0000 7f833ed9d640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8340100790 0x7f8340072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.914+0000 7f833f59e640 1 -- 172.21.15.28:0/1226793422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8340101150 msgr2=0x7f8340073480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.914+0000 7f833f59e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8340101150 0x7f8340073480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.914+0000 7f833f59e640 1 -- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8340100790 msgr2=0x7f8340072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.914+0000 7f833f59e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8340100790 0x7f8340072f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.914+0000 7f833f59e640 1 -- 172.21.15.28:0/1226793422 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834006cdc0 con 0x7f8340106750 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.915+0000 7f833f59e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8340106750 0x7f834006c500 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8330009850 tx=0x7f8330003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.915+0000 7f8327fff640 1 -- 172.21.15.28:0/1226793422 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833003f070 con 0x7f8340106750 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.915+0000 7f834509e640 1 -- 172.21.15.28:0/1226793422 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f834006d050 con 0x7f8340106750 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.915+0000 7f834509e640 1 -- 172.21.15.28:0/1226793422 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8340070c40 con 0x7f8340106750 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.915+0000 7f8327fff640 1 -- 172.21.15.28:0/1226793422 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8330031c90 con 0x7f8340106750 2024-05-09T08:41:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.915+0000 7f8327fff640 1 -- 172.21.15.28:0/1226793422 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8330035030 con 0x7f8340106750 2024-05-09T08:41:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.920+0000 7f8347da9740 1 Processor -- start 2024-05-09T08:41:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.920+0000 7f8347da9740 1 -- start start 2024-05-09T08:41:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.920+0000 7f8347da9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 0x55f3fe3fc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.920+0000 7f8347da9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe4070f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.920+0000 7f8347da9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe409a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.920+0000 7f8347da9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3fe361340 con 0x55f3fe3fcd70 2024-05-09T08:41:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.920+0000 7f8347da9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f3fe309290 con 0x55f3fe407630 2024-05-09T08:41:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.920+0000 7f8347da9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f3fe35d300 con 0x55f3fe3fc450 2024-05-09T08:41:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.923+0000 7f8327fff640 1 -- 172.21.15.28:0/1226793422 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8330042070 con 0x7f8340106750 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.923+0000 7f833ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe4070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.923+0000 7f833ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe4070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36798/0 (socket says 172.21.15.28:36798) 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.923+0000 7f833ed9d640 1 -- 172.21.15.28:0/1001407290 learned_addr learned my addr 172.21.15.28:0/1001407290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.924+0000 7f8327fff640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f83280a2ef0 0x7f83280a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.924+0000 7f833e59c640 1 --2- 172.21.15.28:0/1001407290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 0x55f3fe3fc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.924+0000 7f833e59c640 1 --2- 172.21.15.28:0/1001407290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe409a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.924+0000 7f8327fff640 1 -- 172.21.15.28:0/1226793422 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f83300efb80 con 0x7f8340106750 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.925+0000 7f833ed9d640 1 -- 172.21.15.28:0/1001407290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 msgr2=0x55f3fe3fc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.925+0000 7f833ed9d640 1 --2- 172.21.15.28:0/1001407290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 0x55f3fe3fc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.925+0000 7f833ed9d640 1 -- 172.21.15.28:0/1001407290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 msgr2=0x55f3fe409a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.925+0000 7f833ed9d640 1 --2- 172.21.15.28:0/1001407290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe409a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.925+0000 7f833ed9d640 1 -- 172.21.15.28:0/1001407290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3fe32d8e0 con 0x55f3fe3fcd70 2024-05-09T08:41:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.925+0000 7f833ed9d640 1 --2- 172.21.15.28:0/1001407290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe4070f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f833800aec0 tx=0x7f833800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238cbc2b4afb1a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.927+0000 7f83257fa640 1 -- 172.21.15.28:0/1001407290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338010840 con 0x55f3fe3fcd70 2024-05-09T08:41:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.927+0000 7f83257fa640 1 -- 172.21.15.28:0/1001407290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8338018430 con 0x55f3fe3fcd70 2024-05-09T08:41:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.927+0000 7f83257fa640 1 -- 172.21.15.28:0/1001407290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338017620 con 0x55f3fe3fcd70 2024-05-09T08:41:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.929+0000 7f833f59e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f83280a2ef0 0x7f83280a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.931+0000 7f8347da9740 1 -- 172.21.15.28:0/1001407290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 msgr2=0x55f3fe4070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.931+0000 7f8347da9740 1 --2- 172.21.15.28:0/1001407290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe4070f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f833800aec0 tx=0x7f833800f790 comp rx=0 tx=0).stop 2024-05-09T08:41:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.934+0000 7f833f59e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f83280a2ef0 0x7f83280a53b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8330009930 tx=0x7f833003b460 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.940+0000 7f8347da9740 1 -- 172.21.15.28:0/1001407290 shutdown_connections 2024-05-09T08:41:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.940+0000 7f8347da9740 1 --2- 172.21.15.28:0/1001407290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe409a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.940+0000 7f8347da9740 1 --2- 172.21.15.28:0/1001407290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe4070f0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.940+0000 7f8347da9740 1 --2- 172.21.15.28:0/1001407290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 0x55f3fe3fc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.941+0000 7f8347da9740 1 -- 172.21.15.28:0/1001407290 >> 172.21.15.28:0/1001407290 conn(0x55f3fe3f8960 msgr2=0x55f3fe3f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.948+0000 7f8347da9740 1 -- 172.21.15.28:0/1001407290 shutdown_connections 2024-05-09T08:41:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.951+0000 7f8347da9740 1 -- 172.21.15.28:0/1001407290 wait complete. 2024-05-09T08:41:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.951+0000 7f8347da9740 1 -- start start 2024-05-09T08:41:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.951+0000 7f8347da9740 1 Processor -- start 2024-05-09T08:41:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.953+0000 7f8347da9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 0x55f3fe404570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.953+0000 7f8347da9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe404ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.953+0000 7f8347da9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe404ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.953+0000 7f8347da9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3fe40a260 con 0x55f3fe3fcd70 2024-05-09T08:41:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.953+0000 7f8347da9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f3fe31c790 con 0x55f3fe407630 2024-05-09T08:41:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.953+0000 7f8347da9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f3fe40a3e0 con 0x55f3fe3fc450 2024-05-09T08:41:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.953+0000 7f8347da9740 10 client.0 _ll_register_callbacks cb 0x55f3fe357a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:28.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.954+0000 7f833e59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe404ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.954+0000 7f833e59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 0x55f3fe404570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.954+0000 7f833e59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 0x55f3fe404570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:39926/0 (socket says 172.21.15.28:39926) 2024-05-09T08:41:28.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.954+0000 7f833e59c640 1 -- 172.21.15.28:0/495807736 learned_addr learned my addr 172.21.15.28:0/495807736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:28.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.954+0000 7f833ed9d640 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe404ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.954+0000 7f833ed9d640 1 -- 172.21.15.28:0/495807736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 msgr2=0x55f3fe404570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:28.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.954+0000 7f833ed9d640 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 0x55f3fe404570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.954+0000 7f833ed9d640 1 -- 172.21.15.28:0/495807736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 msgr2=0x55f3fe404ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:28.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.954+0000 7f833ed9d640 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe404ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.955+0000 7f833ed9d640 1 -- 172.21.15.28:0/495807736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3fe075210 con 0x55f3fe3fcd70 2024-05-09T08:41:28.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.955+0000 7f833e59c640 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe404ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:28.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.955+0000 7f833ed9d640 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe404ab0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f833800aec0 tx=0x7f8338004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7529226dee6091 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:28.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.955+0000 7f8322ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:41:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.956+0000 7f8322ff5640 1 -- 172.21.15.28:0/495807736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338026070 con 0x55f3fe3fcd70 2024-05-09T08:41:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.956+0000 7f8347da9740 1 -- 172.21.15.28:0/495807736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8338018430 con 0x55f3fe3fcd70 2024-05-09T08:41:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.956+0000 7f8322ff5640 1 -- 172.21.15.28:0/495807736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f8338003b70 con 0x55f3fe3fcd70 2024-05-09T08:41:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.956+0000 7f8322ff5640 1 -- 172.21.15.28:0/495807736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f8338031050 con 0x55f3fe3fcd70 2024-05-09T08:41:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.957+0000 7f8347da9740 10 client.17178 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.957+0000 7f8347da9740 1 -- 172.21.15.28:0/495807736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55f3fe141620 con 0x55f3fe3fcd70 2024-05-09T08:41:28.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.958+0000 7f8322ff5640 1 -- 172.21.15.28:0/495807736 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8338018430 con 0x55f3fe3fcd70 2024-05-09T08:41:28.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.958+0000 7f8322ff5640 1 -- 172.21.15.28:0/495807736 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f83380050e0 con 0x55f3fe3fcd70 2024-05-09T08:41:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.960+0000 7f8347da9740 10 client.17178 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.960+0000 7f8347da9740 10 client.17178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.960+0000 7f8347da9740 20 client.17178 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.960+0000 7f8347da9740 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f3fe417770 0x55f3fe419bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.960+0000 7f8347da9740 4 client.17178 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.960+0000 7f8347da9740 1 -- 172.21.15.28:0/495807736 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3fe36ea70 con 0x55f3fe417770 2024-05-09T08:41:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.961+0000 7f833e59c640 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f3fe417770 0x55f3fe419bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.964+0000 7f833e59c640 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f3fe417770 0x55f3fe419bd0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fdb8415ca0c3529 server_cookie=336c45bbc4fc1464 in_seq=0 out_seq=0 2024-05-09T08:41:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.964+0000 7f8322ff5640 10 client.17178 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.966+0000 7f8322ff5640 1 -- 172.21.15.28:0/495807736 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3fe36ea70 con 0x55f3fe417770 2024-05-09T08:41:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.966+0000 7f8322ff5640 10 client.17178 handle_command_reply: tid=0 2024-05-09T08:41:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.966+0000 7f8347da9740 1 client.17178 shutdown 2024-05-09T08:41:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.974+0000 7f8347da9740 1 -- 172.21.15.28:0/495807736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 msgr2=0x55f3fe404ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.974+0000 7f8347da9740 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe404ab0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f833800aec0 tx=0x7f8338004c70 comp rx=0 tx=0).stop 2024-05-09T08:41:28.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.978+0000 7f8347da9740 1 -- 172.21.15.28:0/495807736 shutdown_connections 2024-05-09T08:41:28.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.978+0000 7f8347da9740 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f3fe417770 0x55f3fe419bd0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.978+0000 7f8347da9740 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe404ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.978+0000 7f8347da9740 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fcd70 0x55f3fe404ab0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.978+0000 7f8347da9740 1 --2- 172.21.15.28:0/495807736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fc450 0x55f3fe404570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:28.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.978+0000 7f8347da9740 1 -- 172.21.15.28:0/495807736 >> 172.21.15.28:0/495807736 conn(0x55f3fe3f8960 msgr2=0x55f3fe3fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:28.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.978+0000 7f8322ff5640 0 client.17178 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:28.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.979+0000 7f8347da9740 1 -- 172.21.15.28:0/495807736 shutdown_connections 2024-05-09T08:41:28.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.980+0000 7f8347da9740 1 -- 172.21.15.28:0/495807736 wait complete. 2024-05-09T08:41:28.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:28.980+0000 7f8347da9740 20 client.17178 trim_cache size 0 max 16384 2024-05-09T08:41:29.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.011+0000 7f8347da9740 1 Processor -- start 2024-05-09T08:41:29.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.011+0000 7f8347da9740 1 -- start start 2024-05-09T08:41:29.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.012+0000 7f8347da9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe406ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.012+0000 7f8347da9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe3fcd70 0x55f3fe413180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.012+0000 7f8347da9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe407630 0x55f3fe3fee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.012+0000 7f8347da9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f832c010400 con 0x55f3fe3fc450 2024-05-09T08:41:29.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.012+0000 7f8347da9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f832c012180 con 0x55f3fe3fcd70 2024-05-09T08:41:29.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.012+0000 7f8347da9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f832c010b90 con 0x55f3fe407630 2024-05-09T08:41:29.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.017+0000 7f833e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe406ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.017+0000 7f833e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe406ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36808/0 (socket says 172.21.15.28:36808) 2024-05-09T08:41:29.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.017+0000 7f833e59c640 1 -- 172.21.15.28:0/658465903 learned_addr learned my addr 172.21.15.28:0/658465903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:29.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.017+0000 7f833e59c640 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe407630 0x55f3fe3fee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.018+0000 7f833ed9d640 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe3fcd70 0x55f3fe413180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.021+0000 7f833e59c640 1 -- 172.21.15.28:0/658465903 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe407630 msgr2=0x55f3fe3fee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.021+0000 7f833e59c640 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe407630 0x55f3fe3fee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.021+0000 7f833e59c640 1 -- 172.21.15.28:0/658465903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe3fcd70 msgr2=0x55f3fe413180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.021+0000 7f833e59c640 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe3fcd70 0x55f3fe413180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.021+0000 7f833e59c640 1 -- 172.21.15.28:0/658465903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3fe3f6f00 con 0x55f3fe3fc450 2024-05-09T08:41:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.021+0000 7f833ed9d640 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe3fcd70 0x55f3fe413180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.024+0000 7f833e59c640 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe406ca0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8334004130 tx=0x7f8334004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9b5f1966943aed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.030+0000 7f8324ff9640 1 -- 172.21.15.28:0/658465903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334022390 con 0x55f3fe3fc450 2024-05-09T08:41:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.030+0000 7f8324ff9640 1 -- 172.21.15.28:0/658465903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83340123f0 con 0x55f3fe3fc450 2024-05-09T08:41:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.030+0000 7f8324ff9640 1 -- 172.21.15.28:0/658465903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83340113e0 con 0x55f3fe3fc450 2024-05-09T08:41:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.031+0000 7f8347da9740 1 -- 172.21.15.28:0/658465903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 msgr2=0x55f3fe406ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.031+0000 7f8347da9740 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe406ca0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8334004130 tx=0x7f8334004040 comp rx=0 tx=0).stop 2024-05-09T08:41:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.034+0000 7f8347da9740 1 -- 172.21.15.28:0/658465903 shutdown_connections 2024-05-09T08:41:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.034+0000 7f8347da9740 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe407630 0x55f3fe3fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.034+0000 7f8347da9740 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe3fcd70 0x55f3fe413180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.034+0000 7f8347da9740 1 --2- 172.21.15.28:0/658465903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe406ca0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.034+0000 7f8347da9740 1 -- 172.21.15.28:0/658465903 >> 172.21.15.28:0/658465903 conn(0x55f3fe3f8960 msgr2=0x55f3fe402e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.039+0000 7f8347da9740 1 -- 172.21.15.28:0/658465903 shutdown_connections 2024-05-09T08:41:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.043+0000 7f8347da9740 1 -- 172.21.15.28:0/658465903 wait complete. 2024-05-09T08:41:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.043+0000 7f8347da9740 1 -- start start 2024-05-09T08:41:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.043+0000 7f8347da9740 1 Processor -- start 2024-05-09T08:41:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.047+0000 7f8347da9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe40fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.047+0000 7f8347da9740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fcd70 0x55f3fe401880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.047+0000 7f8347da9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe401e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.047+0000 7f8347da9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f832c011ab0 con 0x55f3fe3fc450 2024-05-09T08:41:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.047+0000 7f8347da9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f832c011320 con 0x55f3fe407630 2024-05-09T08:41:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.047+0000 7f8347da9740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f832c00e400 con 0x55f3fe3fcd70 2024-05-09T08:41:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.047+0000 7f8347da9740 10 client.0 _ll_register_callbacks cb 0x7f833800ad40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:29.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.049+0000 7f833e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe40fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.049+0000 7f833e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe40fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36812/0 (socket says 172.21.15.28:36812) 2024-05-09T08:41:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.049+0000 7f833e59c640 1 -- 172.21.15.28:0/1028859323 learned_addr learned my addr 172.21.15.28:0/1028859323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.049+0000 7f833e59c640 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe401e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.050+0000 7f833e59c640 1 -- 172.21.15.28:0/1028859323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fcd70 msgr2=0x55f3fe401880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:41:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.050+0000 7f833e59c640 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fcd70 0x55f3fe401880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.050+0000 7f833e59c640 1 -- 172.21.15.28:0/1028859323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 msgr2=0x55f3fe40fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.050+0000 7f833e59c640 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe40fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.050+0000 7f833e59c640 1 -- 172.21.15.28:0/1028859323 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3fe400800 con 0x55f3fe407630 2024-05-09T08:41:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.050+0000 7f833e59c640 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe401e40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8334012690 tx=0x7f8334004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e9464c08675ace0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.050+0000 7f8322ff5640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:41:29.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.053+0000 7f8322ff5640 1 -- 172.21.15.28:0/1028859323 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83340128f0 con 0x55f3fe407630 2024-05-09T08:41:29.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.054+0000 7f8347da9740 1 -- 172.21.15.28:0/1028859323 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83340123f0 con 0x55f3fe407630 2024-05-09T08:41:29.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.054+0000 7f8322ff5640 1 -- 172.21.15.28:0/1028859323 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f8334012a90 con 0x55f3fe407630 2024-05-09T08:41:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.055+0000 7f8322ff5640 1 -- 172.21.15.28:0/1028859323 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f8334022390 con 0x55f3fe407630 2024-05-09T08:41:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.055+0000 7f8347da9740 10 client.26654 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.055+0000 7f8347da9740 1 -- 172.21.15.28:0/1028859323 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55f3fe47ced0 con 0x55f3fe407630 2024-05-09T08:41:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.062+0000 7f8322ff5640 1 -- 172.21.15.28:0/1028859323 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83340123f0 con 0x55f3fe407630 2024-05-09T08:41:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.063+0000 7f8322ff5640 1 -- 172.21.15.28:0/1028859323 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f8334022ce0 con 0x55f3fe407630 2024-05-09T08:41:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.063+0000 7f8347da9740 10 client.26654 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.063+0000 7f8347da9740 10 client.26654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.063+0000 7f8347da9740 20 client.26654 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.063+0000 7f8347da9740 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f3fe47d590 0x55f3fe47f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.063+0000 7f8347da9740 4 client.26654 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.063+0000 7f8347da9740 1 -- 172.21.15.28:0/1028859323 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f3fe368dc0 con 0x55f3fe47d590 2024-05-09T08:41:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.068+0000 7f833ed9d640 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f3fe47d590 0x55f3fe47f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.079+0000 7f833ed9d640 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f3fe47d590 0x55f3fe47f9b0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b12caf6314b8ea6 server_cookie=769817bc1c1dd23c in_seq=0 out_seq=0 2024-05-09T08:41:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.080+0000 7f8322ff5640 10 client.26654 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.206+0000 7f8322ff5640 1 -- 172.21.15.28:0/1028859323 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f3fe368dc0 con 0x55f3fe47d590 2024-05-09T08:41:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.206+0000 7f8322ff5640 10 client.26654 handle_command_reply: tid=0 2024-05-09T08:41:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 client.26654 shutdown 2024-05-09T08:41:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 -- 172.21.15.28:0/1028859323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 msgr2=0x55f3fe401e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe401e40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8334012690 tx=0x7f8334004730 comp rx=0 tx=0).stop 2024-05-09T08:41:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 -- 172.21.15.28:0/1028859323 shutdown_connections 2024-05-09T08:41:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f3fe47d590 0x55f3fe47f9b0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f3fe407630 0x55f3fe401e40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f3fe3fcd70 0x55f3fe401880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 --2- 172.21.15.28:0/1028859323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fe3fc450 0x55f3fe40fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 -- 172.21.15.28:0/1028859323 >> 172.21.15.28:0/1028859323 conn(0x55f3fe3f8960 msgr2=0x55f3fe3f7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8322ff5640 0 client.26654 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 -- 172.21.15.28:0/1028859323 shutdown_connections 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 1 -- 172.21.15.28:0/1028859323 wait complete. 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.207+0000 7f8347da9740 20 client.26654 trim_cache size 0 max 16384 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:41:29.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:41:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 -- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f83280a2ef0 msgr2=0x7f83280a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f83280a2ef0 0x7f83280a53b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8330009930 tx=0x7f833003b460 comp rx=0 tx=0).stop 2024-05-09T08:41:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 -- 172.21.15.28:0/1226793422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8340106750 msgr2=0x7f834006c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8340106750 0x7f834006c500 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8330009850 tx=0x7f8330003b90 comp rx=0 tx=0).stop 2024-05-09T08:41:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 -- 172.21.15.28:0/1226793422 shutdown_connections 2024-05-09T08:41:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f83280a2ef0 0x7f83280a53b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8340106750 0x7f834006c500 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8340101150 0x7f8340073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 --2- 172.21.15.28:0/1226793422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8340100790 0x7f8340072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 -- 172.21.15.28:0/1226793422 >> 172.21.15.28:0/1226793422 conn(0x7f8340077880 msgr2=0x7f83401033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 -- 172.21.15.28:0/1226793422 shutdown_connections 2024-05-09T08:41:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:29.208+0000 7f834509e640 1 -- 172.21.15.28:0/1226793422 wait complete. 2024-05-09T08:41:29.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:29 smithi165 ceph-mon[34540]: pgmap v486: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-05-09T08:41:29.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:29 smithi184 ceph-mon[34509]: pgmap v486: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-05-09T08:41:29.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:29 smithi028 ceph-mon[30119]: pgmap v486: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-05-09T08:41:31.733 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:31 smithi028 ceph-mon[30119]: pgmap v487: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 771 KiB/s wr, 137 op/s 2024-05-09T08:41:31.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:31 smithi165 ceph-mon[34540]: pgmap v487: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 771 KiB/s wr, 137 op/s 2024-05-09T08:41:31.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:31 smithi184 ceph-mon[34509]: pgmap v487: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 771 KiB/s wr, 137 op/s 2024-05-09T08:41:33.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:33 smithi165 ceph-mon[34540]: pgmap v488: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 906 KiB/s wr, 129 op/s 2024-05-09T08:41:33.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:33 smithi184 ceph-mon[34509]: pgmap v488: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 906 KiB/s wr, 129 op/s 2024-05-09T08:41:33.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:33 smithi028 ceph-mon[30119]: pgmap v488: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 906 KiB/s wr, 129 op/s 2024-05-09T08:41:35.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:35 smithi165 ceph-mon[34540]: pgmap v489: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 650 KiB/s wr, 141 op/s 2024-05-09T08:41:35.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:35 smithi184 ceph-mon[34509]: pgmap v489: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 650 KiB/s wr, 141 op/s 2024-05-09T08:41:35.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:35 smithi028 ceph-mon[30119]: pgmap v489: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 650 KiB/s wr, 141 op/s 2024-05-09T08:41:37.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:37 smithi165 ceph-mon[34540]: pgmap v490: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 433 KiB/s wr, 136 op/s 2024-05-09T08:41:37.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:37 smithi184 ceph-mon[34509]: pgmap v490: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 433 KiB/s wr, 136 op/s 2024-05-09T08:41:37.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:37 smithi028 ceph-mon[30119]: pgmap v490: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 433 KiB/s wr, 136 op/s 2024-05-09T08:41:39.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:39 smithi165 ceph-mon[34540]: pgmap v491: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 888 KiB/s wr, 140 op/s 2024-05-09T08:41:39.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:39 smithi184 ceph-mon[34509]: pgmap v491: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 888 KiB/s wr, 140 op/s 2024-05-09T08:41:39.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:39 smithi028 ceph-mon[30119]: pgmap v491: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 888 KiB/s wr, 140 op/s 2024-05-09T08:41:41.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:41 smithi165 ceph-mon[34540]: pgmap v492: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 896 KiB/s wr, 145 op/s 2024-05-09T08:41:41.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:41 smithi184 ceph-mon[34509]: pgmap v492: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 896 KiB/s wr, 145 op/s 2024-05-09T08:41:41.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:41 smithi028 ceph-mon[30119]: pgmap v492: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 896 KiB/s wr, 145 op/s 2024-05-09T08:41:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.379+0000 7f62cff16640 1 -- 172.21.15.28:0/4151172283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8110400 msgr2=0x7f62c8108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.379+0000 7f62cff16640 1 --2- 172.21.15.28:0/4151172283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8110400 0x7f62c8108d70 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009930 tx=0x7f62bc02a4f0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.383+0000 7f62cff16640 1 -- 172.21.15.28:0/4151172283 shutdown_connections 2024-05-09T08:41:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.383+0000 7f62cff16640 1 --2- 172.21.15.28:0/4151172283 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c811b010 0x7f62c811d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.383+0000 7f62cff16640 1 --2- 172.21.15.28:0/4151172283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8110400 0x7f62c8108d70 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.383+0000 7f62cff16640 1 --2- 172.21.15.28:0/4151172283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62c8110020 0x7f62c8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.383+0000 7f62cff16640 1 -- 172.21.15.28:0/4151172283 >> 172.21.15.28:0/4151172283 conn(0x7f62c806c900 msgr2=0x7f62c806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.391+0000 7f62cff16640 1 -- 172.21.15.28:0/4151172283 shutdown_connections 2024-05-09T08:41:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.393+0000 7f62cff16640 1 -- 172.21.15.28:0/4151172283 wait complete. 2024-05-09T08:41:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.394+0000 7f62cff16640 1 Processor -- start 2024-05-09T08:41:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.395+0000 7f62cff16640 1 -- start start 2024-05-09T08:41:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.396+0000 7f62cff16640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62c8110020 0x7f62c81a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.396+0000 7f62cff16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c811b010 0x7f62c81a7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.396+0000 7f62cff16640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c81ab880 0x7f62c81abd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.396+0000 7f62cdcb2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62c8110020 0x7f62c81a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.396+0000 7f62cdcb2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62c8110020 0x7f62c81a6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:41652/0 (socket says 172.21.15.28:41652) 2024-05-09T08:41:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.396+0000 7f62cdcb2640 1 -- 172.21.15.28:0/630547366 learned_addr learned my addr 172.21.15.28:0/630547366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.396+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62c8120b00 con 0x7f62c811b010 2024-05-09T08:41:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.396+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f62c8120980 con 0x7f62c8110020 2024-05-09T08:41:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.397+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62c8120c80 con 0x7f62c81ab880 2024-05-09T08:41:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.398+0000 7f62ce4b3640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c81ab880 0x7f62c81abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.399+0000 7f62cdcb2640 1 -- 172.21.15.28:0/630547366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c81ab880 msgr2=0x7f62c81abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.399+0000 7f62cdcb2640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c81ab880 0x7f62c81abd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.399+0000 7f62cdcb2640 1 -- 172.21.15.28:0/630547366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c811b010 msgr2=0x7f62c81a7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.399+0000 7f62cdcb2640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c811b010 0x7f62c81a7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.399+0000 7f62cdcb2640 1 -- 172.21.15.28:0/630547366 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c81ac370 con 0x7f62c8110020 2024-05-09T08:41:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.402+0000 7f62cdcb2640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62c8110020 0x7f62c81a6fc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62c400b450 tx=0x7f62c400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.403+0000 7f62baffd640 1 -- 172.21.15.28:0/630547366 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4015020 con 0x7f62c8110020 2024-05-09T08:41:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.403+0000 7f62baffd640 1 -- 172.21.15.28:0/630547366 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c4011d70 con 0x7f62c8110020 2024-05-09T08:41:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.403+0000 7f62baffd640 1 -- 172.21.15.28:0/630547366 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c400f340 con 0x7f62c8110020 2024-05-09T08:41:42.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.403+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c81ac660 con 0x7f62c8110020 2024-05-09T08:41:42.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.403+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c81b3e00 con 0x7f62c8110020 2024-05-09T08:41:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.408+0000 7f62d2c1c740 1 Processor -- start 2024-05-09T08:41:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.412+0000 7f62baffd640 1 -- 172.21.15.28:0/630547366 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f62c4018020 con 0x7f62c8110020 2024-05-09T08:41:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.412+0000 7f62d2c1c740 1 -- start start 2024-05-09T08:41:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.412+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e67470830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.412+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e67470d70 0x557e6747b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.412+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 0x557e6747da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.412+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557e673d5340 con 0x557e67470450 2024-05-09T08:41:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.412+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557e6737d290 con 0x557e6747b630 2024-05-09T08:41:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.412+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557e673d1300 con 0x557e67470d70 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.413+0000 7f62ce4b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e67470830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.413+0000 7f62ce4b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e67470830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39276/0 (socket says 172.21.15.28:39276) 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.413+0000 7f62ce4b3640 1 -- 172.21.15.28:0/1809700238 learned_addr learned my addr 172.21.15.28:0/1809700238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.414+0000 7f62ce4b3640 1 -- 172.21.15.28:0/1809700238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e67470d70 msgr2=0x557e6747b0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.414+0000 7f62baffd640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62b40a2fd0 0x7f62b40a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.414+0000 7f62baffd640 1 -- 172.21.15.28:0/630547366 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f62c40cf360 con 0x7f62c8110020 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.414+0000 7f62ce4b3640 1 --2- 172.21.15.28:0/1809700238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e67470d70 0x557e6747b0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.414+0000 7f62ce4b3640 1 -- 172.21.15.28:0/1809700238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 msgr2=0x557e6747da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.414+0000 7f62ce4b3640 1 --2- 172.21.15.28:0/1809700238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 0x557e6747da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.414+0000 7f62ce4b3640 1 -- 172.21.15.28:0/1809700238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e673a18e0 con 0x557e67470450 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.415+0000 7f62ce4b3640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62b40a2fd0 0x7f62b40a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.415+0000 7f62ce4b3640 1 --2- 172.21.15.28:0/1809700238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e67470830 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f62c0008cd0 tx=0x7f62c000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5e1e30b2a83b4d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.416+0000 7f62b3fff640 1 -- 172.21.15.28:0/1809700238 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c001f070 con 0x557e67470450 2024-05-09T08:41:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.416+0000 7f62b3fff640 1 -- 172.21.15.28:0/1809700238 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c0019430 con 0x557e67470450 2024-05-09T08:41:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.416+0000 7f62b3fff640 1 -- 172.21.15.28:0/1809700238 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c000e610 con 0x557e67470450 2024-05-09T08:41:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.417+0000 7f62d2c1c740 1 -- 172.21.15.28:0/1809700238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 msgr2=0x557e67470830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.417+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/1809700238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e67470830 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f62c0008cd0 tx=0x7f62c000d710 comp rx=0 tx=0).stop 2024-05-09T08:41:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.417+0000 7f62d2c1c740 1 -- 172.21.15.28:0/1809700238 shutdown_connections 2024-05-09T08:41:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.417+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/1809700238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 0x557e6747da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.417+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/1809700238 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e67470d70 0x557e6747b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.417+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/1809700238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e67470830 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.417+0000 7f62d2c1c740 1 -- 172.21.15.28:0/1809700238 >> 172.21.15.28:0/1809700238 conn(0x557e6746c960 msgr2=0x557e6746cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.418+0000 7f62ce4b3640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62b40a2fd0 0x7f62b40a5490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f62c001c000 tx=0x7f62c000eda0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.424+0000 7f62d2c1c740 1 -- 172.21.15.28:0/1809700238 shutdown_connections 2024-05-09T08:41:42.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.428+0000 7f62d2c1c740 1 -- 172.21.15.28:0/1809700238 wait complete. 2024-05-09T08:41:42.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.429+0000 7f62d2c1c740 1 -- start start 2024-05-09T08:41:42.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.429+0000 7f62d2c1c740 1 Processor -- start 2024-05-09T08:41:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.430+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e674784a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.430+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e67470d70 0x557e674789e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.430+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 0x557e67478f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.430+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557e6747e1a0 con 0x557e67470450 2024-05-09T08:41:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.430+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557e67390790 con 0x557e6747b630 2024-05-09T08:41:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.430+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557e6747e320 con 0x557e67470d70 2024-05-09T08:41:42.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.430+0000 7f62d2c1c740 10 client.0 _ll_register_callbacks cb 0x557e673cba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.434+0000 7f62ce4b3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 0x557e67478f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.434+0000 7f62ce4b3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 0x557e67478f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:41662/0 (socket says 172.21.15.28:41662) 2024-05-09T08:41:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.434+0000 7f62ce4b3640 1 -- 172.21.15.28:0/2311419289 learned_addr learned my addr 172.21.15.28:0/2311419289 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.434+0000 7f62cd4b1640 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e67470d70 0x557e674789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.434+0000 7f62cdcb2640 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e674784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.435+0000 7f62cdcb2640 1 -- 172.21.15.28:0/2311419289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e67470d70 msgr2=0x557e674789e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:42.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.435+0000 7f62cdcb2640 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e67470d70 0x557e674789e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.435+0000 7f62cdcb2640 1 -- 172.21.15.28:0/2311419289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 msgr2=0x557e67478f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:42.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.435+0000 7f62cdcb2640 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 0x557e67478f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.435+0000 7f62cdcb2640 1 -- 172.21.15.28:0/2311419289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e670e9210 con 0x557e67470450 2024-05-09T08:41:42.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.435+0000 7f62b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:41:42.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.436+0000 7f62cdcb2640 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e674784a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f62c40cc650 tx=0x7f62c409bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3522645af54c4d1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:42.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.436+0000 7f62b17fa640 1 -- 172.21.15.28:0/2311419289 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c40b4060 con 0x557e67470450 2024-05-09T08:41:42.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.436+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2311419289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c0019430 con 0x557e67470450 2024-05-09T08:41:42.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.436+0000 7f62b17fa640 1 -- 172.21.15.28:0/2311419289 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f62c40103e0 con 0x557e67470450 2024-05-09T08:41:42.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.437+0000 7f62b17fa640 1 -- 172.21.15.28:0/2311419289 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f62c40b4060 con 0x557e67470450 2024-05-09T08:41:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.443+0000 7f62d2c1c740 10 client.17193 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.443+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2311419289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x557e671b5620 con 0x557e67470450 2024-05-09T08:41:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.444+0000 7f62b17fa640 1 -- 172.21.15.28:0/2311419289 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c0019430 con 0x557e67470450 2024-05-09T08:41:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.444+0000 7f62b17fa640 1 -- 172.21.15.28:0/2311419289 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f62c40b4200 con 0x557e67470450 2024-05-09T08:41:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.448+0000 7f62d2c1c740 10 client.17193 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.448+0000 7f62d2c1c740 10 client.17193 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:42.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.448+0000 7f62d2c1c740 20 client.17193 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.448+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557e6748b220 0x557e6748d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.448+0000 7f62d2c1c740 4 client.17193 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.448+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2311419289 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e673e2a70 con 0x557e6748b220 2024-05-09T08:41:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.449+0000 7f62cd4b1640 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557e6748b220 0x557e6748d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.451+0000 7f62cd4b1640 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557e6748b220 0x557e6748d6e0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69a15d4d1b6513da server_cookie=f9a9ddda17a4813e in_seq=0 out_seq=0 2024-05-09T08:41:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.451+0000 7f62b17fa640 10 client.17193 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.452+0000 7f62b17fa640 1 -- 172.21.15.28:0/2311419289 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557e673e2a70 con 0x557e6748b220 2024-05-09T08:41:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.452+0000 7f62b17fa640 10 client.17193 handle_command_reply: tid=0 2024-05-09T08:41:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.453+0000 7f62d2c1c740 1 client.17193 shutdown 2024-05-09T08:41:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.454+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2311419289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 msgr2=0x557e674784a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.454+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e674784a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f62c40cc650 tx=0x7f62c409bc60 comp rx=0 tx=0).stop 2024-05-09T08:41:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.463+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2311419289 shutdown_connections 2024-05-09T08:41:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.463+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557e6748b220 0x557e6748d6e0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.463+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e6747b630 0x557e67478f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.463+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e67470d70 0x557e674789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.463+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2311419289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e674784a0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.463+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2311419289 >> 172.21.15.28:0/2311419289 conn(0x557e6746c960 msgr2=0x557e67472690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.463+0000 7f62b17fa640 0 client.17193 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:42.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.468+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2311419289 shutdown_connections 2024-05-09T08:41:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.473+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2311419289 wait complete. 2024-05-09T08:41:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.473+0000 7f62d2c1c740 20 client.17193 trim_cache size 0 max 16384 2024-05-09T08:41:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.502+0000 7f62d2c1c740 1 Processor -- start 2024-05-09T08:41:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.504+0000 7f62d2c1c740 1 -- start start 2024-05-09T08:41:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.504+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e6747abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.504+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470d70 0x557e6746b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.504+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e6747b630 0x557e67494060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.504+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81d7a80 con 0x557e67470450 2024-05-09T08:41:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.504+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81d9800 con 0x557e67470d70 2024-05-09T08:41:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.504+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81d8210 con 0x557e6747b630 2024-05-09T08:41:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.505+0000 7f62cd4b1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470d70 0x557e6746b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.505+0000 7f62cd4b1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470d70 0x557e6746b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:41676/0 (socket says 172.21.15.28:41676) 2024-05-09T08:41:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.505+0000 7f62cd4b1640 1 -- 172.21.15.28:0/2847227578 learned_addr learned my addr 172.21.15.28:0/2847227578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.506+0000 7f62cd4b1640 1 -- 172.21.15.28:0/2847227578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e6747b630 msgr2=0x557e67494060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:41:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.506+0000 7f62cd4b1640 1 --2- 172.21.15.28:0/2847227578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e6747b630 0x557e67494060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.506+0000 7f62cd4b1640 1 -- 172.21.15.28:0/2847227578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 msgr2=0x557e6747abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:42.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.506+0000 7f62cd4b1640 1 --2- 172.21.15.28:0/2847227578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e6747abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.506+0000 7f62cd4b1640 1 -- 172.21.15.28:0/2847227578 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e67470ba0 con 0x557e67470d70 2024-05-09T08:41:42.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.506+0000 7f62cd4b1640 1 --2- 172.21.15.28:0/2847227578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470d70 0x557e6746b880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009850 tx=0x7f62bc009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8848d9a8acabecd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:42.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.507+0000 7f62b37fe640 1 -- 172.21.15.28:0/2847227578 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc035040 con 0x557e67470d70 2024-05-09T08:41:42.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.507+0000 7f62b37fe640 1 -- 172.21.15.28:0/2847227578 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62bc0053e0 con 0x557e67470d70 2024-05-09T08:41:42.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.508+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2847227578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470d70 msgr2=0x557e6746b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:42.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.508+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2847227578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470d70 0x557e6746b880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009850 tx=0x7f62bc009680 comp rx=0 tx=0).stop 2024-05-09T08:41:42.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.508+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2847227578 shutdown_connections 2024-05-09T08:41:42.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.509+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2847227578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e6747b630 0x557e67494060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.509+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2847227578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470d70 0x557e6746b880 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.509+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2847227578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470450 0x557e6747abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.509+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2847227578 >> 172.21.15.28:0/2847227578 conn(0x557e6746c960 msgr2=0x557e6746b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:42.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.513+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2847227578 shutdown_connections 2024-05-09T08:41:42.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.516+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2847227578 wait complete. 2024-05-09T08:41:42.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.516+0000 7f62d2c1c740 1 -- start start 2024-05-09T08:41:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.516+0000 7f62d2c1c740 1 Processor -- start 2024-05-09T08:41:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.518+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470450 0x557e67475570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.518+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470d70 0x557e67475ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.519+0000 7f62d2c1c740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e6747b630 0x557e67481b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.519+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81d9130 con 0x557e67470d70 2024-05-09T08:41:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.519+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81d89a0 con 0x557e67470450 2024-05-09T08:41:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.519+0000 7f62d2c1c740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62c8118a00 con 0x557e6747b630 2024-05-09T08:41:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.519+0000 7f62d2c1c740 10 client.0 _ll_register_callbacks cb 0x557e6747b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.519+0000 7f62cd4b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470d70 0x557e67475ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.519+0000 7f62cd4b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470d70 0x557e67475ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39292/0 (socket says 172.21.15.28:39292) 2024-05-09T08:41:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.519+0000 7f62cd4b1640 1 -- 172.21.15.28:0/2417241000 learned_addr learned my addr 172.21.15.28:0/2417241000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.523+0000 7f62ce4b3640 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e6747b630 0x557e67481b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.523+0000 7f62cd4b1640 1 -- 172.21.15.28:0/2417241000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e6747b630 msgr2=0x557e67481b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.523+0000 7f62cd4b1640 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e6747b630 0x557e67481b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.523+0000 7f62cd4b1640 1 -- 172.21.15.28:0/2417241000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470450 msgr2=0x557e67475570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:41:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.523+0000 7f62cd4b1640 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470450 0x557e67475570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.523+0000 7f62cd4b1640 1 -- 172.21.15.28:0/2417241000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e674820c0 con 0x557e67470d70 2024-05-09T08:41:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.533+0000 7f62cd4b1640 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470d70 0x557e67475ab0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f62bc044000 tx=0x7f62bc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd2579857a3be2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.534+0000 7f62b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:41:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.538+0000 7f62b17fa640 1 -- 172.21.15.28:0/2417241000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc04e020 con 0x557e67470d70 2024-05-09T08:41:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.538+0000 7f62b17fa640 1 -- 172.21.15.28:0/2417241000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f62bc031c00 con 0x557e67470d70 2024-05-09T08:41:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.541+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2417241000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62bc0053e0 con 0x557e67470d70 2024-05-09T08:41:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.547+0000 7f62b17fa640 1 -- 172.21.15.28:0/2417241000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f62bc035040 con 0x557e67470d70 2024-05-09T08:41:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.547+0000 7f62d2c1c740 10 client.17196 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.547+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2417241000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x557e674f0fd0 con 0x557e67470d70 2024-05-09T08:41:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.550+0000 7f62b17fa640 1 -- 172.21.15.28:0/2417241000 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62bc0053e0 con 0x557e67470d70 2024-05-09T08:41:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.550+0000 7f62b17fa640 1 -- 172.21.15.28:0/2417241000 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f62bc0035f0 con 0x557e67470d70 2024-05-09T08:41:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.551+0000 7f62d2c1c740 10 client.17196 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.551+0000 7f62d2c1c740 10 client.17196 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.551+0000 7f62d2c1c740 20 client.17196 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.551+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557e674f1710 0x557e674f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.551+0000 7f62d2c1c740 4 client.17196 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.551+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2417241000 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557e673dcdc0 con 0x557e674f1710 2024-05-09T08:41:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.554+0000 7f62cdcb2640 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557e674f1710 0x557e674f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.559+0000 7f62cdcb2640 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557e674f1710 0x557e674f3b30 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fba0f75c1b78e9 server_cookie=697964d81bb362df in_seq=0 out_seq=0 2024-05-09T08:41:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.559+0000 7f62b17fa640 10 client.17196 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.637+0000 7f62b17fa640 1 -- 172.21.15.28:0/2417241000 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557e673dcdc0 con 0x557e674f1710 2024-05-09T08:41:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.637+0000 7f62b17fa640 10 client.17196 handle_command_reply: tid=0 2024-05-09T08:41:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.637+0000 7f62d2c1c740 1 client.17196 shutdown 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.637+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2417241000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470d70 msgr2=0x557e67475ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.637+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470d70 0x557e67475ab0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f62bc044000 tx=0x7f62bc005fd0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2417241000 shutdown_connections 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557e674f1710 0x557e674f3b30 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557e6747b630 0x557e67481b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e67470d70 0x557e67475ab0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62d2c1c740 1 --2- 172.21.15.28:0/2417241000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557e67470450 0x557e67475570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62b17fa640 0 client.17196 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2417241000 >> 172.21.15.28:0/2417241000 conn(0x557e6746c960 msgr2=0x557e67471e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2417241000 shutdown_connections 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62d2c1c740 1 -- 172.21.15.28:0/2417241000 wait complete. 2024-05-09T08:41:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.638+0000 7f62d2c1c740 20 client.17196 trim_cache size 0 max 16384 2024-05-09T08:41:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:41:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:41:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:41:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:41:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62b40a2fd0 msgr2=0x7f62b40a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62b40a2fd0 0x7f62b40a5490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f62c001c000 tx=0x7f62c000eda0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62c8110020 msgr2=0x7f62c81a6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62c8110020 0x7f62c81a6fc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62c400b450 tx=0x7f62c400b910 comp rx=0 tx=0).stop 2024-05-09T08:41:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 shutdown_connections 2024-05-09T08:41:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f62b40a2fd0 0x7f62b40a5490 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c81ab880 0x7f62c81abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c811b010 0x7f62c81a7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 --2- 172.21.15.28:0/630547366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62c8110020 0x7f62c81a6fc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 >> 172.21.15.28:0/630547366 conn(0x7f62c806c900 msgr2=0x7f62c810dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 shutdown_connections 2024-05-09T08:41:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:42.639+0000 7f62cff16640 1 -- 172.21.15.28:0/630547366 wait complete. 2024-05-09T08:41:43.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:43 smithi165 ceph-mon[34540]: pgmap v493: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 893 KiB/s wr, 142 op/s 2024-05-09T08:41:43.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:43 smithi184 ceph-mon[34509]: pgmap v493: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 893 KiB/s wr, 142 op/s 2024-05-09T08:41:43.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:43 smithi028 ceph-mon[30119]: pgmap v493: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 893 KiB/s wr, 142 op/s 2024-05-09T08:41:45.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:45 smithi165 ceph-mon[34540]: pgmap v494: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 733 KiB/s wr, 156 op/s 2024-05-09T08:41:45.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:45 smithi184 ceph-mon[34509]: pgmap v494: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 733 KiB/s wr, 156 op/s 2024-05-09T08:41:45.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:45 smithi028 ceph-mon[30119]: pgmap v494: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 733 KiB/s wr, 156 op/s 2024-05-09T08:41:47.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:47 smithi165 ceph-mon[34540]: pgmap v495: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 854 KiB/s wr, 145 op/s 2024-05-09T08:41:47.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:47 smithi184 ceph-mon[34509]: pgmap v495: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 854 KiB/s wr, 145 op/s 2024-05-09T08:41:47.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:47 smithi028 ceph-mon[30119]: pgmap v495: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 854 KiB/s wr, 145 op/s 2024-05-09T08:41:49.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:49 smithi165 ceph-mon[34540]: pgmap v496: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1014 KiB/s wr, 158 op/s 2024-05-09T08:41:49.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:49 smithi184 ceph-mon[34509]: pgmap v496: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1014 KiB/s wr, 158 op/s 2024-05-09T08:41:49.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:49 smithi028 ceph-mon[30119]: pgmap v496: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1014 KiB/s wr, 158 op/s 2024-05-09T08:41:51.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:51 smithi165 ceph-mon[34540]: pgmap v497: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 853 KiB/s wr, 152 op/s 2024-05-09T08:41:51.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:51 smithi184 ceph-mon[34509]: pgmap v497: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 853 KiB/s wr, 152 op/s 2024-05-09T08:41:51.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:51 smithi028 ceph-mon[30119]: pgmap v497: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 853 KiB/s wr, 152 op/s 2024-05-09T08:41:53.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:53 smithi165 ceph-mon[34540]: pgmap v498: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 848 KiB/s wr, 139 op/s 2024-05-09T08:41:53.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:53 smithi184 ceph-mon[34509]: pgmap v498: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 848 KiB/s wr, 139 op/s 2024-05-09T08:41:53.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:53 smithi028 ceph-mon[30119]: pgmap v498: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 848 KiB/s wr, 139 op/s 2024-05-09T08:41:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.797+0000 7f07a25d5640 1 -- 172.21.15.28:0/2875953592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1011d0 msgr2=0x7f079c105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:55.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.797+0000 7f07a25d5640 1 --2- 172.21.15.28:0/2875953592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1011d0 0x7f079c105690 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0790009900 tx=0x7f079002a540 comp rx=0 tx=0).stop 2024-05-09T08:41:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.799+0000 7f07a25d5640 1 -- 172.21.15.28:0/2875953592 shutdown_connections 2024-05-09T08:41:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.799+0000 7f07a25d5640 1 --2- 172.21.15.28:0/2875953592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c105bd0 0x7f079c106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.799+0000 7f07a25d5640 1 --2- 172.21.15.28:0/2875953592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1011d0 0x7f079c105690 secure :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0790009900 tx=0x7f079002a540 comp rx=0 tx=0).stop 2024-05-09T08:41:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.799+0000 7f07a25d5640 1 --2- 172.21.15.28:0/2875953592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f079c1008b0 0x7f079c100c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.799+0000 7f07a25d5640 1 -- 172.21.15.28:0/2875953592 >> 172.21.15.28:0/2875953592 conn(0x7f079c0778b0 msgr2=0x7f079c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.801+0000 7f07a25d5640 1 -- 172.21.15.28:0/2875953592 shutdown_connections 2024-05-09T08:41:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.803+0000 7f07a25d5640 1 -- 172.21.15.28:0/2875953592 wait complete. 2024-05-09T08:41:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.804+0000 7f07a25d5640 1 Processor -- start 2024-05-09T08:41:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.804+0000 7f07a25d5640 1 -- start start 2024-05-09T08:41:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.805+0000 7f07a25d5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f079c1008b0 0x7f079c072460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.805+0000 7f07a25d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c105bd0 0x7f079c0729a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.805+0000 7f07a25d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c109f60 0x7f079c10a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.805+0000 7f07a25d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f079c1182b0 con 0x7f079c105bd0 2024-05-09T08:41:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.805+0000 7f07a25d5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f079c118130 con 0x7f079c1008b0 2024-05-09T08:41:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.805+0000 7f07a25d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f079c118430 con 0x7f079c109f60 2024-05-09T08:41:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.807+0000 7f079bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f079c1008b0 0x7f079c072460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.808+0000 7f07a0b72640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c109f60 0x7f079c10a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.808+0000 7f079bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f079c1008b0 0x7f079c072460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:52572/0 (socket says 172.21.15.28:52572) 2024-05-09T08:41:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.808+0000 7f079bfff640 1 -- 172.21.15.28:0/150131411 learned_addr learned my addr 172.21.15.28:0/150131411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.809+0000 7f079b7fe640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c105bd0 0x7f079c0729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.810+0000 7f079bfff640 1 -- 172.21.15.28:0/150131411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c109f60 msgr2=0x7f079c10a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.810+0000 7f079bfff640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c109f60 0x7f079c10a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.810+0000 7f079bfff640 1 -- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c105bd0 msgr2=0x7f079c0729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.810+0000 7f079bfff640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c105bd0 0x7f079c0729a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.810+0000 7f079bfff640 1 -- 172.21.15.28:0/150131411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c10aa00 con 0x7f079c1008b0 2024-05-09T08:41:55.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.814+0000 7f079bfff640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f079c1008b0 0x7f079c072460 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f079400cce0 tx=0x7f079400a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.817+0000 7f07997fa640 1 -- 172.21.15.28:0/150131411 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794004b20 con 0x7f079c1008b0 2024-05-09T08:41:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.817+0000 7f07a25d5640 1 -- 172.21.15.28:0/150131411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f079c10acf0 con 0x7f079c1008b0 2024-05-09T08:41:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.817+0000 7f07a25d5640 1 -- 172.21.15.28:0/150131411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f079c1124f0 con 0x7f079c1008b0 2024-05-09T08:41:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.817+0000 7f07997fa640 1 -- 172.21.15.28:0/150131411 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0794004cc0 con 0x7f079c1008b0 2024-05-09T08:41:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.817+0000 7f07997fa640 1 -- 172.21.15.28:0/150131411 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079400f7d0 con 0x7f079c1008b0 2024-05-09T08:41:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.823+0000 7f07a53c4740 1 Processor -- start 2024-05-09T08:41:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.825+0000 7f07a53c4740 1 -- start start 2024-05-09T08:41:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.825+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a92996830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.825+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 0x562a929a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.825+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a929a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.825+0000 7f07a53c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a928fb340 con 0x562a92996450 2024-05-09T08:41:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.825+0000 7f07a53c4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x562a928a3290 con 0x562a929a1630 2024-05-09T08:41:55.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.825+0000 7f07a53c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562a928f7300 con 0x562a92996d70 2024-05-09T08:41:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.826+0000 7f079bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 0x562a929a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.827+0000 7f079bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 0x562a929a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:60840/0 (socket says 172.21.15.28:60840) 2024-05-09T08:41:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.827+0000 7f07a0b72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a92996830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.827+0000 7f07a0b72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a92996830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34146/0 (socket says 172.21.15.28:34146) 2024-05-09T08:41:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.827+0000 7f07a0b72640 1 -- 172.21.15.28:0/2684461265 learned_addr learned my addr 172.21.15.28:0/2684461265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:55.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.827+0000 7f07997fa640 1 -- 172.21.15.28:0/150131411 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f079400f970 con 0x7f079c1008b0 2024-05-09T08:41:55.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.828+0000 7f07a0b72640 1 -- 172.21.15.28:0/2684461265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 msgr2=0x562a929a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.828+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2684461265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 0x562a929a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.828+0000 7f07a0b72640 1 -- 172.21.15.28:0/2684461265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 msgr2=0x562a929a3a20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.828+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2684461265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a929a3a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.828+0000 7f07a0b72640 1 -- 172.21.15.28:0/2684461265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a928c78e0 con 0x562a92996450 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.828+0000 7f07997fa640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07840a2fd0 0x7f07840a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.828+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2684461265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a92996830 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f078c00b7a0 tx=0x7f078c00bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd72606eca95495 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.828+0000 7f07827fc640 1 -- 172.21.15.28:0/2684461265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c00d950 con 0x562a92996450 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.828+0000 7f07997fa640 1 -- 172.21.15.28:0/150131411 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f07940cfa90 con 0x7f079c1008b0 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.829+0000 7f07827fc640 1 -- 172.21.15.28:0/2684461265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f078c012420 con 0x562a92996450 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.829+0000 7f07827fc640 1 -- 172.21.15.28:0/2684461265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c014630 con 0x562a92996450 2024-05-09T08:41:55.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.829+0000 7f079bfff640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07840a2fd0 0x7f07840a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.830+0000 7f07a53c4740 1 -- 172.21.15.28:0/2684461265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 msgr2=0x562a92996830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:55.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.830+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2684461265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a92996830 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f078c00b7a0 tx=0x7f078c00bc60 comp rx=0 tx=0).stop 2024-05-09T08:41:55.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.830+0000 7f079bfff640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07840a2fd0 0x7f07840a5490 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x562a929979f0 tx=0x7f07940d34c0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.832+0000 7f07a53c4740 1 -- 172.21.15.28:0/2684461265 shutdown_connections 2024-05-09T08:41:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.832+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2684461265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a929a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.832+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2684461265 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 0x562a929a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.832+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2684461265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a92996830 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.832+0000 7f07a53c4740 1 -- 172.21.15.28:0/2684461265 >> 172.21.15.28:0/2684461265 conn(0x562a92992960 msgr2=0x562a92992d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.838+0000 7f07a53c4740 1 -- 172.21.15.28:0/2684461265 shutdown_connections 2024-05-09T08:41:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.840+0000 7f07a53c4740 1 -- 172.21.15.28:0/2684461265 wait complete. 2024-05-09T08:41:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.840+0000 7f07a53c4740 1 -- start start 2024-05-09T08:41:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.840+0000 7f07a53c4740 1 Processor -- start 2024-05-09T08:41:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a9299e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 0x562a9299ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a53c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a929a4260 con 0x562a92996450 2024-05-09T08:41:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a53c4740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x562a928b6790 con 0x562a929a1630 2024-05-09T08:41:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a53c4740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x562a929a43e0 con 0x562a92996d70 2024-05-09T08:41:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a53c4740 10 client.0 _ll_register_callbacks cb 0x562a928f1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a0b72640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a0b72640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:52598/0 (socket says 172.21.15.28:52598) 2024-05-09T08:41:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.841+0000 7f07a0b72640 1 -- 172.21.15.28:0/2649109000 learned_addr learned my addr 172.21.15.28:0/2649109000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.842+0000 7f079b7fe640 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 0x562a9299ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.844+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a9299e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.844+0000 7f07a0b72640 1 -- 172.21.15.28:0/2649109000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 msgr2=0x562a9299ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.844+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 0x562a9299ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.844+0000 7f07a0b72640 1 -- 172.21.15.28:0/2649109000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 msgr2=0x562a9299e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.844+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a9299e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.844+0000 7f07a0b72640 1 -- 172.21.15.28:0/2649109000 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a9260f210 con 0x562a929a1630 2024-05-09T08:41:55.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.846+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299ef80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f078c00b770 tx=0x7f078c012e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89b685af352d8a63 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:55.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.846+0000 7f077b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:41:55.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.847+0000 7f077b7fe640 1 -- 172.21.15.28:0/2649109000 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c027070 con 0x562a929a1630 2024-05-09T08:41:55.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.847+0000 7f077b7fe640 1 -- 172.21.15.28:0/2649109000 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f078c0129e0 con 0x562a929a1630 2024-05-09T08:41:55.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.848+0000 7f07a53c4740 1 -- 172.21.15.28:0/2649109000 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f078c012420 con 0x562a929a1630 2024-05-09T08:41:55.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:55 smithi165 ceph-mon[34540]: pgmap v499: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 1.1 MiB/s wr, 148 op/s 2024-05-09T08:41:55.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:55 smithi184 ceph-mon[34509]: pgmap v499: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 1.1 MiB/s wr, 148 op/s 2024-05-09T08:41:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.855+0000 7f077b7fe640 1 -- 172.21.15.28:0/2649109000 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f078c02c440 con 0x562a929a1630 2024-05-09T08:41:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.857+0000 7f07a53c4740 10 client.26669 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:55.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.857+0000 7f07a53c4740 1 -- 172.21.15.28:0/2649109000 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x562a926db620 con 0x562a929a1630 2024-05-09T08:41:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.861+0000 7f077b7fe640 1 -- 172.21.15.28:0/2649109000 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f078c012420 con 0x562a929a1630 2024-05-09T08:41:55.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.861+0000 7f077b7fe640 1 -- 172.21.15.28:0/2649109000 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f078c02a020 con 0x562a929a1630 2024-05-09T08:41:55.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.863+0000 7f07a53c4740 10 client.26669 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.863+0000 7f07a53c4740 10 client.26669 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.863+0000 7f07a53c4740 20 client.26669 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.863+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562a929b12e0 0x562a929b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.863+0000 7f07a53c4740 4 client.26669 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:55.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.863+0000 7f07a53c4740 1 -- 172.21.15.28:0/2649109000 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a92908a70 con 0x562a929b12e0 2024-05-09T08:41:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.874+0000 7f079b7fe640 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562a929b12e0 0x562a929b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.875+0000 7f079b7fe640 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562a929b12e0 0x562a929b3700 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d91dca89014415cc server_cookie=4f8d30d4bece331f in_seq=0 out_seq=0 2024-05-09T08:41:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.875+0000 7f077b7fe640 10 client.26669 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.879+0000 7f077b7fe640 1 -- 172.21.15.28:0/2649109000 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a92908a70 con 0x562a929b12e0 2024-05-09T08:41:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.879+0000 7f077b7fe640 10 client.26669 handle_command_reply: tid=0 2024-05-09T08:41:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.880+0000 7f07a53c4740 1 client.26669 shutdown 2024-05-09T08:41:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.881+0000 7f07a53c4740 1 -- 172.21.15.28:0/2649109000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 msgr2=0x562a9299ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.881+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299ef80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f078c00b770 tx=0x7f078c012e40 comp rx=0 tx=0).stop 2024-05-09T08:41:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.882+0000 7f07a53c4740 1 -- 172.21.15.28:0/2649109000 shutdown_connections 2024-05-09T08:41:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.882+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562a929b12e0 0x562a929b3700 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.882+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299ef80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.882+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996d70 0x562a9299ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.882+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2649109000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a9299e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.882+0000 7f07a53c4740 1 -- 172.21.15.28:0/2649109000 >> 172.21.15.28:0/2649109000 conn(0x562a92992960 msgr2=0x562a92998690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.882+0000 7f077b7fe640 0 client.26669 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.884+0000 7f07a53c4740 1 -- 172.21.15.28:0/2649109000 shutdown_connections 2024-05-09T08:41:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.886+0000 7f07a53c4740 1 -- 172.21.15.28:0/2649109000 wait complete. 2024-05-09T08:41:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.886+0000 7f07a53c4740 20 client.26669 trim_cache size 0 max 16384 2024-05-09T08:41:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.924+0000 7f07a53c4740 1 Processor -- start 2024-05-09T08:41:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.924+0000 7f07a53c4740 1 -- start start 2024-05-09T08:41:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.924+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a929a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.925+0000 7f07a0b72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a929a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.925+0000 7f07a0b72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a929a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34168/0 (socket says 172.21.15.28:34168) 2024-05-09T08:41:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.925+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a92996d70 0x562a9299eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.925+0000 7f07a0b72640 1 -- 172.21.15.28:0/2449596906 learned_addr learned my addr 172.21.15.28:0/2449596906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.925+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a929a1630 0x562a92998510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.925+0000 7f07a53c4740 1 -- 172.21.15.28:0/2449596906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0788010400 con 0x562a92996450 2024-05-09T08:41:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.925+0000 7f07a53c4740 1 -- 172.21.15.28:0/2449596906 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0788012180 con 0x562a92996d70 2024-05-09T08:41:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.925+0000 7f07a53c4740 1 -- 172.21.15.28:0/2449596906 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0788010b90 con 0x562a929a1630 2024-05-09T08:41:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.926+0000 7f079b7fe640 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a92996d70 0x562a9299eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:55 smithi028 ceph-mon[30119]: pgmap v499: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 1.1 MiB/s wr, 148 op/s 2024-05-09T08:41:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.930+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a929a1630 0x562a92998510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.930+0000 7f07a0b72640 1 -- 172.21.15.28:0/2449596906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a929a1630 msgr2=0x562a92998510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.930+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a929a1630 0x562a92998510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.930+0000 7f07a0b72640 1 -- 172.21.15.28:0/2449596906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a92996d70 msgr2=0x562a9299eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.930+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a92996d70 0x562a9299eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.930+0000 7f07a0b72640 1 -- 172.21.15.28:0/2449596906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a92990f00 con 0x562a92996450 2024-05-09T08:41:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.930+0000 7f07a0b72640 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a929a0c30 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f078c006b00 tx=0x7f078c00b2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0688caf9e39cf15 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.932+0000 7f0781ffb640 1 -- 172.21.15.28:0/2449596906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c027050 con 0x562a92996450 2024-05-09T08:41:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.932+0000 7f0781ffb640 1 -- 172.21.15.28:0/2449596906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f078c00d530 con 0x562a92996450 2024-05-09T08:41:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.932+0000 7f0781ffb640 1 -- 172.21.15.28:0/2449596906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0147f0 con 0x562a92996450 2024-05-09T08:41:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.933+0000 7f07a53c4740 1 -- 172.21.15.28:0/2449596906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 msgr2=0x562a929a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.933+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a929a0c30 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f078c006b00 tx=0x7f078c00b2a0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.936+0000 7f07a53c4740 1 -- 172.21.15.28:0/2449596906 shutdown_connections 2024-05-09T08:41:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.936+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a929a1630 0x562a92998510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.936+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a92996d70 0x562a9299eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.936+0000 7f07a53c4740 1 --2- 172.21.15.28:0/2449596906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996450 0x562a929a0c30 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.936+0000 7f07a53c4740 1 -- 172.21.15.28:0/2449596906 >> 172.21.15.28:0/2449596906 conn(0x562a92992960 msgr2=0x562a92997e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.940+0000 7f07a53c4740 1 -- 172.21.15.28:0/2449596906 shutdown_connections 2024-05-09T08:41:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.942+0000 7f07a53c4740 1 -- 172.21.15.28:0/2449596906 wait complete. 2024-05-09T08:41:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.943+0000 7f07a53c4740 1 -- start start 2024-05-09T08:41:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.943+0000 7f07a53c4740 1 Processor -- start 2024-05-09T08:41:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.947+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996450 0x562a9299b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.947+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996d70 0x562a9299b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.947+0000 7f07a53c4740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.948+0000 7f07a0b72640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996450 0x562a9299b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.948+0000 7f07a0b72640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996450 0x562a9299b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:60866/0 (socket says 172.21.15.28:60866) 2024-05-09T08:41:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.948+0000 7f07a0b72640 1 -- 172.21.15.28:0/1825432624 learned_addr learned my addr 172.21.15.28:0/1825432624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:41:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.948+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0788011ab0 con 0x562a92996d70 2024-05-09T08:41:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.948+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0788011320 con 0x562a929a1630 2024-05-09T08:41:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.948+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f078800e400 con 0x562a92996450 2024-05-09T08:41:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.948+0000 7f07a53c4740 10 client.0 _ll_register_callbacks cb 0x7f07940d3390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.949+0000 7f07a0b72640 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.949+0000 7f079b7fe640 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996d70 0x562a9299b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.949+0000 7f07a0b72640 1 -- 172.21.15.28:0/1825432624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 msgr2=0x562a9299bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.949+0000 7f07a0b72640 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.949+0000 7f07a0b72640 1 -- 172.21.15.28:0/1825432624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996d70 msgr2=0x562a9299b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.949+0000 7f07a0b72640 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996d70 0x562a9299b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.949+0000 7f07a0b72640 1 -- 172.21.15.28:0/1825432624 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a92a168b0 con 0x562a92996450 2024-05-09T08:41:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.952+0000 7f07a0b72640 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996450 0x562a9299b210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f078c00bca0 tx=0x7f078c0143f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97c2fc2d256a94de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.952+0000 7f077b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:41:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.956+0000 7f077b7fe640 1 -- 172.21.15.28:0/1825432624 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c027040 con 0x562a92996450 2024-05-09T08:41:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.956+0000 7f077b7fe640 1 -- 172.21.15.28:0/1825432624 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9815+0+0 (secure 0 0 0) 0x7f078c012e20 con 0x562a92996450 2024-05-09T08:41:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.956+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f078c00d530 con 0x562a92996450 2024-05-09T08:41:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.958+0000 7f077b7fe640 1 -- 172.21.15.28:0/1825432624 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f078c040050 con 0x562a92996450 2024-05-09T08:41:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.959+0000 7f07a53c4740 10 client.26512 fetch_fsmap learned FSMap version 67 2024-05-09T08:41:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.959+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x562a92a17bc0 con 0x562a92996450 2024-05-09T08:41:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.962+0000 7f077b7fe640 1 -- 172.21.15.28:0/1825432624 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f078c004510 con 0x562a92996450 2024-05-09T08:41:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.962+0000 7f077b7fe640 1 -- 172.21.15.28:0/1825432624 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f078c034040 con 0x562a92996450 2024-05-09T08:41:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.962+0000 7f07a53c4740 10 client.26512 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:41:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.962+0000 7f07a53c4740 10 client.26512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:41:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.962+0000 7f07a53c4740 20 client.26512 populate_metadata read hostname 'smithi028' 2024-05-09T08:41:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.962+0000 7f07a53c4740 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562a92a18280 0x562a92a1a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.962+0000 7f07a53c4740 4 client.26512 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.962+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f078c00d530 con 0x562a92a18280 2024-05-09T08:41:55.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.962+0000 7f079b7fe640 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562a92a18280 0x562a92a1a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:55.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.963+0000 7f077b7fe640 10 client.26512 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:55.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:55.964+0000 7f079b7fe640 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562a92a18280 0x562a92a1a6a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30187b3a6e23430b server_cookie=e0f00fb55d71e22f in_seq=0 out_seq=0 2024-05-09T08:41:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.077+0000 7f077b7fe640 1 -- 172.21.15.28:0/1825432624 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f078c00d530 con 0x562a92a18280 2024-05-09T08:41:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.077+0000 7f077b7fe640 10 client.26512 handle_command_reply: tid=0 2024-05-09T08:41:56.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.077+0000 7f07a53c4740 1 client.26512 shutdown 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.078+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996450 msgr2=0x562a9299b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.078+0000 7f07a53c4740 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996450 0x562a9299b210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f078c00bca0 tx=0x7f078c0143f0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.078+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 shutdown_connections 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.078+0000 7f07a53c4740 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x562a92a18280 0x562a92a1a6a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.078+0000 7f07a53c4740 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562a929a1630 0x562a9299bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.078+0000 7f077b7fe640 0 client.26512 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.078+0000 7f07a53c4740 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a92996d70 0x562a9299b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.078+0000 7f07a53c4740 1 --2- 172.21.15.28:0/1825432624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x562a92996450 0x562a9299b210 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.078+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 >> 172.21.15.28:0/1825432624 conn(0x562a92992960 msgr2=0x562a92998640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.079+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 shutdown_connections 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.079+0000 7f07a53c4740 1 -- 172.21.15.28:0/1825432624 wait complete. 2024-05-09T08:41:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.079+0000 7f07a53c4740 20 client.26512 trim_cache size 0 max 16384 2024-05-09T08:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:41:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 -- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07840a2fd0 msgr2=0x7f07840a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07840a2fd0 0x7f07840a5490 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x562a929979f0 tx=0x7f07940d34c0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 -- 172.21.15.28:0/150131411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f079c1008b0 msgr2=0x7f079c072460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f079c1008b0 0x7f079c072460 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f079400cce0 tx=0x7f079400a620 comp rx=0 tx=0).stop 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 -- 172.21.15.28:0/150131411 shutdown_connections 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07840a2fd0 0x7f07840a5490 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f079c109f60 0x7f079c10a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c105bd0 0x7f079c0729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 --2- 172.21.15.28:0/150131411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f079c1008b0 0x7f079c072460 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 -- 172.21.15.28:0/150131411 >> 172.21.15.28:0/150131411 conn(0x7f079c0778b0 msgr2=0x7f079c1038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 -- 172.21.15.28:0/150131411 shutdown_connections 2024-05-09T08:41:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:41:56.080+0000 7f07a25d5640 1 -- 172.21.15.28:0/150131411 wait complete. 2024-05-09T08:41:57.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:57 smithi184 ceph-mon[34509]: pgmap v500: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-05-09T08:41:57.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:57 smithi028 ceph-mon[30119]: pgmap v500: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-05-09T08:41:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:57 smithi165 ceph-mon[34540]: pgmap v500: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-05-09T08:41:59.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:41:59 smithi184 ceph-mon[34509]: pgmap v501: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-05-09T08:41:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:41:59 smithi028 ceph-mon[30119]: pgmap v501: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-05-09T08:42:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:41:59 smithi165 ceph-mon[34540]: pgmap v501: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-05-09T08:42:01.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:01 smithi028 ceph-mon[30119]: pgmap v502: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-05-09T08:42:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:01 smithi165 ceph-mon[34540]: pgmap v502: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-05-09T08:42:02.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:01 smithi184 ceph-mon[34509]: pgmap v502: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 126 op/s 2024-05-09T08:42:03.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:03 smithi165 ceph-mon[34540]: pgmap v503: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-05-09T08:42:03.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:03 smithi028 ceph-mon[30119]: pgmap v503: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-05-09T08:42:04.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:03 smithi184 ceph-mon[34509]: pgmap v503: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-05-09T08:42:05.908 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:05 smithi184 ceph-mon[34509]: pgmap v504: 129 pgs: 63 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 45 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 136 op/s 2024-05-09T08:42:05.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:05 smithi028 ceph-mon[30119]: pgmap v504: 129 pgs: 63 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 45 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 136 op/s 2024-05-09T08:42:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:05 smithi165 ceph-mon[34540]: pgmap v504: 129 pgs: 63 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 45 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 136 op/s 2024-05-09T08:42:07.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:07 smithi028 ceph-mon[30119]: pgmap v505: 129 pgs: 63 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 45 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-05-09T08:42:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:07 smithi165 ceph-mon[34540]: pgmap v505: 129 pgs: 63 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 45 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-05-09T08:42:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:07 smithi184 ceph-mon[34509]: pgmap v505: 129 pgs: 63 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 45 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-05-09T08:42:09.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.254+0000 7fbe53288640 1 -- 172.21.15.28:0/3683507358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe4c104280 msgr2=0x7fbe4c104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.254+0000 7fbe53288640 1 --2- 172.21.15.28:0/3683507358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe4c104280 0x7fbe4c104730 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009900 tx=0x7fbe3c02a4b0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.257+0000 7fbe53288640 1 -- 172.21.15.28:0/3683507358 shutdown_connections 2024-05-09T08:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.257+0000 7fbe53288640 1 --2- 172.21.15.28:0/3683507358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe4c104280 0x7fbe4c104730 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.257+0000 7fbe53288640 1 --2- 172.21.15.28:0/3683507358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe4c106c00 0x7fbe4c06a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.257+0000 7fbe53288640 1 --2- 172.21.15.28:0/3683507358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe4c106820 0x7fbe4c06a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.257+0000 7fbe53288640 1 -- 172.21.15.28:0/3683507358 >> 172.21.15.28:0/3683507358 conn(0x7fbe4c076530 msgr2=0x7fbe4c076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.262+0000 7fbe53288640 1 -- 172.21.15.28:0/3683507358 shutdown_connections 2024-05-09T08:42:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.264+0000 7fbe53288640 1 -- 172.21.15.28:0/3683507358 wait complete. 2024-05-09T08:42:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.265+0000 7fbe53288640 1 Processor -- start 2024-05-09T08:42:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.266+0000 7fbe53288640 1 -- start start 2024-05-09T08:42:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.266+0000 7fbe53288640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe4c104280 0x7fbe4c072180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.266+0000 7fbe53288640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe4c106820 0x7fbe4c0726c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.266+0000 7fbe53288640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe4c106c00 0x7fbe4c10a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.266+0000 7fbe53288640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4c118230 con 0x7fbe4c106820 2024-05-09T08:42:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.266+0000 7fbe53288640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4c1180b0 con 0x7fbe4c106c00 2024-05-09T08:42:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.266+0000 7fbe53288640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4c1183b0 con 0x7fbe4c104280 2024-05-09T08:42:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe50823640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe4c106820 0x7fbe4c0726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe51825640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe4c106c00 0x7fbe4c10a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe50823640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe4c106820 0x7fbe4c0726c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33388/0 (socket says 172.21.15.28:33388) 2024-05-09T08:42:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe50823640 1 -- 172.21.15.28:0/57862488 learned_addr learned my addr 172.21.15.28:0/57862488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe51825640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe4c106c00 0x7fbe4c10a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:54548/0 (socket says 172.21.15.28:54548) 2024-05-09T08:42:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe51024640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe4c104280 0x7fbe4c072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe50823640 1 -- 172.21.15.28:0/57862488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe4c104280 msgr2=0x7fbe4c072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe50823640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe4c104280 0x7fbe4c072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe50823640 1 -- 172.21.15.28:0/57862488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe4c106c00 msgr2=0x7fbe4c10a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe50823640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe4c106c00 0x7fbe4c10a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.267+0000 7fbe50823640 1 -- 172.21.15.28:0/57862488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4c10a800 con 0x7fbe4c106820 2024-05-09T08:42:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.268+0000 7fbe50823640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe4c106820 0x7fbe4c0726c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400d770 tx=0x7fbe4400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.275+0000 7fbe427fc640 1 -- 172.21.15.28:0/57862488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe44017070 con 0x7fbe4c106820 2024-05-09T08:42:09.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.275+0000 7fbe427fc640 1 -- 172.21.15.28:0/57862488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe44009c30 con 0x7fbe4c106820 2024-05-09T08:42:09.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.275+0000 7fbe427fc640 1 -- 172.21.15.28:0/57862488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4400f6b0 con 0x7fbe4c106820 2024-05-09T08:42:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.275+0000 7fbe53288640 1 -- 172.21.15.28:0/57862488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe4c10aaf0 con 0x7fbe4c106820 2024-05-09T08:42:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.275+0000 7fbe53288640 1 -- 172.21.15.28:0/57862488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe4c1123c0 con 0x7fbe4c106820 2024-05-09T08:42:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.283+0000 7fbe55f91740 1 Processor -- start 2024-05-09T08:42:09.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.285+0000 7fbe427fc640 1 -- 172.21.15.28:0/57862488 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbe4401a070 con 0x7fbe4c106820 2024-05-09T08:42:09.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.286+0000 7fbe427fc640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fbe380a2fd0 0x7fbe380a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.286+0000 7fbe427fc640 1 -- 172.21.15.28:0/57862488 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9879+0+0 (secure 0 0 0) 0x7fbe440cf2a0 con 0x7fbe4c106820 2024-05-09T08:42:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.290+0000 7fbe55f91740 1 -- start start 2024-05-09T08:42:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.290+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a68698450 0x557a68698830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.290+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.290+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 0x557a686a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.290+0000 7fbe55f91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a685fd340 con 0x557a68698d70 2024-05-09T08:42:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.290+0000 7fbe55f91740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557a685a5290 con 0x557a68698450 2024-05-09T08:42:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.290+0000 7fbe55f91740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557a685f9300 con 0x557a686a3630 2024-05-09T08:42:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.291+0000 7fbe51024640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a68698450 0x557a68698830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.293+0000 7fbe51825640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fbe380a2fd0 0x7fbe380a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.293+0000 7fbe50823640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.293+0000 7fbe50823640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33402/0 (socket says 172.21.15.28:33402) 2024-05-09T08:42:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.293+0000 7fbe50823640 1 -- 172.21.15.28:0/4004579539 learned_addr learned my addr 172.21.15.28:0/4004579539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.293+0000 7fbe51825640 1 --2- 172.21.15.28:0/4004579539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 0x557a686a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.294+0000 7fbe50823640 1 -- 172.21.15.28:0/4004579539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 msgr2=0x557a686a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.294+0000 7fbe50823640 1 --2- 172.21.15.28:0/4004579539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 0x557a686a5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.294+0000 7fbe50823640 1 -- 172.21.15.28:0/4004579539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a68698450 msgr2=0x557a68698830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.294+0000 7fbe50823640 1 --2- 172.21.15.28:0/4004579539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a68698450 0x557a68698830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.294+0000 7fbe50823640 1 -- 172.21.15.28:0/4004579539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a685c98e0 con 0x557a68698d70 2024-05-09T08:42:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.295+0000 7fbe51825640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fbe380a2fd0 0x7fbe380a5490 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0096c0 tx=0x7fbe3c004310 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.296+0000 7fbe50823640 1 --2- 172.21.15.28:0/4004579539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a30f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fbe44097be0 tx=0x7fbe4409ae80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d3be140b79e537 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.298+0000 7fbe377fe640 1 -- 172.21.15.28:0/4004579539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe44021070 con 0x557a68698d70 2024-05-09T08:42:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.299+0000 7fbe377fe640 1 -- 172.21.15.28:0/4004579539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe440b9e50 con 0x557a68698d70 2024-05-09T08:42:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.299+0000 7fbe55f91740 1 -- 172.21.15.28:0/4004579539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 msgr2=0x557a686a30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.300+0000 7fbe55f91740 1 --2- 172.21.15.28:0/4004579539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a30f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fbe44097be0 tx=0x7fbe4409ae80 comp rx=0 tx=0).stop 2024-05-09T08:42:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.303+0000 7fbe55f91740 1 -- 172.21.15.28:0/4004579539 shutdown_connections 2024-05-09T08:42:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.303+0000 7fbe55f91740 1 --2- 172.21.15.28:0/4004579539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 0x557a686a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.303+0000 7fbe55f91740 1 --2- 172.21.15.28:0/4004579539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a30f0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.303+0000 7fbe55f91740 1 --2- 172.21.15.28:0/4004579539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a68698450 0x557a68698830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.303+0000 7fbe55f91740 1 -- 172.21.15.28:0/4004579539 >> 172.21.15.28:0/4004579539 conn(0x557a68694960 msgr2=0x557a68694d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.309+0000 7fbe55f91740 1 -- 172.21.15.28:0/4004579539 shutdown_connections 2024-05-09T08:42:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.311+0000 7fbe55f91740 1 -- 172.21.15.28:0/4004579539 wait complete. 2024-05-09T08:42:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.311+0000 7fbe55f91740 1 -- start start 2024-05-09T08:42:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.312+0000 7fbe55f91740 1 Processor -- start 2024-05-09T08:42:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.316+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a68698450 0x557a686a0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.316+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.316+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686a3630 0x557a686a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.316+0000 7fbe55f91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a686a61a0 con 0x557a68698d70 2024-05-09T08:42:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.316+0000 7fbe55f91740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557a685b8790 con 0x557a686a3630 2024-05-09T08:42:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.316+0000 7fbe55f91740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557a686a6320 con 0x557a68698450 2024-05-09T08:42:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.316+0000 7fbe55f91740 10 client.0 _ll_register_callbacks cb 0x557a685f3a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.317+0000 7fbe51024640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a68698450 0x557a686a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.317+0000 7fbe51024640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a68698450 0x557a686a0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:57736/0 (socket says 172.21.15.28:57736) 2024-05-09T08:42:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.317+0000 7fbe51024640 1 -- 172.21.15.28:0/1572059981 learned_addr learned my addr 172.21.15.28:0/1572059981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.318+0000 7fbe50823640 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.319+0000 7fbe50823640 1 -- 172.21.15.28:0/1572059981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a68698450 msgr2=0x557a686a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.319+0000 7fbe50823640 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a68698450 0x557a686a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.319+0000 7fbe50823640 1 -- 172.21.15.28:0/1572059981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686a3630 msgr2=0x557a686a0f90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.319+0000 7fbe50823640 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686a3630 0x557a686a0f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.319+0000 7fbe50823640 1 -- 172.21.15.28:0/1572059981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a68311210 con 0x557a68698d70 2024-05-09T08:42:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.320+0000 7fbe50823640 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a0a50 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fbe44097be0 tx=0x7fbe4400df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590f8c53334eac97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.320+0000 7fbe34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:42:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.320+0000 7fbe34ff9640 1 -- 172.21.15.28:0/1572059981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe44021070 con 0x557a68698d70 2024-05-09T08:42:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.321+0000 7fbe34ff9640 1 -- 172.21.15.28:0/1572059981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9879+0+0 (secure 0 0 0) 0x7fbe4409f320 con 0x557a68698d70 2024-05-09T08:42:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.323+0000 7fbe55f91740 1 -- 172.21.15.28:0/1572059981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe440b9e50 con 0x557a68698d70 2024-05-09T08:42:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.323+0000 7fbe34ff9640 1 -- 172.21.15.28:0/1572059981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fbe440b4200 con 0x557a68698d70 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.324+0000 7fbe55f91740 10 client.17226 fetch_fsmap learned FSMap version 67 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.324+0000 7fbe55f91740 1 -- 172.21.15.28:0/1572059981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x557a683dd620 con 0x557a68698d70 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.324+0000 7fbe34ff9640 1 -- 172.21.15.28:0/1572059981 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe440b9e50 con 0x557a68698d70 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.324+0000 7fbe34ff9640 1 -- 172.21.15.28:0/1572059981 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fbe4409a0d0 con 0x557a68698d70 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.324+0000 7fbe55f91740 10 client.17226 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.324+0000 7fbe55f91740 10 client.17226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.324+0000 7fbe55f91740 20 client.17226 populate_metadata read hostname 'smithi028' 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.325+0000 7fbe55f91740 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557a686b3510 0x557a686b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.325+0000 7fbe55f91740 4 client.17226 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.325+0000 7fbe55f91740 1 -- 172.21.15.28:0/1572059981 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a6860aa70 con 0x557a686b3510 2024-05-09T08:42:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.325+0000 7fbe51024640 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557a686b3510 0x557a686b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.325+0000 7fbe51024640 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557a686b3510 0x557a686b5930 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97ac65428fa8b3d8 server_cookie=4e56f5d7a4f1adbf in_seq=0 out_seq=0 2024-05-09T08:42:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.326+0000 7fbe34ff9640 10 client.17226 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.330+0000 7fbe34ff9640 1 -- 172.21.15.28:0/1572059981 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a6860aa70 con 0x557a686b3510 2024-05-09T08:42:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.330+0000 7fbe34ff9640 10 client.17226 handle_command_reply: tid=0 2024-05-09T08:42:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.330+0000 7fbe55f91740 1 client.17226 shutdown 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe55f91740 1 -- 172.21.15.28:0/1572059981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 msgr2=0x557a686a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe55f91740 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a0a50 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fbe44097be0 tx=0x7fbe4400df80 comp rx=0 tx=0).stop 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe55f91740 1 -- 172.21.15.28:0/1572059981 shutdown_connections 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe55f91740 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557a686b3510 0x557a686b5930 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe55f91740 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686a3630 0x557a686a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe55f91740 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698d70 0x557a686a0a50 secure :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fbe44097be0 tx=0x7fbe4400df80 comp rx=0 tx=0).stop 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe55f91740 1 --2- 172.21.15.28:0/1572059981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a68698450 0x557a686a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe55f91740 1 -- 172.21.15.28:0/1572059981 >> 172.21.15.28:0/1572059981 conn(0x557a68694960 msgr2=0x557a6869a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe34ff9640 0 client.17226 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.335+0000 7fbe55f91740 1 -- 172.21.15.28:0/1572059981 shutdown_connections 2024-05-09T08:42:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.336+0000 7fbe55f91740 1 -- 172.21.15.28:0/1572059981 wait complete. 2024-05-09T08:42:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.336+0000 7fbe55f91740 20 client.17226 trim_cache size 0 max 16384 2024-05-09T08:42:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe55f91740 1 Processor -- start 2024-05-09T08:42:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe55f91740 1 -- start start 2024-05-09T08:42:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 0x557a686af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686a3630 0x557a686a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686bb1c0 0x557a686bb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe51024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 0x557a686af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe55f91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30010410 con 0x557a68698450 2024-05-09T08:42:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe55f91740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30012190 con 0x557a686a3630 2024-05-09T08:42:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe55f91740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30010ba0 con 0x557a686bb1c0 2024-05-09T08:42:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.370+0000 7fbe51825640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686bb1c0 0x557a686bb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.371+0000 7fbe51825640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686bb1c0 0x557a686bb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:57740/0 (socket says 172.21.15.28:57740) 2024-05-09T08:42:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.371+0000 7fbe51825640 1 -- 172.21.15.28:0/4170868888 learned_addr learned my addr 172.21.15.28:0/4170868888 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.371+0000 7fbe51825640 1 -- 172.21.15.28:0/4170868888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686a3630 msgr2=0x557a686a2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.371+0000 7fbe51825640 1 --2- 172.21.15.28:0/4170868888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686a3630 0x557a686a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.371+0000 7fbe51825640 1 -- 172.21.15.28:0/4170868888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 msgr2=0x557a686af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.371+0000 7fbe51825640 1 --2- 172.21.15.28:0/4170868888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 0x557a686af0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.371+0000 7fbe51825640 1 -- 172.21.15.28:0/4170868888 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a68697c50 con 0x557a686bb1c0 2024-05-09T08:42:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.375+0000 7fbe51825640 1 --2- 172.21.15.28:0/4170868888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686bb1c0 0x557a686bb600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c033000 tx=0x7fbe3c040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57ea3385c211d783 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.375+0000 7fbe36ffd640 1 -- 172.21.15.28:0/4170868888 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c04c070 con 0x557a686bb1c0 2024-05-09T08:42:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.375+0000 7fbe36ffd640 1 -- 172.21.15.28:0/4170868888 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe3c004ba0 con 0x557a686bb1c0 2024-05-09T08:42:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.375+0000 7fbe36ffd640 1 -- 172.21.15.28:0/4170868888 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c035050 con 0x557a686bb1c0 2024-05-09T08:42:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.376+0000 7fbe55f91740 1 -- 172.21.15.28:0/4170868888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686bb1c0 msgr2=0x557a686bb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.376+0000 7fbe55f91740 1 --2- 172.21.15.28:0/4170868888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686bb1c0 0x557a686bb600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c033000 tx=0x7fbe3c040000 comp rx=0 tx=0).stop 2024-05-09T08:42:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.376+0000 7fbe55f91740 1 -- 172.21.15.28:0/4170868888 shutdown_connections 2024-05-09T08:42:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.376+0000 7fbe55f91740 1 --2- 172.21.15.28:0/4170868888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686bb1c0 0x557a686bb600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.376+0000 7fbe55f91740 1 --2- 172.21.15.28:0/4170868888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686a3630 0x557a686a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.376+0000 7fbe55f91740 1 --2- 172.21.15.28:0/4170868888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 0x557a686af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.376+0000 7fbe55f91740 1 -- 172.21.15.28:0/4170868888 >> 172.21.15.28:0/4170868888 conn(0x557a68694960 msgr2=0x557a68693a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.379+0000 7fbe55f91740 1 -- 172.21.15.28:0/4170868888 shutdown_connections 2024-05-09T08:42:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.384+0000 7fbe55f91740 1 -- 172.21.15.28:0/4170868888 wait complete. 2024-05-09T08:42:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.384+0000 7fbe55f91740 1 -- start start 2024-05-09T08:42:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.384+0000 7fbe55f91740 1 Processor -- start 2024-05-09T08:42:09.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.385+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 0x557a686a7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.385+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 0x557a686a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.385+0000 7fbe55f91740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686bb1c0 0x557a686a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.385+0000 7fbe55f91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30011ac0 con 0x557a68698450 2024-05-09T08:42:09.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.385+0000 7fbe55f91740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30011330 con 0x557a686bb1c0 2024-05-09T08:42:09.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.385+0000 7fbe55f91740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3000e410 con 0x557a686a3630 2024-05-09T08:42:09.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.385+0000 7fbe55f91740 10 client.0 _ll_register_callbacks cb 0x7fbe480096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.385+0000 7fbe51825640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686bb1c0 0x557a686a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.387+0000 7fbe51024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 0x557a686a7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.387+0000 7fbe51024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 0x557a686a7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33420/0 (socket says 172.21.15.28:33420) 2024-05-09T08:42:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.387+0000 7fbe51024640 1 -- 172.21.15.28:0/3996026416 learned_addr learned my addr 172.21.15.28:0/3996026416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.387+0000 7fbe50823640 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 0x557a686a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.387+0000 7fbe50823640 1 -- 172.21.15.28:0/3996026416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686bb1c0 msgr2=0x557a686a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.387+0000 7fbe50823640 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686bb1c0 0x557a686a7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.387+0000 7fbe50823640 1 -- 172.21.15.28:0/3996026416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 msgr2=0x557a686a7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.387+0000 7fbe50823640 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 0x557a686a7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.387+0000 7fbe50823640 1 -- 172.21.15.28:0/3996026416 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a68718cd0 con 0x557a686a3630 2024-05-09T08:42:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.393+0000 7fbe50823640 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 0x557a686a77c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x557a686a15d0 tx=0x7fbe4400b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49279d706b0565a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.393+0000 7fbe34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:42:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.395+0000 7fbe34ff9640 1 -- 172.21.15.28:0/3996026416 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe440974b0 con 0x557a686a3630 2024-05-09T08:42:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.395+0000 7fbe34ff9640 1 -- 172.21.15.28:0/3996026416 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9879+0+0 (secure 0 0 0) 0x7fbe4409f320 con 0x557a686a3630 2024-05-09T08:42:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.395+0000 7fbe55f91740 1 -- 172.21.15.28:0/3996026416 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe3c004ba0 con 0x557a686a3630 2024-05-09T08:42:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.398+0000 7fbe34ff9640 1 -- 172.21.15.28:0/3996026416 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fbe4409b910 con 0x557a686a3630 2024-05-09T08:42:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.402+0000 7fbe55f91740 10 client.26524 fetch_fsmap learned FSMap version 67 2024-05-09T08:42:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.402+0000 7fbe55f91740 1 -- 172.21.15.28:0/3996026416 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x557a68718cd0 con 0x557a686a3630 2024-05-09T08:42:09.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.404+0000 7fbe34ff9640 1 -- 172.21.15.28:0/3996026416 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe3c004ba0 con 0x557a686a3630 2024-05-09T08:42:09.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.404+0000 7fbe34ff9640 1 -- 172.21.15.28:0/3996026416 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fbe440ccb90 con 0x557a686a3630 2024-05-09T08:42:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.404+0000 7fbe55f91740 10 client.26524 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:42:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.404+0000 7fbe55f91740 10 client.26524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:42:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.404+0000 7fbe55f91740 20 client.26524 populate_metadata read hostname 'smithi028' 2024-05-09T08:42:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.404+0000 7fbe55f91740 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557a6871a310 0x557a6871c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.404+0000 7fbe55f91740 4 client.26524 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:42:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.404+0000 7fbe55f91740 1 -- 172.21.15.28:0/3996026416 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a68604dc0 con 0x557a6871a310 2024-05-09T08:42:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.405+0000 7fbe51024640 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557a6871a310 0x557a6871c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.409+0000 7fbe34ff9640 10 client.26524 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:09.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.409+0000 7fbe51024640 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557a6871a310 0x557a6871c730 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=890c2e2c592834d6 server_cookie=9e90a9d95cbdd390 in_seq=0 out_seq=0 2024-05-09T08:42:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe34ff9640 1 -- 172.21.15.28:0/3996026416 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a68604dc0 con 0x557a6871a310 2024-05-09T08:42:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe34ff9640 10 client.26524 handle_command_reply: tid=0 2024-05-09T08:42:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe55f91740 1 client.26524 shutdown 2024-05-09T08:42:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe55f91740 1 -- 172.21.15.28:0/3996026416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 msgr2=0x557a686a77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe55f91740 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 0x557a686a77c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x557a686a15d0 tx=0x7fbe4400b480 comp rx=0 tx=0).stop 2024-05-09T08:42:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe55f91740 1 -- 172.21.15.28:0/3996026416 shutdown_connections 2024-05-09T08:42:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe55f91740 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557a6871a310 0x557a6871c730 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe55f91740 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557a686bb1c0 0x557a686a7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe55f91740 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557a686a3630 0x557a686a77c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe55f91740 1 --2- 172.21.15.28:0/3996026416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a68698450 0x557a686a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.804+0000 7fbe55f91740 1 -- 172.21.15.28:0/3996026416 >> 172.21.15.28:0/3996026416 conn(0x557a68694960 msgr2=0x557a686a04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.805+0000 7fbe34ff9640 0 client.26524 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.805+0000 7fbe55f91740 1 -- 172.21.15.28:0/3996026416 shutdown_connections 2024-05-09T08:42:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.805+0000 7fbe55f91740 1 -- 172.21.15.28:0/3996026416 wait complete. 2024-05-09T08:42:09.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.805+0000 7fbe55f91740 20 client.26524 trim_cache size 0 max 16384 2024-05-09T08:42:09.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:42:09.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:42:09.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:42:09.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:42:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 -- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fbe380a2fd0 msgr2=0x7fbe380a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fbe380a2fd0 0x7fbe380a5490 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0096c0 tx=0x7fbe3c004310 comp rx=0 tx=0).stop 2024-05-09T08:42:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 -- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe4c106820 msgr2=0x7fbe4c0726c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe4c106820 0x7fbe4c0726c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400d770 tx=0x7fbe4400dc30 comp rx=0 tx=0).stop 2024-05-09T08:42:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 -- 172.21.15.28:0/57862488 shutdown_connections 2024-05-09T08:42:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe4c106c00 0x7fbe4c10a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe4c106820 0x7fbe4c0726c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fbe380a2fd0 0x7fbe380a5490 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 --2- 172.21.15.28:0/57862488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbe4c104280 0x7fbe4c072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 -- 172.21.15.28:0/57862488 >> 172.21.15.28:0/57862488 conn(0x7fbe4c076530 msgr2=0x7fbe4c1016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 -- 172.21.15.28:0/57862488 shutdown_connections 2024-05-09T08:42:09.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:09.806+0000 7fbe53288640 1 -- 172.21.15.28:0/57862488 wait complete. 2024-05-09T08:42:09.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:09 smithi028 ceph-mon[30119]: pgmap v506: 129 pgs: 63 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 45 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-05-09T08:42:09.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:09 smithi028 ceph-mon[30119]: osdmap e123: 12 total, 12 up, 12 in 2024-05-09T08:42:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:09 smithi165 ceph-mon[34540]: pgmap v506: 129 pgs: 63 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 45 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-05-09T08:42:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:09 smithi165 ceph-mon[34540]: osdmap e123: 12 total, 12 up, 12 in 2024-05-09T08:42:10.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:09 smithi184 ceph-mon[34509]: pgmap v506: 129 pgs: 63 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 45 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-05-09T08:42:10.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:09 smithi184 ceph-mon[34509]: osdmap e123: 12 total, 12 up, 12 in 2024-05-09T08:42:12.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:11 smithi165 ceph-mon[34540]: pgmap v508: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-05-09T08:42:12.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:11 smithi165 ceph-mon[34540]: osdmap e124: 12 total, 12 up, 12 in 2024-05-09T08:42:12.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:11 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:11 smithi184 ceph-mon[34509]: pgmap v508: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-05-09T08:42:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:11 smithi184 ceph-mon[34509]: osdmap e124: 12 total, 12 up, 12 in 2024-05-09T08:42:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:11 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:12.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:11 smithi028 ceph-mon[30119]: pgmap v508: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-05-09T08:42:12.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:11 smithi028 ceph-mon[30119]: osdmap e124: 12 total, 12 up, 12 in 2024-05-09T08:42:12.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:11 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:13.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:13.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:13.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:42:13.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:13.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:13.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:42:13.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:13.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:13.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:42:14.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:13 smithi165 ceph-mon[34540]: pgmap v510: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 131 op/s 2024-05-09T08:42:14.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:13 smithi184 ceph-mon[34509]: pgmap v510: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 131 op/s 2024-05-09T08:42:14.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:13 smithi028 ceph-mon[30119]: pgmap v510: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 131 op/s 2024-05-09T08:42:16.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:15 smithi165 ceph-mon[34540]: pgmap v511: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-05-09T08:42:16.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:15 smithi184 ceph-mon[34509]: pgmap v511: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-05-09T08:42:16.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:15 smithi028 ceph-mon[30119]: pgmap v511: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-05-09T08:42:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:17 smithi165 ceph-mon[34540]: pgmap v512: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 171 op/s 2024-05-09T08:42:18.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:17 smithi184 ceph-mon[34509]: pgmap v512: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 171 op/s 2024-05-09T08:42:18.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:17 smithi028 ceph-mon[30119]: pgmap v512: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 171 op/s 2024-05-09T08:42:18.931 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:42:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:42:18.673+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:42:18.932 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:42:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:42:18.673+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:42:20.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:19 smithi165 ceph-mon[34540]: pgmap v513: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 934 KiB/s wr, 168 op/s 2024-05-09T08:42:20.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:19 smithi184 ceph-mon[34509]: pgmap v513: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 934 KiB/s wr, 168 op/s 2024-05-09T08:42:20.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:19 smithi028 ceph-mon[30119]: pgmap v513: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 934 KiB/s wr, 168 op/s 2024-05-09T08:42:22.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:21 smithi165 ceph-mon[34540]: pgmap v514: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-09T08:42:22.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:21 smithi184 ceph-mon[34509]: pgmap v514: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-09T08:42:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:21 smithi028 ceph-mon[30119]: pgmap v514: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-05-09T08:42:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.975+0000 7f85ca122640 1 -- 172.21.15.28:0/3447103700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85c411b010 msgr2=0x7f85c411d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.975+0000 7f85ca122640 1 --2- 172.21.15.28:0/3447103700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85c411b010 0x7f85c411d400 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f85bc009be0 tx=0x7f85bc02a790 comp rx=0 tx=0).stop 2024-05-09T08:42:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.982+0000 7f85ca122640 1 -- 172.21.15.28:0/3447103700 shutdown_connections 2024-05-09T08:42:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.982+0000 7f85ca122640 1 --2- 172.21.15.28:0/3447103700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85c411b010 0x7f85c411d400 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.982+0000 7f85ca122640 1 --2- 172.21.15.28:0/3447103700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85c4110400 0x7f85c4108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.982+0000 7f85ca122640 1 --2- 172.21.15.28:0/3447103700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c4110020 0x7f85c4108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.982+0000 7f85ca122640 1 -- 172.21.15.28:0/3447103700 >> 172.21.15.28:0/3447103700 conn(0x7f85c406c900 msgr2=0x7f85c406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:22.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.985+0000 7f85ca122640 1 -- 172.21.15.28:0/3447103700 shutdown_connections 2024-05-09T08:42:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.989+0000 7f85ca122640 1 -- 172.21.15.28:0/3447103700 wait complete. 2024-05-09T08:42:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.990+0000 7f85ca122640 1 Processor -- start 2024-05-09T08:42:22.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.993+0000 7f85ca122640 1 -- start start 2024-05-09T08:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.993+0000 7f85ca122640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c4110020 0x7f85c4118590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.993+0000 7f85ca122640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85c411b010 0x7f85c4111660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.993+0000 7f85ca122640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85c4118be0 0x7f85c4111c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.993+0000 7f85ca122640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85c4120a30 con 0x7f85c411b010 2024-05-09T08:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.993+0000 7f85ca122640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f85c41208b0 con 0x7f85c4110020 2024-05-09T08:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.993+0000 7f85ca122640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f85c4120bb0 con 0x7f85c4118be0 2024-05-09T08:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.995+0000 7f85c3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85c4118be0 0x7f85c4111c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.995+0000 7f85c3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85c4118be0 0x7f85c4111c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37696/0 (socket says 172.21.15.28:37696) 2024-05-09T08:42:22.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.995+0000 7f85c3fff640 1 -- 172.21.15.28:0/1216619532 learned_addr learned my addr 172.21.15.28:0/1216619532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.996+0000 7f85c37fe640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c4110020 0x7f85c4118590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.997+0000 7f85c2ffd640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85c411b010 0x7f85c4111660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.998+0000 7f85c2ffd640 1 -- 172.21.15.28:0/1216619532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85c4118be0 msgr2=0x7f85c4111c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.998+0000 7f85c2ffd640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85c4118be0 0x7f85c4111c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.998+0000 7f85c2ffd640 1 -- 172.21.15.28:0/1216619532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c4110020 msgr2=0x7f85c4118590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.998+0000 7f85c2ffd640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c4110020 0x7f85c4118590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.998+0000 7f85c2ffd640 1 -- 172.21.15.28:0/1216619532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c41160b0 con 0x7f85c411b010 2024-05-09T08:42:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.998+0000 7f85c2ffd640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85c411b010 0x7f85c4111660 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f85b400d6a0 tx=0x7f85b400db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.999+0000 7f85c0ff9640 1 -- 172.21.15.28:0/1216619532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4017070 con 0x7f85c411b010 2024-05-09T08:42:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.999+0000 7f85ca122640 1 -- 172.21.15.28:0/1216619532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c41163a0 con 0x7f85c411b010 2024-05-09T08:42:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:22.999+0000 7f85ca122640 1 -- 172.21.15.28:0/1216619532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c41168f0 con 0x7f85c411b010 2024-05-09T08:42:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.002+0000 7f85c0ff9640 1 -- 172.21.15.28:0/1216619532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85b4009820 con 0x7f85c411b010 2024-05-09T08:42:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.002+0000 7f85c0ff9640 1 -- 172.21.15.28:0/1216619532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4004ce0 con 0x7f85c411b010 2024-05-09T08:42:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.002+0000 7f85cce32740 1 Processor -- start 2024-05-09T08:42:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.007+0000 7f85cce32740 1 -- start start 2024-05-09T08:42:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.008+0000 7f85cce32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09a9450 0x557cd09a9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.008+0000 7f85cce32740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9d70 0x557cd09b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.008+0000 7f85cce32740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09b4630 0x557cd09b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.008+0000 7f85cce32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557cd090e340 con 0x557cd09a9450 2024-05-09T08:42:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.008+0000 7f85cce32740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557cd08b6290 con 0x557cd09b4630 2024-05-09T08:42:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.008+0000 7f85cce32740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557cd090a300 con 0x557cd09a9d70 2024-05-09T08:42:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.012+0000 7f85c2ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09b4630 0x557cd09b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.012+0000 7f85c2ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09b4630 0x557cd09b6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:41416/0 (socket says 172.21.15.28:41416) 2024-05-09T08:42:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.012+0000 7f85c2ffd640 1 -- 172.21.15.28:0/870546005 learned_addr learned my addr 172.21.15.28:0/870546005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.013+0000 7f85c3fff640 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09a9450 0x557cd09a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.014+0000 7f85c3fff640 1 -- 172.21.15.28:0/870546005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9d70 msgr2=0x557cd09b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:23.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.012+0000 7f85c37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9d70 0x557cd09b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.015+0000 7f85c3fff640 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9d70 0x557cd09b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.015+0000 7f85c3fff640 1 -- 172.21.15.28:0/870546005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09b4630 msgr2=0x557cd09b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:23.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.015+0000 7f85c3fff640 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09b4630 0x557cd09b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.015+0000 7f85c3fff640 1 -- 172.21.15.28:0/870546005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd08da8e0 con 0x557cd09a9450 2024-05-09T08:42:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.016+0000 7f85c2ffd640 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09b4630 0x557cd09b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.017+0000 7f85c3fff640 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09a9450 0x557cd09a9830 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f85bc034040 tx=0x7f85bc0303a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8720588b1cab0f54 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.017+0000 7f85c37fe640 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9d70 0x557cd09b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.019+0000 7f85a9ffb640 1 -- 172.21.15.28:0/870546005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc040030 con 0x557cd09a9450 2024-05-09T08:42:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.019+0000 7f85a9ffb640 1 -- 172.21.15.28:0/870546005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85bc003e50 con 0x557cd09a9450 2024-05-09T08:42:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.019+0000 7f85c0ff9640 1 -- 172.21.15.28:0/1216619532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f85b40053b0 con 0x7f85c411b010 2024-05-09T08:42:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.020+0000 7f85cce32740 1 -- 172.21.15.28:0/870546005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09a9450 msgr2=0x557cd09a9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.020+0000 7f85cce32740 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09a9450 0x557cd09a9830 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f85bc034040 tx=0x7f85bc0303a0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.021+0000 7f85c0ff9640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f85ac0a2fd0 0x7f85ac0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.022+0000 7f85c0ff9640 1 -- 172.21.15.28:0/1216619532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9803+0+0 (secure 0 0 0) 0x7f85b40d6250 con 0x7f85c411b010 2024-05-09T08:42:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.024+0000 7f85c3fff640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f85ac0a2fd0 0x7f85ac0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.026+0000 7f85c3fff640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f85ac0a2fd0 0x7f85ac0a5490 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f85bc030bd0 tx=0x7f85bc030f50 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.027+0000 7f85cce32740 1 -- 172.21.15.28:0/870546005 shutdown_connections 2024-05-09T08:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.027+0000 7f85cce32740 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09b4630 0x557cd09b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.027+0000 7f85cce32740 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9d70 0x557cd09b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.027+0000 7f85cce32740 1 --2- 172.21.15.28:0/870546005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09a9450 0x557cd09a9830 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.027+0000 7f85cce32740 1 -- 172.21.15.28:0/870546005 >> 172.21.15.28:0/870546005 conn(0x557cd09a5960 msgr2=0x557cd09a5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.028+0000 7f85cce32740 1 -- 172.21.15.28:0/870546005 shutdown_connections 2024-05-09T08:42:23.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.031+0000 7f85cce32740 1 -- 172.21.15.28:0/870546005 wait complete. 2024-05-09T08:42:23.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.031+0000 7f85cce32740 1 -- start start 2024-05-09T08:42:23.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.031+0000 7f85cce32740 1 Processor -- start 2024-05-09T08:42:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.032+0000 7f85cce32740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9450 0x557cd09b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.032+0000 7f85cce32740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.032+0000 7f85cce32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 0x557cd09b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.032+0000 7f85c37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9450 0x557cd09b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.032+0000 7f85c37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9450 0x557cd09b40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37708/0 (socket says 172.21.15.28:37708) 2024-05-09T08:42:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.033+0000 7f85c37fe640 1 -- 172.21.15.28:0/2228166296 learned_addr learned my addr 172.21.15.28:0/2228166296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.033+0000 7f85c2ffd640 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.033+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557cd09b7200 con 0x557cd09b4630 2024-05-09T08:42:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.033+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x557cd08c9790 con 0x557cd09a9d70 2024-05-09T08:42:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.033+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557cd09b7380 con 0x557cd09a9450 2024-05-09T08:42:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.033+0000 7f85cce32740 10 client.0 _ll_register_callbacks cb 0x557cd0904a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.034+0000 7f85c3fff640 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 0x557cd09b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.034+0000 7f85c37fe640 1 -- 172.21.15.28:0/2228166296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 msgr2=0x557cd09b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.034+0000 7f85c37fe640 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09b1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.034+0000 7f85c37fe640 1 -- 172.21.15.28:0/2228166296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 msgr2=0x557cd09b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.034+0000 7f85c37fe640 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 0x557cd09b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.034+0000 7f85c37fe640 1 -- 172.21.15.28:0/2228166296 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd0622210 con 0x557cd09a9450 2024-05-09T08:42:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.035+0000 7f85c37fe640 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9450 0x557cd09b40a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f85b80126d0 tx=0x7f85b8012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8fb61149b5b5ef4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.036+0000 7f85a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:42:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.036+0000 7f85a2ffd640 1 -- 172.21.15.28:0/2228166296 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b801f060 con 0x557cd09a9450 2024-05-09T08:42:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.036+0000 7f85a2ffd640 1 -- 172.21.15.28:0/2228166296 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9803+0+0 (secure 0 0 0) 0x7f85b8010dc0 con 0x557cd09a9450 2024-05-09T08:42:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.036+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85bc003e50 con 0x557cd09a9450 2024-05-09T08:42:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.037+0000 7f85a2ffd640 1 -- 172.21.15.28:0/2228166296 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f85b801a430 con 0x557cd09a9450 2024-05-09T08:42:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.038+0000 7f85cce32740 10 client.26533 fetch_fsmap learned FSMap version 67 2024-05-09T08:42:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.038+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x557cd06ee620 con 0x557cd09a9450 2024-05-09T08:42:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.040+0000 7f85a2ffd640 1 -- 172.21.15.28:0/2228166296 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85bc003e50 con 0x557cd09a9450 2024-05-09T08:42:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.040+0000 7f85a2ffd640 1 -- 172.21.15.28:0/2228166296 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f85b80235a0 con 0x557cd09a9450 2024-05-09T08:42:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.042+0000 7f85cce32740 10 client.26533 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:42:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.042+0000 7f85cce32740 10 client.26533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:42:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.042+0000 7f85cce32740 20 client.26533 populate_metadata read hostname 'smithi028' 2024-05-09T08:42:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.042+0000 7f85cce32740 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557cd09c42a0 0x557cd09c6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.042+0000 7f85cce32740 4 client.26533 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.042+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cd091ba70 con 0x557cd09c42a0 2024-05-09T08:42:23.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.044+0000 7f85c2ffd640 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557cd09c42a0 0x557cd09c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.052+0000 7f85c2ffd640 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557cd09c42a0 0x557cd09c6700 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c92898afec0f38 server_cookie=df8b80a7012c1b10 in_seq=0 out_seq=0 2024-05-09T08:42:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.052+0000 7f85a2ffd640 10 client.26533 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.056+0000 7f85a2ffd640 1 -- 172.21.15.28:0/2228166296 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557cd091ba70 con 0x557cd09c42a0 2024-05-09T08:42:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.056+0000 7f85a2ffd640 10 client.26533 handle_command_reply: tid=0 2024-05-09T08:42:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.058+0000 7f85cce32740 1 client.26533 shutdown 2024-05-09T08:42:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.062+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9450 msgr2=0x557cd09b40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.062+0000 7f85cce32740 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9450 0x557cd09b40a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f85b80126d0 tx=0x7f85b8012b90 comp rx=0 tx=0).stop 2024-05-09T08:42:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.062+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 shutdown_connections 2024-05-09T08:42:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.062+0000 7f85cce32740 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557cd09c42a0 0x557cd09c6700 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.062+0000 7f85cce32740 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 0x557cd09b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.062+0000 7f85cce32740 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.062+0000 7f85cce32740 1 --2- 172.21.15.28:0/2228166296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09a9450 0x557cd09b40a0 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f85b80126d0 tx=0x7f85b8012b90 comp rx=0 tx=0).stop 2024-05-09T08:42:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.062+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 >> 172.21.15.28:0/2228166296 conn(0x557cd09a5960 msgr2=0x557cd09ab690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:23.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.063+0000 7f85a2ffd640 0 client.26533 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:23.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.066+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 shutdown_connections 2024-05-09T08:42:23.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.070+0000 7f85cce32740 1 -- 172.21.15.28:0/2228166296 wait complete. 2024-05-09T08:42:23.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.073+0000 7f85cce32740 20 client.26533 trim_cache size 0 max 16384 2024-05-09T08:42:23.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.107+0000 7f85cce32740 1 Processor -- start 2024-05-09T08:42:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.108+0000 7f85cce32740 1 -- start start 2024-05-09T08:42:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.108+0000 7f85cce32740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.108+0000 7f85cce32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 0x557cd09a5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.108+0000 7f85cce32740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09cc010 0x557cd09cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.108+0000 7f85cce32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85b00103c0 con 0x557cd09b4630 2024-05-09T08:42:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.108+0000 7f85cce32740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f85b0012140 con 0x557cd09a9d70 2024-05-09T08:42:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.108+0000 7f85cce32740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f85b0010b50 con 0x557cd09cc010 2024-05-09T08:42:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.109+0000 7f85c3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09cc010 0x557cd09cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.109+0000 7f85c3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09cc010 0x557cd09cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37724/0 (socket says 172.21.15.28:37724) 2024-05-09T08:42:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.109+0000 7f85c3fff640 1 -- 172.21.15.28:0/1559884477 learned_addr learned my addr 172.21.15.28:0/1559884477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.109+0000 7f85c2ffd640 1 --2- 172.21.15.28:0/1559884477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 0x557cd09a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.112+0000 7f85c3fff640 1 -- 172.21.15.28:0/1559884477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 msgr2=0x557cd09b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.112+0000 7f85c3fff640 1 --2- 172.21.15.28:0/1559884477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.112+0000 7f85c3fff640 1 -- 172.21.15.28:0/1559884477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 msgr2=0x557cd09a5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.112+0000 7f85c3fff640 1 --2- 172.21.15.28:0/1559884477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 0x557cd09a5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.112+0000 7f85c3fff640 1 -- 172.21.15.28:0/1559884477 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd09b13d0 con 0x557cd09cc010 2024-05-09T08:42:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.113+0000 7f85c3fff640 1 --2- 172.21.15.28:0/1559884477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09cc010 0x557cd09cc450 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f85bc030410 tx=0x7f85bc04b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=639bd82a8aabea34 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.116+0000 7f85a97fa640 1 -- 172.21.15.28:0/1559884477 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc05a070 con 0x557cd09cc010 2024-05-09T08:42:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.116+0000 7f85a97fa640 1 -- 172.21.15.28:0/1559884477 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85bc041040 con 0x557cd09cc010 2024-05-09T08:42:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.116+0000 7f85a97fa640 1 -- 172.21.15.28:0/1559884477 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc040040 con 0x557cd09cc010 2024-05-09T08:42:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.118+0000 7f85cce32740 1 -- 172.21.15.28:0/1559884477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09cc010 msgr2=0x557cd09cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.118+0000 7f85cce32740 1 --2- 172.21.15.28:0/1559884477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09cc010 0x557cd09cc450 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f85bc030410 tx=0x7f85bc04b000 comp rx=0 tx=0).stop 2024-05-09T08:42:23.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.124+0000 7f85cce32740 1 -- 172.21.15.28:0/1559884477 shutdown_connections 2024-05-09T08:42:23.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.124+0000 7f85cce32740 1 --2- 172.21.15.28:0/1559884477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09cc010 0x557cd09cc450 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.124+0000 7f85cce32740 1 --2- 172.21.15.28:0/1559884477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09b4630 0x557cd09a5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.124+0000 7f85cce32740 1 --2- 172.21.15.28:0/1559884477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.124+0000 7f85cce32740 1 -- 172.21.15.28:0/1559884477 >> 172.21.15.28:0/1559884477 conn(0x557cd09a5960 msgr2=0x557cd09a5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.129+0000 7f85cce32740 1 -- 172.21.15.28:0/1559884477 shutdown_connections 2024-05-09T08:42:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.131+0000 7f85cce32740 1 -- 172.21.15.28:0/1559884477 wait complete. 2024-05-09T08:42:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.131+0000 7f85cce32740 1 -- start start 2024-05-09T08:42:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.131+0000 7f85cce32740 1 Processor -- start 2024-05-09T08:42:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.135+0000 7f85cce32740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.135+0000 7f85cce32740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09b4630 0x557cd09ae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.135+0000 7f85cce32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09cc010 0x557cd09aed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.135+0000 7f85cce32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85b0011a70 con 0x557cd09cc010 2024-05-09T08:42:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.135+0000 7f85cce32740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f85b00112e0 con 0x557cd09a9d70 2024-05-09T08:42:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.135+0000 7f85cce32740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f85b000e3a0 con 0x557cd09b4630 2024-05-09T08:42:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.135+0000 7f85cce32740 10 client.0 _ll_register_callbacks cb 0x7f85b8009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:23.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.136+0000 7f85c2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09b4630 0x557cd09ae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.136+0000 7f85c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09cc010 0x557cd09aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.136+0000 7f85c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09cc010 0x557cd09aed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47336/0 (socket says 172.21.15.28:47336) 2024-05-09T08:42:23.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.136+0000 7f85c3fff640 1 -- 172.21.15.28:0/3261936088 learned_addr learned my addr 172.21.15.28:0/3261936088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:23.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.137+0000 7f85c3fff640 1 -- 172.21.15.28:0/3261936088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09b4630 msgr2=0x557cd09ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:23.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.137+0000 7f85c3fff640 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09b4630 0x557cd09ae7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.137+0000 7f85c3fff640 1 -- 172.21.15.28:0/3261936088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 msgr2=0x557cd09ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:23.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.137+0000 7f85c3fff640 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.137+0000 7f85c3fff640 1 -- 172.21.15.28:0/3261936088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd09b1ee0 con 0x557cd09cc010 2024-05-09T08:42:23.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.138+0000 7f85c3fff640 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09cc010 0x557cd09aed30 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f85bc02a7d0 tx=0x7f85bc030e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab4a200271d72b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:23.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.140+0000 7f85a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:42:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.142+0000 7f85a2ffd640 1 -- 172.21.15.28:0/3261936088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc063060 con 0x557cd09cc010 2024-05-09T08:42:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.142+0000 7f85cce32740 1 -- 172.21.15.28:0/3261936088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85bc041040 con 0x557cd09cc010 2024-05-09T08:42:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.142+0000 7f85a2ffd640 1 -- 172.21.15.28:0/3261936088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9803+0+0 (secure 0 0 0) 0x7f85bc004630 con 0x557cd09cc010 2024-05-09T08:42:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.148+0000 7f85a2ffd640 1 -- 172.21.15.28:0/3261936088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f85bc05a070 con 0x557cd09cc010 2024-05-09T08:42:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.148+0000 7f85cce32740 10 client.17247 fetch_fsmap learned FSMap version 67 2024-05-09T08:42:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.148+0000 7f85cce32740 1 -- 172.21.15.28:0/3261936088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x557cd0a2aa90 con 0x557cd09cc010 2024-05-09T08:42:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.149+0000 7f85a2ffd640 1 -- 172.21.15.28:0/3261936088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85bc041040 con 0x557cd09cc010 2024-05-09T08:42:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.149+0000 7f85a2ffd640 1 -- 172.21.15.28:0/3261936088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f85bc04f070 con 0x557cd09cc010 2024-05-09T08:42:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.149+0000 7f85cce32740 10 client.17247 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:42:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.149+0000 7f85cce32740 10 client.17247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:42:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.149+0000 7f85cce32740 20 client.17247 populate_metadata read hostname 'smithi028' 2024-05-09T08:42:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.149+0000 7f85cce32740 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557cd0a2b0f0 0x557cd0a2d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.149+0000 7f85cce32740 4 client.17247 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:42:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.149+0000 7f85cce32740 1 -- 172.21.15.28:0/3261936088 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557cd0915dc0 con 0x557cd0a2b0f0 2024-05-09T08:42:23.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.152+0000 7f85c37fe640 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557cd0a2b0f0 0x557cd0a2d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.160+0000 7f85c37fe640 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557cd0a2b0f0 0x557cd0a2d510 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=180a977ade9414de server_cookie=45d73b95d3b607f3 in_seq=0 out_seq=0 2024-05-09T08:42:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.161+0000 7f85a2ffd640 10 client.17247 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85a2ffd640 1 -- 172.21.15.28:0/3261936088 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557cd0915dc0 con 0x557cd0a2b0f0 2024-05-09T08:42:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85a2ffd640 10 client.17247 handle_command_reply: tid=0 2024-05-09T08:42:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85cce32740 1 client.17247 shutdown 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85cce32740 1 -- 172.21.15.28:0/3261936088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09cc010 msgr2=0x557cd09aed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85cce32740 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09cc010 0x557cd09aed30 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f85bc02a7d0 tx=0x7f85bc030e00 comp rx=0 tx=0).stop 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85cce32740 1 -- 172.21.15.28:0/3261936088 shutdown_connections 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85cce32740 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x557cd0a2b0f0 0x557cd0a2d510 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85cce32740 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557cd09cc010 0x557cd09aed30 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85cce32740 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557cd09b4630 0x557cd09ae7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85cce32740 1 --2- 172.21.15.28:0/3261936088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557cd09a9d70 0x557cd09ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.517+0000 7f85cce32740 1 -- 172.21.15.28:0/3261936088 >> 172.21.15.28:0/3261936088 conn(0x557cd09a5960 msgr2=0x557cd09aae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.518+0000 7f85a2ffd640 0 client.17247 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.518+0000 7f85cce32740 1 -- 172.21.15.28:0/3261936088 shutdown_connections 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.518+0000 7f85cce32740 1 -- 172.21.15.28:0/3261936088 wait complete. 2024-05-09T08:42:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.518+0000 7f85cce32740 20 client.17247 trim_cache size 0 max 16384 2024-05-09T08:42:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:42:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:42:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:42:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:42:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 -- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f85ac0a2fd0 msgr2=0x7f85ac0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f85ac0a2fd0 0x7f85ac0a5490 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f85bc030bd0 tx=0x7f85bc030f50 comp rx=0 tx=0).stop 2024-05-09T08:42:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 -- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85c411b010 msgr2=0x7f85c4111660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85c411b010 0x7f85c4111660 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f85b400d6a0 tx=0x7f85b400db60 comp rx=0 tx=0).stop 2024-05-09T08:42:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 -- 172.21.15.28:0/1216619532 shutdown_connections 2024-05-09T08:42:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f85ac0a2fd0 0x7f85ac0a5490 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85c4118be0 0x7f85c4111c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85c411b010 0x7f85c4111660 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 --2- 172.21.15.28:0/1216619532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c4110020 0x7f85c4118590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 -- 172.21.15.28:0/1216619532 >> 172.21.15.28:0/1216619532 conn(0x7f85c406c900 msgr2=0x7f85c410dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 -- 172.21.15.28:0/1216619532 shutdown_connections 2024-05-09T08:42:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:23.519+0000 7f85ca122640 1 -- 172.21.15.28:0/1216619532 wait complete. 2024-05-09T08:42:24.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:23 smithi184 ceph-mon[34509]: pgmap v515: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 896 KiB/s wr, 156 op/s 2024-05-09T08:42:24.109 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:23 smithi165 ceph-mon[34540]: pgmap v515: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 896 KiB/s wr, 156 op/s 2024-05-09T08:42:24.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:23 smithi028 ceph-mon[30119]: pgmap v515: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 896 KiB/s wr, 156 op/s 2024-05-09T08:42:26.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:25 smithi165 ceph-mon[34540]: pgmap v516: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 171 op/s 2024-05-09T08:42:26.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:25 smithi165 ceph-mon[34540]: osdmap e125: 12 total, 12 up, 12 in 2024-05-09T08:42:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:25 smithi184 ceph-mon[34509]: pgmap v516: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 171 op/s 2024-05-09T08:42:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:25 smithi184 ceph-mon[34509]: osdmap e125: 12 total, 12 up, 12 in 2024-05-09T08:42:26.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:25 smithi028 ceph-mon[30119]: pgmap v516: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 171 op/s 2024-05-09T08:42:26.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:25 smithi028 ceph-mon[30119]: osdmap e125: 12 total, 12 up, 12 in 2024-05-09T08:42:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:27 smithi165 ceph-mon[34540]: pgmap v518: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 163 op/s 2024-05-09T08:42:28.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:27 smithi184 ceph-mon[34509]: pgmap v518: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 163 op/s 2024-05-09T08:42:28.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:27 smithi028 ceph-mon[30119]: pgmap v518: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 163 op/s 2024-05-09T08:42:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:29 smithi165 ceph-mon[34540]: pgmap v519: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-09T08:42:30.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:29 smithi184 ceph-mon[34509]: pgmap v519: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-09T08:42:30.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:29 smithi028 ceph-mon[30119]: pgmap v519: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-09T08:42:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:31 smithi165 ceph-mon[34540]: pgmap v520: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 199 op/s 2024-05-09T08:42:32.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:31 smithi184 ceph-mon[34509]: pgmap v520: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 199 op/s 2024-05-09T08:42:32.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:31 smithi028 ceph-mon[30119]: pgmap v520: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 199 op/s 2024-05-09T08:42:34.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:33 smithi165 ceph-mon[34540]: pgmap v521: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 207 op/s 2024-05-09T08:42:34.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:33 smithi184 ceph-mon[34509]: pgmap v521: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 207 op/s 2024-05-09T08:42:34.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:33 smithi028 ceph-mon[30119]: pgmap v521: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 207 op/s 2024-05-09T08:42:36.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:35 smithi165 ceph-mon[34540]: pgmap v522: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 869 KiB/s wr, 213 op/s 2024-05-09T08:42:36.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:35 smithi184 ceph-mon[34509]: pgmap v522: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 869 KiB/s wr, 213 op/s 2024-05-09T08:42:36.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:35 smithi028 ceph-mon[30119]: pgmap v522: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 869 KiB/s wr, 213 op/s 2024-05-09T08:42:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.679+0000 7fc436858640 1 -- 172.21.15.28:0/584982917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43011b010 msgr2=0x7fc43011d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.679+0000 7fc436858640 1 --2- 172.21.15.28:0/584982917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43011b010 0x7fc43011d400 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc420009890 tx=0x7fc42002a440 comp rx=0 tx=0).stop 2024-05-09T08:42:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.683+0000 7fc436858640 1 -- 172.21.15.28:0/584982917 shutdown_connections 2024-05-09T08:42:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.683+0000 7fc436858640 1 --2- 172.21.15.28:0/584982917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43011b010 0x7fc43011d400 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.683+0000 7fc436858640 1 --2- 172.21.15.28:0/584982917 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc430110400 0x7fc430108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.683+0000 7fc436858640 1 --2- 172.21.15.28:0/584982917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc430110020 0x7fc430108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.683+0000 7fc436858640 1 -- 172.21.15.28:0/584982917 >> 172.21.15.28:0/584982917 conn(0x7fc43006c900 msgr2=0x7fc43006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.686+0000 7fc436858640 1 -- 172.21.15.28:0/584982917 shutdown_connections 2024-05-09T08:42:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.690+0000 7fc436858640 1 -- 172.21.15.28:0/584982917 wait complete. 2024-05-09T08:42:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.691+0000 7fc436858640 1 Processor -- start 2024-05-09T08:42:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.691+0000 7fc436858640 1 -- start start 2024-05-09T08:42:36.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.695+0000 7fc436858640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc430110020 0x7fc4301a6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.695+0000 7fc436858640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43011b010 0x7fc4301a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.695+0000 7fc436858640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4301ab7c0 0x7fc4301abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.695+0000 7fc436858640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc430120a80 con 0x7fc43011b010 2024-05-09T08:42:36.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.696+0000 7fc436858640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc430120900 con 0x7fc430110020 2024-05-09T08:42:36.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.696+0000 7fc434df5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4301ab7c0 0x7fc4301abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.696+0000 7fc434df5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4301ab7c0 0x7fc4301abc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:60026/0 (socket says 172.21.15.28:60026) 2024-05-09T08:42:36.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.696+0000 7fc434df5640 1 -- 172.21.15.28:0/1455838181 learned_addr learned my addr 172.21.15.28:0/1455838181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:36.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.696+0000 7fc436858640 1 -- 172.21.15.28:0/1455838181 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc430120c00 con 0x7fc4301ab7c0 2024-05-09T08:42:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.700+0000 7fc42ffff640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc430110020 0x7fc4301a6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.701+0000 7fc42f7fe640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43011b010 0x7fc4301a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.702+0000 7fc434df5640 1 -- 172.21.15.28:0/1455838181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc430110020 msgr2=0x7fc4301a6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.702+0000 7fc434df5640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc430110020 0x7fc4301a6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.702+0000 7fc434df5640 1 -- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43011b010 msgr2=0x7fc4301a7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.702+0000 7fc434df5640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43011b010 0x7fc4301a7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.702+0000 7fc434df5640 1 -- 172.21.15.28:0/1455838181 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4301ac2b0 con 0x7fc4301ab7c0 2024-05-09T08:42:36.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.705+0000 7fc434df5640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4301ab7c0 0x7fc4301abc70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc420030940 tx=0x7fc420030970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.708+0000 7fc42d7fa640 1 -- 172.21.15.28:0/1455838181 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420005520 con 0x7fc4301ab7c0 2024-05-09T08:42:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.708+0000 7fc42d7fa640 1 -- 172.21.15.28:0/1455838181 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc420031400 con 0x7fc4301ab7c0 2024-05-09T08:42:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.708+0000 7fc42d7fa640 1 -- 172.21.15.28:0/1455838181 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420004920 con 0x7fc4301ab7c0 2024-05-09T08:42:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.710+0000 7fc436858640 1 -- 172.21.15.28:0/1455838181 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4301ac540 con 0x7fc4301ab7c0 2024-05-09T08:42:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.710+0000 7fc436858640 1 -- 172.21.15.28:0/1455838181 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4301b3c50 con 0x7fc4301ab7c0 2024-05-09T08:42:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.716+0000 7fc439564740 1 Processor -- start 2024-05-09T08:42:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.717+0000 7fc439564740 1 -- start start 2024-05-09T08:42:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.717+0000 7fc439564740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73450 0x563937a73830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.717+0000 7fc439564740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73d70 0x563937a7e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.717+0000 7fc439564740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a7e630 0x563937a80a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.717+0000 7fc439564740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5639379d8340 con 0x563937a7e630 2024-05-09T08:42:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.717+0000 7fc439564740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x563937980290 con 0x563937a73d70 2024-05-09T08:42:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.717+0000 7fc439564740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5639379d4300 con 0x563937a73450 2024-05-09T08:42:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.719+0000 7fc42ffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73d70 0x563937a7e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.719+0000 7fc42d7fa640 1 -- 172.21.15.28:0/1455838181 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc420039040 con 0x7fc4301ab7c0 2024-05-09T08:42:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.719+0000 7fc42ffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73d70 0x563937a7e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:42220/0 (socket says 172.21.15.28:42220) 2024-05-09T08:42:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.719+0000 7fc42ffff640 1 -- 172.21.15.28:0/707516091 learned_addr learned my addr 172.21.15.28:0/707516091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.720+0000 7fc42ffff640 1 -- 172.21.15.28:0/707516091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73450 msgr2=0x563937a73830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.720+0000 7fc42ffff640 1 --2- 172.21.15.28:0/707516091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73450 0x563937a73830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.720+0000 7fc42ffff640 1 -- 172.21.15.28:0/707516091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a7e630 msgr2=0x563937a80a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:42:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.720+0000 7fc42ffff640 1 --2- 172.21.15.28:0/707516091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a7e630 0x563937a80a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.720+0000 7fc42ffff640 1 -- 172.21.15.28:0/707516091 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639379a48e0 con 0x563937a73d70 2024-05-09T08:42:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.720+0000 7fc42d7fa640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4180a2fc0 0x7fc4180a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.721+0000 7fc42ffff640 1 --2- 172.21.15.28:0/707516091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73d70 0x563937a7e0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc42400baa0 tx=0x7fc42400bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ece11ccaaad1b726 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.721+0000 7fc42d7fa640 1 -- 172.21.15.28:0/1455838181 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9787+0+0 (secure 0 0 0) 0x7fc4200f0880 con 0x7fc4301ab7c0 2024-05-09T08:42:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.721+0000 7fc42f7fe640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4180a2fc0 0x7fc4180a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.722+0000 7fc4167fc640 1 -- 172.21.15.28:0/707516091 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc424017420 con 0x563937a73d70 2024-05-09T08:42:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.722+0000 7fc4167fc640 1 -- 172.21.15.28:0/707516091 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc424017aa0 con 0x563937a73d70 2024-05-09T08:42:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc4167fc640 1 -- 172.21.15.28:0/707516091 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42400fc30 con 0x563937a73d70 2024-05-09T08:42:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc439564740 1 -- 172.21.15.28:0/707516091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73d70 msgr2=0x563937a7e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc42f7fe640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4180a2fc0 0x7fc4180a5480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x563937a70d80 tx=0x7fc428009820 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc439564740 1 --2- 172.21.15.28:0/707516091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73d70 0x563937a7e0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc42400baa0 tx=0x7fc42400bf60 comp rx=0 tx=0).stop 2024-05-09T08:42:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc439564740 1 -- 172.21.15.28:0/707516091 shutdown_connections 2024-05-09T08:42:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc439564740 1 --2- 172.21.15.28:0/707516091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a7e630 0x563937a80a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc439564740 1 --2- 172.21.15.28:0/707516091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73d70 0x563937a7e0f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc439564740 1 --2- 172.21.15.28:0/707516091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73450 0x563937a73830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc439564740 1 -- 172.21.15.28:0/707516091 >> 172.21.15.28:0/707516091 conn(0x563937a6f960 msgr2=0x563937a6fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc439564740 1 -- 172.21.15.28:0/707516091 shutdown_connections 2024-05-09T08:42:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.723+0000 7fc439564740 1 -- 172.21.15.28:0/707516091 wait complete. 2024-05-09T08:42:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.724+0000 7fc439564740 1 -- start start 2024-05-09T08:42:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.724+0000 7fc439564740 1 Processor -- start 2024-05-09T08:42:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.724+0000 7fc439564740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73450 0x563937a7b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc439564740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73d70 0x563937a7ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc439564740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a7e630 0x563937a7bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc439564740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563937a81260 con 0x563937a73d70 2024-05-09T08:42:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc439564740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x563937993790 con 0x563937a73450 2024-05-09T08:42:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc439564740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563937a813e0 con 0x563937a7e630 2024-05-09T08:42:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc439564740 10 client.0 _ll_register_callbacks cb 0x5639379cea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc42f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73d70 0x563937a7ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc42f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73d70 0x563937a7ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38560/0 (socket says 172.21.15.28:38560) 2024-05-09T08:42:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc42f7fe640 1 -- 172.21.15.28:0/3503425200 learned_addr learned my addr 172.21.15.28:0/3503425200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.725+0000 7fc434df5640 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a7e630 0x563937a7bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.726+0000 7fc42f7fe640 1 -- 172.21.15.28:0/3503425200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a7e630 msgr2=0x563937a7bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.726+0000 7fc42f7fe640 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a7e630 0x563937a7bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.726+0000 7fc42f7fe640 1 -- 172.21.15.28:0/3503425200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73450 msgr2=0x563937a7b530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:42:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.726+0000 7fc42f7fe640 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73450 0x563937a7b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.726+0000 7fc42f7fe640 1 -- 172.21.15.28:0/3503425200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639376ec210 con 0x563937a73d70 2024-05-09T08:42:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.727+0000 7fc42f7fe640 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73d70 0x563937a7ba70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc42800edd0 tx=0x7fc428004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db52b3b9b7e3933a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.727+0000 7fc40f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:42:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.727+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3503425200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42801bb20 con 0x563937a73d70 2024-05-09T08:42:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.728+0000 7fc439564740 1 -- 172.21.15.28:0/3503425200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc424017aa0 con 0x563937a73d70 2024-05-09T08:42:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.728+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3503425200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9787+0+0 (secure 0 0 0) 0x7fc42801bcc0 con 0x563937a73d70 2024-05-09T08:42:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.728+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3503425200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fc428030050 con 0x563937a73d70 2024-05-09T08:42:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.731+0000 7fc439564740 10 client.17256 fetch_fsmap learned FSMap version 67 2024-05-09T08:42:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.731+0000 7fc439564740 1 -- 172.21.15.28:0/3503425200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5639377b8620 con 0x563937a73d70 2024-05-09T08:42:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.736+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3503425200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc424017aa0 con 0x563937a73d70 2024-05-09T08:42:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.736+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3503425200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fc42800b040 con 0x563937a73d70 2024-05-09T08:42:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.736+0000 7fc439564740 10 client.17256 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:42:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.736+0000 7fc439564740 10 client.17256 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:42:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.736+0000 7fc439564740 20 client.17256 populate_metadata read hostname 'smithi028' 2024-05-09T08:42:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.736+0000 7fc439564740 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563937a9f8e0 0x563937aa1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.736+0000 7fc439564740 4 client.17256 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.736+0000 7fc439564740 1 -- 172.21.15.28:0/3503425200 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639379e5a70 con 0x563937a9f8e0 2024-05-09T08:42:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.739+0000 7fc42ffff640 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563937a9f8e0 0x563937aa1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.742+0000 7fc42ffff640 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563937a9f8e0 0x563937aa1da0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77eceab3669775a5 server_cookie=6d8d72210c4b9928 in_seq=0 out_seq=0 2024-05-09T08:42:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.743+0000 7fc40f7fe640 10 client.17256 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.748+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3503425200 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639379e5a70 con 0x563937a9f8e0 2024-05-09T08:42:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.748+0000 7fc40f7fe640 10 client.17256 handle_command_reply: tid=0 2024-05-09T08:42:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.748+0000 7fc439564740 1 client.17256 shutdown 2024-05-09T08:42:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.749+0000 7fc439564740 1 -- 172.21.15.28:0/3503425200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73d70 msgr2=0x563937a7ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.749+0000 7fc439564740 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73d70 0x563937a7ba70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc42800edd0 tx=0x7fc428004ab0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.749+0000 7fc439564740 1 -- 172.21.15.28:0/3503425200 shutdown_connections 2024-05-09T08:42:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.749+0000 7fc439564740 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563937a9f8e0 0x563937aa1da0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.749+0000 7fc439564740 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a7e630 0x563937a7bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.749+0000 7fc439564740 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73d70 0x563937a7ba70 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.749+0000 7fc439564740 1 --2- 172.21.15.28:0/3503425200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a73450 0x563937a7b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.749+0000 7fc439564740 1 -- 172.21.15.28:0/3503425200 >> 172.21.15.28:0/3503425200 conn(0x563937a6f960 msgr2=0x563937a75690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.750+0000 7fc40f7fe640 0 client.17256 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.750+0000 7fc439564740 1 -- 172.21.15.28:0/3503425200 shutdown_connections 2024-05-09T08:42:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.750+0000 7fc439564740 1 -- 172.21.15.28:0/3503425200 wait complete. 2024-05-09T08:42:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.751+0000 7fc439564740 20 client.17256 trim_cache size 0 max 16384 2024-05-09T08:42:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.791+0000 7fc439564740 1 Processor -- start 2024-05-09T08:42:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.791+0000 7fc439564740 1 -- start start 2024-05-09T08:42:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.791+0000 7fc439564740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73450 0x563937a8a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.791+0000 7fc439564740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73d70 0x563937a7a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.791+0000 7fc439564740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a7e630 0x563937a6f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.791+0000 7fc439564740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301b7cc0 con 0x563937a73450 2024-05-09T08:42:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.791+0000 7fc439564740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301b57b0 con 0x563937a7e630 2024-05-09T08:42:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.791+0000 7fc439564740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301b7530 con 0x563937a73d70 2024-05-09T08:42:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.792+0000 7fc434df5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a7e630 0x563937a6f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.792+0000 7fc434df5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a7e630 0x563937a6f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:42222/0 (socket says 172.21.15.28:42222) 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.792+0000 7fc434df5640 1 -- 172.21.15.28:0/2024032480 learned_addr learned my addr 172.21.15.28:0/2024032480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.792+0000 7fc434df5640 1 -- 172.21.15.28:0/2024032480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73d70 msgr2=0x563937a7a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.792+0000 7fc434df5640 1 --2- 172.21.15.28:0/2024032480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73d70 0x563937a7a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.792+0000 7fc434df5640 1 -- 172.21.15.28:0/2024032480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73450 msgr2=0x563937a8a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.792+0000 7fc434df5640 1 --2- 172.21.15.28:0/2024032480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73450 0x563937a8a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.792+0000 7fc434df5640 1 -- 172.21.15.28:0/2024032480 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563937a80d90 con 0x563937a7e630 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.792+0000 7fc434df5640 1 --2- 172.21.15.28:0/2024032480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a7e630 0x563937a6f180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc420004650 tx=0x7fc4200bd090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=511f3cdd6f2e4a56 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.793+0000 7fc415ffb640 1 -- 172.21.15.28:0/2024032480 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4200da070 con 0x563937a7e630 2024-05-09T08:42:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.793+0000 7fc415ffb640 1 -- 172.21.15.28:0/2024032480 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4200bab50 con 0x563937a7e630 2024-05-09T08:42:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.794+0000 7fc439564740 1 -- 172.21.15.28:0/2024032480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a7e630 msgr2=0x563937a6f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.794+0000 7fc439564740 1 --2- 172.21.15.28:0/2024032480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a7e630 0x563937a6f180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc420004650 tx=0x7fc4200bd090 comp rx=0 tx=0).stop 2024-05-09T08:42:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.794+0000 7fc439564740 1 -- 172.21.15.28:0/2024032480 shutdown_connections 2024-05-09T08:42:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.794+0000 7fc439564740 1 --2- 172.21.15.28:0/2024032480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a7e630 0x563937a6f180 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.794+0000 7fc439564740 1 --2- 172.21.15.28:0/2024032480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73d70 0x563937a7a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.794+0000 7fc439564740 1 --2- 172.21.15.28:0/2024032480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a73450 0x563937a8a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.794+0000 7fc439564740 1 -- 172.21.15.28:0/2024032480 >> 172.21.15.28:0/2024032480 conn(0x563937a6f960 msgr2=0x563937a74e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.798+0000 7fc439564740 1 -- 172.21.15.28:0/2024032480 shutdown_connections 2024-05-09T08:42:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.803+0000 7fc439564740 1 -- 172.21.15.28:0/2024032480 wait complete. 2024-05-09T08:42:36.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.803+0000 7fc439564740 1 -- start start 2024-05-09T08:42:36.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.803+0000 7fc439564740 1 Processor -- start 2024-05-09T08:42:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.805+0000 7fc439564740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73d70 0x563937a86680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.805+0000 7fc439564740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a7e630 0x563937a86bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.805+0000 7fc439564740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a778a0 0x563937a77d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.805+0000 7fc439564740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301b66d0 con 0x563937a7e630 2024-05-09T08:42:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.805+0000 7fc439564740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301b5f40 con 0x563937a778a0 2024-05-09T08:42:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.805+0000 7fc439564740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301b6e60 con 0x563937a73d70 2024-05-09T08:42:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.805+0000 7fc439564740 10 client.0 _ll_register_callbacks cb 0x7fc4280095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.806+0000 7fc42ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73d70 0x563937a86680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.806+0000 7fc434df5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a778a0 0x563937a77d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.806+0000 7fc434df5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a778a0 0x563937a77d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:42236/0 (socket says 172.21.15.28:42236) 2024-05-09T08:42:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.806+0000 7fc434df5640 1 -- 172.21.15.28:0/3543305530 learned_addr learned my addr 172.21.15.28:0/3543305530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc434df5640 1 -- 172.21.15.28:0/3543305530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73d70 msgr2=0x563937a86680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc434df5640 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73d70 0x563937a86680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc434df5640 1 -- 172.21.15.28:0/3543305530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a7e630 msgr2=0x563937a86bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc434df5640 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a7e630 0x563937a86bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc434df5640 1 -- 172.21.15.28:0/3543305530 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563937a78410 con 0x563937a778a0 2024-05-09T08:42:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc434df5640 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a778a0 0x563937a77d20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc4200b9220 tx=0x7fc420031b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12d9ca90e59d15e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc40f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:42:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3543305530 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4200da070 con 0x563937a778a0 2024-05-09T08:42:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc439564740 1 -- 172.21.15.28:0/3543305530 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4200bab50 con 0x563937a778a0 2024-05-09T08:42:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.810+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3543305530 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9787+0+0 (secure 0 0 0) 0x7fc4200edd00 con 0x563937a778a0 2024-05-09T08:42:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.811+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3543305530 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fc4200edf00 con 0x563937a778a0 2024-05-09T08:42:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.811+0000 7fc439564740 10 client.26708 fetch_fsmap learned FSMap version 67 2024-05-09T08:42:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.811+0000 7fc439564740 1 -- 172.21.15.28:0/3543305530 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x563937b05d70 con 0x563937a778a0 2024-05-09T08:42:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.812+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3543305530 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4200bab50 con 0x563937a778a0 2024-05-09T08:42:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.812+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3543305530 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fc42004c060 con 0x563937a778a0 2024-05-09T08:42:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.812+0000 7fc439564740 10 client.26708 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:42:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.812+0000 7fc439564740 10 client.26708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:42:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.812+0000 7fc439564740 20 client.26708 populate_metadata read hostname 'smithi028' 2024-05-09T08:42:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.812+0000 7fc439564740 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563937b062d0 0x563937b086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.812+0000 7fc439564740 4 client.26708 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:42:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.812+0000 7fc439564740 1 -- 172.21.15.28:0/3543305530 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5639379dfdc0 con 0x563937b062d0 2024-05-09T08:42:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.812+0000 7fc42ffff640 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563937b062d0 0x563937b086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.815+0000 7fc42ffff640 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563937b062d0 0x563937b086f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6d1bbb74bb24453 server_cookie=9fbd63d696d8508c in_seq=0 out_seq=0 2024-05-09T08:42:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.815+0000 7fc40f7fe640 10 client.26708 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:36.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc40f7fe640 1 -- 172.21.15.28:0/3543305530 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5639379dfdc0 con 0x563937b062d0 2024-05-09T08:42:36.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc40f7fe640 10 client.26708 handle_command_reply: tid=0 2024-05-09T08:42:36.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 client.26708 shutdown 2024-05-09T08:42:36.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 -- 172.21.15.28:0/3543305530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a778a0 msgr2=0x563937a77d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a778a0 0x563937a77d20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc4200b9220 tx=0x7fc420031b70 comp rx=0 tx=0).stop 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 -- 172.21.15.28:0/3543305530 shutdown_connections 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563937b062d0 0x563937b086f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563937a778a0 0x563937a77d20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563937a7e630 0x563937a86bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 --2- 172.21.15.28:0/3543305530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563937a73d70 0x563937a86680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 -- 172.21.15.28:0/3543305530 >> 172.21.15.28:0/3543305530 conn(0x563937a6f960 msgr2=0x563937a75690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc40f7fe640 0 client.26708 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 -- 172.21.15.28:0/3543305530 shutdown_connections 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.976+0000 7fc439564740 1 -- 172.21.15.28:0/3543305530 wait complete. 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.977+0000 7fc439564740 20 client.26708 trim_cache size 0 max 16384 2024-05-09T08:42:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:42:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:42:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:42:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:42:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 -- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4180a2fc0 msgr2=0x7fc4180a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4180a2fc0 0x7fc4180a5480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x563937a70d80 tx=0x7fc428009820 comp rx=0 tx=0).stop 2024-05-09T08:42:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 -- 172.21.15.28:0/1455838181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4301ab7c0 msgr2=0x7fc4301abc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4301ab7c0 0x7fc4301abc70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc420030940 tx=0x7fc420030970 comp rx=0 tx=0).stop 2024-05-09T08:42:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 -- 172.21.15.28:0/1455838181 shutdown_connections 2024-05-09T08:42:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4180a2fc0 0x7fc4180a5480 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4301ab7c0 0x7fc4301abc70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43011b010 0x7fc4301a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 --2- 172.21.15.28:0/1455838181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc430110020 0x7fc4301a6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.978+0000 7fc436858640 1 -- 172.21.15.28:0/1455838181 >> 172.21.15.28:0/1455838181 conn(0x7fc43006c900 msgr2=0x7fc43010dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.980+0000 7fc436858640 1 -- 172.21.15.28:0/1455838181 shutdown_connections 2024-05-09T08:42:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:36.982+0000 7fc436858640 1 -- 172.21.15.28:0/1455838181 wait complete. 2024-05-09T08:42:38.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:37 smithi165 ceph-mon[34540]: pgmap v523: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 932 KiB/s wr, 222 op/s 2024-05-09T08:42:38.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:37 smithi184 ceph-mon[34509]: pgmap v523: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 932 KiB/s wr, 222 op/s 2024-05-09T08:42:38.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:37 smithi028 ceph-mon[30119]: pgmap v523: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 932 KiB/s wr, 222 op/s 2024-05-09T08:42:40.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:39 smithi165 ceph-mon[34540]: pgmap v524: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 817 KiB/s wr, 207 op/s 2024-05-09T08:42:40.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:39 smithi184 ceph-mon[34509]: pgmap v524: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 817 KiB/s wr, 207 op/s 2024-05-09T08:42:40.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:39 smithi028 ceph-mon[30119]: pgmap v524: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 817 KiB/s wr, 207 op/s 2024-05-09T08:42:42.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:41 smithi165 ceph-mon[34540]: pgmap v525: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 477 KiB/s wr, 204 op/s 2024-05-09T08:42:42.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:41 smithi184 ceph-mon[34509]: pgmap v525: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 477 KiB/s wr, 204 op/s 2024-05-09T08:42:42.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:41 smithi028 ceph-mon[30119]: pgmap v525: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 477 KiB/s wr, 204 op/s 2024-05-09T08:42:44.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:43 smithi165 ceph-mon[34540]: pgmap v526: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 727 KiB/s wr, 205 op/s 2024-05-09T08:42:44.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:43 smithi184 ceph-mon[34509]: pgmap v526: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 727 KiB/s wr, 205 op/s 2024-05-09T08:42:44.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:43 smithi028 ceph-mon[30119]: pgmap v526: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 727 KiB/s wr, 205 op/s 2024-05-09T08:42:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:45 smithi165 ceph-mon[34540]: pgmap v527: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 720 KiB/s wr, 216 op/s 2024-05-09T08:42:46.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:45 smithi184 ceph-mon[34509]: pgmap v527: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 720 KiB/s wr, 216 op/s 2024-05-09T08:42:46.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:45 smithi028 ceph-mon[30119]: pgmap v527: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 720 KiB/s wr, 216 op/s 2024-05-09T08:42:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:47 smithi165 ceph-mon[34540]: pgmap v528: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 860 KiB/s wr, 217 op/s 2024-05-09T08:42:48.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:47 smithi184 ceph-mon[34509]: pgmap v528: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 860 KiB/s wr, 217 op/s 2024-05-09T08:42:48.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:47 smithi028 ceph-mon[30119]: pgmap v528: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 860 KiB/s wr, 217 op/s 2024-05-09T08:42:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:49 smithi165 ceph-mon[34540]: pgmap v529: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 742 KiB/s wr, 217 op/s 2024-05-09T08:42:50.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:49 smithi184 ceph-mon[34509]: pgmap v529: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 742 KiB/s wr, 217 op/s 2024-05-09T08:42:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.154+0000 7f62a6ca5640 1 -- 172.21.15.28:0/925342745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 msgr2=0x7f62a0108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.154+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/925342745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 0x7f62a0108470 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6288009900 tx=0x7f628802a410 comp rx=0 tx=0).stop 2024-05-09T08:42:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.157+0000 7f62a6ca5640 1 -- 172.21.15.28:0/925342745 shutdown_connections 2024-05-09T08:42:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.157+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/925342745 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62a010d160 0x7f62a0114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.157+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/925342745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a0108a40 0x7f62a010ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.157+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/925342745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 0x7f62a0108470 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.157+0000 7f62a6ca5640 1 -- 172.21.15.28:0/925342745 >> 172.21.15.28:0/925342745 conn(0x7f62a0075dc0 msgr2=0x7f62a00781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:50.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.164+0000 7f62a6ca5640 1 -- 172.21.15.28:0/925342745 shutdown_connections 2024-05-09T08:42:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.166+0000 7f62a6ca5640 1 -- 172.21.15.28:0/925342745 wait complete. 2024-05-09T08:42:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.167+0000 7f62a6ca5640 1 Processor -- start 2024-05-09T08:42:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.168+0000 7f62a6ca5640 1 -- start start 2024-05-09T08:42:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.168+0000 7f62a6ca5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 0x7f62a0072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.168+0000 7f62a6ca5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62a010d160 0x7f62a0073470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.168+0000 7f62a6ca5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006c0a0 0x7f62a006c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.168+0000 7f62a6ca5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62a0118240 con 0x7f62a006c0a0 2024-05-09T08:42:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.168+0000 7f62a6ca5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f62a01180c0 con 0x7f62a0108090 2024-05-09T08:42:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.168+0000 7f62a6ca5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62a01183c0 con 0x7f62a010d160 2024-05-09T08:42:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.169+0000 7f62a4a41640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 0x7f62a0072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.169+0000 7f62a4a41640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 0x7f62a0072f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:53944/0 (socket says 172.21.15.28:53944) 2024-05-09T08:42:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.169+0000 7f62a4a41640 1 -- 172.21.15.28:0/2477535327 learned_addr learned my addr 172.21.15.28:0/2477535327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.171+0000 7f629ffff640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62a010d160 0x7f62a0073470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.171+0000 7f62a4a41640 1 -- 172.21.15.28:0/2477535327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62a010d160 msgr2=0x7f62a0073470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:50.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.171+0000 7f62a4a41640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62a010d160 0x7f62a0073470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.171+0000 7f62a4a41640 1 -- 172.21.15.28:0/2477535327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006c0a0 msgr2=0x7f62a006c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:50.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.171+0000 7f62a4a41640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006c0a0 0x7f62a006c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.171+0000 7f62a4a41640 1 -- 172.21.15.28:0/2477535327 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a006cde0 con 0x7f62a0108090 2024-05-09T08:42:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.174+0000 7f62a4a41640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 0x7f62a0072f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f628802a980 tx=0x7f6288003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.178+0000 7f629dffb640 1 -- 172.21.15.28:0/2477535327 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628803f070 con 0x7f62a0108090 2024-05-09T08:42:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.178+0000 7f629dffb640 1 -- 172.21.15.28:0/2477535327 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6288031c20 con 0x7f62a0108090 2024-05-09T08:42:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.178+0000 7f629dffb640 1 -- 172.21.15.28:0/2477535327 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288035030 con 0x7f62a0108090 2024-05-09T08:42:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.180+0000 7f62a6ca5640 1 -- 172.21.15.28:0/2477535327 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a0070ab0 con 0x7f62a0108090 2024-05-09T08:42:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.180+0000 7f62a6ca5640 1 -- 172.21.15.28:0/2477535327 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a0070f50 con 0x7f62a0108090 2024-05-09T08:42:50.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:49 smithi028 ceph-mon[30119]: pgmap v529: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 742 KiB/s wr, 217 op/s 2024-05-09T08:42:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.184+0000 7f62a99b3740 1 Processor -- start 2024-05-09T08:42:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.185+0000 7f62a99b3740 1 -- start start 2024-05-09T08:42:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.186+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c946830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.186+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946d70 0x555e4c9510f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.186+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c953a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.186+0000 7f62a99b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e4c8ab340 con 0x555e4c946450 2024-05-09T08:42:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.186+0000 7f62a99b3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x555e4c853290 con 0x555e4c946d70 2024-05-09T08:42:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.186+0000 7f62a99b3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555e4c8a7300 con 0x555e4c951630 2024-05-09T08:42:50.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.189+0000 7f629dffb640 1 -- 172.21.15.28:0/2477535327 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6288042070 con 0x7f62a0108090 2024-05-09T08:42:50.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.189+0000 7f629ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c953a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.189+0000 7f629ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c953a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:44852/0 (socket says 172.21.15.28:44852) 2024-05-09T08:42:50.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.189+0000 7f629ffff640 1 -- 172.21.15.28:0/900151066 learned_addr learned my addr 172.21.15.28:0/900151066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.190+0000 7f629ffff640 1 -- 172.21.15.28:0/900151066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946d70 msgr2=0x555e4c9510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.191+0000 7f62a5242640 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c946830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.193+0000 7f629dffb640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f628c0a2fd0 0x7f628c0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.196+0000 7f62a4a41640 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946d70 0x555e4c9510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.200+0000 7f629ffff640 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946d70 0x555e4c9510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.200+0000 7f629ffff640 1 -- 172.21.15.28:0/900151066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 msgr2=0x555e4c946830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.200+0000 7f629ffff640 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c946830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.200+0000 7f629ffff640 1 -- 172.21.15.28:0/900151066 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e4c8778e0 con 0x555e4c951630 2024-05-09T08:42:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.200+0000 7f629dffb640 1 -- 172.21.15.28:0/2477535327 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9787+0+0 (secure 0 0 0) 0x7f62880f0170 con 0x7f62a0108090 2024-05-09T08:42:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.201+0000 7f62a5242640 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c946830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.201+0000 7f62a5242640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f628c0a2fd0 0x7f628c0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.201+0000 7f62a5242640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f628c0a2fd0 0x7f628c0a5490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6298005c40 tx=0x7f6298004a40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.202+0000 7f62a4a41640 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946d70 0x555e4c9510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.205+0000 7f629ffff640 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c953a20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6290009a60 tx=0x7f6290009f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aedb0b6d222cd50e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.209+0000 7f6296ffd640 1 -- 172.21.15.28:0/900151066 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629000ec50 con 0x555e4c951630 2024-05-09T08:42:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.209+0000 7f6296ffd640 1 -- 172.21.15.28:0/900151066 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f629000edf0 con 0x555e4c951630 2024-05-09T08:42:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.209+0000 7f6296ffd640 1 -- 172.21.15.28:0/900151066 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62900116d0 con 0x555e4c951630 2024-05-09T08:42:50.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.211+0000 7f62a99b3740 1 -- 172.21.15.28:0/900151066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 msgr2=0x555e4c953a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:50.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.211+0000 7f62a99b3740 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c953a20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6290009a60 tx=0x7f6290009f20 comp rx=0 tx=0).stop 2024-05-09T08:42:50.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.213+0000 7f62a99b3740 1 -- 172.21.15.28:0/900151066 shutdown_connections 2024-05-09T08:42:50.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.213+0000 7f62a99b3740 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c953a20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.213+0000 7f62a99b3740 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946d70 0x555e4c9510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.213+0000 7f62a99b3740 1 --2- 172.21.15.28:0/900151066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c946830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.213+0000 7f62a99b3740 1 -- 172.21.15.28:0/900151066 >> 172.21.15.28:0/900151066 conn(0x555e4c942960 msgr2=0x555e4c942d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:50.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.216+0000 7f62a99b3740 1 -- 172.21.15.28:0/900151066 shutdown_connections 2024-05-09T08:42:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.219+0000 7f62a99b3740 1 -- 172.21.15.28:0/900151066 wait complete. 2024-05-09T08:42:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.219+0000 7f62a99b3740 1 -- start start 2024-05-09T08:42:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.219+0000 7f62a99b3740 1 Processor -- start 2024-05-09T08:42:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.220+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 0x555e4c94e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.220+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c946d70 0x555e4c94ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.220+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c951630 0x555e4c94ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.220+0000 7f62a99b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e4c954260 con 0x555e4c951630 2024-05-09T08:42:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.220+0000 7f62a99b3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x555e4c866790 con 0x555e4c946450 2024-05-09T08:42:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.220+0000 7f62a99b3740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x555e4c9543e0 con 0x555e4c946d70 2024-05-09T08:42:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.220+0000 7f62a99b3740 10 client.0 _ll_register_callbacks cb 0x555e4c8a1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.221+0000 7f62a5242640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c951630 0x555e4c94ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.221+0000 7f62a5242640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c951630 0x555e4c94ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50316/0 (socket says 172.21.15.28:50316) 2024-05-09T08:42:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.221+0000 7f62a5242640 1 -- 172.21.15.28:0/1584675380 learned_addr learned my addr 172.21.15.28:0/1584675380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.221+0000 7f62a5242640 1 -- 172.21.15.28:0/1584675380 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c946d70 msgr2=0x555e4c94ea50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.221+0000 7f62a5242640 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c946d70 0x555e4c94ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.221+0000 7f62a5242640 1 -- 172.21.15.28:0/1584675380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 msgr2=0x555e4c94e510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.221+0000 7f62a5242640 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 0x555e4c94e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.221+0000 7f62a5242640 1 -- 172.21.15.28:0/1584675380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e4c5bf210 con 0x555e4c951630 2024-05-09T08:42:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.222+0000 7f62a5242640 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c951630 0x555e4c94ef90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f62980116a0 tx=0x7f6298009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2a8804b3e6f179 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:50.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.222+0000 7f627ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:42:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.226+0000 7f627ffff640 1 -- 172.21.15.28:0/1584675380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298021070 con 0x555e4c951630 2024-05-09T08:42:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.227+0000 7f62a99b3740 1 -- 172.21.15.28:0/1584675380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f629000edf0 con 0x555e4c951630 2024-05-09T08:42:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.227+0000 7f627ffff640 1 -- 172.21.15.28:0/1584675380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9787+0+0 (secure 0 0 0) 0x7f62980109a0 con 0x555e4c951630 2024-05-09T08:42:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.227+0000 7f627ffff640 1 -- 172.21.15.28:0/1584675380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f629802d050 con 0x555e4c951630 2024-05-09T08:42:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.229+0000 7f62a99b3740 10 client.17262 fetch_fsmap learned FSMap version 67 2024-05-09T08:42:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.229+0000 7f62a99b3740 1 -- 172.21.15.28:0/1584675380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x555e4c68b620 con 0x555e4c951630 2024-05-09T08:42:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.231+0000 7f627ffff640 1 -- 172.21.15.28:0/1584675380 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f629000edf0 con 0x555e4c951630 2024-05-09T08:42:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.231+0000 7f627ffff640 1 -- 172.21.15.28:0/1584675380 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f629801b3c0 con 0x555e4c951630 2024-05-09T08:42:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.235+0000 7f62a99b3740 10 client.17262 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:42:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.235+0000 7f62a99b3740 10 client.17262 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:42:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.235+0000 7f62a99b3740 20 client.17262 populate_metadata read hostname 'smithi028' 2024-05-09T08:42:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.235+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555e4c961320 0x555e4c9637c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.235+0000 7f62a99b3740 4 client.17262 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.235+0000 7f62a99b3740 1 -- 172.21.15.28:0/1584675380 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e4c8b8a70 con 0x555e4c961320 2024-05-09T08:42:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.239+0000 7f62a4a41640 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555e4c961320 0x555e4c9637c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.243+0000 7f62a4a41640 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555e4c961320 0x555e4c9637c0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79a564a1cc4fc1da server_cookie=eef7c3ac5cc39c9f in_seq=0 out_seq=0 2024-05-09T08:42:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.244+0000 7f627ffff640 10 client.17262 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:50.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.248+0000 7f627ffff640 1 -- 172.21.15.28:0/1584675380 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e4c8b8a70 con 0x555e4c961320 2024-05-09T08:42:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.248+0000 7f627ffff640 10 client.17262 handle_command_reply: tid=0 2024-05-09T08:42:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.249+0000 7f62a99b3740 1 client.17262 shutdown 2024-05-09T08:42:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.258+0000 7f62a99b3740 1 -- 172.21.15.28:0/1584675380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c951630 msgr2=0x555e4c94ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.258+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c951630 0x555e4c94ef90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f62980116a0 tx=0x7f6298009260 comp rx=0 tx=0).stop 2024-05-09T08:42:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.262+0000 7f62a99b3740 1 -- 172.21.15.28:0/1584675380 shutdown_connections 2024-05-09T08:42:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.262+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555e4c961320 0x555e4c9637c0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.262+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c951630 0x555e4c94ef90 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.262+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c946d70 0x555e4c94ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.262+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1584675380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 0x555e4c94e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.262+0000 7f62a99b3740 1 -- 172.21.15.28:0/1584675380 >> 172.21.15.28:0/1584675380 conn(0x555e4c942960 msgr2=0x555e4c9486e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.262+0000 7f627ffff640 0 client.17262 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.263+0000 7f62a99b3740 1 -- 172.21.15.28:0/1584675380 shutdown_connections 2024-05-09T08:42:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.263+0000 7f62a99b3740 1 -- 172.21.15.28:0/1584675380 wait complete. 2024-05-09T08:42:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.263+0000 7f62a99b3740 20 client.17262 trim_cache size 0 max 16384 2024-05-09T08:42:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.299+0000 7f62a99b3740 1 Processor -- start 2024-05-09T08:42:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a99b3740 1 -- start start 2024-05-09T08:42:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 0x555e4c950c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946d70 0x555e4c95d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c94d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f629ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946d70 0x555e4c95d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a4a41640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 0x555e4c950c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a4a41640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 0x555e4c950c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:53978/0 (socket says 172.21.15.28:53978) 2024-05-09T08:42:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f629ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946d70 0x555e4c95d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50318/0 (socket says 172.21.15.28:50318) 2024-05-09T08:42:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a4a41640 1 -- 172.21.15.28:0/1172106843 learned_addr learned my addr 172.21.15.28:0/1172106843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a99b3740 1 -- 172.21.15.28:0/1172106843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62a01c66f0 con 0x555e4c946d70 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a99b3740 1 -- 172.21.15.28:0/1172106843 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f62a01c8470 con 0x555e4c946450 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.300+0000 7f62a99b3740 1 -- 172.21.15.28:0/1172106843 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62a01c6e80 con 0x555e4c951630 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.301+0000 7f62a5242640 1 --2- 172.21.15.28:0/1172106843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c94d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.301+0000 7f62a4a41640 1 -- 172.21.15.28:0/1172106843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 msgr2=0x555e4c94d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.301+0000 7f62a4a41640 1 --2- 172.21.15.28:0/1172106843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c94d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.301+0000 7f62a4a41640 1 -- 172.21.15.28:0/1172106843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946d70 msgr2=0x555e4c95d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.301+0000 7f62a4a41640 1 --2- 172.21.15.28:0/1172106843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946d70 0x555e4c95d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.301+0000 7f62a4a41640 1 -- 172.21.15.28:0/1172106843 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e4c9474b0 con 0x555e4c946450 2024-05-09T08:42:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.301+0000 7f62a4a41640 1 --2- 172.21.15.28:0/1172106843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 0x555e4c950c40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6288004650 tx=0x7f6288009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ee77b0942e08322 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.304+0000 7f62967fc640 1 -- 172.21.15.28:0/1172106843 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62880c4db0 con 0x555e4c946450 2024-05-09T08:42:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.304+0000 7f62967fc640 1 -- 172.21.15.28:0/1172106843 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62880ba440 con 0x555e4c946450 2024-05-09T08:42:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.304+0000 7f62967fc640 1 -- 172.21.15.28:0/1172106843 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62880ed5f0 con 0x555e4c946450 2024-05-09T08:42:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.306+0000 7f62a99b3740 1 -- 172.21.15.28:0/1172106843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 msgr2=0x555e4c950c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.306+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1172106843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 0x555e4c950c40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6288004650 tx=0x7f6288009260 comp rx=0 tx=0).stop 2024-05-09T08:42:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.307+0000 7f62a99b3740 1 -- 172.21.15.28:0/1172106843 shutdown_connections 2024-05-09T08:42:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.307+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1172106843 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c951630 0x555e4c94d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.307+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1172106843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946d70 0x555e4c95d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.307+0000 7f62a99b3740 1 --2- 172.21.15.28:0/1172106843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c946450 0x555e4c950c40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.308+0000 7f62a99b3740 1 -- 172.21.15.28:0/1172106843 >> 172.21.15.28:0/1172106843 conn(0x555e4c942960 msgr2=0x555e4c940f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:50.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.309+0000 7f62a99b3740 1 -- 172.21.15.28:0/1172106843 shutdown_connections 2024-05-09T08:42:50.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.309+0000 7f62a99b3740 1 -- 172.21.15.28:0/1172106843 wait complete. 2024-05-09T08:42:50.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.310+0000 7f62a99b3740 1 -- start start 2024-05-09T08:42:50.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.310+0000 7f62a99b3740 1 Processor -- start 2024-05-09T08:42:50.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.310+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c94b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.310+0000 7f62a99b3740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c946d70 0x555e4c94b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.311+0000 7f62a4a41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c94b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.311+0000 7f62a4a41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c94b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50324/0 (socket says 172.21.15.28:50324) 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.311+0000 7f62a4a41640 1 -- 172.21.15.28:0/3189447575 learned_addr learned my addr 172.21.15.28:0/3189447575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.311+0000 7f62a99b3740 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c951630 0x555e4c94bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.311+0000 7f629ffff640 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c946d70 0x555e4c94b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.311+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62a01c7da0 con 0x555e4c946450 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.311+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f62a01c7610 con 0x555e4c951630 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.311+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62a00817f0 con 0x555e4c946d70 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.312+0000 7f62a99b3740 10 client.0 _ll_register_callbacks cb 0x7f6298009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.312+0000 7f62a4a41640 1 -- 172.21.15.28:0/3189447575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c946d70 msgr2=0x555e4c94b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.312+0000 7f62a4a41640 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c946d70 0x555e4c94b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.312+0000 7f62a4a41640 1 -- 172.21.15.28:0/3189447575 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c951630 msgr2=0x555e4c94bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.312+0000 7f62a4a41640 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c951630 0x555e4c94bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.312+0000 7f62a4a41640 1 -- 172.21.15.28:0/3189447575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e4c9c6940 con 0x555e4c946450 2024-05-09T08:42:50.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.312+0000 7f62a4a41640 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c94b0f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f62880bbe40 tx=0x7f628802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb285b32a058f772 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:50.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.312+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62880ba440 con 0x555e4c946450 2024-05-09T08:42:50.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.314+0000 7f627ffff640 10 client.17268 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:42:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.314+0000 7f627ffff640 1 -- 172.21.15.28:0/3189447575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62880d5060 con 0x555e4c946450 2024-05-09T08:42:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.314+0000 7f627ffff640 1 -- 172.21.15.28:0/3189447575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9787+0+0 (secure 0 0 0) 0x7f62880433b0 con 0x555e4c946450 2024-05-09T08:42:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.314+0000 7f627ffff640 1 -- 172.21.15.28:0/3189447575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f62880435b0 con 0x555e4c946450 2024-05-09T08:42:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.315+0000 7f62a99b3740 10 client.17268 fetch_fsmap learned FSMap version 67 2024-05-09T08:42:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.315+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=67}) -- 0x555e4c9c7a30 con 0x555e4c946450 2024-05-09T08:42:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.319+0000 7f627ffff640 1 -- 172.21.15.28:0/3189447575 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f62880ed5f0 con 0x555e4c946450 2024-05-09T08:42:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.321+0000 7f62a99b3740 10 client.17268 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:42:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.321+0000 7f62a99b3740 10 client.17268 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:42:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.321+0000 7f62a99b3740 20 client.17268 populate_metadata read hostname 'smithi028' 2024-05-09T08:42:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.321+0000 7f62a99b3740 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555e4c9c80a0 0x555e4c9ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.321+0000 7f62a99b3740 4 client.17268 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:42:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.321+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f62880ba440 con 0x555e4c9c80a0 2024-05-09T08:42:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.329+0000 7f629ffff640 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555e4c9c80a0 0x555e4c9ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.336+0000 7f629ffff640 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555e4c9c80a0 0x555e4c9ca4c0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e3e82775ad9689 server_cookie=e04f358ddf509b9f in_seq=0 out_seq=0 2024-05-09T08:42:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.337+0000 7f627ffff640 10 client.17268 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:50.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.586+0000 7f627ffff640 1 -- 172.21.15.28:0/3189447575 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f62880ba440 con 0x555e4c9c80a0 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.586+0000 7f627ffff640 10 client.17268 handle_command_reply: tid=0 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.586+0000 7f62a99b3740 1 client.17268 shutdown 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 msgr2=0x555e4c94b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f62a99b3740 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c94b0f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f62880bbe40 tx=0x7f628802a410 comp rx=0 tx=0).stop 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 shutdown_connections 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f62a99b3740 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x555e4c9c80a0 0x555e4c9ca4c0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f62a99b3740 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555e4c951630 0x555e4c94bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f62a99b3740 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x555e4c946d70 0x555e4c94b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f62a99b3740 1 --2- 172.21.15.28:0/3189447575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e4c946450 0x555e4c94b0f0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 >> 172.21.15.28:0/3189447575 conn(0x555e4c942960 msgr2=0x555e4c9486e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f627ffff640 0 client.17268 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:42:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.587+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 shutdown_connections 2024-05-09T08:42:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.588+0000 7f62a99b3740 1 -- 172.21.15.28:0/3189447575 wait complete. 2024-05-09T08:42:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.588+0000 7f62a99b3740 20 client.17268 trim_cache size 0 max 16384 2024-05-09T08:42:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:42:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:42:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:42:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 -- 172.21.15.28:0/2477535327 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f628c0a2fd0 msgr2=0x7f628c0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f628c0a2fd0 0x7f628c0a5490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6298005c40 tx=0x7f6298004a40 comp rx=0 tx=0).stop 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 -- 172.21.15.28:0/2477535327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 msgr2=0x7f62a0072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 0x7f62a0072f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f628802a980 tx=0x7f6288003b90 comp rx=0 tx=0).stop 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 -- 172.21.15.28:0/2477535327 shutdown_connections 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006c0a0 0x7f62a006c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f628c0a2fd0 0x7f628c0a5490 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62a010d160 0x7f62a0073470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 --2- 172.21.15.28:0/2477535327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62a0108090 0x7f62a0072f30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 -- 172.21.15.28:0/2477535327 >> 172.21.15.28:0/2477535327 conn(0x7f62a0075dc0 msgr2=0x7f62a0078080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 -- 172.21.15.28:0/2477535327 shutdown_connections 2024-05-09T08:42:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:42:50.589+0000 7f62a6ca5640 1 -- 172.21.15.28:0/2477535327 wait complete. 2024-05-09T08:42:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:51 smithi165 ceph-mon[34540]: pgmap v530: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 946 KiB/s wr, 240 op/s 2024-05-09T08:42:52.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:51 smithi184 ceph-mon[34509]: pgmap v530: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 946 KiB/s wr, 240 op/s 2024-05-09T08:42:52.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:51 smithi028 ceph-mon[30119]: pgmap v530: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 946 KiB/s wr, 240 op/s 2024-05-09T08:42:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:53 smithi165 ceph-mon[34540]: pgmap v531: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 235 op/s 2024-05-09T08:42:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:53 smithi184 ceph-mon[34509]: pgmap v531: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 235 op/s 2024-05-09T08:42:54.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:53 smithi028 ceph-mon[30119]: pgmap v531: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 235 op/s 2024-05-09T08:42:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:55 smithi165 ceph-mon[34540]: pgmap v532: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 840 KiB/s wr, 247 op/s 2024-05-09T08:42:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:55 smithi184 ceph-mon[34509]: pgmap v532: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 840 KiB/s wr, 247 op/s 2024-05-09T08:42:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:55 smithi028 ceph-mon[30119]: pgmap v532: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 840 KiB/s wr, 247 op/s 2024-05-09T08:42:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:57 smithi165 ceph-mon[34540]: pgmap v533: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-05-09T08:42:58.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:57 smithi184 ceph-mon[34509]: pgmap v533: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-05-09T08:42:58.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:57 smithi028 ceph-mon[30119]: pgmap v533: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-05-09T08:43:00.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:42:59 smithi028 ceph-mon[30119]: pgmap v534: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 918 KiB/s wr, 235 op/s 2024-05-09T08:43:00.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:42:59 smithi165 ceph-mon[34540]: pgmap v534: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 918 KiB/s wr, 235 op/s 2024-05-09T08:43:00.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:42:59 smithi184 ceph-mon[34509]: pgmap v534: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 918 KiB/s wr, 235 op/s 2024-05-09T08:43:02.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:01 smithi028 ceph-mon[30119]: pgmap v535: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 247 op/s 2024-05-09T08:43:02.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:01 smithi165 ceph-mon[34540]: pgmap v535: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 247 op/s 2024-05-09T08:43:02.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:01 smithi184 ceph-mon[34509]: pgmap v535: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 247 op/s 2024-05-09T08:43:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.748+0000 7fc4be329640 1 -- 172.21.15.28:0/2804572624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4b8100aa0 msgr2=0x7fc4b8104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.748+0000 7fc4be329640 1 --2- 172.21.15.28:0/2804572624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4b8100aa0 0x7fc4b8104ae0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009be0 tx=0x7fc4a802b910 comp rx=0 tx=0).stop 2024-05-09T08:43:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.749+0000 7fc4be329640 1 -- 172.21.15.28:0/2804572624 shutdown_connections 2024-05-09T08:43:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.749+0000 7fc4be329640 1 --2- 172.21.15.28:0/2804572624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b81051c0 0x7fc4b810cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.749+0000 7fc4be329640 1 --2- 172.21.15.28:0/2804572624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4b8100aa0 0x7fc4b8104ae0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.749+0000 7fc4be329640 1 --2- 172.21.15.28:0/2804572624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b81000f0 0x7fc4b81004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.749+0000 7fc4be329640 1 -- 172.21.15.28:0/2804572624 >> 172.21.15.28:0/2804572624 conn(0x7fc4b8076cc0 msgr2=0x7fc4b80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.754+0000 7fc4be329640 1 -- 172.21.15.28:0/2804572624 shutdown_connections 2024-05-09T08:43:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.755+0000 7fc4be329640 1 -- 172.21.15.28:0/2804572624 wait complete. 2024-05-09T08:43:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.756+0000 7fc4be329640 1 Processor -- start 2024-05-09T08:43:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.756+0000 7fc4be329640 1 -- start start 2024-05-09T08:43:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.756+0000 7fc4be329640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4b81000f0 0x7fc4b8108170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.756+0000 7fc4be329640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b81051c0 0x7fc4b81086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.756+0000 7fc4be329640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b8108bf0 0x7fc4b8114920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.756+0000 7fc4be329640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b8071820 con 0x7fc4b81051c0 2024-05-09T08:43:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.756+0000 7fc4be329640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b80716a0 con 0x7fc4b8108bf0 2024-05-09T08:43:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.756+0000 7fc4be329640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b80719a0 con 0x7fc4b81000f0 2024-05-09T08:43:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.757+0000 7fc4bc8c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b8108bf0 0x7fc4b8114920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.757+0000 7fc4bc8c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b8108bf0 0x7fc4b8114920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:32892/0 (socket says 172.21.15.28:32892) 2024-05-09T08:43:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.757+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/1285087006 learned_addr learned my addr 172.21.15.28:0/1285087006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.757+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/1285087006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4b81000f0 msgr2=0x7fc4b8108170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:43:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.757+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4b81000f0 0x7fc4b8108170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.757+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/1285087006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b81051c0 msgr2=0x7fc4b81086b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:43:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.757+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b81051c0 0x7fc4b81086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.757+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/1285087006 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b8115020 con 0x7fc4b8108bf0 2024-05-09T08:43:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.758+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b8108bf0 0x7fc4b8114920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc4a001e920 tx=0x7fc4a001ede0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.760+0000 7fc4b57fa640 1 -- 172.21.15.28:0/1285087006 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0028070 con 0x7fc4b8108bf0 2024-05-09T08:43:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.760+0000 7fc4b57fa640 1 -- 172.21.15.28:0/1285087006 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a0005020 con 0x7fc4b8108bf0 2024-05-09T08:43:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.760+0000 7fc4b57fa640 1 -- 172.21.15.28:0/1285087006 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0004cc0 con 0x7fc4b8108bf0 2024-05-09T08:43:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.762+0000 7fc4be329640 1 -- 172.21.15.28:0/1285087006 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b8115310 con 0x7fc4b8108bf0 2024-05-09T08:43:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.762+0000 7fc4be329640 1 -- 172.21.15.28:0/1285087006 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b81157e0 con 0x7fc4b8108bf0 2024-05-09T08:43:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.766+0000 7fc4b57fa640 1 -- 172.21.15.28:0/1285087006 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc4a00051d0 con 0x7fc4b8108bf0 2024-05-09T08:43:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.767+0000 7fc4b57fa640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4a40a2fc0 0x7fc4a40a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.767+0000 7fc4b57fa640 1 -- 172.21.15.28:0/1285087006 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9787+0+0 (secure 0 0 0) 0x7fc4a00e0640 con 0x7fc4b8108bf0 2024-05-09T08:43:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.769+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4a40a2fc0 0x7fc4a40a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.770+0000 7fc4c103b740 1 Processor -- start 2024-05-09T08:43:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.770+0000 7fc4c103b740 1 -- start start 2024-05-09T08:43:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.773+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 0x55ced5ca8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.773+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 0x55ced5cb30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.773+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.773+0000 7fc4b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 0x55ced5cb30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.773+0000 7fc4b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 0x55ced5cb30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38544/0 (socket says 172.21.15.28:38544) 2024-05-09T08:43:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.773+0000 7fc4b77fe640 1 -- 172.21.15.28:0/3145134617 learned_addr learned my addr 172.21.15.28:0/3145134617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.773+0000 7fc4c103b740 1 -- 172.21.15.28:0/3145134617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ced5c0d340 con 0x55ced5ca8d70 2024-05-09T08:43:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.773+0000 7fc4c103b740 1 -- 172.21.15.28:0/3145134617 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ced5bb5290 con 0x55ced5ca8450 2024-05-09T08:43:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.773+0000 7fc4c103b740 1 -- 172.21.15.28:0/3145134617 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ced5c09300 con 0x55ced5cb3630 2024-05-09T08:43:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.775+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/3145134617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.776+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/3145134617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 0x55ced5ca8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.779+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4a40a2fc0 0x7fc4a40a5480 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009700 tx=0x7fc4b0009b00 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:03.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.780+0000 7fc4b7fff640 1 -- 172.21.15.28:0/3145134617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 msgr2=0x55ced5cb5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.780+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/3145134617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.780+0000 7fc4b7fff640 1 -- 172.21.15.28:0/3145134617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 msgr2=0x55ced5cb30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.780+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/3145134617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 0x55ced5cb30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.780+0000 7fc4b7fff640 1 -- 172.21.15.28:0/3145134617 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ced5bd98e0 con 0x55ced5ca8450 2024-05-09T08:43:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.780+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/3145134617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 0x55ced5ca8830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000b0e0 tx=0x7fc4b0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19db5011b35eb761 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.782+0000 7fc4ae7fc640 1 -- 172.21.15.28:0/3145134617 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b000f040 con 0x55ced5ca8450 2024-05-09T08:43:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.782+0000 7fc4ae7fc640 1 -- 172.21.15.28:0/3145134617 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4b0022070 con 0x55ced5ca8450 2024-05-09T08:43:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.782+0000 7fc4ae7fc640 1 -- 172.21.15.28:0/3145134617 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b001de50 con 0x55ced5ca8450 2024-05-09T08:43:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.783+0000 7fc4c103b740 1 -- 172.21.15.28:0/3145134617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 msgr2=0x55ced5ca8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.783+0000 7fc4c103b740 1 --2- 172.21.15.28:0/3145134617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 0x55ced5ca8830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000b0e0 tx=0x7fc4b0016000 comp rx=0 tx=0).stop 2024-05-09T08:43:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.784+0000 7fc4c103b740 1 -- 172.21.15.28:0/3145134617 shutdown_connections 2024-05-09T08:43:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.784+0000 7fc4c103b740 1 --2- 172.21.15.28:0/3145134617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.784+0000 7fc4c103b740 1 --2- 172.21.15.28:0/3145134617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 0x55ced5cb30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.784+0000 7fc4c103b740 1 --2- 172.21.15.28:0/3145134617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 0x55ced5ca8830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.784+0000 7fc4c103b740 1 -- 172.21.15.28:0/3145134617 >> 172.21.15.28:0/3145134617 conn(0x55ced5ca4960 msgr2=0x55ced5ca4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.789+0000 7fc4c103b740 1 -- 172.21.15.28:0/3145134617 shutdown_connections 2024-05-09T08:43:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.789+0000 7fc4c103b740 1 -- 172.21.15.28:0/3145134617 wait complete. 2024-05-09T08:43:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.789+0000 7fc4c103b740 1 -- start start 2024-05-09T08:43:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.791+0000 7fc4c103b740 1 Processor -- start 2024-05-09T08:43:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.791+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 0x55ced5cb05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.791+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 0x55ced5cb0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.791+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.792+0000 7fc4c103b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ced5cb6260 con 0x55ced5ca8d70 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.792+0000 7fc4c103b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55ced5bc8790 con 0x55ced5ca8450 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.792+0000 7fc4c103b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ced5cb63e0 con 0x55ced5cb3630 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.792+0000 7fc4bc8c6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.792+0000 7fc4bc8c6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38626/0 (socket says 172.21.15.28:38626) 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.792+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/2820459686 learned_addr learned my addr 172.21.15.28:0/2820459686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.792+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 0x55ced5cb05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.792+0000 7fc4b77fe640 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 0x55ced5cb0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.792+0000 7fc4c103b740 10 client.0 _ll_register_callbacks cb 0x55ced5c03a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.793+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/2820459686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 msgr2=0x55ced5cb05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.793+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 0x55ced5cb05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.793+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/2820459686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 msgr2=0x55ced5cb0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.793+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 0x55ced5cb0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.793+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/2820459686 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ced5921210 con 0x55ced5cb3630 2024-05-09T08:43:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.794+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb1040 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4a001a620 tx=0x7fc4a00acd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb9d70c44c39f3f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.794+0000 7fc4977fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:43:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.794+0000 7fc4977fe640 1 -- 172.21.15.28:0/2820459686 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00c5020 con 0x55ced5cb3630 2024-05-09T08:43:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.794+0000 7fc4977fe640 1 -- 172.21.15.28:0/2820459686 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9787+0+0 (secure 0 0 0) 0x7fc4a00c0bb0 con 0x55ced5cb3630 2024-05-09T08:43:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.795+0000 7fc4c103b740 1 -- 172.21.15.28:0/2820459686 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4b0022070 con 0x55ced5cb3630 2024-05-09T08:43:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.795+0000 7fc4977fe640 1 -- 172.21.15.28:0/2820459686 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fc4a00ddcc0 con 0x55ced5cb3630 2024-05-09T08:43:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.795+0000 7fc4c103b740 10 client.26569 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.795+0000 7fc4c103b740 1 -- 172.21.15.28:0/2820459686 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55ced59ed620 con 0x55ced5cb3630 2024-05-09T08:43:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.796+0000 7fc4977fe640 1 -- 172.21.15.28:0/2820459686 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a00b5300 con 0x55ced5cb3630 2024-05-09T08:43:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.796+0000 7fc4977fe640 1 -- 172.21.15.28:0/2820459686 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fc4a001f040 con 0x55ced5cb3630 2024-05-09T08:43:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.796+0000 7fc4c103b740 10 client.26569 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.796+0000 7fc4c103b740 10 client.26569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.796+0000 7fc4c103b740 20 client.26569 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.796+0000 7fc4c103b740 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ced5cc3640 0x55ced5cc5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.796+0000 7fc4c103b740 4 client.26569 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.796+0000 7fc4c103b740 1 -- 172.21.15.28:0/2820459686 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc4b0022070 con 0x55ced5cc3640 2024-05-09T08:43:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.798+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ced5cc3640 0x55ced5cc5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.798+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ced5cc3640 0x55ced5cc5a30 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8576642f297f5707 server_cookie=f0b575e86fcdd7c0 in_seq=0 out_seq=0 2024-05-09T08:43:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.798+0000 7fc4977fe640 10 client.26569 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.801+0000 7fc4977fe640 1 -- 172.21.15.28:0/2820459686 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fc4b0022070 con 0x55ced5cc3640 2024-05-09T08:43:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.801+0000 7fc4977fe640 10 client.26569 handle_command_reply: tid=0 2024-05-09T08:43:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.801+0000 7fc4c103b740 1 client.26569 shutdown 2024-05-09T08:43:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.804+0000 7fc4c103b740 1 -- 172.21.15.28:0/2820459686 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 msgr2=0x55ced5cb1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.804+0000 7fc4c103b740 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb1040 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4a001a620 tx=0x7fc4a00acd90 comp rx=0 tx=0).stop 2024-05-09T08:43:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.804+0000 7fc4c103b740 1 -- 172.21.15.28:0/2820459686 shutdown_connections 2024-05-09T08:43:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.804+0000 7fc4c103b740 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ced5cc3640 0x55ced5cc5a30 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.804+0000 7fc4c103b740 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5cb1040 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.804+0000 7fc4c103b740 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8d70 0x55ced5cb0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.804+0000 7fc4c103b740 1 --2- 172.21.15.28:0/2820459686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8450 0x55ced5cb05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.804+0000 7fc4c103b740 1 -- 172.21.15.28:0/2820459686 >> 172.21.15.28:0/2820459686 conn(0x55ced5ca4960 msgr2=0x55ced5caa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.804+0000 7fc4977fe640 0 client.26569 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.805+0000 7fc4c103b740 1 -- 172.21.15.28:0/2820459686 shutdown_connections 2024-05-09T08:43:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.805+0000 7fc4c103b740 1 -- 172.21.15.28:0/2820459686 wait complete. 2024-05-09T08:43:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.805+0000 7fc4c103b740 20 client.26569 trim_cache size 0 max 16384 2024-05-09T08:43:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.834+0000 7fc4c103b740 1 Processor -- start 2024-05-09T08:43:03.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.834+0000 7fc4c103b740 1 -- start start 2024-05-09T08:43:03.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 0x55ced5cbf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8d70 0x55ced5ccb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5ccb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4c103b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b81a45b0 con 0x55ced5ca8450 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4c103b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b81a6330 con 0x55ced5ca8d70 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4c103b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b81a4d40 con 0x55ced5cb3630 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4b77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8d70 0x55ced5ccb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 0x55ced5cbf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4b77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8d70 0x55ced5ccb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:32932/0 (socket says 172.21.15.28:32932) 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4b77fe640 1 -- 172.21.15.28:0/3897558015 learned_addr learned my addr 172.21.15.28:0/3897558015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5ccb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4b77fe640 1 -- 172.21.15.28:0/3897558015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 msgr2=0x55ced5ccb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4b77fe640 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5ccb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4b77fe640 1 -- 172.21.15.28:0/3897558015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 msgr2=0x55ced5cbf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4b77fe640 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 0x55ced5cbf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.835+0000 7fc4b77fe640 1 -- 172.21.15.28:0/3897558015 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ced5ca9820 con 0x55ced5ca8d70 2024-05-09T08:43:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.836+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5ccb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.836+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 0x55ced5cbf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.836+0000 7fc4b77fe640 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8d70 0x55ced5ccb120 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8005f00 tx=0x7fc4a8030bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de3d5541a05f62e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.836+0000 7fc4adffb640 1 -- 172.21.15.28:0/3897558015 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8030d70 con 0x55ced5ca8d70 2024-05-09T08:43:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.836+0000 7fc4adffb640 1 -- 172.21.15.28:0/3897558015 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a8003e50 con 0x55ced5ca8d70 2024-05-09T08:43:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.837+0000 7fc4c103b740 1 -- 172.21.15.28:0/3897558015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8d70 msgr2=0x55ced5ccb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.837+0000 7fc4c103b740 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8d70 0x55ced5ccb120 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8005f00 tx=0x7fc4a8030bc0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.837+0000 7fc4c103b740 1 -- 172.21.15.28:0/3897558015 shutdown_connections 2024-05-09T08:43:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.837+0000 7fc4c103b740 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5cb3630 0x55ced5ccb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.837+0000 7fc4c103b740 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5ca8d70 0x55ced5ccb120 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.837+0000 7fc4c103b740 1 --2- 172.21.15.28:0/3897558015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 0x55ced5cbf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.838+0000 7fc4c103b740 1 -- 172.21.15.28:0/3897558015 >> 172.21.15.28:0/3897558015 conn(0x55ced5ca4960 msgr2=0x55ced5ca3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.838+0000 7fc4c103b740 1 -- 172.21.15.28:0/3897558015 shutdown_connections 2024-05-09T08:43:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.838+0000 7fc4c103b740 1 -- 172.21.15.28:0/3897558015 wait complete. 2024-05-09T08:43:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.838+0000 7fc4c103b740 1 -- start start 2024-05-09T08:43:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.839+0000 7fc4c103b740 1 Processor -- start 2024-05-09T08:43:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.839+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 0x55ced5cad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.839+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5ca8d70 0x55ced5cad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.839+0000 7fc4c103b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5cb3630 0x55ced5cade10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.839+0000 7fc4c103b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b81a5c60 con 0x55ced5ca8450 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.839+0000 7fc4c103b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b81a54d0 con 0x55ced5cb3630 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.839+0000 7fc4c103b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b81015c0 con 0x55ced5ca8d70 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.839+0000 7fc4c103b740 10 client.0 _ll_register_callbacks cb 0x7fc4a00a89b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.840+0000 7fc4b77fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5ca8d70 0x55ced5cad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.840+0000 7fc4bc8c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5cb3630 0x55ced5cade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.840+0000 7fc4b77fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5ca8d70 0x55ced5cad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:38642/0 (socket says 172.21.15.28:38642) 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.840+0000 7fc4b77fe640 1 -- 172.21.15.28:0/829466866 learned_addr learned my addr 172.21.15.28:0/829466866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.840+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 0x55ced5cad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/829466866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5ca8d70 msgr2=0x55ced5cad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5ca8d70 0x55ced5cad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/829466866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 msgr2=0x55ced5cad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 0x55ced5cad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4bc8c6640 1 -- 172.21.15.28:0/829466866 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ced5d28a60 con 0x55ced5cb3630 2024-05-09T08:43:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4bc8c6640 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5cb3630 0x55ced5cade10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00acf40 tx=0x7fc4a0005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaad0de3e42572c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4977fe640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:43:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4977fe640 1 -- 172.21.15.28:0/829466866 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a002ada0 con 0x55ced5cb3630 2024-05-09T08:43:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4c103b740 1 -- 172.21.15.28:0/829466866 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4a8003e50 con 0x55ced5cb3630 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.841+0000 7fc4977fe640 1 -- 172.21.15.28:0/829466866 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9787+0+0 (secure 0 0 0) 0x7fc4a00e0120 con 0x55ced5cb3630 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4977fe640 1 -- 172.21.15.28:0/829466866 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fc4a00b5300 con 0x55ced5cb3630 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4c103b740 10 client.26729 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4c103b740 1 -- 172.21.15.28:0/829466866 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55ced5d29d70 con 0x55ced5cb3630 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4977fe640 1 -- 172.21.15.28:0/829466866 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a8003e50 con 0x55ced5cb3630 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4977fe640 1 -- 172.21.15.28:0/829466866 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fc4a00ab550 con 0x55ced5cb3630 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4c103b740 10 client.26729 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4c103b740 10 client.26729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4c103b740 20 client.26729 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4c103b740 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ced5d2a580 0x55ced5d2c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4c103b740 4 client.26729 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.842+0000 7fc4c103b740 1 -- 172.21.15.28:0/829466866 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ced5c1aa70 con 0x55ced5d2a580 2024-05-09T08:43:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.843+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ced5d2a580 0x55ced5d2c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.843+0000 7fc4b7fff640 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ced5d2a580 0x55ced5d2c9a0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17fa1a12e9a39cbd server_cookie=38105fab15b0e53c in_seq=0 out_seq=0 2024-05-09T08:43:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.843+0000 7fc4977fe640 10 client.26729 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.960+0000 7fc4977fe640 1 -- 172.21.15.28:0/829466866 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ced5c1aa70 con 0x55ced5d2a580 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.960+0000 7fc4977fe640 10 client.26729 handle_command_reply: tid=0 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.960+0000 7fc4c103b740 1 client.26729 shutdown 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.960+0000 7fc4c103b740 1 -- 172.21.15.28:0/829466866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5cb3630 msgr2=0x55ced5cade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.960+0000 7fc4c103b740 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5cb3630 0x55ced5cade10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00acf40 tx=0x7fc4a0005f10 comp rx=0 tx=0).stop 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.960+0000 7fc4c103b740 1 -- 172.21.15.28:0/829466866 shutdown_connections 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.960+0000 7fc4c103b740 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55ced5d2a580 0x55ced5d2c9a0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.960+0000 7fc4c103b740 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ced5cb3630 0x55ced5cade10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.961+0000 7fc4c103b740 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ced5ca8d70 0x55ced5cad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.961+0000 7fc4c103b740 1 --2- 172.21.15.28:0/829466866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ced5ca8450 0x55ced5cad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.961+0000 7fc4c103b740 1 -- 172.21.15.28:0/829466866 >> 172.21.15.28:0/829466866 conn(0x55ced5ca4960 msgr2=0x55ced5ca9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.961+0000 7fc4977fe640 0 client.26729 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.961+0000 7fc4c103b740 1 -- 172.21.15.28:0/829466866 shutdown_connections 2024-05-09T08:43:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.961+0000 7fc4c103b740 1 -- 172.21.15.28:0/829466866 wait complete. 2024-05-09T08:43:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.961+0000 7fc4c103b740 20 client.26729 trim_cache size 0 max 16384 2024-05-09T08:43:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:43:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:43:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:43:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:43:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 -- 172.21.15.28:0/1285087006 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4a40a2fc0 msgr2=0x7fc4a40a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4a40a2fc0 0x7fc4a40a5480 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009700 tx=0x7fc4b0009b00 comp rx=0 tx=0).stop 2024-05-09T08:43:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 -- 172.21.15.28:0/1285087006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b8108bf0 msgr2=0x7fc4b8114920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b8108bf0 0x7fc4b8114920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc4a001e920 tx=0x7fc4a001ede0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 -- 172.21.15.28:0/1285087006 shutdown_connections 2024-05-09T08:43:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b8108bf0 0x7fc4b8114920 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b81051c0 0x7fc4b81086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc4a40a2fc0 0x7fc4a40a5480 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 --2- 172.21.15.28:0/1285087006 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4b81000f0 0x7fc4b8108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 -- 172.21.15.28:0/1285087006 >> 172.21.15.28:0/1285087006 conn(0x7fc4b8076cc0 msgr2=0x7fc4b80fc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 -- 172.21.15.28:0/1285087006 shutdown_connections 2024-05-09T08:43:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:03.962+0000 7fc4be329640 1 -- 172.21.15.28:0/1285087006 wait complete. 2024-05-09T08:43:04.152 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:03 smithi165 ceph-mon[34540]: pgmap v536: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.1 MiB/s wr, 220 op/s 2024-05-09T08:43:04.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:03 smithi028 ceph-mon[30119]: pgmap v536: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.1 MiB/s wr, 220 op/s 2024-05-09T08:43:04.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:03 smithi184 ceph-mon[34509]: pgmap v536: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.1 MiB/s wr, 220 op/s 2024-05-09T08:43:05.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:04 smithi028 ceph-mon[30119]: pgmap v537: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 234 op/s 2024-05-09T08:43:05.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:04 smithi184 ceph-mon[34509]: pgmap v537: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 234 op/s 2024-05-09T08:43:05.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:04 smithi165 ceph-mon[34540]: pgmap v537: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 234 op/s 2024-05-09T08:43:07.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:07 smithi165 ceph-mon[34540]: pgmap v538: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 222 op/s 2024-05-09T08:43:07.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:07 smithi184 ceph-mon[34509]: pgmap v538: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 222 op/s 2024-05-09T08:43:07.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:07 smithi028 ceph-mon[30119]: pgmap v538: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 222 op/s 2024-05-09T08:43:09.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:09 smithi165 ceph-mon[34540]: pgmap v539: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.1 MiB/s wr, 191 op/s 2024-05-09T08:43:09.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:09 smithi165 ceph-mon[34540]: osdmap e126: 12 total, 12 up, 12 in 2024-05-09T08:43:09.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:09 smithi184 ceph-mon[34509]: pgmap v539: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.1 MiB/s wr, 191 op/s 2024-05-09T08:43:09.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:09 smithi184 ceph-mon[34509]: osdmap e126: 12 total, 12 up, 12 in 2024-05-09T08:43:09.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:09 smithi028 ceph-mon[30119]: pgmap v539: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.1 MiB/s wr, 191 op/s 2024-05-09T08:43:09.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:09 smithi028 ceph-mon[30119]: osdmap e126: 12 total, 12 up, 12 in 2024-05-09T08:43:11.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:11 smithi165 ceph-mon[34540]: pgmap v541: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 880 KiB/s wr, 192 op/s 2024-05-09T08:43:11.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:11 smithi184 ceph-mon[34509]: pgmap v541: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 880 KiB/s wr, 192 op/s 2024-05-09T08:43:11.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:11 smithi028 ceph-mon[30119]: pgmap v541: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 880 KiB/s wr, 192 op/s 2024-05-09T08:43:12.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:12.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:12.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:13.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:13 smithi165 ceph-mon[34540]: pgmap v542: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-05-09T08:43:13.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:13 smithi184 ceph-mon[34509]: pgmap v542: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-05-09T08:43:13.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:13 smithi028 ceph-mon[30119]: pgmap v542: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-05-09T08:43:14.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:14 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:14.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:14 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:14.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:14 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:43:14.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:14 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:14.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:14 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:14.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:14 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:43:14.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:14 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:43:14.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:14 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:43:14.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:14 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:43:15.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:15 smithi165 ceph-mon[34540]: pgmap v543: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 1.1 MiB/s wr, 161 op/s 2024-05-09T08:43:15.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:15 smithi184 ceph-mon[34509]: pgmap v543: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 1.1 MiB/s wr, 161 op/s 2024-05-09T08:43:15.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:15 smithi028 ceph-mon[30119]: pgmap v543: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 1.1 MiB/s wr, 161 op/s 2024-05-09T08:43:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.119+0000 7f76fb2b2640 1 -- 172.21.15.28:0/1799750195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76f4105bd0 msgr2=0x7f76f4106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.119+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/1799750195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76f4105bd0 0x7f76f4106060 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009900 tx=0x7f76e402a560 comp rx=0 tx=0).stop 2024-05-09T08:43:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.122+0000 7f76fb2b2640 1 -- 172.21.15.28:0/1799750195 shutdown_connections 2024-05-09T08:43:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.122+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/1799750195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76f4105bd0 0x7f76f4106060 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.122+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/1799750195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76f41011d0 0x7f76f4105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.122+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/1799750195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f41008b0 0x7f76f4100c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.122+0000 7f76fb2b2640 1 -- 172.21.15.28:0/1799750195 >> 172.21.15.28:0/1799750195 conn(0x7f76f40778b0 msgr2=0x7f76f4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.126+0000 7f76fb2b2640 1 -- 172.21.15.28:0/1799750195 shutdown_connections 2024-05-09T08:43:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.127+0000 7f76fb2b2640 1 -- 172.21.15.28:0/1799750195 wait complete. 2024-05-09T08:43:17.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76fb2b2640 1 Processor -- start 2024-05-09T08:43:17.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76fb2b2640 1 -- start start 2024-05-09T08:43:17.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76fb2b2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76f41008b0 0x7f76f4072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76fb2b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f41011d0 0x7f76f406c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76fb2b2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76f4105bd0 0x7f76f406c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76fb2b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41182e0 con 0x7f76f41011d0 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76fb2b2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f76f4118160 con 0x7f76f41008b0 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76fb2b2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f76f4118460 con 0x7f76f4105bd0 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76f884d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f41011d0 0x7f76f406c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.128+0000 7f76f884d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f41011d0 0x7f76f406c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52298/0 (socket says 172.21.15.28:52298) 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.129+0000 7f76f884d640 1 -- 172.21.15.28:0/826940922 learned_addr learned my addr 172.21.15.28:0/826940922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.129+0000 7f76f884d640 1 -- 172.21.15.28:0/826940922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76f4105bd0 msgr2=0x7f76f406c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.129+0000 7f76f884d640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76f4105bd0 0x7f76f406c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.129+0000 7f76f884d640 1 -- 172.21.15.28:0/826940922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76f41008b0 msgr2=0x7f76f4072ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.129+0000 7f76f884d640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76f41008b0 0x7f76f4072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.129+0000 7f76f884d640 1 -- 172.21.15.28:0/826940922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f406cb20 con 0x7f76f41011d0 2024-05-09T08:43:17.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.129+0000 7f76f884d640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f41011d0 0x7f76f406c070 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00ba50 tx=0x7f76ec00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.130+0000 7f76ea7fc640 1 -- 172.21.15.28:0/826940922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec019070 con 0x7f76f41011d0 2024-05-09T08:43:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.133+0000 7f76ea7fc640 1 -- 172.21.15.28:0/826940922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ec00cc10 con 0x7f76f41011d0 2024-05-09T08:43:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.133+0000 7f76ea7fc640 1 -- 172.21.15.28:0/826940922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec007660 con 0x7f76f41011d0 2024-05-09T08:43:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.134+0000 7f76fb2b2640 1 -- 172.21.15.28:0/826940922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f406ce10 con 0x7f76f41011d0 2024-05-09T08:43:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.134+0000 7f76fb2b2640 1 -- 172.21.15.28:0/826940922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f4070db0 con 0x7f76f41011d0 2024-05-09T08:43:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.140+0000 7f76fdfbd740 1 Processor -- start 2024-05-09T08:43:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.141+0000 7f76fdfbd740 1 -- start start 2024-05-09T08:43:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.141+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c54a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.141+0000 7f76ea7fc640 1 -- 172.21.15.28:0/826940922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f76ec01c070 con 0x7f76f41011d0 2024-05-09T08:43:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.141+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c5550f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.141+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c557a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.141+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e64c4af340 con 0x55e64c54a450 2024-05-09T08:43:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.141+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e64c457290 con 0x55e64c54ad70 2024-05-09T08:43:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.141+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e64c4ab300 con 0x55e64c555630 2024-05-09T08:43:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.142+0000 7f76f884d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c557a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.142+0000 7f76f884d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c557a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:39716/0 (socket says 172.21.15.28:39716) 2024-05-09T08:43:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.142+0000 7f76f884d640 1 -- 172.21.15.28:0/3869297093 learned_addr learned my addr 172.21.15.28:0/3869297093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:17.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.142+0000 7f76f984f640 1 --2- 172.21.15.28:0/3869297093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c54a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.143+0000 7f76f984f640 1 -- 172.21.15.28:0/3869297093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 msgr2=0x55e64c557a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.143+0000 7f76f984f640 1 --2- 172.21.15.28:0/3869297093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c557a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.143+0000 7f76f984f640 1 -- 172.21.15.28:0/3869297093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 msgr2=0x55e64c5550f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:43:17.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.143+0000 7f76f984f640 1 --2- 172.21.15.28:0/3869297093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c5550f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.143+0000 7f76f984f640 1 -- 172.21.15.28:0/3869297093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e64c47b8e0 con 0x55e64c54a450 2024-05-09T08:43:17.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.143+0000 7f76f984f640 1 --2- 172.21.15.28:0/3869297093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c54a830 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f76e402ab50 tx=0x7f76e4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63bbfb740604ba14 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.144+0000 7f76df7fe640 1 -- 172.21.15.28:0/3869297093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e403e030 con 0x55e64c54a450 2024-05-09T08:43:17.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.144+0000 7f76ea7fc640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76e00a2ef0 0x7f76e00a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.144+0000 7f76df7fe640 1 -- 172.21.15.28:0/3869297093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e4004210 con 0x55e64c54a450 2024-05-09T08:43:17.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.144+0000 7f76ea7fc640 1 -- 172.21.15.28:0/826940922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9911+0+0 (secure 0 0 0) 0x7f76ec0cee90 con 0x7f76f41011d0 2024-05-09T08:43:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.145+0000 7f76f904e640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76e00a2ef0 0x7f76e00a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.145+0000 7f76fdfbd740 1 -- 172.21.15.28:0/3869297093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 msgr2=0x55e64c54a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.145+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/3869297093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c54a830 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f76e402ab50 tx=0x7f76e4009680 comp rx=0 tx=0).stop 2024-05-09T08:43:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.145+0000 7f76fdfbd740 1 -- 172.21.15.28:0/3869297093 shutdown_connections 2024-05-09T08:43:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.145+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/3869297093 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c557a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.145+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/3869297093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c5550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.145+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/3869297093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c54a830 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.145+0000 7f76fdfbd740 1 -- 172.21.15.28:0/3869297093 >> 172.21.15.28:0/3869297093 conn(0x55e64c546960 msgr2=0x55e64c546d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.149+0000 7f76f904e640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76e00a2ef0 0x7f76e00a53b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55e64c54b9f0 tx=0x7f76f0006c90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.149+0000 7f76fdfbd740 1 -- 172.21.15.28:0/3869297093 shutdown_connections 2024-05-09T08:43:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.151+0000 7f76fdfbd740 1 -- 172.21.15.28:0/3869297093 wait complete. 2024-05-09T08:43:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.152+0000 7f76fdfbd740 1 -- start start 2024-05-09T08:43:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.152+0000 7f76fdfbd740 1 Processor -- start 2024-05-09T08:43:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.153+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c5524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.153+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c552a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.153+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c552f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.153+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e64c558250 con 0x55e64c54a450 2024-05-09T08:43:17.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.153+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55e64c46a790 con 0x55e64c54ad70 2024-05-09T08:43:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.153+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55e64c5583d0 con 0x55e64c555630 2024-05-09T08:43:17.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.153+0000 7f76fdfbd740 10 client.0 _ll_register_callbacks cb 0x55e64c4a5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.155+0000 7f76f884d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c552a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.155+0000 7f76f884d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c552a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:35028/0 (socket says 172.21.15.28:35028) 2024-05-09T08:43:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.155+0000 7f76f884d640 1 -- 172.21.15.28:0/98990271 learned_addr learned my addr 172.21.15.28:0/98990271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.155+0000 7f76f984f640 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c552f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.156+0000 7f76f884d640 1 -- 172.21.15.28:0/98990271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 msgr2=0x55e64c552f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.156+0000 7f76f884d640 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c552f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.156+0000 7f76f884d640 1 -- 172.21.15.28:0/98990271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 msgr2=0x55e64c5524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.156+0000 7f76f884d640 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c5524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.156+0000 7f76f884d640 1 -- 172.21.15.28:0/98990271 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e64c1c3210 con 0x55e64c54ad70 2024-05-09T08:43:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.159+0000 7f76f884d640 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c552a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f76ec097350 tx=0x7f76ec09a910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd94013dc96e89b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.161+0000 7f76dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:43:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.161+0000 7f76dcff9640 1 -- 172.21.15.28:0/98990271 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec09aef0 con 0x55e64c54ad70 2024-05-09T08:43:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.161+0000 7f76dcff9640 1 -- 172.21.15.28:0/98990271 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9911+0+0 (secure 0 0 0) 0x7f76ec097560 con 0x55e64c54ad70 2024-05-09T08:43:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.164+0000 7f76fdfbd740 1 -- 172.21.15.28:0/98990271 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76e4004210 con 0x55e64c54ad70 2024-05-09T08:43:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.168+0000 7f76dcff9640 1 -- 172.21.15.28:0/98990271 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f76ec09f2a0 con 0x55e64c54ad70 2024-05-09T08:43:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.169+0000 7f76fdfbd740 10 client.26732 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:17.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.169+0000 7f76fdfbd740 1 -- 172.21.15.28:0/98990271 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55e64c28f620 con 0x55e64c54ad70 2024-05-09T08:43:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.174+0000 7f76dcff9640 1 -- 172.21.15.28:0/98990271 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e4004210 con 0x55e64c54ad70 2024-05-09T08:43:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.174+0000 7f76dcff9640 1 -- 172.21.15.28:0/98990271 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f76ec0b9ca0 con 0x55e64c54ad70 2024-05-09T08:43:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.175+0000 7f76fdfbd740 10 client.26732 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.175+0000 7f76fdfbd740 10 client.26732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.175+0000 7f76fdfbd740 20 client.26732 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.175+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e64c5653c0 0x55e64c567840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.175+0000 7f76fdfbd740 4 client.26732 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.175+0000 7f76fdfbd740 1 -- 172.21.15.28:0/98990271 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e64c4bca70 con 0x55e64c5653c0 2024-05-09T08:43:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.175+0000 7f76f904e640 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e64c5653c0 0x55e64c567840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.176+0000 7f76f904e640 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e64c5653c0 0x55e64c567840 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e64f35a91b5ebc3a server_cookie=ca7f02ca07278c97 in_seq=0 out_seq=0 2024-05-09T08:43:17.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.179+0000 7f76dcff9640 10 client.26732 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.183+0000 7f76dcff9640 1 -- 172.21.15.28:0/98990271 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e64c4bca70 con 0x55e64c5653c0 2024-05-09T08:43:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.183+0000 7f76dcff9640 10 client.26732 handle_command_reply: tid=0 2024-05-09T08:43:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.185+0000 7f76fdfbd740 1 client.26732 shutdown 2024-05-09T08:43:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.188+0000 7f76fdfbd740 1 -- 172.21.15.28:0/98990271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 msgr2=0x55e64c552a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.188+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c552a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f76ec097350 tx=0x7f76ec09a910 comp rx=0 tx=0).stop 2024-05-09T08:43:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.190+0000 7f76fdfbd740 1 -- 172.21.15.28:0/98990271 shutdown_connections 2024-05-09T08:43:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.190+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e64c5653c0 0x55e64c567840 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.190+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c552f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.190+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c552a00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.190+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/98990271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c5524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.190+0000 7f76fdfbd740 1 -- 172.21.15.28:0/98990271 >> 172.21.15.28:0/98990271 conn(0x55e64c546960 msgr2=0x55e64c54c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.191+0000 7f76dcff9640 0 client.26732 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.199+0000 7f76fdfbd740 1 -- 172.21.15.28:0/98990271 shutdown_connections 2024-05-09T08:43:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.200+0000 7f76fdfbd740 1 -- 172.21.15.28:0/98990271 wait complete. 2024-05-09T08:43:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.200+0000 7f76fdfbd740 20 client.26732 trim_cache size 0 max 16384 2024-05-09T08:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.238+0000 7f76fdfbd740 1 Processor -- start 2024-05-09T08:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.238+0000 7f76fdfbd740 1 -- start start 2024-05-09T08:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.238+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c554bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.238+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c5611c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.238+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c54ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.238+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76d80103a0 con 0x55e64c54a450 2024-05-09T08:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.238+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8012120 con 0x55e64c54ad70 2024-05-09T08:43:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.238+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8010b30 con 0x55e64c555630 2024-05-09T08:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.240+0000 7f76f984f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c54ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.240+0000 7f76f984f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c54ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:39726/0 (socket says 172.21.15.28:39726) 2024-05-09T08:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.240+0000 7f76f984f640 1 -- 172.21.15.28:0/2318201565 learned_addr learned my addr 172.21.15.28:0/2318201565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.241+0000 7f76f884d640 1 --2- 172.21.15.28:0/2318201565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c5611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.243+0000 7f76f884d640 1 -- 172.21.15.28:0/2318201565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 msgr2=0x55e64c54ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.243+0000 7f76f884d640 1 --2- 172.21.15.28:0/2318201565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c54ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.243+0000 7f76f884d640 1 -- 172.21.15.28:0/2318201565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 msgr2=0x55e64c554bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.243+0000 7f76f884d640 1 --2- 172.21.15.28:0/2318201565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c554bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.243+0000 7f76f884d640 1 -- 172.21.15.28:0/2318201565 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e64c54aba0 con 0x55e64c54ad70 2024-05-09T08:43:17.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.248+0000 7f76f884d640 1 --2- 172.21.15.28:0/2318201565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c5611c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00ba20 tx=0x7f76ec09a980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63f8864512bf4bf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.250+0000 7f76deffd640 1 -- 172.21.15.28:0/2318201565 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec07e030 con 0x55e64c54ad70 2024-05-09T08:43:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.250+0000 7f76deffd640 1 -- 172.21.15.28:0/2318201565 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ec09aef0 con 0x55e64c54ad70 2024-05-09T08:43:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.250+0000 7f76deffd640 1 -- 172.21.15.28:0/2318201565 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec009040 con 0x55e64c54ad70 2024-05-09T08:43:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.251+0000 7f76fdfbd740 1 -- 172.21.15.28:0/2318201565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 msgr2=0x55e64c5611c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.251+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/2318201565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c5611c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00ba20 tx=0x7f76ec09a980 comp rx=0 tx=0).stop 2024-05-09T08:43:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.251+0000 7f76fdfbd740 1 -- 172.21.15.28:0/2318201565 shutdown_connections 2024-05-09T08:43:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.251+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/2318201565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c54ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.251+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/2318201565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54ad70 0x55e64c5611c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.251+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/2318201565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c54a450 0x55e64c554bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.251+0000 7f76fdfbd740 1 -- 172.21.15.28:0/2318201565 >> 172.21.15.28:0/2318201565 conn(0x55e64c546960 msgr2=0x55e64c545a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.256+0000 7f76fdfbd740 1 -- 172.21.15.28:0/2318201565 shutdown_connections 2024-05-09T08:43:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.257+0000 7f76fdfbd740 1 -- 172.21.15.28:0/2318201565 wait complete. 2024-05-09T08:43:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.258+0000 7f76fdfbd740 1 -- start start 2024-05-09T08:43:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.258+0000 7f76fdfbd740 1 Processor -- start 2024-05-09T08:43:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.258+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54a450 0x55e64c55d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.258+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c558a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.258+0000 7f76fdfbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c558fc0 0x55e64c559440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.258+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8011a50 con 0x55e64c558fc0 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.258+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f76d80112c0 con 0x55e64c54a450 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.258+0000 7f76fdfbd740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f76d800e430 con 0x55e64c555630 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.258+0000 7f76fdfbd740 10 client.0 _ll_register_callbacks cb 0x7f76ec09ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.259+0000 7f76f904e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54a450 0x55e64c55d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.259+0000 7f76f984f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c558fc0 0x55e64c559440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.259+0000 7f76f984f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c558fc0 0x55e64c559440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52340/0 (socket says 172.21.15.28:52340) 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.259+0000 7f76f984f640 1 -- 172.21.15.28:0/4034696673 learned_addr learned my addr 172.21.15.28:0/4034696673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.260+0000 7f76f904e640 1 -- 172.21.15.28:0/4034696673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 msgr2=0x55e64c558a80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.260+0000 7f76f904e640 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c558a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.260+0000 7f76f904e640 1 -- 172.21.15.28:0/4034696673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c558fc0 msgr2=0x55e64c559440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.260+0000 7f76f904e640 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c558fc0 0x55e64c559440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.260+0000 7f76f904e640 1 -- 172.21.15.28:0/4034696673 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e64c55dfb0 con 0x55e64c54a450 2024-05-09T08:43:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.264+0000 7f76f904e640 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54a450 0x55e64c55d7b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55e64c568090 tx=0x7f76f0013b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec1d3268b13fd009 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.264+0000 7f76dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:43:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.264+0000 7f76dcff9640 1 -- 172.21.15.28:0/4034696673 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f000bb50 con 0x55e64c54a450 2024-05-09T08:43:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.265+0000 7f76fdfbd740 1 -- 172.21.15.28:0/4034696673 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76ec09aef0 con 0x55e64c54a450 2024-05-09T08:43:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.265+0000 7f76dcff9640 1 -- 172.21.15.28:0/4034696673 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9911+0+0 (secure 0 0 0) 0x7f76f000bcf0 con 0x55e64c54a450 2024-05-09T08:43:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.265+0000 7f76dcff9640 1 -- 172.21.15.28:0/4034696673 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f76f0031050 con 0x55e64c54a450 2024-05-09T08:43:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.266+0000 7f76fdfbd740 10 client.26738 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.266+0000 7f76fdfbd740 1 -- 172.21.15.28:0/4034696673 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55e64c5edb30 con 0x55e64c54a450 2024-05-09T08:43:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.268+0000 7f76dcff9640 1 -- 172.21.15.28:0/4034696673 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ec09aef0 con 0x55e64c54a450 2024-05-09T08:43:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.269+0000 7f76dcff9640 1 -- 172.21.15.28:0/4034696673 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f76f0012070 con 0x55e64c54a450 2024-05-09T08:43:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.270+0000 7f76fdfbd740 10 client.26738 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.270+0000 7f76fdfbd740 10 client.26738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.270+0000 7f76fdfbd740 20 client.26738 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.270+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e64c5ee2f0 0x55e64c5f0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.270+0000 7f76fdfbd740 4 client.26738 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:43:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.270+0000 7f76fdfbd740 1 -- 172.21.15.28:0/4034696673 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e64c4b6dc0 con 0x55e64c5ee2f0 2024-05-09T08:43:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.275+0000 7f76f884d640 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e64c5ee2f0 0x55e64c5f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.281+0000 7f76f884d640 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e64c5ee2f0 0x55e64c5f0710 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=407e1552a95caa19 server_cookie=36874a99fe4919bf in_seq=0 out_seq=0 2024-05-09T08:43:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.281+0000 7f76dcff9640 10 client.26738 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76dcff9640 1 -- 172.21.15.28:0/4034696673 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e64c4b6dc0 con 0x55e64c5ee2f0 2024-05-09T08:43:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76dcff9640 10 client.26738 handle_command_reply: tid=0 2024-05-09T08:43:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76fdfbd740 1 client.26738 shutdown 2024-05-09T08:43:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76fdfbd740 1 -- 172.21.15.28:0/4034696673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54a450 msgr2=0x55e64c55d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54a450 0x55e64c55d7b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55e64c568090 tx=0x7f76f0013b50 comp rx=0 tx=0).stop 2024-05-09T08:43:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76fdfbd740 1 -- 172.21.15.28:0/4034696673 shutdown_connections 2024-05-09T08:43:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55e64c5ee2f0 0x55e64c5f0710 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e64c558fc0 0x55e64c559440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55e64c555630 0x55e64c558a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.424+0000 7f76fdfbd740 1 --2- 172.21.15.28:0/4034696673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e64c54a450 0x55e64c55d7b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.425+0000 7f76fdfbd740 1 -- 172.21.15.28:0/4034696673 >> 172.21.15.28:0/4034696673 conn(0x55e64c546960 msgr2=0x55e64c550c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.425+0000 7f76dcff9640 0 client.26738 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.425+0000 7f76fdfbd740 1 -- 172.21.15.28:0/4034696673 shutdown_connections 2024-05-09T08:43:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.425+0000 7f76fdfbd740 1 -- 172.21.15.28:0/4034696673 wait complete. 2024-05-09T08:43:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.425+0000 7f76fdfbd740 20 client.26738 trim_cache size 0 max 16384 2024-05-09T08:43:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:43:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:43:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:43:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:43:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 -- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76e00a2ef0 msgr2=0x7f76e00a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76e00a2ef0 0x7f76e00a53b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55e64c54b9f0 tx=0x7f76f0006c90 comp rx=0 tx=0).stop 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 -- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f41011d0 msgr2=0x7f76f406c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f41011d0 0x7f76f406c070 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00ba50 tx=0x7f76ec00bf10 comp rx=0 tx=0).stop 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 -- 172.21.15.28:0/826940922 shutdown_connections 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f76e00a2ef0 0x7f76e00a53b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76f4105bd0 0x7f76f406c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f41011d0 0x7f76f406c070 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 --2- 172.21.15.28:0/826940922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76f41008b0 0x7f76f4072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 -- 172.21.15.28:0/826940922 >> 172.21.15.28:0/826940922 conn(0x7f76f40778b0 msgr2=0x7f76f4103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 -- 172.21.15.28:0/826940922 shutdown_connections 2024-05-09T08:43:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:17.426+0000 7f76fb2b2640 1 -- 172.21.15.28:0/826940922 wait complete. 2024-05-09T08:43:17.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:17 smithi165 ceph-mon[34540]: pgmap v544: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 961 KiB/s wr, 162 op/s 2024-05-09T08:43:17.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:17 smithi184 ceph-mon[34509]: pgmap v544: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 961 KiB/s wr, 162 op/s 2024-05-09T08:43:17.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:17 smithi028 ceph-mon[30119]: pgmap v544: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 961 KiB/s wr, 162 op/s 2024-05-09T08:43:19.182 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:43:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:43:18.704+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:43:19.182 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:43:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:43:18.705+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:43:19.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:19 smithi184 ceph-mon[34509]: pgmap v545: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-09T08:43:19.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:19 smithi028 ceph-mon[30119]: pgmap v545: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-09T08:43:20.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:19 smithi165 ceph-mon[34540]: pgmap v545: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-09T08:43:22.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:21 smithi165 ceph-mon[34540]: pgmap v546: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-05-09T08:43:22.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:21 smithi165 ceph-mon[34540]: osdmap e127: 12 total, 12 up, 12 in 2024-05-09T08:43:22.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:21 smithi184 ceph-mon[34509]: pgmap v546: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-05-09T08:43:22.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:21 smithi184 ceph-mon[34509]: osdmap e127: 12 total, 12 up, 12 in 2024-05-09T08:43:22.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:21 smithi028 ceph-mon[30119]: pgmap v546: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 172 op/s 2024-05-09T08:43:22.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:21 smithi028 ceph-mon[30119]: osdmap e127: 12 total, 12 up, 12 in 2024-05-09T08:43:24.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:23 smithi165 ceph-mon[34540]: pgmap v548: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-05-09T08:43:24.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:23 smithi184 ceph-mon[34509]: pgmap v548: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-05-09T08:43:24.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:23 smithi028 ceph-mon[30119]: pgmap v548: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-05-09T08:43:26.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:25 smithi165 ceph-mon[34540]: pgmap v549: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 181 op/s 2024-05-09T08:43:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:25 smithi184 ceph-mon[34509]: pgmap v549: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 181 op/s 2024-05-09T08:43:26.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:25 smithi028 ceph-mon[30119]: pgmap v549: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 181 op/s 2024-05-09T08:43:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:27 smithi165 ceph-mon[34540]: pgmap v550: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 168 op/s 2024-05-09T08:43:28.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:27 smithi184 ceph-mon[34509]: pgmap v550: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 168 op/s 2024-05-09T08:43:28.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:27 smithi028 ceph-mon[30119]: pgmap v550: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 168 op/s 2024-05-09T08:43:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:29 smithi165 ceph-mon[34540]: pgmap v551: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-09T08:43:30.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:29 smithi184 ceph-mon[34509]: pgmap v551: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-09T08:43:30.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:29 smithi028 ceph-mon[30119]: pgmap v551: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-05-09T08:43:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.595+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/2757662369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec110400 msgr2=0x7f1fec108d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.595+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/2757662369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec110400 0x7f1fec108d70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009900 tx=0x7f1fd802a4b0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.596+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/2757662369 shutdown_connections 2024-05-09T08:43:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.596+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/2757662369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1fec11b010 0x7f1fec11d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.596+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/2757662369 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec110400 0x7f1fec108d70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.596+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/2757662369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec110020 0x7f1fec108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.596+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/2757662369 >> 172.21.15.28:0/2757662369 conn(0x7f1fec06c900 msgr2=0x7f1fec06cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.603+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/2757662369 shutdown_connections 2024-05-09T08:43:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.606+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/2757662369 wait complete. 2024-05-09T08:43:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.606+0000 7f1ff1ad4640 1 Processor -- start 2024-05-09T08:43:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.607+0000 7f1ff1ad4640 1 -- start start 2024-05-09T08:43:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.607+0000 7f1ff1ad4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1fec110020 0x7f1fec118500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.607+0000 7f1ff1ad4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec11b010 0x7f1fec118a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.607+0000 7f1ff1ad4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec111670 0x7f1fec111b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.607+0000 7f1ff1ad4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec120a80 con 0x7f1fec111670 2024-05-09T08:43:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.607+0000 7f1ff1ad4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec120900 con 0x7f1fec110020 2024-05-09T08:43:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.607+0000 7f1ff1ad4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec120c00 con 0x7f1fec11b010 2024-05-09T08:43:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.608+0000 7f1feaffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec11b010 0x7f1fec118a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.608+0000 7f1feaffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec11b010 0x7f1fec118a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51326/0 (socket says 172.21.15.28:51326) 2024-05-09T08:43:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.608+0000 7f1feaffd640 1 -- 172.21.15.28:0/547370174 learned_addr learned my addr 172.21.15.28:0/547370174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.609+0000 7f1feaffd640 1 -- 172.21.15.28:0/547370174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1fec110020 msgr2=0x7f1fec118500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:43:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.609+0000 7f1feaffd640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1fec110020 0x7f1fec118500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.609+0000 7f1feaffd640 1 -- 172.21.15.28:0/547370174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec111670 msgr2=0x7f1fec111b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.609+0000 7f1feaffd640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec111670 0x7f1fec111b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.609+0000 7f1feaffd640 1 -- 172.21.15.28:0/547370174 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec116020 con 0x7f1fec11b010 2024-05-09T08:43:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.610+0000 7f1feaffd640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec11b010 0x7f1fec118a40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1fd802aa20 tx=0x7f1fd8003b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.611+0000 7f1fe8ff9640 1 -- 172.21.15.28:0/547370174 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd803f070 con 0x7f1fec11b010 2024-05-09T08:43:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.611+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/547370174 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fec1162b0 con 0x7f1fec11b010 2024-05-09T08:43:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.611+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/547370174 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fec1167a0 con 0x7f1fec11b010 2024-05-09T08:43:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.612+0000 7f1fe8ff9640 1 -- 172.21.15.28:0/547370174 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd8031c90 con 0x7f1fec11b010 2024-05-09T08:43:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.612+0000 7f1fe8ff9640 1 -- 172.21.15.28:0/547370174 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8035030 con 0x7f1fec11b010 2024-05-09T08:43:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.617+0000 7f1fe8ff9640 1 -- 172.21.15.28:0/547370174 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1fd8042070 con 0x7f1fec11b010 2024-05-09T08:43:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.618+0000 7f1ff47e5740 1 Processor -- start 2024-05-09T08:43:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.620+0000 7f1fe8ff9640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1fd00a2fd0 0x7f1fd00a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.620+0000 7f1fe8ff9640 1 -- 172.21.15.28:0/547370174 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9835+0+0 (secure 0 0 0) 0x7f1fd80f11c0 con 0x7f1fec11b010 2024-05-09T08:43:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.620+0000 7f1febfff640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1fd00a2fd0 0x7f1fd00a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.621+0000 7f1ff47e5740 1 -- start start 2024-05-09T08:43:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.621+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019e450 0x55a53019e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.621+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 0x55a5301a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.621+0000 7f1febfff640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1fd00a2fd0 0x7f1fd00a5490 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1fe40092f0 tx=0x7f1fe4009220 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.621+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301aba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.621+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a530103340 con 0x55a53019ed70 2024-05-09T08:43:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.621+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a5300ab290 con 0x55a53019e450 2024-05-09T08:43:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.621+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a5300ff300 con 0x55a5301a9630 2024-05-09T08:43:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.622+0000 7f1febfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301aba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.622+0000 7f1febfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301aba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51336/0 (socket says 172.21.15.28:51336) 2024-05-09T08:43:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.622+0000 7f1febfff640 1 -- 172.21.15.28:0/2105474095 learned_addr learned my addr 172.21.15.28:0/2105474095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.623+0000 7f1febfff640 1 -- 172.21.15.28:0/2105474095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019e450 msgr2=0x55a53019e830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:43:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.623+0000 7f1febfff640 1 --2- 172.21.15.28:0/2105474095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019e450 0x55a53019e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.623+0000 7f1febfff640 1 -- 172.21.15.28:0/2105474095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 msgr2=0x55a5301a90f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:43:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.623+0000 7f1febfff640 1 --2- 172.21.15.28:0/2105474095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 0x55a5301a90f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.623+0000 7f1febfff640 1 -- 172.21.15.28:0/2105474095 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5300cf8e0 con 0x55a5301a9630 2024-05-09T08:43:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.627+0000 7f1febfff640 1 --2- 172.21.15.28:0/2105474095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301aba20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1fe40093f0 tx=0x7f1fe4011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=446c9297d6673313 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.631+0000 7f1fe1ffb640 1 -- 172.21.15.28:0/2105474095 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe401e640 con 0x55a5301a9630 2024-05-09T08:43:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.631+0000 7f1fe1ffb640 1 -- 172.21.15.28:0/2105474095 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe4023070 con 0x55a5301a9630 2024-05-09T08:43:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.631+0000 7f1fe1ffb640 1 -- 172.21.15.28:0/2105474095 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4010be0 con 0x55a5301a9630 2024-05-09T08:43:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.632+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2105474095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 msgr2=0x55a5301aba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.632+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/2105474095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301aba20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1fe40093f0 tx=0x7f1fe4011000 comp rx=0 tx=0).stop 2024-05-09T08:43:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.633+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2105474095 shutdown_connections 2024-05-09T08:43:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.633+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/2105474095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301aba20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.633+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/2105474095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 0x55a5301a90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.633+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/2105474095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019e450 0x55a53019e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.633+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2105474095 >> 172.21.15.28:0/2105474095 conn(0x55a53019a960 msgr2=0x55a53019ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.641+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2105474095 shutdown_connections 2024-05-09T08:43:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.642+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2105474095 wait complete. 2024-05-09T08:43:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.642+0000 7f1ff47e5740 1 -- start start 2024-05-09T08:43:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.642+0000 7f1ff47e5740 1 Processor -- start 2024-05-09T08:43:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.646+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a53019e450 0x55a5301a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.646+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019ed70 0x55a5301a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.646+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5301a9630 0x55a5301a7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.646+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5301ac200 con 0x55a5301a9630 2024-05-09T08:43:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.646+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55a5300be790 con 0x55a53019ed70 2024-05-09T08:43:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.646+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a5301ac380 con 0x55a53019e450 2024-05-09T08:43:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.646+0000 7f1ff47e5740 10 client.0 _ll_register_callbacks cb 0x55a5300f9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.649+0000 7f1febfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5301a9630 0x55a5301a7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.649+0000 7f1febfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5301a9630 0x55a5301a7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49544/0 (socket says 172.21.15.28:49544) 2024-05-09T08:43:30.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.649+0000 7f1febfff640 1 -- 172.21.15.28:0/3514429456 learned_addr learned my addr 172.21.15.28:0/3514429456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.651+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a53019e450 0x55a5301a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.655+0000 7f1febfff640 1 -- 172.21.15.28:0/3514429456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a53019e450 msgr2=0x55a5301a65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.655+0000 7f1febfff640 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a53019e450 0x55a5301a65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.655+0000 7f1febfff640 1 -- 172.21.15.28:0/3514429456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019ed70 msgr2=0x55a5301a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.655+0000 7f1feaffd640 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019ed70 0x55a5301a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.655+0000 7f1febfff640 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019ed70 0x55a5301a6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.655+0000 7f1febfff640 1 -- 172.21.15.28:0/3514429456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a52fe17210 con 0x55a5301a9630 2024-05-09T08:43:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.657+0000 7f1febfff640 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5301a9630 0x55a5301a7030 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4004a80 tx=0x7f1fe4010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723da905135227df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.658+0000 7f1fceffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:43:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.661+0000 7f1fceffd640 1 -- 172.21.15.28:0/3514429456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe401e850 con 0x55a5301a9630 2024-05-09T08:43:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.661+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3514429456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fe4023070 con 0x55a5301a9630 2024-05-09T08:43:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.661+0000 7f1fceffd640 1 -- 172.21.15.28:0/3514429456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9835+0+0 (secure 0 0 0) 0x7f1fe401e9f0 con 0x55a5301a9630 2024-05-09T08:43:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.664+0000 7f1fceffd640 1 -- 172.21.15.28:0/3514429456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f1fe403c050 con 0x55a5301a9630 2024-05-09T08:43:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.664+0000 7f1ff47e5740 10 client.17298 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.664+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3514429456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55a52fee3620 con 0x55a5301a9630 2024-05-09T08:43:30.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.668+0000 7f1fceffd640 1 -- 172.21.15.28:0/3514429456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe4023070 con 0x55a5301a9630 2024-05-09T08:43:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.670+0000 7f1fceffd640 1 -- 172.21.15.28:0/3514429456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f1fe400f040 con 0x55a5301a9630 2024-05-09T08:43:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.670+0000 7f1ff47e5740 10 client.17298 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.670+0000 7f1ff47e5740 10 client.17298 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.670+0000 7f1ff47e5740 20 client.17298 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.670+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a5301b9710 0x55a5301bbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.670+0000 7f1ff47e5740 4 client.17298 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.670+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3514429456 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a530110a70 con 0x55a5301b9710 2024-05-09T08:43:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.670+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a5301b9710 0x55a5301bbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.671+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a5301b9710 0x55a5301bbb90 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e89946efaba7888 server_cookie=7ccf18628d597de3 in_seq=0 out_seq=0 2024-05-09T08:43:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.671+0000 7f1fceffd640 10 client.17298 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.672+0000 7f1fceffd640 1 -- 172.21.15.28:0/3514429456 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a530110a70 con 0x55a5301b9710 2024-05-09T08:43:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.673+0000 7f1fceffd640 10 client.17298 handle_command_reply: tid=0 2024-05-09T08:43:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.674+0000 7f1ff47e5740 1 client.17298 shutdown 2024-05-09T08:43:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.678+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3514429456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5301a9630 msgr2=0x55a5301a7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.678+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5301a9630 0x55a5301a7030 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4004a80 tx=0x7f1fe4010e70 comp rx=0 tx=0).stop 2024-05-09T08:43:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.682+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3514429456 shutdown_connections 2024-05-09T08:43:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.682+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a5301b9710 0x55a5301bbb90 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.682+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5301a9630 0x55a5301a7030 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.682+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019ed70 0x55a5301a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.682+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3514429456 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a53019e450 0x55a5301a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.682+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3514429456 >> 172.21.15.28:0/3514429456 conn(0x55a53019a960 msgr2=0x55a5301a06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.683+0000 7f1fceffd640 0 client.17298 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:30.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.689+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3514429456 shutdown_connections 2024-05-09T08:43:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.690+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3514429456 wait complete. 2024-05-09T08:43:30.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.691+0000 7f1ff47e5740 20 client.17298 trim_cache size 0 max 16384 2024-05-09T08:43:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.730+0000 7f1ff47e5740 1 Processor -- start 2024-05-09T08:43:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.731+0000 7f1ff47e5740 1 -- start start 2024-05-09T08:43:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.731+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019e450 0x55a5301b5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.731+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019ed70 0x55a53019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.731+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.731+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd40102f0 con 0x55a53019e450 2024-05-09T08:43:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.731+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd4012070 con 0x55a53019ed70 2024-05-09T08:43:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.731+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd4010a80 con 0x55a5301a9630 2024-05-09T08:43:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.732+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019e450 0x55a5301b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.732+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019e450 0x55a5301b5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49560/0 (socket says 172.21.15.28:49560) 2024-05-09T08:43:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.732+0000 7f1feb7fe640 1 -- 172.21.15.28:0/2455342822 learned_addr learned my addr 172.21.15.28:0/2455342822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.735+0000 7f1febfff640 1 --2- 172.21.15.28:0/2455342822 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.735+0000 7f1feb7fe640 1 -- 172.21.15.28:0/2455342822 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 msgr2=0x55a5301a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.735+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/2455342822 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.735+0000 7f1feb7fe640 1 -- 172.21.15.28:0/2455342822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019ed70 msgr2=0x55a53019ad40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.735+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/2455342822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019ed70 0x55a53019ad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.736+0000 7f1feb7fe640 1 -- 172.21.15.28:0/2455342822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5301a55c0 con 0x55a53019e450 2024-05-09T08:43:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.740+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/2455342822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019e450 0x55a5301b5120 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0098e0 tx=0x7f1fdc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c34c94b0a45ec0a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.742+0000 7f1fe17fa640 1 -- 172.21.15.28:0/2455342822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0128a0 con 0x55a53019e450 2024-05-09T08:43:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.742+0000 7f1fe17fa640 1 -- 172.21.15.28:0/2455342822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fdc01c030 con 0x55a53019e450 2024-05-09T08:43:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.742+0000 7f1fe17fa640 1 -- 172.21.15.28:0/2455342822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc020630 con 0x55a53019e450 2024-05-09T08:43:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.745+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2455342822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019e450 msgr2=0x55a5301b5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.745+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/2455342822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019e450 0x55a5301b5120 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0098e0 tx=0x7f1fdc005850 comp rx=0 tx=0).stop 2024-05-09T08:43:30.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.746+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2455342822 shutdown_connections 2024-05-09T08:43:30.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.746+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/2455342822 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a9630 0x55a5301a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.746+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/2455342822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a53019ed70 0x55a53019ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.746+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/2455342822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019e450 0x55a5301b5120 secure :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0098e0 tx=0x7f1fdc005850 comp rx=0 tx=0).stop 2024-05-09T08:43:30.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.746+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2455342822 >> 172.21.15.28:0/2455342822 conn(0x55a53019a960 msgr2=0x55a530199a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.753+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2455342822 shutdown_connections 2024-05-09T08:43:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.754+0000 7f1ff47e5740 1 -- 172.21.15.28:0/2455342822 wait complete. 2024-05-09T08:43:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.754+0000 7f1ff47e5740 1 -- start start 2024-05-09T08:43:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.755+0000 7f1ff47e5740 1 Processor -- start 2024-05-09T08:43:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.757+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 0x55a5301b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.757+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a5301a9630 0x55a5301a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.757+0000 7f1ff47e5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a3500 0x55a5301a3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.757+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd40119a0 con 0x55a53019ed70 2024-05-09T08:43:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.757+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd4011210 con 0x55a5301a9630 2024-05-09T08:43:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.757+0000 7f1ff47e5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd400e380 con 0x55a5301a3500 2024-05-09T08:43:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.757+0000 7f1ff47e5740 10 client.0 _ll_register_callbacks cb 0x7f1fe4009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.757+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 0x55a5301b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.758+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 0x55a5301b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49568/0 (socket says 172.21.15.28:49568) 2024-05-09T08:43:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.758+0000 7f1feb7fe640 1 -- 172.21.15.28:0/3351188825 learned_addr learned my addr 172.21.15.28:0/3351188825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.758+0000 7f1feaffd640 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a5301a9630 0x55a5301a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.758+0000 7f1febfff640 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a3500 0x55a5301a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.758+0000 7f1feaffd640 1 -- 172.21.15.28:0/3351188825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a3500 msgr2=0x55a5301a3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.758+0000 7f1feaffd640 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a3500 0x55a5301a3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.758+0000 7f1feaffd640 1 -- 172.21.15.28:0/3351188825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 msgr2=0x55a5301b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.758+0000 7f1feaffd640 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 0x55a5301b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.758+0000 7f1feaffd640 1 -- 172.21.15.28:0/3351188825 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53021ede0 con 0x55a5301a9630 2024-05-09T08:43:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.759+0000 7f1feaffd640 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a5301a9630 0x55a5301a2fc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80ee390 tx=0x7f1fd80bcd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e766d7d68433fad server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.759+0000 7f1fceffd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:43:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.759+0000 7f1fceffd640 1 -- 172.21.15.28:0/3351188825 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8049040 con 0x55a5301a9630 2024-05-09T08:43:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.759+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3351188825 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fdc01c030 con 0x55a5301a9630 2024-05-09T08:43:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.759+0000 7f1fceffd640 1 -- 172.21.15.28:0/3351188825 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9835+0+0 (secure 0 0 0) 0x7f1fd80dd0c0 con 0x55a5301a9630 2024-05-09T08:43:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.760+0000 7f1fceffd640 1 -- 172.21.15.28:0/3351188825 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f1fd80bd160 con 0x55a5301a9630 2024-05-09T08:43:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.762+0000 7f1ff47e5740 10 client.26744 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.762+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3351188825 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55a530220140 con 0x55a5301a9630 2024-05-09T08:43:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.766+0000 7f1fceffd640 1 -- 172.21.15.28:0/3351188825 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fdc01c030 con 0x55a5301a9630 2024-05-09T08:43:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.766+0000 7f1fceffd640 1 -- 172.21.15.28:0/3351188825 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f1fd80c1120 con 0x55a5301a9630 2024-05-09T08:43:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.768+0000 7f1ff47e5740 10 client.26744 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.768+0000 7f1ff47e5740 10 client.26744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.768+0000 7f1ff47e5740 20 client.26744 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.768+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a530220910 0x55a530222d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.768+0000 7f1ff47e5740 4 client.26744 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:43:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.768+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3351188825 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a53010adc0 con 0x55a530220910 2024-05-09T08:43:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.769+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a530220910 0x55a530222d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:30.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.786+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a530220910 0x55a530222d30 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88da570249a5847f server_cookie=cf86f1310a713086 in_seq=0 out_seq=0 2024-05-09T08:43:30.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.787+0000 7f1fceffd640 10 client.26744 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1fceffd640 1 -- 172.21.15.28:0/3351188825 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a53010adc0 con 0x55a530220910 2024-05-09T08:43:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1fceffd640 10 client.26744 handle_command_reply: tid=0 2024-05-09T08:43:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 client.26744 shutdown 2024-05-09T08:43:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3351188825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a5301a9630 msgr2=0x55a5301a2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a5301a9630 0x55a5301a2fc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80ee390 tx=0x7f1fd80bcd60 comp rx=0 tx=0).stop 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3351188825 shutdown_connections 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55a530220910 0x55a530222d30 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a5301a3500 0x55a5301a3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a5301a9630 0x55a5301a2fc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 --2- 172.21.15.28:0/3351188825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a53019ed70 0x55a5301b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3351188825 >> 172.21.15.28:0/3351188825 conn(0x55a53019a960 msgr2=0x55a530199a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1fceffd640 0 client.26744 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3351188825 shutdown_connections 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.884+0000 7f1ff47e5740 1 -- 172.21.15.28:0/3351188825 wait complete. 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.885+0000 7f1ff47e5740 20 client.26744 trim_cache size 0 max 16384 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:43:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:43:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:43:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:43:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.887+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/547370174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1fd00a2fd0 msgr2=0x7f1fd00a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.887+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1fd00a2fd0 0x7f1fd00a5490 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1fe40092f0 tx=0x7f1fe4009220 comp rx=0 tx=0).stop 2024-05-09T08:43:30.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.887+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/547370174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec11b010 msgr2=0x7f1fec118a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.887+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec11b010 0x7f1fec118a40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1fd802aa20 tx=0x7f1fd8003b70 comp rx=0 tx=0).stop 2024-05-09T08:43:30.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.887+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/547370174 shutdown_connections 2024-05-09T08:43:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.887+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec111670 0x7f1fec111b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.887+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f1fd00a2fd0 0x7f1fd00a5490 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.888+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fec11b010 0x7f1fec118a40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.888+0000 7f1ff1ad4640 1 --2- 172.21.15.28:0/547370174 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1fec110020 0x7f1fec118500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.888+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/547370174 >> 172.21.15.28:0/547370174 conn(0x7f1fec06c900 msgr2=0x7f1fec10dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.889+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/547370174 shutdown_connections 2024-05-09T08:43:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:30.891+0000 7f1ff1ad4640 1 -- 172.21.15.28:0/547370174 wait complete. 2024-05-09T08:43:31.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:31 smithi028 ceph-mon[30119]: pgmap v552: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-09T08:43:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:31 smithi165 ceph-mon[34540]: pgmap v552: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-09T08:43:32.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:31 smithi184 ceph-mon[34509]: pgmap v552: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-05-09T08:43:34.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:33 smithi165 ceph-mon[34540]: pgmap v553: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 155 op/s 2024-05-09T08:43:34.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:33 smithi184 ceph-mon[34509]: pgmap v553: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 155 op/s 2024-05-09T08:43:34.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:33 smithi028 ceph-mon[30119]: pgmap v553: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 155 op/s 2024-05-09T08:43:36.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:35 smithi165 ceph-mon[34540]: pgmap v554: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 161 op/s 2024-05-09T08:43:36.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:35 smithi184 ceph-mon[34509]: pgmap v554: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 161 op/s 2024-05-09T08:43:36.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:35 smithi028 ceph-mon[30119]: pgmap v554: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 161 op/s 2024-05-09T08:43:38.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:37 smithi165 ceph-mon[34540]: pgmap v555: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-05-09T08:43:38.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:37 smithi184 ceph-mon[34509]: pgmap v555: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-05-09T08:43:38.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:37 smithi028 ceph-mon[30119]: pgmap v555: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-05-09T08:43:40.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:39 smithi165 ceph-mon[34540]: pgmap v556: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-05-09T08:43:40.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:39 smithi184 ceph-mon[34509]: pgmap v556: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-05-09T08:43:40.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:39 smithi028 ceph-mon[30119]: pgmap v556: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 146 op/s 2024-05-09T08:43:42.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:41 smithi165 ceph-mon[34540]: pgmap v557: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 166 op/s 2024-05-09T08:43:42.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:41 smithi184 ceph-mon[34509]: pgmap v557: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 166 op/s 2024-05-09T08:43:42.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:41 smithi028 ceph-mon[30119]: pgmap v557: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 166 op/s 2024-05-09T08:43:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.063+0000 7f581c9a0640 1 -- 172.21.15.28:0/1243315580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58181102d0 msgr2=0x7f581810dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.063+0000 7f581c9a0640 1 --2- 172.21.15.28:0/1243315580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58181102d0 0x7f581810dee0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5810009ca0 tx=0x7f581002a980 comp rx=0 tx=0).stop 2024-05-09T08:43:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.065+0000 7f581c9a0640 1 -- 172.21.15.28:0/1243315580 shutdown_connections 2024-05-09T08:43:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.065+0000 7f581c9a0640 1 --2- 172.21.15.28:0/1243315580 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f581810e500 0x7f581811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.065+0000 7f581c9a0640 1 --2- 172.21.15.28:0/1243315580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58181102d0 0x7f581810dee0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.065+0000 7f581c9a0640 1 --2- 172.21.15.28:0/1243315580 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f581810fef0 0x7f581810d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.065+0000 7f581c9a0640 1 -- 172.21.15.28:0/1243315580 >> 172.21.15.28:0/1243315580 conn(0x7f581806c770 msgr2=0x7f581806cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:44.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.072+0000 7f581c9a0640 1 -- 172.21.15.28:0/1243315580 shutdown_connections 2024-05-09T08:43:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.074+0000 7f581c9a0640 1 -- 172.21.15.28:0/1243315580 wait complete. 2024-05-09T08:43:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.075+0000 7f581c9a0640 1 Processor -- start 2024-05-09T08:43:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.077+0000 7f581c9a0640 1 -- start start 2024-05-09T08:43:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.077+0000 7f581c9a0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f581810e500 0x7f581811c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.077+0000 7f581c9a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581810fef0 0x7f58181156e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.077+0000 7f581c9a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58181102d0 0x7f5818115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.077+0000 7f581c9a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5818120970 con 0x7f581810fef0 2024-05-09T08:43:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.077+0000 7f581c9a0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f58181207f0 con 0x7f581810e500 2024-05-09T08:43:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.077+0000 7f581c9a0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5818120af0 con 0x7f58181102d0 2024-05-09T08:43:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.080+0000 7f5816ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581810fef0 0x7f58181156e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.080+0000 7f5816ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581810fef0 0x7f58181156e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46962/0 (socket says 172.21.15.28:46962) 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.080+0000 7f5816ffd640 1 -- 172.21.15.28:0/4090513902 learned_addr learned my addr 172.21.15.28:0/4090513902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.080+0000 7f5816ffd640 1 -- 172.21.15.28:0/4090513902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58181102d0 msgr2=0x7f5818115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.080+0000 7f5816ffd640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58181102d0 0x7f5818115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.080+0000 7f5816ffd640 1 -- 172.21.15.28:0/4090513902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f581810e500 msgr2=0x7f581811c660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.080+0000 7f5816ffd640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f581810e500 0x7f581811c660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.080+0000 7f5816ffd640 1 -- 172.21.15.28:0/4090513902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58181161b0 con 0x7f581810fef0 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.081+0000 7f5816ffd640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581810fef0 0x7f58181156e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5810006010 tx=0x7f58100308b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.081+0000 7f5814ff9640 1 -- 172.21.15.28:0/4090513902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5810034050 con 0x7f581810fef0 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.081+0000 7f581c9a0640 1 -- 172.21.15.28:0/4090513902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5818116440 con 0x7f581810fef0 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.081+0000 7f581c9a0640 1 -- 172.21.15.28:0/4090513902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f581811a3c0 con 0x7f581810fef0 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.081+0000 7f5814ff9640 1 -- 172.21.15.28:0/4090513902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5810004040 con 0x7f581810fef0 2024-05-09T08:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.081+0000 7f5814ff9640 1 -- 172.21.15.28:0/4090513902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5810039760 con 0x7f581810fef0 2024-05-09T08:43:44.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.086+0000 7f58209ea740 1 Processor -- start 2024-05-09T08:43:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.086+0000 7f58209ea740 1 -- start start 2024-05-09T08:43:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f58209ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1092235d0 0x55f1092239b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f58209ea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f109223ef0 0x55f10922e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f58209ea740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f10922e7b0 0x55f109230ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f58209ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f109188340 con 0x55f1092235d0 2024-05-09T08:43:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f58209ea740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f109130290 con 0x55f10922e7b0 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f58209ea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f109184300 con 0x55f109223ef0 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f5814ff9640 1 -- 172.21.15.28:0/4090513902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5810037020 con 0x7f581810fef0 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f5816ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f10922e7b0 0x55f109230ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f5816ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f10922e7b0 0x55f109230ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60814/0 (socket says 172.21.15.28:60814) 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.087+0000 7f5816ffd640 1 -- 172.21.15.28:0/1762441250 learned_addr learned my addr 172.21.15.28:0/1762441250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.088+0000 7f5816ffd640 1 -- 172.21.15.28:0/1762441250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f109223ef0 msgr2=0x55f10922e270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.088+0000 7f5816ffd640 1 --2- 172.21.15.28:0/1762441250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f109223ef0 0x55f10922e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.088+0000 7f5816ffd640 1 -- 172.21.15.28:0/1762441250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1092235d0 msgr2=0x55f1092239b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.088+0000 7f5816ffd640 1 --2- 172.21.15.28:0/1762441250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1092235d0 0x55f1092239b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.088+0000 7f5816ffd640 1 -- 172.21.15.28:0/1762441250 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1091548e0 con 0x55f10922e7b0 2024-05-09T08:43:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.088+0000 7f5816ffd640 1 --2- 172.21.15.28:0/1762441250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f10922e7b0 0x55f109230ba0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f58100ed980 tx=0x7f58100f6e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=336676784a3ea05b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.089+0000 7f5814ff9640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f58000a2f00 0x7f58000a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.089+0000 7f5817fff640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f58000a2f00 0x7f58000a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.090+0000 7f5817fff640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f58000a2f00 0x7f58000a53c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5808009650 tx=0x7f5808006cb0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.090+0000 7f5814ff9640 1 -- 172.21.15.28:0/4090513902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9835+0+0 (secure 0 0 0) 0x7f58100f0870 con 0x7f581810fef0 2024-05-09T08:43:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.090+0000 7f57fdffb640 1 -- 172.21.15.28:0/1762441250 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5810100020 con 0x55f10922e7b0 2024-05-09T08:43:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.090+0000 7f57fdffb640 1 -- 172.21.15.28:0/1762441250 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58100f8d50 con 0x55f10922e7b0 2024-05-09T08:43:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.091+0000 7f57fdffb640 1 -- 172.21.15.28:0/1762441250 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58100f2500 con 0x55f10922e7b0 2024-05-09T08:43:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.092+0000 7f58209ea740 1 -- 172.21.15.28:0/1762441250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f10922e7b0 msgr2=0x55f109230ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:44.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.092+0000 7f58209ea740 1 --2- 172.21.15.28:0/1762441250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f10922e7b0 0x55f109230ba0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f58100ed980 tx=0x7f58100f6e90 comp rx=0 tx=0).stop 2024-05-09T08:43:44.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.093+0000 7f58209ea740 1 -- 172.21.15.28:0/1762441250 shutdown_connections 2024-05-09T08:43:44.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.093+0000 7f58209ea740 1 --2- 172.21.15.28:0/1762441250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f10922e7b0 0x55f109230ba0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.093+0000 7f58209ea740 1 --2- 172.21.15.28:0/1762441250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f109223ef0 0x55f10922e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.093+0000 7f58209ea740 1 --2- 172.21.15.28:0/1762441250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1092235d0 0x55f1092239b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.093+0000 7f58209ea740 1 -- 172.21.15.28:0/1762441250 >> 172.21.15.28:0/1762441250 conn(0x55f10921fae0 msgr2=0x55f10921fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:44.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.098+0000 7f58209ea740 1 -- 172.21.15.28:0/1762441250 shutdown_connections 2024-05-09T08:43:44.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:43 smithi165 ceph-mon[34540]: pgmap v558: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-05-09T08:43:44.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:43 smithi184 ceph-mon[34509]: pgmap v558: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-05-09T08:43:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.107+0000 7f58209ea740 1 -- 172.21.15.28:0/1762441250 wait complete. 2024-05-09T08:43:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.107+0000 7f58209ea740 1 -- start start 2024-05-09T08:43:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.107+0000 7f58209ea740 1 Processor -- start 2024-05-09T08:43:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.108+0000 7f58209ea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 0x55f10922b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.108+0000 7f58209ea740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f109223ef0 0x55f10922bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.108+0000 7f58209ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.108+0000 7f58209ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f109130290 con 0x55f10922e7b0 2024-05-09T08:43:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.108+0000 7f58209ea740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f109143790 con 0x55f109223ef0 2024-05-09T08:43:44.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.108+0000 7f58209ea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f1092311a0 con 0x55f1092235d0 2024-05-09T08:43:44.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.108+0000 7f58209ea740 10 client.0 _ll_register_callbacks cb 0x55f10917ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:44.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.109+0000 7f5816ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f109223ef0 0x55f10922bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.109+0000 7f58177fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 0x55f10922b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.109+0000 7f58177fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 0x55f10922b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:36712/0 (socket says 172.21.15.28:36712) 2024-05-09T08:43:44.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.109+0000 7f58177fe640 1 -- 172.21.15.28:0/96328529 learned_addr learned my addr 172.21.15.28:0/96328529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.110+0000 7f5817fff640 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.110+0000 7f58177fe640 1 -- 172.21.15.28:0/96328529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f109223ef0 msgr2=0x55f10922bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.110+0000 7f58177fe640 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f109223ef0 0x55f10922bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.110+0000 7f58177fe640 1 -- 172.21.15.28:0/96328529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 msgr2=0x55f10922c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.111+0000 7f58177fe640 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.111+0000 7f58177fe640 1 -- 172.21.15.28:0/96328529 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f108e9c210 con 0x55f1092235d0 2024-05-09T08:43:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.111+0000 7f58177fe640 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 0x55f10922b730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f580c00c2d0 tx=0x7f580c00c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4107efd9dc3c761 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.115+0000 7f57f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:43:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.115+0000 7f57f6ffd640 1 -- 172.21.15.28:0/96328529 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c00f860 con 0x55f1092235d0 2024-05-09T08:43:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.115+0000 7f57f6ffd640 1 -- 172.21.15.28:0/96328529 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9835+0+0 (secure 0 0 0) 0x7f580c00fa00 con 0x55f1092235d0 2024-05-09T08:43:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.117+0000 7f58209ea740 1 -- 172.21.15.28:0/96328529 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58100f8d50 con 0x55f1092235d0 2024-05-09T08:43:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.118+0000 7f57f6ffd640 1 -- 172.21.15.28:0/96328529 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f580c00fc00 con 0x55f1092235d0 2024-05-09T08:43:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.123+0000 7f58209ea740 10 client.26602 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.123+0000 7f58209ea740 1 -- 172.21.15.28:0/96328529 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55f108f68620 con 0x55f1092235d0 2024-05-09T08:43:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.124+0000 7f57f6ffd640 1 -- 172.21.15.28:0/96328529 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58100f8d50 con 0x55f1092235d0 2024-05-09T08:43:44.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.125+0000 7f57f6ffd640 1 -- 172.21.15.28:0/96328529 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f580c018490 con 0x55f1092235d0 2024-05-09T08:43:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.125+0000 7f58209ea740 10 client.26602 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.125+0000 7f58209ea740 10 client.26602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.126+0000 7f58209ea740 20 client.26602 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.126+0000 7f58209ea740 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f10923e210 0x55f1092406a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.126+0000 7f58209ea740 4 client.26602 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.126+0000 7f58209ea740 1 -- 172.21.15.28:0/96328529 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f109195a70 con 0x55f10923e210 2024-05-09T08:43:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.130+0000 7f5816ffd640 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f10923e210 0x55f1092406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.132+0000 7f5816ffd640 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f10923e210 0x55f1092406a0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c50a3078abc306ef server_cookie=5ddbbed6942e9606 in_seq=0 out_seq=0 2024-05-09T08:43:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.132+0000 7f57f6ffd640 10 client.26602 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.135+0000 7f57f6ffd640 1 -- 172.21.15.28:0/96328529 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f109195a70 con 0x55f10923e210 2024-05-09T08:43:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.135+0000 7f57f6ffd640 10 client.26602 handle_command_reply: tid=0 2024-05-09T08:43:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.137+0000 7f58209ea740 1 client.26602 shutdown 2024-05-09T08:43:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.146+0000 7f58209ea740 1 -- 172.21.15.28:0/96328529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 msgr2=0x55f10922b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.146+0000 7f58209ea740 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 0x55f10922b730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f580c00c2d0 tx=0x7f580c00c790 comp rx=0 tx=0).stop 2024-05-09T08:43:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.149+0000 7f58209ea740 1 -- 172.21.15.28:0/96328529 shutdown_connections 2024-05-09T08:43:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.149+0000 7f58209ea740 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f10923e210 0x55f1092406a0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.149+0000 7f58209ea740 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.149+0000 7f58209ea740 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f109223ef0 0x55f10922bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.149+0000 7f58209ea740 1 --2- 172.21.15.28:0/96328529 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 0x55f10922b730 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.149+0000 7f58209ea740 1 -- 172.21.15.28:0/96328529 >> 172.21.15.28:0/96328529 conn(0x55f10921fae0 msgr2=0x55f109225860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:44.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.151+0000 7f57f6ffd640 0 client.26602 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:44.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.153+0000 7f58209ea740 1 -- 172.21.15.28:0/96328529 shutdown_connections 2024-05-09T08:43:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.155+0000 7f58209ea740 1 -- 172.21.15.28:0/96328529 wait complete. 2024-05-09T08:43:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.157+0000 7f58209ea740 20 client.26602 trim_cache size 0 max 16384 2024-05-09T08:43:44.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:43 smithi028 ceph-mon[30119]: pgmap v558: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-05-09T08:43:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.193+0000 7f58209ea740 1 Processor -- start 2024-05-09T08:43:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.193+0000 7f58209ea740 1 -- start start 2024-05-09T08:43:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.193+0000 7f58209ea740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 0x55f10922de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f58209ea740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f109223ef0 0x55f109239ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f58209ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f58209ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58040102a0 con 0x55f10922e7b0 2024-05-09T08:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f58209ea740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5804012020 con 0x55f109223ef0 2024-05-09T08:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f58209ea740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5804010a30 con 0x55f1092235d0 2024-05-09T08:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f5817fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f5817fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47000/0 (socket says 172.21.15.28:47000) 2024-05-09T08:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f5817fff640 1 -- 172.21.15.28:0/1762153593 learned_addr learned my addr 172.21.15.28:0/1762153593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f5817fff640 1 -- 172.21.15.28:0/1762153593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 msgr2=0x55f10922de60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f5817fff640 1 --2- 172.21.15.28:0/1762153593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 0x55f10922de60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f5817fff640 1 -- 172.21.15.28:0/1762153593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f109223ef0 msgr2=0x55f109239ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f5817fff640 1 --2- 172.21.15.28:0/1762153593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f109223ef0 0x55f109239ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.194+0000 7f5817fff640 1 -- 172.21.15.28:0/1762153593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f10922a6e0 con 0x55f10922e7b0 2024-05-09T08:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.195+0000 7f5817fff640 1 --2- 172.21.15.28:0/1762153593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922a000 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f58080139c0 tx=0x7f5808012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908d3a4e8dab6602 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.198+0000 7f57fd7fa640 1 -- 172.21.15.28:0/1762153593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580800bcf0 con 0x55f10922e7b0 2024-05-09T08:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.198+0000 7f57fd7fa640 1 -- 172.21.15.28:0/1762153593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5808024070 con 0x55f10922e7b0 2024-05-09T08:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.199+0000 7f58209ea740 1 -- 172.21.15.28:0/1762153593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 msgr2=0x55f10922a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.199+0000 7f58209ea740 1 --2- 172.21.15.28:0/1762153593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922a000 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f58080139c0 tx=0x7f5808012000 comp rx=0 tx=0).stop 2024-05-09T08:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.200+0000 7f58209ea740 1 -- 172.21.15.28:0/1762153593 shutdown_connections 2024-05-09T08:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.200+0000 7f58209ea740 1 --2- 172.21.15.28:0/1762153593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f10922e7b0 0x55f10922a000 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.200+0000 7f58209ea740 1 --2- 172.21.15.28:0/1762153593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f109223ef0 0x55f109239ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.200+0000 7f58209ea740 1 --2- 172.21.15.28:0/1762153593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f1092235d0 0x55f10922de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.200+0000 7f58209ea740 1 -- 172.21.15.28:0/1762153593 >> 172.21.15.28:0/1762153593 conn(0x55f10921fae0 msgr2=0x55f10921eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.201+0000 7f58209ea740 1 -- 172.21.15.28:0/1762153593 shutdown_connections 2024-05-09T08:43:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.201+0000 7f58209ea740 1 -- 172.21.15.28:0/1762153593 wait complete. 2024-05-09T08:43:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.202+0000 7f58209ea740 1 -- start start 2024-05-09T08:43:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.202+0000 7f58209ea740 1 Processor -- start 2024-05-09T08:43:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.203+0000 7f58209ea740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f1092235d0 0x55f1092363f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.203+0000 7f58209ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109223ef0 0x55f109236930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.203+0000 7f58177fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f1092235d0 0x55f1092363f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.203+0000 7f58177fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f1092235d0 0x55f1092363f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60828/0 (socket says 172.21.15.28:60828) 2024-05-09T08:43:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.203+0000 7f58177fe640 1 -- 172.21.15.28:0/132758770 learned_addr learned my addr 172.21.15.28:0/132758770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.203+0000 7f58209ea740 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f10922e7b0 0x55f1092333c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.203+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5804011950 con 0x55f109223ef0 2024-05-09T08:43:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.204+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f58040111c0 con 0x55f1092235d0 2024-05-09T08:43:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.204+0000 7f5817fff640 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f10922e7b0 0x55f1092333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.204+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f580400e330 con 0x55f10922e7b0 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.204+0000 7f58209ea740 10 client.0 _ll_register_callbacks cb 0x7f580c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.204+0000 7f58177fe640 1 -- 172.21.15.28:0/132758770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f10922e7b0 msgr2=0x55f1092333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.204+0000 7f58177fe640 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f10922e7b0 0x55f1092333c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.204+0000 7f58177fe640 1 -- 172.21.15.28:0/132758770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109223ef0 msgr2=0x55f109236930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.204+0000 7f58177fe640 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109223ef0 0x55f109236930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.204+0000 7f58177fe640 1 -- 172.21.15.28:0/132758770 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f109233bd0 con 0x55f1092235d0 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.205+0000 7f58177fe640 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f1092235d0 0x55f1092363f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f580c00b010 tx=0x7f580c00abc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d62b851c2db57bea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.205+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5808024070 con 0x55f1092235d0 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.205+0000 7f57f6ffd640 10 client.26753 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:43:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.205+0000 7f57f6ffd640 1 -- 172.21.15.28:0/132758770 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c030060 con 0x55f1092235d0 2024-05-09T08:43:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.205+0000 7f57f6ffd640 1 -- 172.21.15.28:0/132758770 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9835+0+0 (secure 0 0 0) 0x7f580c00f660 con 0x55f1092235d0 2024-05-09T08:43:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.206+0000 7f57f6ffd640 1 -- 172.21.15.28:0/132758770 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f580c00f8e0 con 0x55f1092235d0 2024-05-09T08:43:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.207+0000 7f58209ea740 10 client.26753 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.207+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({fsmap=67}) -- 0x55f1092a5180 con 0x55f1092235d0 2024-05-09T08:43:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.210+0000 7f57f6ffd640 1 -- 172.21.15.28:0/132758770 <== mon.2 v2:172.21.15.165:3300/0 4 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f580c02b480 con 0x55f1092235d0 2024-05-09T08:43:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.212+0000 7f58209ea740 10 client.26753 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.212+0000 7f58209ea740 10 client.26753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.212+0000 7f58209ea740 20 client.26753 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.212+0000 7f58209ea740 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f1092a58b0 0x55f1092a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.212+0000 7f58209ea740 4 client.26753 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:43:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.212+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5808024070 con 0x55f1092a58b0 2024-05-09T08:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.214+0000 7f5816ffd640 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f1092a58b0 0x55f1092a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.218+0000 7f5816ffd640 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f1092a58b0 0x55f1092a7d30 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cd04f648cd811d1 server_cookie=7890ffd50cd5f83 in_seq=0 out_seq=0 2024-05-09T08:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.219+0000 7f57f6ffd640 10 client.26753 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.376+0000 7f57f6ffd640 1 -- 172.21.15.28:0/132758770 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5808024070 con 0x55f1092a58b0 2024-05-09T08:43:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.376+0000 7f57f6ffd640 10 client.26753 handle_command_reply: tid=0 2024-05-09T08:43:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 client.26753 shutdown 2024-05-09T08:43:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f1092235d0 msgr2=0x55f1092363f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f1092235d0 0x55f1092363f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f580c00b010 tx=0x7f580c00abc0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 shutdown_connections 2024-05-09T08:43:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f1092a58b0 0x55f1092a7d30 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f10922e7b0 0x55f1092333c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109223ef0 0x55f109236930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 --2- 172.21.15.28:0/132758770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f1092235d0 0x55f1092363f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 >> 172.21.15.28:0/132758770 conn(0x55f10921fae0 msgr2=0x55f10921e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f57f6ffd640 0 client.26753 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 shutdown_connections 2024-05-09T08:43:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 1 -- 172.21.15.28:0/132758770 wait complete. 2024-05-09T08:43:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.377+0000 7f58209ea740 20 client.26753 trim_cache size 0 max 16384 2024-05-09T08:43:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.378+0000 7f581c9a0640 1 -- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f58000a2f00 msgr2=0x7f58000a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.378+0000 7f581c9a0640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f58000a2f00 0x7f58000a53c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5808009650 tx=0x7f5808006cb0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.378+0000 7f581c9a0640 1 -- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581810fef0 msgr2=0x7f58181156e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.378+0000 7f581c9a0640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581810fef0 0x7f58181156e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5810006010 tx=0x7f58100308b0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.379+0000 7f581c9a0640 1 -- 172.21.15.28:0/4090513902 shutdown_connections 2024-05-09T08:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.379+0000 7f581c9a0640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f58000a2f00 0x7f58000a53c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.379+0000 7f581c9a0640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58181102d0 0x7f5818115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.379+0000 7f581c9a0640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581810fef0 0x7f58181156e0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.379+0000 7f581c9a0640 1 --2- 172.21.15.28:0/4090513902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f581810e500 0x7f581811c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.379+0000 7f581c9a0640 1 -- 172.21.15.28:0/4090513902 >> 172.21.15.28:0/4090513902 conn(0x7f581806c770 msgr2=0x7f5818070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.379+0000 7f581c9a0640 1 -- 172.21.15.28:0/4090513902 shutdown_connections 2024-05-09T08:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:44.379+0000 7f581c9a0640 1 -- 172.21.15.28:0/4090513902 wait complete. 2024-05-09T08:43:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:45 smithi165 ceph-mon[34540]: pgmap v559: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-05-09T08:43:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:45 smithi165 ceph-mon[34540]: osdmap e128: 12 total, 12 up, 12 in 2024-05-09T08:43:46.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:45 smithi184 ceph-mon[34509]: pgmap v559: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-05-09T08:43:46.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:45 smithi184 ceph-mon[34509]: osdmap e128: 12 total, 12 up, 12 in 2024-05-09T08:43:46.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:45 smithi028 ceph-mon[30119]: pgmap v559: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-05-09T08:43:46.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:45 smithi028 ceph-mon[30119]: osdmap e128: 12 total, 12 up, 12 in 2024-05-09T08:43:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:47 smithi165 ceph-mon[34540]: pgmap v561: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 183 op/s 2024-05-09T08:43:48.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:47 smithi184 ceph-mon[34509]: pgmap v561: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 183 op/s 2024-05-09T08:43:48.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:47 smithi028 ceph-mon[30119]: pgmap v561: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 183 op/s 2024-05-09T08:43:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:49 smithi165 ceph-mon[34540]: pgmap v562: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-05-09T08:43:50.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:49 smithi184 ceph-mon[34509]: pgmap v562: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-05-09T08:43:50.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:49 smithi028 ceph-mon[30119]: pgmap v562: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-05-09T08:43:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:51 smithi165 ceph-mon[34540]: pgmap v563: 129 pgs: 68 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 39 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 945 KiB/s wr, 179 op/s 2024-05-09T08:43:52.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:51 smithi184 ceph-mon[34509]: pgmap v563: 129 pgs: 68 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 39 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 945 KiB/s wr, 179 op/s 2024-05-09T08:43:52.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:51 smithi028 ceph-mon[30119]: pgmap v563: 129 pgs: 68 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 39 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 945 KiB/s wr, 179 op/s 2024-05-09T08:43:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:53 smithi165 ceph-mon[34540]: pgmap v564: 129 pgs: 68 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 39 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 941 KiB/s wr, 176 op/s 2024-05-09T08:43:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:53 smithi184 ceph-mon[34509]: pgmap v564: 129 pgs: 68 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 39 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 941 KiB/s wr, 176 op/s 2024-05-09T08:43:54.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:53 smithi028 ceph-mon[30119]: pgmap v564: 129 pgs: 68 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 39 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 941 KiB/s wr, 176 op/s 2024-05-09T08:43:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:55 smithi165 ceph-mon[34540]: pgmap v565: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 167 op/s 2024-05-09T08:43:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:55 smithi184 ceph-mon[34509]: pgmap v565: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 167 op/s 2024-05-09T08:43:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:55 smithi028 ceph-mon[30119]: pgmap v565: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 167 op/s 2024-05-09T08:43:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.539+0000 7fdd96c38640 1 -- 172.21.15.28:0/3431644981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8809acf0 msgr2=0x7fdd88006560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.539+0000 7fdd96c38640 1 --2- 172.21.15.28:0/3431644981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8809acf0 0x7fdd88006560 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fdd90066fa0 tx=0x7fdd900962b0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.544+0000 7fdd96c38640 1 -- 172.21.15.28:0/3431644981 shutdown_connections 2024-05-09T08:43:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.544+0000 7fdd96c38640 1 --2- 172.21.15.28:0/3431644981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd880986e0 0x7fdd88098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.544+0000 7fdd96c38640 1 --2- 172.21.15.28:0/3431644981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd88006aa0 0x7fdd88006f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.544+0000 7fdd96c38640 1 --2- 172.21.15.28:0/3431644981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8809acf0 0x7fdd88006560 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.544+0000 7fdd96c38640 1 -- 172.21.15.28:0/3431644981 >> 172.21.15.28:0/3431644981 conn(0x7fdd8800a740 msgr2=0x7fdd8800ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.545+0000 7fdd96c38640 1 -- 172.21.15.28:0/3431644981 shutdown_connections 2024-05-09T08:43:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.545+0000 7fdd96c38640 1 -- 172.21.15.28:0/3431644981 wait complete. 2024-05-09T08:43:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.546+0000 7fdd96c38640 1 Processor -- start 2024-05-09T08:43:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.546+0000 7fdd96c38640 1 -- start start 2024-05-09T08:43:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd96c38640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd88006aa0 0x7fdd88138620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd96c38640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd880986e0 0x7fdd88138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd96c38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8809acf0 0x7fdd881326e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd96c38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd880ac5e0 con 0x7fdd8809acf0 2024-05-09T08:43:57.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd96c38640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdd880ac460 con 0x7fdd88006aa0 2024-05-09T08:43:57.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd96c38640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdd880ac760 con 0x7fdd880986e0 2024-05-09T08:43:57.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd95435640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd880986e0 0x7fdd88138b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd95435640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd880986e0 0x7fdd88138b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:46148/0 (socket says 172.21.15.28:46148) 2024-05-09T08:43:57.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd95435640 1 -- 172.21.15.28:0/1084591759 learned_addr learned my addr 172.21.15.28:0/1084591759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:57.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.547+0000 7fdd96437640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8809acf0 0x7fdd881326e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.548+0000 7fdd95435640 1 -- 172.21.15.28:0/1084591759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd88006aa0 msgr2=0x7fdd88138620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:43:57.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.548+0000 7fdd95435640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd88006aa0 0x7fdd88138620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.548+0000 7fdd95435640 1 -- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8809acf0 msgr2=0x7fdd881326e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:57.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.548+0000 7fdd95435640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8809acf0 0x7fdd881326e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.548+0000 7fdd95435640 1 -- 172.21.15.28:0/1084591759 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd88132eb0 con 0x7fdd880986e0 2024-05-09T08:43:57.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.548+0000 7fdd95435640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd880986e0 0x7fdd88138b60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdd8000b450 tx=0x7fdd8000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:57.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.549+0000 7fdd86ffd640 1 -- 172.21.15.28:0/1084591759 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd80015020 con 0x7fdd880986e0 2024-05-09T08:43:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.549+0000 7fdd96c38640 1 -- 172.21.15.28:0/1084591759 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd881331a0 con 0x7fdd880986e0 2024-05-09T08:43:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.549+0000 7fdd86ffd640 1 -- 172.21.15.28:0/1084591759 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd80011d70 con 0x7fdd880986e0 2024-05-09T08:43:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.549+0000 7fdd86ffd640 1 -- 172.21.15.28:0/1084591759 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8000f340 con 0x7fdd880986e0 2024-05-09T08:43:57.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.553+0000 7fdd96c38640 1 -- 172.21.15.28:0/1084591759 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd8809d480 con 0x7fdd880986e0 2024-05-09T08:43:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.558+0000 7fdd9aba6740 1 Processor -- start 2024-05-09T08:43:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.560+0000 7fdd9aba6740 1 -- start start 2024-05-09T08:43:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.561+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358abb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.561+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358abbf20 0x559358ac62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.561+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358ac67e0 0x559358ac8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.561+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559358a20340 con 0x559358abb600 2024-05-09T08:43:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.561+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5593589c8290 con 0x559358ac67e0 2024-05-09T08:43:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.561+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559358a1c300 con 0x559358abbf20 2024-05-09T08:43:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.562+0000 7fdd86ffd640 1 -- 172.21.15.28:0/1084591759 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdd80018020 con 0x7fdd880986e0 2024-05-09T08:43:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.563+0000 7fdd86ffd640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd7c0a2e40 0x7fdd7c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.563+0000 7fdd95c36640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358abbf20 0x559358ac62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.563+0000 7fdd95c36640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358abbf20 0x559358ac62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:46162/0 (socket says 172.21.15.28:46162) 2024-05-09T08:43:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.563+0000 7fdd86ffd640 1 -- 172.21.15.28:0/1084591759 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9819+0+0 (secure 0 0 0) 0x7fdd800cf330 con 0x7fdd880986e0 2024-05-09T08:43:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.563+0000 7fdd95c36640 1 -- 172.21.15.28:0/3608159037 learned_addr learned my addr 172.21.15.28:0/3608159037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.564+0000 7fdd95435640 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358ac67e0 0x559358ac8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.565+0000 7fdd96437640 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358abb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.565+0000 7fdd95435640 1 -- 172.21.15.28:0/3608159037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358abbf20 msgr2=0x559358ac62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.565+0000 7fdd95435640 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358abbf20 0x559358ac62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.565+0000 7fdd95435640 1 -- 172.21.15.28:0/3608159037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 msgr2=0x559358abb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.565+0000 7fdd95435640 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358abb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.565+0000 7fdd95435640 1 -- 172.21.15.28:0/3608159037 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593589ec8e0 con 0x559358ac67e0 2024-05-09T08:43:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.565+0000 7fdd95c36640 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358abbf20 0x559358ac62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.566+0000 7fdd96437640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd7c0a2e40 0x7fdd7c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.566+0000 7fdd95435640 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358ac67e0 0x559358ac8bd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd800cc620 tx=0x7fdd8009ae10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f866f5227d53067 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.566+0000 7fdd7bfff640 1 -- 172.21.15.28:0/3608159037 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd800b4070 con 0x559358ac67e0 2024-05-09T08:43:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.567+0000 7fdd7bfff640 1 -- 172.21.15.28:0/3608159037 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd8009f7a0 con 0x559358ac67e0 2024-05-09T08:43:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.568+0000 7fdd9aba6740 1 -- 172.21.15.28:0/3608159037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358ac67e0 msgr2=0x559358ac8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.568+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358ac67e0 0x559358ac8bd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd800cc620 tx=0x7fdd8009ae10 comp rx=0 tx=0).stop 2024-05-09T08:43:57.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.570+0000 7fdd96437640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd7c0a2e40 0x7fdd7c0a5300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800abd0 tx=0x7fdd8c00b2e0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.573+0000 7fdd9aba6740 1 -- 172.21.15.28:0/3608159037 shutdown_connections 2024-05-09T08:43:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.573+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358ac67e0 0x559358ac8bd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.573+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358abbf20 0x559358ac62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.573+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/3608159037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358abb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.573+0000 7fdd9aba6740 1 -- 172.21.15.28:0/3608159037 >> 172.21.15.28:0/3608159037 conn(0x559358ab7b10 msgr2=0x559358ab7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.581+0000 7fdd9aba6740 1 -- 172.21.15.28:0/3608159037 shutdown_connections 2024-05-09T08:43:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.585+0000 7fdd9aba6740 1 -- 172.21.15.28:0/3608159037 wait complete. 2024-05-09T08:43:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.585+0000 7fdd9aba6740 1 -- start start 2024-05-09T08:43:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.585+0000 7fdd9aba6740 1 Processor -- start 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.586+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abb600 0x559358ac3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.586+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abbf20 0x559358ac3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.586+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ac40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.586+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559358ac9410 con 0x559358abbf20 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.586+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5593589db790 con 0x559358abb600 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.586+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559358ac9590 con 0x559358ac67e0 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.587+0000 7fdd9aba6740 10 client.0 _ll_register_callbacks cb 0x559358a16a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.587+0000 7fdd96437640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ac40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.587+0000 7fdd96437640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ac40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:46174/0 (socket says 172.21.15.28:46174) 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.587+0000 7fdd96437640 1 -- 172.21.15.28:0/749957097 learned_addr learned my addr 172.21.15.28:0/749957097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.587+0000 7fdd95435640 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abbf20 0x559358ac3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.588+0000 7fdd96437640 1 -- 172.21.15.28:0/749957097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abb600 msgr2=0x559358ac3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.588+0000 7fdd96437640 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abb600 0x559358ac3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.588+0000 7fdd96437640 1 -- 172.21.15.28:0/749957097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abbf20 msgr2=0x559358ac3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.588+0000 7fdd96437640 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abbf20 0x559358ac3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.588+0000 7fdd96437640 1 -- 172.21.15.28:0/749957097 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559358734210 con 0x559358ac67e0 2024-05-09T08:43:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.588+0000 7fdd96437640 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ac40f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00bf60 tx=0x7fdd8c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=818a4cf5375dee3d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.588+0000 7fdd797fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:43:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.590+0000 7fdd797fa640 1 -- 172.21.15.28:0/749957097 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c046070 con 0x559358ac67e0 2024-05-09T08:43:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.590+0000 7fdd9aba6740 1 -- 172.21.15.28:0/749957097 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd8009f7a0 con 0x559358ac67e0 2024-05-09T08:43:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.590+0000 7fdd797fa640 1 -- 172.21.15.28:0/749957097 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9819+0+0 (secure 0 0 0) 0x7fdd8c00fa50 con 0x559358ac67e0 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.592+0000 7fdd797fa640 1 -- 172.21.15.28:0/749957097 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fdd8c00fc50 con 0x559358ac67e0 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.592+0000 7fdd9aba6740 10 client.26614 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.592+0000 7fdd9aba6740 1 -- 172.21.15.28:0/749957097 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x559358800620 con 0x559358ac67e0 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.592+0000 7fdd797fa640 1 -- 172.21.15.28:0/749957097 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd8009f7a0 con 0x559358ac67e0 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.593+0000 7fdd797fa640 1 -- 172.21.15.28:0/749957097 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fdd8c0413a0 con 0x559358ac67e0 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.593+0000 7fdd9aba6740 10 client.26614 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.593+0000 7fdd9aba6740 10 client.26614 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.593+0000 7fdd9aba6740 20 client.26614 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.593+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559358ad40f0 0x559358ad65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.593+0000 7fdd9aba6740 4 client.26614 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.593+0000 7fdd9aba6740 1 -- 172.21.15.28:0/749957097 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdd80097bf0 con 0x559358ad40f0 2024-05-09T08:43:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.594+0000 7fdd95c36640 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559358ad40f0 0x559358ad65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.599+0000 7fdd95c36640 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559358ad40f0 0x559358ad65b0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8e25ccdc6ef0862 server_cookie=bdd4fbfbdc859ac1 in_seq=0 out_seq=0 2024-05-09T08:43:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.600+0000 7fdd797fa640 10 client.26614 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.602+0000 7fdd797fa640 1 -- 172.21.15.28:0/749957097 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fdd80097bf0 con 0x559358ad40f0 2024-05-09T08:43:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.602+0000 7fdd797fa640 10 client.26614 handle_command_reply: tid=0 2024-05-09T08:43:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.602+0000 7fdd9aba6740 1 client.26614 shutdown 2024-05-09T08:43:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.606+0000 7fdd9aba6740 1 -- 172.21.15.28:0/749957097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 msgr2=0x559358ac40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.606+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ac40f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00bf60 tx=0x7fdd8c0046c0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.607+0000 7fdd9aba6740 1 -- 172.21.15.28:0/749957097 shutdown_connections 2024-05-09T08:43:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.607+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559358ad40f0 0x559358ad65b0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.607+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ac40f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.607+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abbf20 0x559358ac3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.607+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/749957097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abb600 0x559358ac3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.607+0000 7fdd9aba6740 1 -- 172.21.15.28:0/749957097 >> 172.21.15.28:0/749957097 conn(0x559358ab7b10 msgr2=0x559358abd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.607+0000 7fdd797fa640 0 client.26614 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.608+0000 7fdd9aba6740 1 -- 172.21.15.28:0/749957097 shutdown_connections 2024-05-09T08:43:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.608+0000 7fdd9aba6740 1 -- 172.21.15.28:0/749957097 wait complete. 2024-05-09T08:43:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.609+0000 7fdd9aba6740 20 client.26614 trim_cache size 0 max 16384 2024-05-09T08:43:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.646+0000 7fdd9aba6740 1 Processor -- start 2024-05-09T08:43:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.646+0000 7fdd9aba6740 1 -- start start 2024-05-09T08:43:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.646+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358ad2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.646+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 0x559358abe010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.646+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ac27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd881521c0 con 0x559358abb600 2024-05-09T08:43:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdd88153f40 con 0x559358abbf20 2024-05-09T08:43:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdd88152950 con 0x559358ac67e0 2024-05-09T08:43:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd95c36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358ad2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd95c36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358ad2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35126/0 (socket says 172.21.15.28:35126) 2024-05-09T08:43:57.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd95c36640 1 -- 172.21.15.28:0/1287369738 learned_addr learned my addr 172.21.15.28:0/1287369738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd95c36640 1 -- 172.21.15.28:0/1287369738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 msgr2=0x559358ac27d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd95c36640 1 --2- 172.21.15.28:0/1287369738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ac27d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd95c36640 1 -- 172.21.15.28:0/1287369738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 msgr2=0x559358abe010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd95c36640 1 --2- 172.21.15.28:0/1287369738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 0x559358abe010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.647+0000 7fdd95c36640 1 -- 172.21.15.28:0/1287369738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559358ac1bc0 con 0x559358abb600 2024-05-09T08:43:57.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.648+0000 7fdd95c36640 1 --2- 172.21.15.28:0/1287369738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358ad2320 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fdd90050b40 tx=0x7fdd900962b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814ac2f8a163d57f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.651+0000 7fdd7b7fe640 1 -- 172.21.15.28:0/1287369738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90073300 con 0x559358abb600 2024-05-09T08:43:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.651+0000 7fdd7b7fe640 1 -- 172.21.15.28:0/1287369738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd9009d540 con 0x559358abb600 2024-05-09T08:43:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.651+0000 7fdd7b7fe640 1 -- 172.21.15.28:0/1287369738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9006d650 con 0x559358abb600 2024-05-09T08:43:57.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.652+0000 7fdd9aba6740 1 -- 172.21.15.28:0/1287369738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 msgr2=0x559358ad2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.652+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/1287369738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358ad2320 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fdd90050b40 tx=0x7fdd900962b0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.653+0000 7fdd9aba6740 1 -- 172.21.15.28:0/1287369738 shutdown_connections 2024-05-09T08:43:57.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.653+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/1287369738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ac27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.653+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/1287369738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 0x559358abe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.653+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/1287369738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358ad2320 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.653+0000 7fdd9aba6740 1 -- 172.21.15.28:0/1287369738 >> 172.21.15.28:0/1287369738 conn(0x559358ab7b10 msgr2=0x559358ab6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.655+0000 7fdd9aba6740 1 -- 172.21.15.28:0/1287369738 shutdown_connections 2024-05-09T08:43:57.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.655+0000 7fdd9aba6740 1 -- 172.21.15.28:0/1287369738 wait complete. 2024-05-09T08:43:57.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.655+0000 7fdd9aba6740 1 -- start start 2024-05-09T08:43:57.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.655+0000 7fdd9aba6740 1 Processor -- start 2024-05-09T08:43:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358ae2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 0x559358ae30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd9aba6740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ae3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd88153870 con 0x559358abb600 2024-05-09T08:43:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdd881530e0 con 0x559358abbf20 2024-05-09T08:43:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd9aba6740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdd88095220 con 0x559358ac67e0 2024-05-09T08:43:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd9aba6740 10 client.0 _ll_register_callbacks cb 0x7fdd8c00ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd95435640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 0x559358ae30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd95435640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 0x559358ae30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:39724/0 (socket says 172.21.15.28:39724) 2024-05-09T08:43:57.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd95435640 1 -- 172.21.15.28:0/2859472582 learned_addr learned my addr 172.21.15.28:0/2859472582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:43:57.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.656+0000 7fdd96437640 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ae3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.657+0000 7fdd96437640 1 -- 172.21.15.28:0/2859472582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 msgr2=0x559358ae30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:57.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.657+0000 7fdd96437640 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 0x559358ae30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.657+0000 7fdd96437640 1 -- 172.21.15.28:0/2859472582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 msgr2=0x559358ae2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:57.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.657+0000 7fdd96437640 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358ae2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.657+0000 7fdd96437640 1 -- 172.21.15.28:0/2859472582 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559358accce0 con 0x559358ac67e0 2024-05-09T08:43:57.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.658+0000 7fdd96437640 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ae3610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c000c00 tx=0x7fdd8c005d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9128524c97c7500d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:57.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.658+0000 7fdd797fa640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:43:57.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.658+0000 7fdd797fa640 1 -- 172.21.15.28:0/2859472582 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c046040 con 0x559358ac67e0 2024-05-09T08:43:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.658+0000 7fdd797fa640 1 -- 172.21.15.28:0/2859472582 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9819+0+0 (secure 0 0 0) 0x7fdd8c0419e0 con 0x559358ac67e0 2024-05-09T08:43:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.659+0000 7fdd9aba6740 1 -- 172.21.15.28:0/2859472582 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd9009d540 con 0x559358ac67e0 2024-05-09T08:43:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.659+0000 7fdd797fa640 1 -- 172.21.15.28:0/2859472582 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fdd8c00feb0 con 0x559358ac67e0 2024-05-09T08:43:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.660+0000 7fdd9aba6740 10 client.26617 fetch_fsmap learned FSMap version 67 2024-05-09T08:43:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.660+0000 7fdd9aba6740 1 -- 172.21.15.28:0/2859472582 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x559358b3c800 con 0x559358ac67e0 2024-05-09T08:43:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.661+0000 7fdd797fa640 1 -- 172.21.15.28:0/2859472582 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd9009d540 con 0x559358ac67e0 2024-05-09T08:43:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.661+0000 7fdd797fa640 1 -- 172.21.15.28:0/2859472582 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fdd8c04b9f0 con 0x559358ac67e0 2024-05-09T08:43:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.661+0000 7fdd9aba6740 10 client.26617 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:43:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.661+0000 7fdd9aba6740 10 client.26617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:43:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.661+0000 7fdd9aba6740 20 client.26617 populate_metadata read hostname 'smithi028' 2024-05-09T08:43:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.661+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559358b3cf80 0x559358b3f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.661+0000 7fdd9aba6740 4 client.26617 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:43:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.661+0000 7fdd9aba6740 1 -- 172.21.15.28:0/2859472582 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559358a2da70 con 0x559358b3cf80 2024-05-09T08:43:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.661+0000 7fdd95c36640 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559358b3cf80 0x559358b3f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.668+0000 7fdd95c36640 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559358b3cf80 0x559358b3f3a0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0771782537809f5 server_cookie=876ed98c5f5fdcee in_seq=0 out_seq=0 2024-05-09T08:43:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.669+0000 7fdd797fa640 10 client.26617 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:57.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd797fa640 1 -- 172.21.15.28:0/2859472582 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559358a2da70 con 0x559358b3cf80 2024-05-09T08:43:57.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd797fa640 10 client.26617 handle_command_reply: tid=0 2024-05-09T08:43:57.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 client.26617 shutdown 2024-05-09T08:43:57.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 -- 172.21.15.28:0/2859472582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 msgr2=0x559358ae3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:57.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ae3610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c000c00 tx=0x7fdd8c005d20 comp rx=0 tx=0).stop 2024-05-09T08:43:57.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 -- 172.21.15.28:0/2859472582 shutdown_connections 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559358b3cf80 0x559358b3f3a0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559358ac67e0 0x559358ae3610 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559358abbf20 0x559358ae30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 --2- 172.21.15.28:0/2859472582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559358abb600 0x559358ae2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 -- 172.21.15.28:0/2859472582 >> 172.21.15.28:0/2859472582 conn(0x559358ab7b10 msgr2=0x559358ab6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd797fa640 0 client.26617 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 -- 172.21.15.28:0/2859472582 shutdown_connections 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.879+0000 7fdd9aba6740 1 -- 172.21.15.28:0/2859472582 wait complete. 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.880+0000 7fdd9aba6740 20 client.26617 trim_cache size 0 max 16384 2024-05-09T08:43:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:43:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:43:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:43:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:43:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.880+0000 7fdd96c38640 1 -- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd7c0a2e40 msgr2=0x7fdd7c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd7c0a2e40 0x7fdd7c0a5300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800abd0 tx=0x7fdd8c00b2e0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 -- 172.21.15.28:0/1084591759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd880986e0 msgr2=0x7fdd88138b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd880986e0 0x7fdd88138b60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdd8000b450 tx=0x7fdd8000b910 comp rx=0 tx=0).stop 2024-05-09T08:43:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 -- 172.21.15.28:0/1084591759 shutdown_connections 2024-05-09T08:43:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd8809acf0 0x7fdd881326e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fdd7c0a2e40 0x7fdd7c0a5300 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdd880986e0 0x7fdd88138b60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 --2- 172.21.15.28:0/1084591759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd88006aa0 0x7fdd88138620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 -- 172.21.15.28:0/1084591759 >> 172.21.15.28:0/1084591759 conn(0x7fdd8800a740 msgr2=0x7fdd88092090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.881+0000 7fdd96c38640 1 -- 172.21.15.28:0/1084591759 shutdown_connections 2024-05-09T08:43:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:43:57.882+0000 7fdd96c38640 1 -- 172.21.15.28:0/1084591759 wait complete. 2024-05-09T08:43:58.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:57 smithi165 ceph-mon[34540]: pgmap v566: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 145 op/s 2024-05-09T08:43:58.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:57 smithi184 ceph-mon[34509]: pgmap v566: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 145 op/s 2024-05-09T08:43:58.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:57 smithi028 ceph-mon[30119]: pgmap v566: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 145 op/s 2024-05-09T08:44:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:43:59 smithi165 ceph-mon[34540]: pgmap v567: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-09T08:44:00.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:43:59 smithi184 ceph-mon[34509]: pgmap v567: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-09T08:44:00.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:43:59 smithi028 ceph-mon[30119]: pgmap v567: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-05-09T08:44:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:01 smithi165 ceph-mon[34540]: pgmap v568: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 843 KiB/s wr, 168 op/s 2024-05-09T08:44:02.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:01 smithi184 ceph-mon[34509]: pgmap v568: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 843 KiB/s wr, 168 op/s 2024-05-09T08:44:02.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:01 smithi028 ceph-mon[30119]: pgmap v568: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 843 KiB/s wr, 168 op/s 2024-05-09T08:44:04.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:03 smithi165 ceph-mon[34540]: pgmap v569: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-05-09T08:44:04.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:03 smithi184 ceph-mon[34509]: pgmap v569: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-05-09T08:44:04.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:03 smithi028 ceph-mon[30119]: pgmap v569: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-05-09T08:44:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:05 smithi165 ceph-mon[34540]: pgmap v570: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 156 op/s 2024-05-09T08:44:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:05 smithi184 ceph-mon[34509]: pgmap v570: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 156 op/s 2024-05-09T08:44:06.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:05 smithi028 ceph-mon[30119]: pgmap v570: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 156 op/s 2024-05-09T08:44:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:07 smithi165 ceph-mon[34540]: pgmap v571: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 864 KiB/s wr, 182 op/s 2024-05-09T08:44:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:07 smithi184 ceph-mon[34509]: pgmap v571: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 864 KiB/s wr, 182 op/s 2024-05-09T08:44:08.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:07 smithi028 ceph-mon[30119]: pgmap v571: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 864 KiB/s wr, 182 op/s 2024-05-09T08:44:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:09 smithi165 ceph-mon[34540]: pgmap v572: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 843 KiB/s wr, 190 op/s 2024-05-09T08:44:10.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:09 smithi165 ceph-mon[34540]: osdmap e129: 12 total, 12 up, 12 in 2024-05-09T08:44:10.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:09 smithi184 ceph-mon[34509]: pgmap v572: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 843 KiB/s wr, 190 op/s 2024-05-09T08:44:10.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:09 smithi184 ceph-mon[34509]: osdmap e129: 12 total, 12 up, 12 in 2024-05-09T08:44:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:09 smithi028 ceph-mon[30119]: pgmap v572: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 843 KiB/s wr, 190 op/s 2024-05-09T08:44:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:09 smithi028 ceph-mon[30119]: osdmap e129: 12 total, 12 up, 12 in 2024-05-09T08:44:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.035+0000 7fabd54d4640 1 -- 172.21.15.28:0/3902816841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabd0109150 msgr2=0x7fabd0114440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.035+0000 7fabd54d4640 1 --2- 172.21.15.28:0/3902816841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabd0109150 0x7fabd0114440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009930 tx=0x7fabc002a510 comp rx=0 tx=0).stop 2024-05-09T08:44:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.035+0000 7fabd54d4640 1 -- 172.21.15.28:0/3902816841 shutdown_connections 2024-05-09T08:44:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.035+0000 7fabd54d4640 1 --2- 172.21.15.28:0/3902816841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabd0114980 0x7fabd0116d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.035+0000 7fabd54d4640 1 --2- 172.21.15.28:0/3902816841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabd0109150 0x7fabd0114440 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.035+0000 7fabd54d4640 1 --2- 172.21.15.28:0/3902816841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd0108830 0x7fabd0108c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.035+0000 7fabd54d4640 1 -- 172.21.15.28:0/3902816841 >> 172.21.15.28:0/3902816841 conn(0x7fabd006c900 msgr2=0x7fabd006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:11.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.036+0000 7fabd54d4640 1 -- 172.21.15.28:0/3902816841 shutdown_connections 2024-05-09T08:44:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.040+0000 7fabd54d4640 1 -- 172.21.15.28:0/3902816841 wait complete. 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.040+0000 7fabd54d4640 1 Processor -- start 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.041+0000 7fabd54d4640 1 -- start start 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.041+0000 7fabd54d4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabd0108830 0x7fabd01a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.041+0000 7fabd54d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabd0109150 0x7fabd019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.041+0000 7fabd54d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd0114980 0x7fabd019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.041+0000 7fabd54d4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabd0118340 con 0x7fabd0114980 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.041+0000 7fabd54d4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fabd01181c0 con 0x7fabd0108830 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.041+0000 7fabd54d4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fabd01184c0 con 0x7fabd0109150 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.041+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabd0109150 0x7fabd019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.042+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabd0109150 0x7fabd019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:49166/0 (socket says 172.21.15.28:49166) 2024-05-09T08:44:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.042+0000 7fabce7fc640 1 -- 172.21.15.28:0/1384739726 learned_addr learned my addr 172.21.15.28:0/1384739726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.042+0000 7fabceffd640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabd0108830 0x7fabd01a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.042+0000 7fabce7fc640 1 -- 172.21.15.28:0/1384739726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabd0108830 msgr2=0x7fabd01a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.042+0000 7fabce7fc640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabd0108830 0x7fabd01a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.042+0000 7fabce7fc640 1 -- 172.21.15.28:0/1384739726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd0114980 msgr2=0x7fabd019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.042+0000 7fabce7fc640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd0114980 0x7fabd019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.042+0000 7fabce7fc640 1 -- 172.21.15.28:0/1384739726 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabd019f180 con 0x7fabd0109150 2024-05-09T08:44:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.043+0000 7fabce7fc640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabd0109150 0x7fabd019e680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009900 tx=0x7fabc0003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.044+0000 7fabb7fff640 1 -- 172.21.15.28:0/1384739726 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc003f070 con 0x7fabd0109150 2024-05-09T08:44:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.044+0000 7fabb7fff640 1 -- 172.21.15.28:0/1384739726 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabc003b460 con 0x7fabd0109150 2024-05-09T08:44:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.044+0000 7fabb7fff640 1 -- 172.21.15.28:0/1384739726 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc0035030 con 0x7fabd0109150 2024-05-09T08:44:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.046+0000 7fabd54d4640 1 -- 172.21.15.28:0/1384739726 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabd019f410 con 0x7fabd0109150 2024-05-09T08:44:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.046+0000 7fabd54d4640 1 -- 172.21.15.28:0/1384739726 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabd01a3360 con 0x7fabd0109150 2024-05-09T08:44:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.049+0000 7fabd81e8740 1 Processor -- start 2024-05-09T08:44:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.052+0000 7fabb7fff640 1 -- 172.21.15.28:0/1384739726 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fabc0042070 con 0x7fabd0109150 2024-05-09T08:44:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.053+0000 7fabb7fff640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fabb80a2fd0 0x7fabb80a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.053+0000 7fabb7fff640 1 -- 172.21.15.28:0/1384739726 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9911+0+0 (secure 0 0 0) 0x7fabc00f00a0 con 0x7fabd0109150 2024-05-09T08:44:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.054+0000 7fabd81e8740 1 -- start start 2024-05-09T08:44:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.054+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa789830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.054+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa7940f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.054+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa794630 0x5564fa796a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.054+0000 7fabd81e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564fa6ee340 con 0x5564fa789450 2024-05-09T08:44:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.054+0000 7fabd81e8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5564fa696290 con 0x5564fa794630 2024-05-09T08:44:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.054+0000 7fabd81e8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5564fa6ea300 con 0x5564fa789d70 2024-05-09T08:44:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.055+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fabb80a2fd0 0x7fabb80a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.055+0000 7fabcf7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa794630 0x5564fa796a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.055+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa7940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.055+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa7940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:49176/0 (socket says 172.21.15.28:49176) 2024-05-09T08:44:11.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.055+0000 7fabce7fc640 1 -- 172.21.15.28:0/4114891573 learned_addr learned my addr 172.21.15.28:0/4114891573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:11.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.055+0000 7fabceffd640 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa789830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.057+0000 7fabce7fc640 1 -- 172.21.15.28:0/4114891573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa794630 msgr2=0x5564fa796a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:11.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.057+0000 7fabce7fc640 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa794630 0x5564fa796a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.057+0000 7fabce7fc640 1 -- 172.21.15.28:0/4114891573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 msgr2=0x5564fa789830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:11.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.057+0000 7fabce7fc640 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa789830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.057+0000 7fabce7fc640 1 -- 172.21.15.28:0/4114891573 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564fa6ba8e0 con 0x5564fa789d70 2024-05-09T08:44:11.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.057+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa794630 0x5564fa796a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:44:11.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.057+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fabb80a2fd0 0x7fabb80a5490 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fabc800e600 tx=0x7fabc800aa10 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.060+0000 7fabceffd640 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa789830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:44:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.060+0000 7fabce7fc640 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa7940f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fabc00ed280 tx=0x7fabc00bbbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d842910693812ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.063+0000 7fabb57fa640 1 -- 172.21.15.28:0/4114891573 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc00dc0e0 con 0x5564fa789d70 2024-05-09T08:44:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.063+0000 7fabb57fa640 1 -- 172.21.15.28:0/4114891573 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabc00bbff0 con 0x5564fa789d70 2024-05-09T08:44:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.064+0000 7fabd81e8740 1 -- 172.21.15.28:0/4114891573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 msgr2=0x5564fa7940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.064+0000 7fabd81e8740 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa7940f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fabc00ed280 tx=0x7fabc00bbbf0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.066+0000 7fabd81e8740 1 -- 172.21.15.28:0/4114891573 shutdown_connections 2024-05-09T08:44:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.066+0000 7fabd81e8740 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa794630 0x5564fa796a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.066+0000 7fabd81e8740 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa7940f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.066+0000 7fabd81e8740 1 --2- 172.21.15.28:0/4114891573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa789830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.066+0000 7fabd81e8740 1 -- 172.21.15.28:0/4114891573 >> 172.21.15.28:0/4114891573 conn(0x5564fa785960 msgr2=0x5564fa785d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:11.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.071+0000 7fabd81e8740 1 -- 172.21.15.28:0/4114891573 shutdown_connections 2024-05-09T08:44:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.077+0000 7fabd81e8740 1 -- 172.21.15.28:0/4114891573 wait complete. 2024-05-09T08:44:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.077+0000 7fabd81e8740 1 -- start start 2024-05-09T08:44:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.077+0000 7fabd81e8740 1 Processor -- start 2024-05-09T08:44:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.077+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 0x5564fa791510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.077+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789d70 0x5564fa791a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.077+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.077+0000 7fabd81e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564fa6ea300 con 0x5564fa789d70 2024-05-09T08:44:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.078+0000 7fabd81e8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x5564fa6a9790 con 0x5564fa789450 2024-05-09T08:44:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.078+0000 7fabd81e8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5564fa797020 con 0x5564fa794630 2024-05-09T08:44:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.078+0000 7fabd81e8740 10 client.0 _ll_register_callbacks cb 0x5564fa6e4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.078+0000 7fabcf7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.081+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789d70 0x5564fa791a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.081+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789d70 0x5564fa791a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57348/0 (socket says 172.21.15.28:57348) 2024-05-09T08:44:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.081+0000 7fabce7fc640 1 -- 172.21.15.28:0/1604230902 learned_addr learned my addr 172.21.15.28:0/1604230902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.082+0000 7fabceffd640 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 0x5564fa791510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.086+0000 7fabcf7fe640 1 -- 172.21.15.28:0/1604230902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 msgr2=0x5564fa791510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.086+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 0x5564fa791510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.086+0000 7fabcf7fe640 1 -- 172.21.15.28:0/1604230902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789d70 msgr2=0x5564fa791a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.086+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789d70 0x5564fa791a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.086+0000 7fabcf7fe640 1 -- 172.21.15.28:0/1604230902 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564fa402210 con 0x5564fa794630 2024-05-09T08:44:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.087+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791f90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x5564fa786d80 tx=0x7fabc8004bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14e1d2e3183175b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.087+0000 7fabb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:44:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.088+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1604230902 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8006e60 con 0x5564fa794630 2024-05-09T08:44:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.088+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1604230902 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9911+0+0 (secure 0 0 0) 0x7fabc800e070 con 0x5564fa794630 2024-05-09T08:44:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.088+0000 7fabd81e8740 1 -- 172.21.15.28:0/1604230902 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabc00bbff0 con 0x5564fa794630 2024-05-09T08:44:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.089+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1604230902 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fabc800e270 con 0x5564fa794630 2024-05-09T08:44:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.089+0000 7fabd81e8740 10 client.26626 fetch_fsmap learned FSMap version 67 2024-05-09T08:44:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.089+0000 7fabd81e8740 1 -- 172.21.15.28:0/1604230902 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5564fa4ce620 con 0x5564fa794630 2024-05-09T08:44:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.094+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1604230902 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabc00bbff0 con 0x5564fa794630 2024-05-09T08:44:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.094+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1604230902 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fabc801a070 con 0x5564fa794630 2024-05-09T08:44:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.095+0000 7fabd81e8740 10 client.26626 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:44:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.095+0000 7fabd81e8740 10 client.26626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:44:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.095+0000 7fabd81e8740 20 client.26626 populate_metadata read hostname 'smithi028' 2024-05-09T08:44:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.095+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564fa7c63a0 0x5564fa7c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.095+0000 7fabd81e8740 4 client.26626 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.095+0000 7fabd81e8740 1 -- 172.21.15.28:0/1604230902 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564fa6fba70 con 0x5564fa7c63a0 2024-05-09T08:44:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.100+0000 7fabceffd640 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564fa7c63a0 0x5564fa7c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.109+0000 7fabceffd640 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564fa7c63a0 0x5564fa7c87e0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebde650cfc02a0d6 server_cookie=30ccf2d6fb82ec9 in_seq=0 out_seq=0 2024-05-09T08:44:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.109+0000 7fabb2ff5640 10 client.26626 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:11.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.110+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1604230902 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564fa6fba70 con 0x5564fa7c63a0 2024-05-09T08:44:11.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.110+0000 7fabb2ff5640 10 client.26626 handle_command_reply: tid=0 2024-05-09T08:44:11.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.110+0000 7fabd81e8740 1 client.26626 shutdown 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.110+0000 7fabd81e8740 1 -- 172.21.15.28:0/1604230902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 msgr2=0x5564fa791f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.110+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791f90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x5564fa786d80 tx=0x7fabc8004bd0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.111+0000 7fabd81e8740 1 -- 172.21.15.28:0/1604230902 shutdown_connections 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.111+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564fa7c63a0 0x5564fa7c87e0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.111+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791f90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.111+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789d70 0x5564fa791a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.111+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1604230902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 0x5564fa791510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.111+0000 7fabd81e8740 1 -- 172.21.15.28:0/1604230902 >> 172.21.15.28:0/1604230902 conn(0x5564fa785960 msgr2=0x5564fa78b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.111+0000 7fabb2ff5640 0 client.26626 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:11.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.111+0000 7fabd81e8740 1 -- 172.21.15.28:0/1604230902 shutdown_connections 2024-05-09T08:44:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.115+0000 7fabd81e8740 1 -- 172.21.15.28:0/1604230902 wait complete. 2024-05-09T08:44:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.116+0000 7fabd81e8740 20 client.26626 trim_cache size 0 max 16384 2024-05-09T08:44:11.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.148+0000 7fabd81e8740 1 Processor -- start 2024-05-09T08:44:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.150+0000 7fabd81e8740 1 -- start start 2024-05-09T08:44:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.150+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa790670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.150+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789d70 0x5564fa78be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.150+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.150+0000 7fabd81e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabd01af990 con 0x5564fa789450 2024-05-09T08:44:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.151+0000 7fabd81e8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fabd01b1710 con 0x5564fa789d70 2024-05-09T08:44:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.151+0000 7fabd81e8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fabd01b0120 con 0x5564fa794630 2024-05-09T08:44:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa790670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa790670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57360/0 (socket says 172.21.15.28:57360) 2024-05-09T08:44:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabceffd640 1 -- 172.21.15.28:0/2200924973 learned_addr learned my addr 172.21.15.28:0/2200924973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:11.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/2200924973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabce7fc640 1 --2- 172.21.15.28:0/2200924973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789d70 0x5564fa78be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabcf7fe640 1 -- 172.21.15.28:0/2200924973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789d70 msgr2=0x5564fa78be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:11.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/2200924973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789d70 0x5564fa78be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabcf7fe640 1 -- 172.21.15.28:0/2200924973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 msgr2=0x5564fa790670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:11.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/2200924973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa790670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.153+0000 7fabcf7fe640 1 -- 172.21.15.28:0/2200924973 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564fa789ba0 con 0x5564fa794630 2024-05-09T08:44:11.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.155+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/2200924973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fabc800c470 tx=0x7fabc800a790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5ea011f543a3965 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:11.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.157+0000 7fabb4ff9640 1 -- 172.21.15.28:0/2200924973 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8006e60 con 0x5564fa794630 2024-05-09T08:44:11.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.157+0000 7fabb4ff9640 1 -- 172.21.15.28:0/2200924973 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabc8032070 con 0x5564fa794630 2024-05-09T08:44:11.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.157+0000 7fabb4ff9640 1 -- 172.21.15.28:0/2200924973 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc800be10 con 0x5564fa794630 2024-05-09T08:44:11.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.158+0000 7fabd81e8740 1 -- 172.21.15.28:0/2200924973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 msgr2=0x5564fa791db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:11.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.158+0000 7fabd81e8740 1 --2- 172.21.15.28:0/2200924973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fabc800c470 tx=0x7fabc800a790 comp rx=0 tx=0).stop 2024-05-09T08:44:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.161+0000 7fabd81e8740 1 -- 172.21.15.28:0/2200924973 shutdown_connections 2024-05-09T08:44:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.161+0000 7fabd81e8740 1 --2- 172.21.15.28:0/2200924973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa794630 0x5564fa791db0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.161+0000 7fabd81e8740 1 --2- 172.21.15.28:0/2200924973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789d70 0x5564fa78be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.161+0000 7fabd81e8740 1 --2- 172.21.15.28:0/2200924973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa789450 0x5564fa790670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.161+0000 7fabd81e8740 1 -- 172.21.15.28:0/2200924973 >> 172.21.15.28:0/2200924973 conn(0x5564fa785960 msgr2=0x5564fa792c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.161+0000 7fabd81e8740 1 -- 172.21.15.28:0/2200924973 shutdown_connections 2024-05-09T08:44:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.161+0000 7fabd81e8740 1 -- 172.21.15.28:0/2200924973 wait complete. 2024-05-09T08:44:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabd81e8740 1 -- start start 2024-05-09T08:44:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabd81e8740 1 Processor -- start 2024-05-09T08:44:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 0x5564fa79c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa78ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabd81e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa794630 0x5564fa78d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabd81e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabd01b1040 con 0x5564fa794630 2024-05-09T08:44:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabd81e8740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fabd01b08b0 con 0x5564fa789450 2024-05-09T08:44:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabd81e8740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fabd01ad990 con 0x5564fa789d70 2024-05-09T08:44:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabd81e8740 10 client.0 _ll_register_callbacks cb 0x7fabc800ad60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.162+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa78ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.163+0000 7fabce7fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa78ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:49194/0 (socket says 172.21.15.28:49194) 2024-05-09T08:44:11.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.163+0000 7fabce7fc640 1 -- 172.21.15.28:0/1706619409 learned_addr learned my addr 172.21.15.28:0/1706619409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:11.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.163+0000 7fabcf7fe640 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa794630 0x5564fa78d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.163+0000 7fabceffd640 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 0x5564fa79c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.163+0000 7fabce7fc640 1 -- 172.21.15.28:0/1706619409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 msgr2=0x5564fa79c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.163+0000 7fabce7fc640 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 0x5564fa79c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.163+0000 7fabce7fc640 1 -- 172.21.15.28:0/1706619409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa794630 msgr2=0x5564fa78d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.163+0000 7fabce7fc640 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa794630 0x5564fa78d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.163+0000 7fabce7fc640 1 -- 172.21.15.28:0/1706619409 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564fa78dcc0 con 0x5564fa789d70 2024-05-09T08:44:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.164+0000 7fabce7fc640 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa78ecb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fabc0000c00 tx=0x7fabc00bbc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c9374101706185d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.164+0000 7fabb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:44:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.165+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1706619409 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc00ed6a0 con 0x5564fa789d70 2024-05-09T08:44:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.165+0000 7fabd81e8740 1 -- 172.21.15.28:0/1706619409 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabc8032070 con 0x5564fa789d70 2024-05-09T08:44:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.165+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1706619409 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9911+0+0 (secure 0 0 0) 0x7fabc00bc250 con 0x5564fa789d70 2024-05-09T08:44:11.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.168+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1706619409 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fabc00dc0b0 con 0x5564fa789d70 2024-05-09T08:44:11.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.170+0000 7fabd81e8740 10 client.26632 fetch_fsmap learned FSMap version 67 2024-05-09T08:44:11.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.170+0000 7fabd81e8740 1 -- 172.21.15.28:0/1706619409 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5564fa82c010 con 0x5564fa789d70 2024-05-09T08:44:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.175+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1706619409 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabc8032070 con 0x5564fa789d70 2024-05-09T08:44:11.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.175+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1706619409 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fabc00c47d0 con 0x5564fa789d70 2024-05-09T08:44:11.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.175+0000 7fabd81e8740 10 client.26632 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:44:11.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.175+0000 7fabd81e8740 10 client.26632 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:44:11.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.175+0000 7fabd81e8740 20 client.26632 populate_metadata read hostname 'smithi028' 2024-05-09T08:44:11.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.175+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564fa82c920 0x5564fa82ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:11.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.175+0000 7fabd81e8740 4 client.26632 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:44:11.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.175+0000 7fabd81e8740 1 -- 172.21.15.28:0/1706619409 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5564fa6f5dc0 con 0x5564fa82c920 2024-05-09T08:44:11.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.176+0000 7fabceffd640 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564fa82c920 0x5564fa82ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.178+0000 7fabceffd640 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564fa82c920 0x5564fa82ed40 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c163cfdbfa3abece server_cookie=5ce2ab5f5738cdfb in_seq=0 out_seq=0 2024-05-09T08:44:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.178+0000 7fabb2ff5640 10 client.26632 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabb2ff5640 1 -- 172.21.15.28:0/1706619409 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5564fa6f5dc0 con 0x5564fa82c920 2024-05-09T08:44:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabb2ff5640 10 client.26632 handle_command_reply: tid=0 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 client.26632 shutdown 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 -- 172.21.15.28:0/1706619409 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 msgr2=0x5564fa78ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa78ecb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fabc0000c00 tx=0x7fabc00bbc60 comp rx=0 tx=0).stop 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 -- 172.21.15.28:0/1706619409 shutdown_connections 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x5564fa82c920 0x5564fa82ed40 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564fa794630 0x5564fa78d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5564fa789d70 0x5564fa78ecb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 --2- 172.21.15.28:0/1706619409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5564fa789450 0x5564fa79c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 -- 172.21.15.28:0/1706619409 >> 172.21.15.28:0/1706619409 conn(0x5564fa785960 msgr2=0x5564fa784a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabb2ff5640 0 client.26632 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.531+0000 7fabd81e8740 1 -- 172.21.15.28:0/1706619409 shutdown_connections 2024-05-09T08:44:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.532+0000 7fabd81e8740 1 -- 172.21.15.28:0/1706619409 wait complete. 2024-05-09T08:44:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.532+0000 7fabd81e8740 20 client.26632 trim_cache size 0 max 16384 2024-05-09T08:44:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:44:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:44:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:44:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:44:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.532+0000 7fabd54d4640 1 -- 172.21.15.28:0/1384739726 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fabb80a2fd0 msgr2=0x7fabb80a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.532+0000 7fabd54d4640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fabb80a2fd0 0x7fabb80a5490 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fabc800e600 tx=0x7fabc800aa10 comp rx=0 tx=0).stop 2024-05-09T08:44:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.532+0000 7fabd54d4640 1 -- 172.21.15.28:0/1384739726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabd0109150 msgr2=0x7fabd019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.532+0000 7fabd54d4640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabd0109150 0x7fabd019e680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009900 tx=0x7fabc0003cd0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.533+0000 7fabd54d4640 1 -- 172.21.15.28:0/1384739726 shutdown_connections 2024-05-09T08:44:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.533+0000 7fabd54d4640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd0114980 0x7fabd019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.533+0000 7fabd54d4640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fabb80a2fd0 0x7fabb80a5490 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.533+0000 7fabd54d4640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabd0109150 0x7fabd019e680 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.533+0000 7fabd54d4640 1 --2- 172.21.15.28:0/1384739726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabd0108830 0x7fabd01a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.533+0000 7fabd54d4640 1 -- 172.21.15.28:0/1384739726 >> 172.21.15.28:0/1384739726 conn(0x7fabd006c900 msgr2=0x7fabd00704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.533+0000 7fabd54d4640 1 -- 172.21.15.28:0/1384739726 shutdown_connections 2024-05-09T08:44:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:11.533+0000 7fabd54d4640 1 -- 172.21.15.28:0/1384739726 wait complete. 2024-05-09T08:44:12.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:11 smithi184 ceph-mon[34509]: pgmap v574: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 189 op/s 2024-05-09T08:44:12.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:11 smithi028 ceph-mon[30119]: pgmap v574: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 189 op/s 2024-05-09T08:44:12.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:11 smithi165 ceph-mon[34540]: pgmap v574: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 189 op/s 2024-05-09T08:44:14.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:13 smithi028 ceph-mon[30119]: pgmap v575: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 862 KiB/s wr, 190 op/s 2024-05-09T08:44:14.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:13 smithi165 ceph-mon[34540]: pgmap v575: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 862 KiB/s wr, 190 op/s 2024-05-09T08:44:14.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:13 smithi184 ceph-mon[34509]: pgmap v575: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 862 KiB/s wr, 190 op/s 2024-05-09T08:44:15.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:14 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:15.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:14 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:15.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:14 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:44:16.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:15 smithi028 ceph-mon[30119]: pgmap v576: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-09T08:44:16.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:15 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:16.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:15 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:16.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:15 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:44:16.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:15 smithi165 ceph-mon[34540]: pgmap v576: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-09T08:44:16.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:15 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:16.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:15 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:16.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:15 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:44:16.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:15 smithi184 ceph-mon[34509]: pgmap v576: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-05-09T08:44:16.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:15 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:16.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:15 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:16.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:15 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:44:18.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:17 smithi028 ceph-mon[30119]: pgmap v577: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 190 op/s 2024-05-09T08:44:18.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:17 smithi165 ceph-mon[34540]: pgmap v577: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 190 op/s 2024-05-09T08:44:18.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:17 smithi184 ceph-mon[34509]: pgmap v577: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 190 op/s 2024-05-09T08:44:19.182 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:44:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:44:18.737+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:44:19.182 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:44:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:44:18.737+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:44:20.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:19 smithi028 ceph-mon[30119]: pgmap v578: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 191 op/s 2024-05-09T08:44:20.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:19 smithi165 ceph-mon[34540]: pgmap v578: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 191 op/s 2024-05-09T08:44:20.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:19 smithi184 ceph-mon[34509]: pgmap v578: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 191 op/s 2024-05-09T08:44:22.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:21 smithi028 ceph-mon[30119]: pgmap v579: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-05-09T08:44:22.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:21 smithi165 ceph-mon[34540]: pgmap v579: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-05-09T08:44:22.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:21 smithi184 ceph-mon[34509]: pgmap v579: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-05-09T08:44:23.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:22 smithi028 ceph-mon[30119]: pgmap v580: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 189 op/s 2024-05-09T08:44:23.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:22 smithi165 ceph-mon[34540]: pgmap v580: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 189 op/s 2024-05-09T08:44:23.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:22 smithi184 ceph-mon[34509]: pgmap v580: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 189 op/s 2024-05-09T08:44:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.689+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/1595939179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e8102200 msgr2=0x7fd4e810e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.689+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/1595939179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e8102200 0x7fd4e810e710 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009cd0 tx=0x7fd4e002a930 comp rx=0 tx=0).stop 2024-05-09T08:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.690+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/1595939179 shutdown_connections 2024-05-09T08:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.690+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/1595939179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e8102200 0x7fd4e810e710 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.690+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/1595939179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4e8101840 0x7fd4e8101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.690+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/1595939179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4e8107850 0x7fd4e8107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.690+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/1595939179 >> 172.21.15.28:0/1595939179 conn(0x7fd4e8078cc0 msgr2=0x7fd4e80790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.697+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/1595939179 shutdown_connections 2024-05-09T08:44:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.697+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/1595939179 wait complete. 2024-05-09T08:44:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.698+0000 7fd4ee3a5640 1 Processor -- start 2024-05-09T08:44:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.699+0000 7fd4ee3a5640 1 -- start start 2024-05-09T08:44:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.699+0000 7fd4ee3a5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4e8101840 0x7fd4e8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.700+0000 7fd4ee3a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4e8102200 0x7fd4e8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.700+0000 7fd4ee3a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e8107850 0x7fd4e819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.700+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4e8102200 0x7fd4e8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.700+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4e8102200 0x7fd4e8198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:51760/0 (socket says 172.21.15.28:51760) 2024-05-09T08:44:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.700+0000 7fd4e77fe640 1 -- 172.21.15.28:0/564218154 learned_addr learned my addr 172.21.15.28:0/564218154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.700+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e8111d20 con 0x7fd4e8107850 2024-05-09T08:44:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.700+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e8111ba0 con 0x7fd4e8101840 2024-05-09T08:44:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.700+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e8111ea0 con 0x7fd4e8102200 2024-05-09T08:44:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.701+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4e8101840 0x7fd4e8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.701+0000 7fd4ec942640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e8107850 0x7fd4e819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.702+0000 7fd4ec942640 1 -- 172.21.15.28:0/564218154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4e8102200 msgr2=0x7fd4e8198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.702+0000 7fd4ec942640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4e8102200 0x7fd4e8198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.702+0000 7fd4ec942640 1 -- 172.21.15.28:0/564218154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4e8101840 msgr2=0x7fd4e8197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.702+0000 7fd4ec942640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4e8101840 0x7fd4e8197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.702+0000 7fd4ec942640 1 -- 172.21.15.28:0/564218154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e819cfd0 con 0x7fd4e8107850 2024-05-09T08:44:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.703+0000 7fd4ec942640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e8107850 0x7fd4e819c8a0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009bf0 tx=0x7fd4e00316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.704+0000 7fd4e57fa640 1 -- 172.21.15.28:0/564218154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0004040 con 0x7fd4e8107850 2024-05-09T08:44:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.704+0000 7fd4e57fa640 1 -- 172.21.15.28:0/564218154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4e002d070 con 0x7fd4e8107850 2024-05-09T08:44:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.704+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e819d260 con 0x7fd4e8107850 2024-05-09T08:44:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.705+0000 7fd4e57fa640 1 -- 172.21.15.28:0/564218154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e002fc60 con 0x7fd4e8107850 2024-05-09T08:44:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.705+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e81a4b50 con 0x7fd4e8107850 2024-05-09T08:44:24.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.710+0000 7fd4f118d740 1 Processor -- start 2024-05-09T08:44:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.714+0000 7fd4f118d740 1 -- start start 2024-05-09T08:44:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.714+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a2b4f0 0x55b529a2b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.714+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a273e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.714+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a27920 0x55b529a33440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.714+0000 7fd4f118d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b529982340 con 0x55b529a27920 2024-05-09T08:44:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.714+0000 7fd4f118d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55b52992a290 con 0x55b529a26f60 2024-05-09T08:44:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.715+0000 7fd4f118d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b52997e300 con 0x55b529a2b4f0 2024-05-09T08:44:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.716+0000 7fd4e57fa640 1 -- 172.21.15.28:0/564218154 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd4e00041e0 con 0x7fd4e8107850 2024-05-09T08:44:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.717+0000 7fd4e57fa640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd4d00a2fd0 0x7fd4d00a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.717+0000 7fd4e7fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a273e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.717+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a27920 0x55b529a33440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.717+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a27920 0x55b529a33440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59988/0 (socket says 172.21.15.28:59988) 2024-05-09T08:44:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.717+0000 7fd4e77fe640 1 -- 172.21.15.28:0/1934267458 learned_addr learned my addr 172.21.15.28:0/1934267458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.717+0000 7fd4e77fe640 1 --2- 172.21.15.28:0/1934267458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a2b4f0 0x55b529a2b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.718+0000 7fd4e7fff640 1 -- 172.21.15.28:0/1934267458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a2b4f0 msgr2=0x55b529a2b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.718+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/1934267458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a2b4f0 0x55b529a2b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.718+0000 7fd4e7fff640 1 -- 172.21.15.28:0/1934267458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a27920 msgr2=0x55b529a33440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.718+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/1934267458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a27920 0x55b529a33440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.718+0000 7fd4e7fff640 1 -- 172.21.15.28:0/1934267458 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b52994e8e0 con 0x55b529a26f60 2024-05-09T08:44:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.718+0000 7fd4ec942640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd4d00a2fd0 0x7fd4d00a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.718+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/1934267458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a273e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00b760 tx=0x7fd4dc00bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c372c287b8baeaa3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.719+0000 7fd4e57fa640 1 -- 172.21.15.28:0/564218154 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9911+0+0 (secure 0 0 0) 0x7fd4e00f10c0 con 0x7fd4e8107850 2024-05-09T08:44:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.719+0000 7fd4ec942640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd4d00a2fd0 0x7fd4d00a5490 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0049000 tx=0x7fd4e0031aa0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.724+0000 7fd4ce7fc640 1 -- 172.21.15.28:0/1934267458 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc00d8f0 con 0x55b529a26f60 2024-05-09T08:44:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.725+0000 7fd4ce7fc640 1 -- 172.21.15.28:0/1934267458 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4dc019070 con 0x55b529a26f60 2024-05-09T08:44:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.726+0000 7fd4f118d740 1 -- 172.21.15.28:0/1934267458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 msgr2=0x55b529a273e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.726+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1934267458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a273e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00b760 tx=0x7fd4dc00bc20 comp rx=0 tx=0).stop 2024-05-09T08:44:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.729+0000 7fd4f118d740 1 -- 172.21.15.28:0/1934267458 shutdown_connections 2024-05-09T08:44:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.729+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1934267458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a27920 0x55b529a33440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.729+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1934267458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a273e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.729+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1934267458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a2b4f0 0x55b529a2b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.729+0000 7fd4f118d740 1 -- 172.21.15.28:0/1934267458 >> 172.21.15.28:0/1934267458 conn(0x55b529a19410 msgr2=0x55b529a197f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.729+0000 7fd4f118d740 1 -- 172.21.15.28:0/1934267458 shutdown_connections 2024-05-09T08:44:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.729+0000 7fd4f118d740 1 -- 172.21.15.28:0/1934267458 wait complete. 2024-05-09T08:44:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.730+0000 7fd4f118d740 1 -- start start 2024-05-09T08:44:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.731+0000 7fd4f118d740 1 Processor -- start 2024-05-09T08:44:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.732+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a333a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.732+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a30730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.732+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a2b4f0 0x55b529a30d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.733+0000 7fd4f118d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b529a44ab0 con 0x55b529a2b4f0 2024-05-09T08:44:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.733+0000 7fd4f118d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55b52993d790 con 0x55b529a26f60 2024-05-09T08:44:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.733+0000 7fd4f118d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55b529a44c30 con 0x55b529a27920 2024-05-09T08:44:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.733+0000 7fd4f118d740 10 client.0 _ll_register_callbacks cb 0x55b529978a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.734+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a2b4f0 0x55b529a30d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.734+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.734+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a333a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:52824/0 (socket says 172.21.15.28:52824) 2024-05-09T08:44:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.734+0000 7fd4e77fe640 1 -- 172.21.15.28:0/1029456570 learned_addr learned my addr 172.21.15.28:0/1029456570 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.735+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a30730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.735+0000 7fd4e7fff640 1 -- 172.21.15.28:0/1029456570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 msgr2=0x55b529a333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.735+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.735+0000 7fd4e7fff640 1 -- 172.21.15.28:0/1029456570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a2b4f0 msgr2=0x55b529a30d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.735+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a2b4f0 0x55b529a30d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.735+0000 7fd4e7fff640 1 -- 172.21.15.28:0/1029456570 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b529696210 con 0x55b529a27920 2024-05-09T08:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.735+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a30730 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc014a10 tx=0x7fd4dc014660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36a34c4d5b41374 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.735+0000 7fd4c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.736+0000 7fd4c77fe640 1 -- 172.21.15.28:0/1029456570 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc023070 con 0x55b529a27920 2024-05-09T08:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.736+0000 7fd4c77fe640 1 -- 172.21.15.28:0/1029456570 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9911+0+0 (secure 0 0 0) 0x7fd4dc00dd00 con 0x55b529a27920 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.736+0000 7fd4f118d740 1 -- 172.21.15.28:0/1029456570 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4dc019070 con 0x55b529a27920 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.736+0000 7fd4c77fe640 1 -- 172.21.15.28:0/1029456570 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fd4dc02f050 con 0x55b529a27920 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.736+0000 7fd4f118d740 10 client.26641 fetch_fsmap learned FSMap version 67 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.736+0000 7fd4f118d740 1 -- 172.21.15.28:0/1029456570 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55b529762620 con 0x55b529a27920 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.737+0000 7fd4c77fe640 1 -- 172.21.15.28:0/1029456570 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4dc019070 con 0x55b529a27920 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.737+0000 7fd4c77fe640 1 -- 172.21.15.28:0/1029456570 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fd4dc013070 con 0x55b529a27920 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.737+0000 7fd4f118d740 10 client.26641 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.737+0000 7fd4f118d740 10 client.26641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.737+0000 7fd4f118d740 20 client.26641 populate_metadata read hostname 'smithi028' 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.737+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55b529a51ab0 0x55b529a53ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.737+0000 7fd4f118d740 4 client.26641 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.737+0000 7fd4f118d740 1 -- 172.21.15.28:0/1029456570 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b52998fa70 con 0x55b529a51ab0 2024-05-09T08:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.738+0000 7fd4e77fe640 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55b529a51ab0 0x55b529a53ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.738+0000 7fd4e77fe640 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55b529a51ab0 0x55b529a53ea0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28f6dda8aac9f986 server_cookie=18c3b8fe503e101f in_seq=0 out_seq=0 2024-05-09T08:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.738+0000 7fd4c77fe640 10 client.26641 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:24.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.742+0000 7fd4c77fe640 1 -- 172.21.15.28:0/1029456570 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b52998fa70 con 0x55b529a51ab0 2024-05-09T08:44:24.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.742+0000 7fd4c77fe640 10 client.26641 handle_command_reply: tid=0 2024-05-09T08:44:24.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.742+0000 7fd4f118d740 1 client.26641 shutdown 2024-05-09T08:44:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.747+0000 7fd4f118d740 1 -- 172.21.15.28:0/1029456570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 msgr2=0x55b529a30730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.747+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a30730 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc014a10 tx=0x7fd4dc014660 comp rx=0 tx=0).stop 2024-05-09T08:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.750+0000 7fd4f118d740 1 -- 172.21.15.28:0/1029456570 shutdown_connections 2024-05-09T08:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.750+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55b529a51ab0 0x55b529a53ea0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.750+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a2b4f0 0x55b529a30d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.750+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a30730 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.750+0000 7fd4f118d740 1 --2- 172.21.15.28:0/1029456570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a333a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.750+0000 7fd4f118d740 1 -- 172.21.15.28:0/1029456570 >> 172.21.15.28:0/1029456570 conn(0x55b529a19410 msgr2=0x55b529a267e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.751+0000 7fd4c77fe640 0 client.26641 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.758+0000 7fd4f118d740 1 -- 172.21.15.28:0/1029456570 shutdown_connections 2024-05-09T08:44:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.758+0000 7fd4f118d740 1 -- 172.21.15.28:0/1029456570 wait complete. 2024-05-09T08:44:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.759+0000 7fd4f118d740 20 client.26641 trim_cache size 0 max 16384 2024-05-09T08:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.789+0000 7fd4f118d740 1 Processor -- start 2024-05-09T08:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.790+0000 7fd4f118d740 1 -- start start 2024-05-09T08:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.791+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a26f60 0x55b529a28f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.791+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a2a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.791+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a2b4f0 0x55b529a4da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.791+0000 7fd4f118d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e8075470 con 0x55b529a26f60 2024-05-09T08:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.791+0000 7fd4f118d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e81a62a0 con 0x55b529a2b4f0 2024-05-09T08:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.791+0000 7fd4f118d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e81a4d20 con 0x55b529a27920 2024-05-09T08:44:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.795+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a26f60 0x55b529a28f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.795+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a26f60 0x55b529a28f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60008/0 (socket says 172.21.15.28:60008) 2024-05-09T08:44:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.795+0000 7fd4e77fe640 1 -- 172.21.15.28:0/3135746770 learned_addr learned my addr 172.21.15.28:0/3135746770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.795+0000 7fd4e77fe640 1 --2- 172.21.15.28:0/3135746770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a2b4f0 0x55b529a4da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.795+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/3135746770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a2a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.796+0000 7fd4e77fe640 1 -- 172.21.15.28:0/3135746770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 msgr2=0x55b529a2a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.796+0000 7fd4e77fe640 1 --2- 172.21.15.28:0/3135746770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a2a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.796+0000 7fd4e77fe640 1 -- 172.21.15.28:0/3135746770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a2b4f0 msgr2=0x55b529a4da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.796+0000 7fd4e77fe640 1 --2- 172.21.15.28:0/3135746770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a2b4f0 0x55b529a4da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.796+0000 7fd4e77fe640 1 -- 172.21.15.28:0/3135746770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b529a54210 con 0x55b529a26f60 2024-05-09T08:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.800+0000 7fd4e77fe640 1 --2- 172.21.15.28:0/3135746770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a26f60 0x55b529a28f80 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80093b0 tx=0x7fd4d80046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c006528d3e155937 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.802+0000 7fd4cdffb640 1 -- 172.21.15.28:0/3135746770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d801f920 con 0x55b529a26f60 2024-05-09T08:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.802+0000 7fd4cdffb640 1 -- 172.21.15.28:0/3135746770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4d8017d70 con 0x55b529a26f60 2024-05-09T08:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.802+0000 7fd4cdffb640 1 -- 172.21.15.28:0/3135746770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8016d60 con 0x55b529a26f60 2024-05-09T08:44:24.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.803+0000 7fd4f118d740 1 -- 172.21.15.28:0/3135746770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a26f60 msgr2=0x55b529a28f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.803+0000 7fd4f118d740 1 --2- 172.21.15.28:0/3135746770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a26f60 0x55b529a28f80 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80093b0 tx=0x7fd4d80046e0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.812+0000 7fd4f118d740 1 -- 172.21.15.28:0/3135746770 shutdown_connections 2024-05-09T08:44:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.812+0000 7fd4f118d740 1 --2- 172.21.15.28:0/3135746770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a2b4f0 0x55b529a4da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.812+0000 7fd4f118d740 1 --2- 172.21.15.28:0/3135746770 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a2a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.812+0000 7fd4f118d740 1 --2- 172.21.15.28:0/3135746770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a26f60 0x55b529a28f80 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.812+0000 7fd4f118d740 1 -- 172.21.15.28:0/3135746770 >> 172.21.15.28:0/3135746770 conn(0x55b529a19410 msgr2=0x55b529a2f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.816+0000 7fd4f118d740 1 -- 172.21.15.28:0/3135746770 shutdown_connections 2024-05-09T08:44:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.819+0000 7fd4f118d740 1 -- 172.21.15.28:0/3135746770 wait complete. 2024-05-09T08:44:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.819+0000 7fd4f118d740 1 -- start start 2024-05-09T08:44:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.820+0000 7fd4f118d740 1 Processor -- start 2024-05-09T08:44:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.823+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a49f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.823+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a4a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.823+0000 7fd4f118d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a2b4f0 0x55b529a46f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.824+0000 7fd4f118d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e81a5bd0 con 0x55b529a2b4f0 2024-05-09T08:44:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.824+0000 7fd4f118d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e81a5440 con 0x55b529a26f60 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.824+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a49f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.824+0000 7fd4e77fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a49f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:52836/0 (socket says 172.21.15.28:52836) 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.824+0000 7fd4f118d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e8199330 con 0x55b529a27920 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.824+0000 7fd4e77fe640 1 -- 172.21.15.28:0/4201925657 learned_addr learned my addr 172.21.15.28:0/4201925657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.824+0000 7fd4f118d740 10 client.0 _ll_register_callbacks cb 0x7fd4d8009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.824+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a4a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.826+0000 7fd4e7fff640 1 -- 172.21.15.28:0/4201925657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 msgr2=0x55b529a49f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.826+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a49f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.826+0000 7fd4e7fff640 1 -- 172.21.15.28:0/4201925657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a2b4f0 msgr2=0x55b529a46f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.826+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a2b4f0 0x55b529a46f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.826+0000 7fd4e7fff640 1 -- 172.21.15.28:0/4201925657 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b529a47500 con 0x55b529a27920 2024-05-09T08:44:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.827+0000 7fd4e7fff640 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a4a480 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55b529a1a250 tx=0x7fd4dc0148e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d42a107a5a9add07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.827+0000 7fd4c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:44:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.831+0000 7fd4c77fe640 1 -- 172.21.15.28:0/4201925657 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc023070 con 0x55b529a27920 2024-05-09T08:44:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.831+0000 7fd4c77fe640 1 -- 172.21.15.28:0/4201925657 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9911+0+0 (secure 0 0 0) 0x7fd4dc01fda0 con 0x55b529a27920 2024-05-09T08:44:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.832+0000 7fd4f118d740 1 -- 172.21.15.28:0/4201925657 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4d8017d70 con 0x55b529a27920 2024-05-09T08:44:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.835+0000 7fd4c77fe640 1 -- 172.21.15.28:0/4201925657 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fd4dc01f910 con 0x55b529a27920 2024-05-09T08:44:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.835+0000 7fd4f118d740 10 client.26644 fetch_fsmap learned FSMap version 67 2024-05-09T08:44:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.835+0000 7fd4f118d740 1 -- 172.21.15.28:0/4201925657 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55b529ab8820 con 0x55b529a27920 2024-05-09T08:44:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.840+0000 7fd4c77fe640 1 -- 172.21.15.28:0/4201925657 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4dc019040 con 0x55b529a27920 2024-05-09T08:44:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.841+0000 7fd4c77fe640 1 -- 172.21.15.28:0/4201925657 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fd4dc0036b0 con 0x55b529a27920 2024-05-09T08:44:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.842+0000 7fd4f118d740 10 client.26644 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:44:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.842+0000 7fd4f118d740 10 client.26644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:44:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.842+0000 7fd4f118d740 20 client.26644 populate_metadata read hostname 'smithi028' 2024-05-09T08:44:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.842+0000 7fd4f118d740 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55b529ab8f40 0x55b529abb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.842+0000 7fd4f118d740 4 client.26644 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:44:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.842+0000 7fd4f118d740 1 -- 172.21.15.28:0/4201925657 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd4d8017d70 con 0x55b529ab8f40 2024-05-09T08:44:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.843+0000 7fd4e77fe640 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55b529ab8f40 0x55b529abb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.845+0000 7fd4e77fe640 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55b529ab8f40 0x55b529abb360 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a22e8a10b2356a7 server_cookie=2f327828781428eb in_seq=0 out_seq=0 2024-05-09T08:44:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:24.845+0000 7fd4c77fe640 10 client.26644 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.389+0000 7fd4c77fe640 1 -- 172.21.15.28:0/4201925657 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd4d8017d70 con 0x55b529ab8f40 2024-05-09T08:44:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.389+0000 7fd4c77fe640 10 client.26644 handle_command_reply: tid=0 2024-05-09T08:44:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.390+0000 7fd4f118d740 1 client.26644 shutdown 2024-05-09T08:44:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.390+0000 7fd4f118d740 1 -- 172.21.15.28:0/4201925657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 msgr2=0x55b529a4a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.390+0000 7fd4f118d740 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a4a480 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55b529a1a250 tx=0x7fd4dc0148e0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.390+0000 7fd4f118d740 1 -- 172.21.15.28:0/4201925657 shutdown_connections 2024-05-09T08:44:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.390+0000 7fd4f118d740 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55b529ab8f40 0x55b529abb360 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.390+0000 7fd4f118d740 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b529a2b4f0 0x55b529a46f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.390+0000 7fd4c77fe640 0 client.26644 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.390+0000 7fd4f118d740 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55b529a27920 0x55b529a4a480 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.391+0000 7fd4f118d740 1 --2- 172.21.15.28:0/4201925657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b529a26f60 0x55b529a49f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.391+0000 7fd4f118d740 1 -- 172.21.15.28:0/4201925657 >> 172.21.15.28:0/4201925657 conn(0x55b529a19410 msgr2=0x55b529a2ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.391+0000 7fd4f118d740 1 -- 172.21.15.28:0/4201925657 shutdown_connections 2024-05-09T08:44:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.391+0000 7fd4f118d740 1 -- 172.21.15.28:0/4201925657 wait complete. 2024-05-09T08:44:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.391+0000 7fd4f118d740 20 client.26644 trim_cache size 0 max 16384 2024-05-09T08:44:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:44:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:44:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:44:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:44:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd4d00a2fd0 msgr2=0x7fd4d00a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd4d00a2fd0 0x7fd4d00a5490 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0049000 tx=0x7fd4e0031aa0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e8107850 msgr2=0x7fd4e819c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e8107850 0x7fd4e819c8a0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009bf0 tx=0x7fd4e00316b0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 shutdown_connections 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e8107850 0x7fd4e819c8a0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd4d00a2fd0 0x7fd4d00a5490 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4e8102200 0x7fd4e8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 --2- 172.21.15.28:0/564218154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4e8101840 0x7fd4e8197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 >> 172.21.15.28:0/564218154 conn(0x7fd4e8078cc0 msgr2=0x7fd4e80fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 shutdown_connections 2024-05-09T08:44:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:25.392+0000 7fd4ee3a5640 1 -- 172.21.15.28:0/564218154 wait complete. 2024-05-09T08:44:25.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:25 smithi165 ceph-mon[34540]: pgmap v581: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 200 op/s 2024-05-09T08:44:25.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:25 smithi184 ceph-mon[34509]: pgmap v581: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 200 op/s 2024-05-09T08:44:25.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:25 smithi028 ceph-mon[30119]: pgmap v581: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 200 op/s 2024-05-09T08:44:27.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:26 smithi165 ceph-mon[34540]: osdmap e130: 12 total, 12 up, 12 in 2024-05-09T08:44:27.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:26 smithi184 ceph-mon[34509]: osdmap e130: 12 total, 12 up, 12 in 2024-05-09T08:44:27.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:26 smithi028 ceph-mon[30119]: osdmap e130: 12 total, 12 up, 12 in 2024-05-09T08:44:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:27 smithi165 ceph-mon[34540]: pgmap v583: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 204 op/s 2024-05-09T08:44:28.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:27 smithi184 ceph-mon[34509]: pgmap v583: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 204 op/s 2024-05-09T08:44:28.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:27 smithi028 ceph-mon[30119]: pgmap v583: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 204 op/s 2024-05-09T08:44:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:29 smithi165 ceph-mon[34540]: pgmap v584: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 190 op/s 2024-05-09T08:44:30.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:29 smithi184 ceph-mon[34509]: pgmap v584: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 190 op/s 2024-05-09T08:44:30.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:29 smithi028 ceph-mon[30119]: pgmap v584: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 190 op/s 2024-05-09T08:44:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:31 smithi165 ceph-mon[34540]: pgmap v585: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 199 op/s 2024-05-09T08:44:32.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:31 smithi184 ceph-mon[34509]: pgmap v585: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 199 op/s 2024-05-09T08:44:32.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:31 smithi028 ceph-mon[30119]: pgmap v585: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 199 op/s 2024-05-09T08:44:34.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:33 smithi165 ceph-mon[34540]: pgmap v586: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 209 op/s 2024-05-09T08:44:34.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:33 smithi184 ceph-mon[34509]: pgmap v586: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 209 op/s 2024-05-09T08:44:34.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:33 smithi028 ceph-mon[30119]: pgmap v586: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 209 op/s 2024-05-09T08:44:36.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:35 smithi165 ceph-mon[34540]: pgmap v587: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 203 op/s 2024-05-09T08:44:36.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:35 smithi184 ceph-mon[34509]: pgmap v587: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 203 op/s 2024-05-09T08:44:36.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:35 smithi028 ceph-mon[30119]: pgmap v587: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 203 op/s 2024-05-09T08:44:38.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:37 smithi165 ceph-mon[34540]: pgmap v588: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-05-09T08:44:38.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:37 smithi184 ceph-mon[34509]: pgmap v588: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-05-09T08:44:38.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:37 smithi028 ceph-mon[30119]: pgmap v588: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-05-09T08:44:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.553+0000 7fa93b59e640 1 -- 172.21.15.28:0/3801555220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934005100 msgr2=0x7fa9340a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.553+0000 7fa93b59e640 1 --2- 172.21.15.28:0/3801555220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934005100 0x7fa9340a2b90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa93c050b40 tx=0x7fa93c09a700 comp rx=0 tx=0).stop 2024-05-09T08:44:38.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.557+0000 7fa93b59e640 1 -- 172.21.15.28:0/3801555220 shutdown_connections 2024-05-09T08:44:38.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.558+0000 7fa93b59e640 1 --2- 172.21.15.28:0/3801555220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934005100 0x7fa9340a2b90 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.558+0000 7fa93b59e640 1 --2- 172.21.15.28:0/3801555220 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa934004740 0x7fa934004bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.558+0000 7fa93b59e640 1 --2- 172.21.15.28:0/3801555220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa93409be10 0x7fa93409c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.558+0000 7fa93b59e640 1 -- 172.21.15.28:0/3801555220 >> 172.21.15.28:0/3801555220 conn(0x7fa93400a740 msgr2=0x7fa93400ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.559+0000 7fa93b59e640 1 -- 172.21.15.28:0/3801555220 shutdown_connections 2024-05-09T08:44:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.559+0000 7fa93b59e640 1 -- 172.21.15.28:0/3801555220 wait complete. 2024-05-09T08:44:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.560+0000 7fa93b59e640 1 Processor -- start 2024-05-09T08:44:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.560+0000 7fa93b59e640 1 -- start start 2024-05-09T08:44:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.561+0000 7fa93b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934004740 0x7fa934141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.561+0000 7fa93b59e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa934005100 0x7fa9341424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.561+0000 7fa93b59e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa93409be10 0x7fa93413b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.561+0000 7fa93b59e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9340b4e10 con 0x7fa934004740 2024-05-09T08:44:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.561+0000 7fa93b59e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa9340b4c90 con 0x7fa93409be10 2024-05-09T08:44:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.561+0000 7fa93b59e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa9340b4f90 con 0x7fa934005100 2024-05-09T08:44:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.562+0000 7fa93a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934004740 0x7fa934141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.562+0000 7fa93a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934004740 0x7fa934141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50592/0 (socket says 172.21.15.28:50592) 2024-05-09T08:44:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.562+0000 7fa93ad9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa93409be10 0x7fa93413b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.562+0000 7fa93a59c640 1 -- 172.21.15.28:0/1633430338 learned_addr learned my addr 172.21.15.28:0/1633430338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.562+0000 7fa93a59c640 1 -- 172.21.15.28:0/1633430338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa934005100 msgr2=0x7fa9341424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.562+0000 7fa93a59c640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa934005100 0x7fa9341424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.562+0000 7fa93a59c640 1 -- 172.21.15.28:0/1633430338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa93409be10 msgr2=0x7fa93413b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.562+0000 7fa93a59c640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa93409be10 0x7fa93413b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.562+0000 7fa93a59c640 1 -- 172.21.15.28:0/1633430338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa93413bac0 con 0x7fa934004740 2024-05-09T08:44:38.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.563+0000 7fa93a59c640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934004740 0x7fa934141fa0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00c890 tx=0x7fa92c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:38.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.566+0000 7fa92b7fe640 1 -- 172.21.15.28:0/1633430338 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c004220 con 0x7fa934004740 2024-05-09T08:44:38.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.566+0000 7fa92b7fe640 1 -- 172.21.15.28:0/1633430338 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa92c0043c0 con 0x7fa934004740 2024-05-09T08:44:38.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.566+0000 7fa92b7fe640 1 -- 172.21.15.28:0/1633430338 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c0049e0 con 0x7fa934004740 2024-05-09T08:44:38.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.566+0000 7fa93b59e640 1 -- 172.21.15.28:0/1633430338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa93413bdb0 con 0x7fa934004740 2024-05-09T08:44:38.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.566+0000 7fa93b59e640 1 -- 172.21.15.28:0/1633430338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa93413fd00 con 0x7fa934004740 2024-05-09T08:44:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.572+0000 7fa94380f740 1 Processor -- start 2024-05-09T08:44:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.572+0000 7fa92b7fe640 1 -- 172.21.15.28:0/1633430338 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa92c004b80 con 0x7fa934004740 2024-05-09T08:44:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.572+0000 7fa94380f740 1 -- start start 2024-05-09T08:44:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.573+0000 7fa94380f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 0x55c7c12e59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.573+0000 7fa94380f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12e5ef0 0x55c7c12f0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.574+0000 7fa94380f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12f07b0 0x55c7c12f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.574+0000 7fa94380f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c7c124a340 con 0x55c7c12e55d0 2024-05-09T08:44:38.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.574+0000 7fa94380f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55c7c11f2290 con 0x55c7c12f07b0 2024-05-09T08:44:38.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.574+0000 7fa94380f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c7c1246300 con 0x55c7c12e5ef0 2024-05-09T08:44:38.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.575+0000 7fa939d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12f07b0 0x55c7c12f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.575+0000 7fa93a59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12e5ef0 0x55c7c12f0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.575+0000 7fa93ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 0x55c7c12e59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.575+0000 7fa93ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 0x55c7c12e59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50600/0 (socket says 172.21.15.28:50600) 2024-05-09T08:44:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.575+0000 7fa93ad9d640 1 -- 172.21.15.28:0/3702292888 learned_addr learned my addr 172.21.15.28:0/3702292888 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:38.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.575+0000 7fa92b7fe640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9240a2f00 0x7fa9240a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.579+0000 7fa93ad9d640 1 -- 172.21.15.28:0/3702292888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12e5ef0 msgr2=0x55c7c12f0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:38.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.579+0000 7fa93ad9d640 1 --2- 172.21.15.28:0/3702292888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12e5ef0 0x55c7c12f0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.579+0000 7fa93ad9d640 1 -- 172.21.15.28:0/3702292888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12f07b0 msgr2=0x55c7c12f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:38.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.579+0000 7fa93ad9d640 1 --2- 172.21.15.28:0/3702292888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12f07b0 0x55c7c12f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.579+0000 7fa93ad9d640 1 -- 172.21.15.28:0/3702292888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7c12168e0 con 0x55c7c12e55d0 2024-05-09T08:44:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.579+0000 7fa93ad9d640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9240a2f00 0x7fa9240a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.579+0000 7fa92b7fe640 1 -- 172.21.15.28:0/1633430338 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9835+0+0 (secure 0 0 0) 0x7fa92c0cf580 con 0x7fa934004740 2024-05-09T08:44:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.580+0000 7fa93ad9d640 1 --2- 172.21.15.28:0/3702292888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 0x55c7c12e59b0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa93c02d780 tx=0x7fa93c067b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0e8b8c779ffad1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:38.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.580+0000 7fa93ad9d640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9240a2f00 0x7fa9240a53c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa93c067540 tx=0x7fa93c0a3040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.584+0000 7fa928ff9640 1 -- 172.21.15.28:0/3702292888 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c0a6040 con 0x55c7c12e55d0 2024-05-09T08:44:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.584+0000 7fa928ff9640 1 -- 172.21.15.28:0/3702292888 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93c073790 con 0x55c7c12e55d0 2024-05-09T08:44:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.585+0000 7fa94380f740 1 -- 172.21.15.28:0/3702292888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 msgr2=0x55c7c12e59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.585+0000 7fa94380f740 1 --2- 172.21.15.28:0/3702292888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 0x55c7c12e59b0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa93c02d780 tx=0x7fa93c067b00 comp rx=0 tx=0).stop 2024-05-09T08:44:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.590+0000 7fa94380f740 1 -- 172.21.15.28:0/3702292888 shutdown_connections 2024-05-09T08:44:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.590+0000 7fa94380f740 1 --2- 172.21.15.28:0/3702292888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12f07b0 0x55c7c12f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.590+0000 7fa94380f740 1 --2- 172.21.15.28:0/3702292888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12e5ef0 0x55c7c12f0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.590+0000 7fa94380f740 1 --2- 172.21.15.28:0/3702292888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 0x55c7c12e59b0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.590+0000 7fa94380f740 1 -- 172.21.15.28:0/3702292888 >> 172.21.15.28:0/3702292888 conn(0x55c7c12e1ae0 msgr2=0x55c7c12e1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.590+0000 7fa94380f740 1 -- 172.21.15.28:0/3702292888 shutdown_connections 2024-05-09T08:44:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.592+0000 7fa94380f740 1 -- 172.21.15.28:0/3702292888 wait complete. 2024-05-09T08:44:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.592+0000 7fa94380f740 1 -- start start 2024-05-09T08:44:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.592+0000 7fa94380f740 1 Processor -- start 2024-05-09T08:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.594+0000 7fa94380f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 0x55c7c12ed6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.594+0000 7fa94380f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.594+0000 7fa94380f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c12ee160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.594+0000 7fa94380f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c7c12f3380 con 0x55c7c12e55d0 2024-05-09T08:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.594+0000 7fa94380f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55c7c1205790 con 0x55c7c12e5ef0 2024-05-09T08:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.594+0000 7fa94380f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55c7c12f3500 con 0x55c7c12f07b0 2024-05-09T08:44:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.594+0000 7fa94380f740 10 client.0 _ll_register_callbacks cb 0x55c7c1240a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.595+0000 7fa939d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.595+0000 7fa939d9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:38446/0 (socket says 172.21.15.28:38446) 2024-05-09T08:44:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.595+0000 7fa939d9b640 1 -- 172.21.15.28:0/2594982328 learned_addr learned my addr 172.21.15.28:0/2594982328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:38.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.597+0000 7fa93ad9d640 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c12ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.597+0000 7fa939d9b640 1 -- 172.21.15.28:0/2594982328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 msgr2=0x55c7c12ee160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.597+0000 7fa939d9b640 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c12ee160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.597+0000 7fa939d9b640 1 -- 172.21.15.28:0/2594982328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 msgr2=0x55c7c12ed6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.597+0000 7fa939d9b640 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 0x55c7c12ed6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.597+0000 7fa939d9b640 1 -- 172.21.15.28:0/2594982328 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7c0f5e210 con 0x55c7c12e5ef0 2024-05-09T08:44:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.598+0000 7fa939d9b640 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edc20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa93000f670 tx=0x7fa93000af90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12ca732c8775654a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.598+0000 7fa921ffb640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:44:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.602+0000 7fa921ffb640 1 -- 172.21.15.28:0/2594982328 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93000da80 con 0x55c7c12e5ef0 2024-05-09T08:44:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.602+0000 7fa94380f740 1 -- 172.21.15.28:0/2594982328 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa93c073790 con 0x55c7c12e5ef0 2024-05-09T08:44:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.602+0000 7fa921ffb640 1 -- 172.21.15.28:0/2594982328 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9835+0+0 (secure 0 0 0) 0x7fa93000dc20 con 0x55c7c12e5ef0 2024-05-09T08:44:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.607+0000 7fa921ffb640 1 -- 172.21.15.28:0/2594982328 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa93000de20 con 0x55c7c12e5ef0 2024-05-09T08:44:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.608+0000 7fa94380f740 10 client.26804 fetch_fsmap learned FSMap version 67 2024-05-09T08:44:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.608+0000 7fa94380f740 1 -- 172.21.15.28:0/2594982328 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55c7c102a620 con 0x55c7c12e5ef0 2024-05-09T08:44:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.610+0000 7fa921ffb640 1 -- 172.21.15.28:0/2594982328 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93c073790 con 0x55c7c12e5ef0 2024-05-09T08:44:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.610+0000 7fa921ffb640 1 -- 172.21.15.28:0/2594982328 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fa930010720 con 0x55c7c12e5ef0 2024-05-09T08:44:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.610+0000 7fa94380f740 10 client.26804 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:44:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.610+0000 7fa94380f740 10 client.26804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:44:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.610+0000 7fa94380f740 20 client.26804 populate_metadata read hostname 'smithi028' 2024-05-09T08:44:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.610+0000 7fa94380f740 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c7c13006d0 0x55c7c1302b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.610+0000 7fa94380f740 4 client.26804 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.610+0000 7fa94380f740 1 -- 172.21.15.28:0/2594982328 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7c1257a70 con 0x55c7c13006d0 2024-05-09T08:44:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.610+0000 7fa93a59c640 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c7c13006d0 0x55c7c1302b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.618+0000 7fa93a59c640 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c7c13006d0 0x55c7c1302b50 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3abf245a4adb9138 server_cookie=c3d7d12a0c6e4d84 in_seq=0 out_seq=0 2024-05-09T08:44:38.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.619+0000 7fa921ffb640 10 client.26804 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:38.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.621+0000 7fa921ffb640 1 -- 172.21.15.28:0/2594982328 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7c1257a70 con 0x55c7c13006d0 2024-05-09T08:44:38.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.621+0000 7fa921ffb640 10 client.26804 handle_command_reply: tid=0 2024-05-09T08:44:38.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.621+0000 7fa94380f740 1 client.26804 shutdown 2024-05-09T08:44:38.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 -- 172.21.15.28:0/2594982328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e5ef0 msgr2=0x55c7c12edc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:38.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edc20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa93000f670 tx=0x7fa93000af90 comp rx=0 tx=0).stop 2024-05-09T08:44:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 -- 172.21.15.28:0/2594982328 shutdown_connections 2024-05-09T08:44:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c7c13006d0 0x55c7c1302b50 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c12ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edc20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 --2- 172.21.15.28:0/2594982328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e55d0 0x55c7c12ed6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 -- 172.21.15.28:0/2594982328 >> 172.21.15.28:0/2594982328 conn(0x55c7c12e1ae0 msgr2=0x55c7c12e7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa921ffb640 0 client.26804 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 -- 172.21.15.28:0/2594982328 shutdown_connections 2024-05-09T08:44:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.622+0000 7fa94380f740 1 -- 172.21.15.28:0/2594982328 wait complete. 2024-05-09T08:44:38.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.623+0000 7fa94380f740 20 client.26804 trim_cache size 0 max 16384 2024-05-09T08:44:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.652+0000 7fa94380f740 1 Processor -- start 2024-05-09T08:44:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa94380f740 1 -- start start 2024-05-09T08:44:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa94380f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 0x55c7c12fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa94380f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa94380f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c1308420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa94380f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c0104c0 con 0x55c7c12e5ef0 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa94380f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c012240 con 0x55c7c12e55d0 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa94380f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c010c50 con 0x55c7c12f07b0 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa93a59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 0x55c7c12fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa93ad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c1308420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.653+0000 7fa93ad9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c1308420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:37994/0 (socket says 172.21.15.28:37994) 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.654+0000 7fa93ad9d640 1 -- 172.21.15.28:0/3805429991 learned_addr learned my addr 172.21.15.28:0/3805429991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.654+0000 7fa939d9b640 1 --2- 172.21.15.28:0/3805429991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.656+0000 7fa93ad9d640 1 -- 172.21.15.28:0/3805429991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 msgr2=0x55c7c12fc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.656+0000 7fa93ad9d640 1 --2- 172.21.15.28:0/3805429991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 0x55c7c12fc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.656+0000 7fa93ad9d640 1 -- 172.21.15.28:0/3805429991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e5ef0 msgr2=0x55c7c12edf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.656+0000 7fa93ad9d640 1 --2- 172.21.15.28:0/3805429991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.656+0000 7fa93ad9d640 1 -- 172.21.15.28:0/3805429991 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7c12ff0d0 con 0x55c7c12f07b0 2024-05-09T08:44:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.657+0000 7fa93ad9d640 1 --2- 172.21.15.28:0/3805429991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c1308420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa93c050ef0 tx=0x7fa93c067ff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd6182a3a40d8573 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.662+0000 7fa923fff640 1 -- 172.21.15.28:0/3805429991 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c0a6070 con 0x55c7c12f07b0 2024-05-09T08:44:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.662+0000 7fa923fff640 1 -- 172.21.15.28:0/3805429991 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93c04fbc0 con 0x55c7c12f07b0 2024-05-09T08:44:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.662+0000 7fa923fff640 1 -- 172.21.15.28:0/3805429991 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c09e610 con 0x55c7c12f07b0 2024-05-09T08:44:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.663+0000 7fa94380f740 1 -- 172.21.15.28:0/3805429991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 msgr2=0x55c7c1308420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.663+0000 7fa94380f740 1 --2- 172.21.15.28:0/3805429991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c1308420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa93c050ef0 tx=0x7fa93c067ff0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.664+0000 7fa94380f740 1 -- 172.21.15.28:0/3805429991 shutdown_connections 2024-05-09T08:44:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.664+0000 7fa94380f740 1 --2- 172.21.15.28:0/3805429991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12f07b0 0x55c7c1308420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.664+0000 7fa94380f740 1 --2- 172.21.15.28:0/3805429991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12e5ef0 0x55c7c12edf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.664+0000 7fa94380f740 1 --2- 172.21.15.28:0/3805429991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 0x55c7c12fc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.664+0000 7fa94380f740 1 -- 172.21.15.28:0/3805429991 >> 172.21.15.28:0/3805429991 conn(0x55c7c12e1ae0 msgr2=0x55c7c12e7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.665+0000 7fa94380f740 1 -- 172.21.15.28:0/3805429991 shutdown_connections 2024-05-09T08:44:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.669+0000 7fa94380f740 1 -- 172.21.15.28:0/3805429991 wait complete. 2024-05-09T08:44:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.669+0000 7fa94380f740 1 -- start start 2024-05-09T08:44:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.669+0000 7fa94380f740 1 Processor -- start 2024-05-09T08:44:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa94380f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 0x55c7c12f89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa94380f740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12e5ef0 0x55c7c12e9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa94380f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12f07b0 0x55c7c12ea4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa94380f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c011b70 con 0x55c7c12f07b0 2024-05-09T08:44:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa94380f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c0113e0 con 0x55c7c12e55d0 2024-05-09T08:44:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa94380f740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c00e4a0 con 0x55c7c12e5ef0 2024-05-09T08:44:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa94380f740 10 client.0 _ll_register_callbacks cb 0x7fa93000a550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa93ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12f07b0 0x55c7c12ea4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa93ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12f07b0 0x55c7c12ea4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50628/0 (socket says 172.21.15.28:50628) 2024-05-09T08:44:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa93ad9d640 1 -- 172.21.15.28:0/897167356 learned_addr learned my addr 172.21.15.28:0/897167356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.670+0000 7fa93a59c640 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 0x55c7c12f89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.671+0000 7fa93a59c640 1 -- 172.21.15.28:0/897167356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12e5ef0 msgr2=0x55c7c12e9fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:44:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.671+0000 7fa93a59c640 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12e5ef0 0x55c7c12e9fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.671+0000 7fa93a59c640 1 -- 172.21.15.28:0/897167356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12f07b0 msgr2=0x55c7c12ea4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.671+0000 7fa93a59c640 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12f07b0 0x55c7c12ea4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.671+0000 7fa93a59c640 1 -- 172.21.15.28:0/897167356 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7c12eaa30 con 0x55c7c12e55d0 2024-05-09T08:44:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.671+0000 7fa93a59c640 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 0x55c7c12f89c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0cc750 tx=0x7fa92c09fed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=934c47ca9ccbc2ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.671+0000 7fa921ffb640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:44:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.672+0000 7fa921ffb640 1 -- 172.21.15.28:0/897167356 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c01d070 con 0x55c7c12e55d0 2024-05-09T08:44:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.672+0000 7fa94380f740 1 -- 172.21.15.28:0/897167356 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c7c1251dc0 con 0x55c7c12e55d0 2024-05-09T08:44:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.672+0000 7fa921ffb640 1 -- 172.21.15.28:0/897167356 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9835+0+0 (secure 0 0 0) 0x7fa92c09a840 con 0x55c7c12e55d0 2024-05-09T08:44:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.672+0000 7fa921ffb640 1 -- 172.21.15.28:0/897167356 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa92c09bd90 con 0x55c7c12e55d0 2024-05-09T08:44:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.673+0000 7fa94380f740 10 client.26810 fetch_fsmap learned FSMap version 67 2024-05-09T08:44:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.673+0000 7fa94380f740 1 -- 172.21.15.28:0/897167356 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55c7c1366a90 con 0x55c7c12e55d0 2024-05-09T08:44:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.677+0000 7fa921ffb640 1 -- 172.21.15.28:0/897167356 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c7c1251dc0 con 0x55c7c12e55d0 2024-05-09T08:44:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.678+0000 7fa921ffb640 1 -- 172.21.15.28:0/897167356 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fa92c0079b0 con 0x55c7c12e55d0 2024-05-09T08:44:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.678+0000 7fa94380f740 10 client.26810 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:44:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.678+0000 7fa94380f740 10 client.26810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:44:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.678+0000 7fa94380f740 20 client.26810 populate_metadata read hostname 'smithi028' 2024-05-09T08:44:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.678+0000 7fa94380f740 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c7c1367340 0x55c7c1369760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.678+0000 7fa94380f740 4 client.26810 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:44:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.678+0000 7fa94380f740 1 -- 172.21.15.28:0/897167356 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c7c11cf8e0 con 0x55c7c1367340 2024-05-09T08:44:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.679+0000 7fa939d9b640 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c7c1367340 0x55c7c1369760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.681+0000 7fa939d9b640 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c7c1367340 0x55c7c1369760 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f65baa501ea9f9f6 server_cookie=ff0fc8b6ce4981aa in_seq=0 out_seq=0 2024-05-09T08:44:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:38.681+0000 7fa921ffb640 10 client.26810 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa921ffb640 1 -- 172.21.15.28:0/897167356 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c7c11cf8e0 con 0x55c7c1367340 2024-05-09T08:44:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa921ffb640 10 client.26810 handle_command_reply: tid=0 2024-05-09T08:44:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa94380f740 1 client.26810 shutdown 2024-05-09T08:44:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa94380f740 1 -- 172.21.15.28:0/897167356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 msgr2=0x55c7c12f89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa94380f740 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 0x55c7c12f89c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0cc750 tx=0x7fa92c09fed0 comp rx=0 tx=0).stop 2024-05-09T08:44:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa94380f740 1 -- 172.21.15.28:0/897167356 shutdown_connections 2024-05-09T08:44:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa94380f740 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55c7c1367340 0x55c7c1369760 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa94380f740 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7c12f07b0 0x55c7c12ea4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa94380f740 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55c7c12e5ef0 0x55c7c12e9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa94380f740 1 --2- 172.21.15.28:0/897167356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c7c12e55d0 0x55c7c12f89c0 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0cc750 tx=0x7fa92c09fed0 comp rx=0 tx=0).stop 2024-05-09T08:44:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.271+0000 7fa94380f740 1 -- 172.21.15.28:0/897167356 >> 172.21.15.28:0/897167356 conn(0x55c7c12e1ae0 msgr2=0x55c7c12e0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.272+0000 7fa921ffb640 0 client.26810 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.272+0000 7fa94380f740 1 -- 172.21.15.28:0/897167356 shutdown_connections 2024-05-09T08:44:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.272+0000 7fa94380f740 1 -- 172.21.15.28:0/897167356 wait complete. 2024-05-09T08:44:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.272+0000 7fa94380f740 20 client.26810 trim_cache size 0 max 16384 2024-05-09T08:44:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:44:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:44:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:44:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:44:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 -- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9240a2f00 msgr2=0x7fa9240a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9240a2f00 0x7fa9240a53c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa93c067540 tx=0x7fa93c0a3040 comp rx=0 tx=0).stop 2024-05-09T08:44:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 -- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934004740 msgr2=0x7fa934141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934004740 0x7fa934141fa0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00c890 tx=0x7fa92c00cd50 comp rx=0 tx=0).stop 2024-05-09T08:44:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 -- 172.21.15.28:0/1633430338 shutdown_connections 2024-05-09T08:44:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa93409be10 0x7fa93413b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa9240a2f00 0x7fa9240a53c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa934005100 0x7fa9341424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 --2- 172.21.15.28:0/1633430338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa934004740 0x7fa934141fa0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.273+0000 7fa93b59e640 1 -- 172.21.15.28:0/1633430338 >> 172.21.15.28:0/1633430338 conn(0x7fa93400a740 msgr2=0x7fa9340939f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.274+0000 7fa93b59e640 1 -- 172.21.15.28:0/1633430338 shutdown_connections 2024-05-09T08:44:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:39.274+0000 7fa93b59e640 1 -- 172.21.15.28:0/1633430338 wait complete. 2024-05-09T08:44:40.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:39 smithi165 ceph-mon[34540]: pgmap v589: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-05-09T08:44:40.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:39 smithi184 ceph-mon[34509]: pgmap v589: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-05-09T08:44:40.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:39 smithi028 ceph-mon[30119]: pgmap v589: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-05-09T08:44:42.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:41 smithi165 ceph-mon[34540]: pgmap v590: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 199 op/s 2024-05-09T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:41 smithi184 ceph-mon[34509]: pgmap v590: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 199 op/s 2024-05-09T08:44:42.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:41 smithi028 ceph-mon[30119]: pgmap v590: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 199 op/s 2024-05-09T08:44:44.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:43 smithi165 ceph-mon[34540]: pgmap v591: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 931 KiB/s wr, 191 op/s 2024-05-09T08:44:44.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:43 smithi184 ceph-mon[34509]: pgmap v591: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 931 KiB/s wr, 191 op/s 2024-05-09T08:44:44.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:43 smithi028 ceph-mon[30119]: pgmap v591: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 931 KiB/s wr, 191 op/s 2024-05-09T08:44:46.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:45 smithi165 ceph-mon[34540]: pgmap v592: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 931 KiB/s wr, 185 op/s 2024-05-09T08:44:46.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:45 smithi184 ceph-mon[34509]: pgmap v592: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 931 KiB/s wr, 185 op/s 2024-05-09T08:44:46.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:45 smithi028 ceph-mon[30119]: pgmap v592: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 931 KiB/s wr, 185 op/s 2024-05-09T08:44:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:47 smithi165 ceph-mon[34540]: pgmap v593: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 191 op/s 2024-05-09T08:44:48.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:47 smithi184 ceph-mon[34509]: pgmap v593: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 191 op/s 2024-05-09T08:44:48.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:47 smithi028 ceph-mon[30119]: pgmap v593: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 191 op/s 2024-05-09T08:44:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:49 smithi165 ceph-mon[34540]: pgmap v594: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 931 KiB/s wr, 175 op/s 2024-05-09T08:44:50.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:49 smithi184 ceph-mon[34509]: pgmap v594: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 931 KiB/s wr, 175 op/s 2024-05-09T08:44:50.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:49 smithi028 ceph-mon[30119]: pgmap v594: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 931 KiB/s wr, 175 op/s 2024-05-09T08:44:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:51 smithi165 ceph-mon[34540]: pgmap v595: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 917 KiB/s wr, 183 op/s 2024-05-09T08:44:52.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:51 smithi184 ceph-mon[34509]: pgmap v595: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 917 KiB/s wr, 183 op/s 2024-05-09T08:44:52.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:51 smithi028 ceph-mon[30119]: pgmap v595: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 917 KiB/s wr, 183 op/s 2024-05-09T08:44:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.428+0000 7f2f46e5d640 1 -- 172.21.15.28:0/2225608133 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f40101840 msgr2=0x7f2f40101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.428+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/2225608133 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f40101840 0x7f2f40101cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009900 tx=0x7f2f3402a420 comp rx=0 tx=0).stop 2024-05-09T08:44:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.431+0000 7f2f46e5d640 1 -- 172.21.15.28:0/2225608133 shutdown_connections 2024-05-09T08:44:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.431+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/2225608133 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f40102200 0x7f2f4010e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.431+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/2225608133 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f40101840 0x7f2f40101cc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.431+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/2225608133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f40107850 0x7f2f40107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.431+0000 7f2f46e5d640 1 -- 172.21.15.28:0/2225608133 >> 172.21.15.28:0/2225608133 conn(0x7f2f400fd2f0 msgr2=0x7f2f400ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.432+0000 7f2f46e5d640 1 -- 172.21.15.28:0/2225608133 shutdown_connections 2024-05-09T08:44:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.432+0000 7f2f46e5d640 1 -- 172.21.15.28:0/2225608133 wait complete. 2024-05-09T08:44:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.432+0000 7f2f46e5d640 1 Processor -- start 2024-05-09T08:44:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.433+0000 7f2f46e5d640 1 -- start start 2024-05-09T08:44:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.433+0000 7f2f46e5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f40101840 0x7f2f40115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.433+0000 7f2f46e5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f40102200 0x7f2f40116320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.433+0000 7f2f46e5d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f40107850 0x7f2f40110ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.433+0000 7f2f46e5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4010fac0 con 0x7f2f40101840 2024-05-09T08:44:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.433+0000 7f2f46e5d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4010f940 con 0x7f2f40102200 2024-05-09T08:44:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.433+0000 7f2f46e5d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4010fc40 con 0x7f2f40107850 2024-05-09T08:44:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f453fa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f40107850 0x7f2f40110ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f453fa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f40107850 0x7f2f40110ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:50268/0 (socket says 172.21.15.28:50268) 2024-05-09T08:44:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f453fa640 1 -- 172.21.15.28:0/174529282 learned_addr learned my addr 172.21.15.28:0/174529282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f40102200 0x7f2f40116320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f40101840 0x7f2f40115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f3ffff640 1 -- 172.21.15.28:0/174529282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f40107850 msgr2=0x7f2f40110ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f40107850 0x7f2f40110ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f3ffff640 1 -- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f40101840 msgr2=0x7f2f40115de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f40101840 0x7f2f40115de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.435+0000 7f2f3ffff640 1 -- 172.21.15.28:0/174529282 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f40111430 con 0x7f2f40102200 2024-05-09T08:44:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.436+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f40102200 0x7f2f40116320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2f3402aa10 tx=0x7f2f34004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:52.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.441+0000 7f2f3dffb640 1 -- 172.21.15.28:0/174529282 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3403f030 con 0x7f2f40102200 2024-05-09T08:44:52.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.441+0000 7f2f46e5d640 1 -- 172.21.15.28:0/174529282 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f401116c0 con 0x7f2f40102200 2024-05-09T08:44:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.441+0000 7f2f46e5d640 1 -- 172.21.15.28:0/174529282 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f40111bb0 con 0x7f2f40102200 2024-05-09T08:44:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.441+0000 7f2f3dffb640 1 -- 172.21.15.28:0/174529282 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f34003bb0 con 0x7f2f40102200 2024-05-09T08:44:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.441+0000 7f2f3dffb640 1 -- 172.21.15.28:0/174529282 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34035070 con 0x7f2f40102200 2024-05-09T08:44:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.445+0000 7f2f49b6d740 1 Processor -- start 2024-05-09T08:44:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.445+0000 7f2f49b6d740 1 -- start start 2024-05-09T08:44:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.445+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d44450 0x559c64d44830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.445+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d44d70 0x559c64d4f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.445+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d4f630 0x559c64d51a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.445+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559c64ca9340 con 0x559c64d44d70 2024-05-09T08:44:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.445+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x559c64c51290 con 0x559c64d44450 2024-05-09T08:44:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.445+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559c64ca5300 con 0x559c64d4f630 2024-05-09T08:44:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.446+0000 7f2f44bf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d44d70 0x559c64d4f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.446+0000 7f2f44bf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d44d70 0x559c64d4f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34078/0 (socket says 172.21.15.28:34078) 2024-05-09T08:44:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.446+0000 7f2f44bf9640 1 -- 172.21.15.28:0/218018198 learned_addr learned my addr 172.21.15.28:0/218018198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.447+0000 7f2f44bf9640 1 -- 172.21.15.28:0/218018198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d4f630 msgr2=0x559c64d51a20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:44:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.447+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/218018198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d4f630 0x559c64d51a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.447+0000 7f2f44bf9640 1 -- 172.21.15.28:0/218018198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d44450 msgr2=0x559c64d44830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.447+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/218018198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d44450 0x559c64d44830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.447+0000 7f2f44bf9640 1 -- 172.21.15.28:0/218018198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c64c758e0 con 0x559c64d44d70 2024-05-09T08:44:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.448+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/218018198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d44d70 0x559c64d4f0f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00b820 tx=0x7f2f2c00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82166b8293812c34 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.450+0000 7f2f2affd640 1 -- 172.21.15.28:0/218018198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c00d950 con 0x559c64d44d70 2024-05-09T08:44:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.450+0000 7f2f2affd640 1 -- 172.21.15.28:0/218018198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f2c019070 con 0x559c64d44d70 2024-05-09T08:44:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.450+0000 7f2f2affd640 1 -- 172.21.15.28:0/218018198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0123e0 con 0x559c64d44d70 2024-05-09T08:44:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.452+0000 7f2f49b6d740 1 -- 172.21.15.28:0/218018198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d44d70 msgr2=0x559c64d4f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.452+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/218018198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d44d70 0x559c64d4f0f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00b820 tx=0x7f2f2c00bce0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.452+0000 7f2f49b6d740 1 -- 172.21.15.28:0/218018198 shutdown_connections 2024-05-09T08:44:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.452+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/218018198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d4f630 0x559c64d51a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.452+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/218018198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d44d70 0x559c64d4f0f0 secure :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00b820 tx=0x7f2f2c00bce0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.452+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/218018198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d44450 0x559c64d44830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.452+0000 7f2f49b6d740 1 -- 172.21.15.28:0/218018198 >> 172.21.15.28:0/218018198 conn(0x559c64d40960 msgr2=0x559c64d40d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:52.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.455+0000 7f2f3dffb640 1 -- 172.21.15.28:0/174529282 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2f34042030 con 0x7f2f40102200 2024-05-09T08:44:52.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.456+0000 7f2f3dffb640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f300a2fc0 0x7f2f300a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.456+0000 7f2f3dffb640 1 -- 172.21.15.28:0/174529282 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9835+0+0 (secure 0 0 0) 0x7f2f340eff30 con 0x7f2f40102200 2024-05-09T08:44:52.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.456+0000 7f2f49b6d740 1 -- 172.21.15.28:0/218018198 shutdown_connections 2024-05-09T08:44:52.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.456+0000 7f2f49b6d740 1 -- 172.21.15.28:0/218018198 wait complete. 2024-05-09T08:44:52.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.457+0000 7f2f49b6d740 1 -- start start 2024-05-09T08:44:52.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.457+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f300a2fc0 0x7f2f300a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.457+0000 7f2f49b6d740 1 Processor -- start 2024-05-09T08:44:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.457+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 0x559c64d4c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.457+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d4d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.457+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 0x559c64d5bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.458+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559c64d52260 con 0x559c64d4cd10 2024-05-09T08:44:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.458+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x559c64c64790 con 0x559c64d59720 2024-05-09T08:44:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.458+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x559c64d523e0 con 0x559c64d44450 2024-05-09T08:44:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.458+0000 7f2f49b6d740 10 client.0 _ll_register_callbacks cb 0x559c64c9fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.458+0000 7f2f44bf9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 0x559c64d4c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.458+0000 7f2f3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d4d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.458+0000 7f2f3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d4d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34088/0 (socket says 172.21.15.28:34088) 2024-05-09T08:44:52.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.458+0000 7f2f3ffff640 1 -- 172.21.15.28:0/837954851 learned_addr learned my addr 172.21.15.28:0/837954851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.459+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f300a2fc0 0x7f2f300a5480 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c014ae0 tx=0x7f2f2c014570 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.459+0000 7f2f3ffff640 1 -- 172.21.15.28:0/837954851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 msgr2=0x559c64d4c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.459+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 0x559c64d4c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.459+0000 7f2f3ffff640 1 -- 172.21.15.28:0/837954851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 msgr2=0x559c64d5bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.459+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 0x559c64d5bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.459+0000 7f2f3ffff640 1 -- 172.21.15.28:0/837954851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c649bd210 con 0x559c64d4cd10 2024-05-09T08:44:52.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.460+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d4d190 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2f34005ab0 tx=0x7f2f340bbd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c099e1d6eb0f618d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:52.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.462+0000 7f2f1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:44:52.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.463+0000 7f2f1ffff640 1 -- 172.21.15.28:0/837954851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f340d4070 con 0x559c64d4cd10 2024-05-09T08:44:52.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.463+0000 7f2f49b6d740 1 -- 172.21.15.28:0/837954851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f2c019070 con 0x559c64d4cd10 2024-05-09T08:44:52.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.463+0000 7f2f1ffff640 1 -- 172.21.15.28:0/837954851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9835+0+0 (secure 0 0 0) 0x7f2f340b8540 con 0x559c64d4cd10 2024-05-09T08:44:52.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.465+0000 7f2f1ffff640 1 -- 172.21.15.28:0/837954851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f2f34031940 con 0x559c64d4cd10 2024-05-09T08:44:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.466+0000 7f2f49b6d740 10 client.17379 fetch_fsmap learned FSMap version 67 2024-05-09T08:44:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.466+0000 7f2f49b6d740 1 -- 172.21.15.28:0/837954851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x559c64a89620 con 0x559c64d4cd10 2024-05-09T08:44:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.471+0000 7f2f1ffff640 1 -- 172.21.15.28:0/837954851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f2c019070 con 0x559c64d4cd10 2024-05-09T08:44:52.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.472+0000 7f2f1ffff640 1 -- 172.21.15.28:0/837954851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f2f34038350 con 0x559c64d4cd10 2024-05-09T08:44:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.474+0000 7f2f49b6d740 10 client.17379 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:44:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.474+0000 7f2f49b6d740 10 client.17379 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:44:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.474+0000 7f2f49b6d740 20 client.17379 populate_metadata read hostname 'smithi028' 2024-05-09T08:44:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.474+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559c64d5ffb0 0x559c64d623d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.474+0000 7f2f49b6d740 4 client.17379 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.474+0000 7f2f49b6d740 1 -- 172.21.15.28:0/837954851 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c64cb6a70 con 0x559c64d5ffb0 2024-05-09T08:44:52.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.476+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559c64d5ffb0 0x559c64d623d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.476+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559c64d5ffb0 0x559c64d623d0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9454a8eeb2ed85c8 server_cookie=feb9254e5f9fcb9c in_seq=0 out_seq=0 2024-05-09T08:44:52.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.476+0000 7f2f1ffff640 10 client.17379 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.480+0000 7f2f1ffff640 1 -- 172.21.15.28:0/837954851 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c64cb6a70 con 0x559c64d5ffb0 2024-05-09T08:44:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.480+0000 7f2f1ffff640 10 client.17379 handle_command_reply: tid=0 2024-05-09T08:44:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.480+0000 7f2f49b6d740 1 client.17379 shutdown 2024-05-09T08:44:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.484+0000 7f2f49b6d740 1 -- 172.21.15.28:0/837954851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 msgr2=0x559c64d4d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.484+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d4d190 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2f34005ab0 tx=0x7f2f340bbd60 comp rx=0 tx=0).stop 2024-05-09T08:44:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.486+0000 7f2f49b6d740 1 -- 172.21.15.28:0/837954851 shutdown_connections 2024-05-09T08:44:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.486+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559c64d5ffb0 0x559c64d623d0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.486+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 0x559c64d5bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.486+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d4d190 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.486+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/837954851 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 0x559c64d4c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.486+0000 7f2f49b6d740 1 -- 172.21.15.28:0/837954851 >> 172.21.15.28:0/837954851 conn(0x559c64d40960 msgr2=0x559c64d46760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.486+0000 7f2f1ffff640 0 client.17379 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:52.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.490+0000 7f2f49b6d740 1 -- 172.21.15.28:0/837954851 shutdown_connections 2024-05-09T08:44:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.493+0000 7f2f49b6d740 1 -- 172.21.15.28:0/837954851 wait complete. 2024-05-09T08:44:52.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.495+0000 7f2f49b6d740 20 client.17379 trim_cache size 0 max 16384 2024-05-09T08:44:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.529+0000 7f2f49b6d740 1 Processor -- start 2024-05-09T08:44:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.529+0000 7f2f49b6d740 1 -- start start 2024-05-09T08:44:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.529+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 0x559c64d67c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.530+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d681c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.530+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 0x559c64d4c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.530+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f240102a0 con 0x559c64d4cd10 2024-05-09T08:44:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.530+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2f24012020 con 0x559c64d59720 2024-05-09T08:44:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.530+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f24010a30 con 0x559c64d44450 2024-05-09T08:44:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.531+0000 7f2f453fa640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 0x559c64d4c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.531+0000 7f2f453fa640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 0x559c64d4c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:49974/0 (socket says 172.21.15.28:49974) 2024-05-09T08:44:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.531+0000 7f2f453fa640 1 -- 172.21.15.28:0/1853466333 learned_addr learned my addr 172.21.15.28:0/1853466333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.531+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/1853466333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 0x559c64d67c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.531+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/1853466333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d681c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.532+0000 7f2f3ffff640 1 -- 172.21.15.28:0/1853466333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 msgr2=0x559c64d67c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.532+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/1853466333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 0x559c64d67c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.532+0000 7f2f3ffff640 1 -- 172.21.15.28:0/1853466333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 msgr2=0x559c64d4c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.532+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/1853466333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 0x559c64d4c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.532+0000 7f2f3ffff640 1 -- 172.21.15.28:0/1853466333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c64d4b7a0 con 0x559c64d4cd10 2024-05-09T08:44:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.532+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/1853466333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d681c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2f34000c00 tx=0x7f2f340b8170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f438a2d393cb27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.533+0000 7f2f2a7fc640 1 -- 172.21.15.28:0/1853466333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f340d4040 con 0x559c64d4cd10 2024-05-09T08:44:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.533+0000 7f2f2a7fc640 1 -- 172.21.15.28:0/1853466333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f340bc5d0 con 0x559c64d4cd10 2024-05-09T08:44:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.533+0000 7f2f2a7fc640 1 -- 172.21.15.28:0/1853466333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f340b8540 con 0x559c64d4cd10 2024-05-09T08:44:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.535+0000 7f2f49b6d740 1 -- 172.21.15.28:0/1853466333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 msgr2=0x559c64d681c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.535+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/1853466333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d681c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2f34000c00 tx=0x7f2f340b8170 comp rx=0 tx=0).stop 2024-05-09T08:44:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.536+0000 7f2f49b6d740 1 -- 172.21.15.28:0/1853466333 shutdown_connections 2024-05-09T08:44:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.536+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/1853466333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d59720 0x559c64d4c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.536+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/1853466333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d681c0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.536+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/1853466333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d44450 0x559c64d67c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.536+0000 7f2f49b6d740 1 -- 172.21.15.28:0/1853466333 >> 172.21.15.28:0/1853466333 conn(0x559c64d40960 msgr2=0x559c64d40d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.541+0000 7f2f49b6d740 1 -- 172.21.15.28:0/1853466333 shutdown_connections 2024-05-09T08:44:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.541+0000 7f2f49b6d740 1 -- 172.21.15.28:0/1853466333 wait complete. 2024-05-09T08:44:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.542+0000 7f2f49b6d740 1 -- start start 2024-05-09T08:44:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.542+0000 7f2f49b6d740 1 Processor -- start 2024-05-09T08:44:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d44450 0x559c64d58df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d54660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f49b6d740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d59720 0x559c64d54ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f24011950 con 0x559c64d4cd10 2024-05-09T08:44:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2f240111c0 con 0x559c64d44450 2024-05-09T08:44:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f49b6d740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2400e330 con 0x559c64d59720 2024-05-09T08:44:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f49b6d740 10 client.0 _ll_register_callbacks cb 0x7f2f340b7a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d54660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d54660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34104/0 (socket says 172.21.15.28:34104) 2024-05-09T08:44:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f3ffff640 1 -- 172.21.15.28:0/4179186020 learned_addr learned my addr 172.21.15.28:0/4179186020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:44:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.545+0000 7f2f453fa640 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d59720 0x559c64d54ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.546+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d44450 0x559c64d58df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.546+0000 7f2f3ffff640 1 -- 172.21.15.28:0/4179186020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d59720 msgr2=0x559c64d54ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.546+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d59720 0x559c64d54ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.546+0000 7f2f3ffff640 1 -- 172.21.15.28:0/4179186020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d44450 msgr2=0x559c64d58df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.546+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d44450 0x559c64d58df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.546+0000 7f2f3ffff640 1 -- 172.21.15.28:0/4179186020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c64d550e0 con 0x559c64d4cd10 2024-05-09T08:44:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.547+0000 7f2f3ffff640 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d54660 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2f340ed4d0 tx=0x7f2f340ba3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21e4bea7d9db52e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.547+0000 7f2f1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:44:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.547+0000 7f2f1ffff640 1 -- 172.21.15.28:0/4179186020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f340bbed0 con 0x559c64d4cd10 2024-05-09T08:44:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.547+0000 7f2f49b6d740 1 -- 172.21.15.28:0/4179186020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f340bc5d0 con 0x559c64d4cd10 2024-05-09T08:44:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.547+0000 7f2f1ffff640 1 -- 172.21.15.28:0/4179186020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9835+0+0 (secure 0 0 0) 0x7f2f340bb320 con 0x559c64d4cd10 2024-05-09T08:44:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.548+0000 7f2f1ffff640 1 -- 172.21.15.28:0/4179186020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f2f340c0280 con 0x559c64d4cd10 2024-05-09T08:44:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.550+0000 7f2f49b6d740 10 client.17385 fetch_fsmap learned FSMap version 67 2024-05-09T08:44:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.550+0000 7f2f49b6d740 1 -- 172.21.15.28:0/4179186020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x559c64d550e0 con 0x559c64d4cd10 2024-05-09T08:44:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.550+0000 7f2f1ffff640 1 -- 172.21.15.28:0/4179186020 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f340bc5d0 con 0x559c64d4cd10 2024-05-09T08:44:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.551+0000 7f2f1ffff640 1 -- 172.21.15.28:0/4179186020 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f2f340cb070 con 0x559c64d4cd10 2024-05-09T08:44:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.551+0000 7f2f49b6d740 10 client.17385 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:44:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.551+0000 7f2f49b6d740 10 client.17385 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:44:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.551+0000 7f2f49b6d740 20 client.17385 populate_metadata read hostname 'smithi028' 2024-05-09T08:44:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.551+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559c64dc6650 0x559c64dc8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.551+0000 7f2f49b6d740 4 client.17385 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:44:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.551+0000 7f2f49b6d740 1 -- 172.21.15.28:0/4179186020 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559c64cb0dc0 con 0x559c64dc6650 2024-05-09T08:44:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.551+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559c64dc6650 0x559c64dc8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.552+0000 7f2f1ffff640 10 client.17385 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.552+0000 7f2f44bf9640 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559c64dc6650 0x559c64dc8a70 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b3fb9be1c791df server_cookie=7f8974dcb48aa9f3 in_seq=0 out_seq=0 2024-05-09T08:44:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f1ffff640 1 -- 172.21.15.28:0/4179186020 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559c64cb0dc0 con 0x559c64dc6650 2024-05-09T08:44:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f1ffff640 10 client.17385 handle_command_reply: tid=0 2024-05-09T08:44:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 client.17385 shutdown 2024-05-09T08:44:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 -- 172.21.15.28:0/4179186020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 msgr2=0x559c64d54660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d54660 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2f340ed4d0 tx=0x7f2f340ba3e0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 -- 172.21.15.28:0/4179186020 shutdown_connections 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x559c64dc6650 0x559c64dc8a70 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x559c64d59720 0x559c64d54ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c64d4cd10 0x559c64d54660 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 --2- 172.21.15.28:0/4179186020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c64d44450 0x559c64d58df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 -- 172.21.15.28:0/4179186020 >> 172.21.15.28:0/4179186020 conn(0x559c64d40960 msgr2=0x559c64d4c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f1ffff640 0 client.17385 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 -- 172.21.15.28:0/4179186020 shutdown_connections 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.659+0000 7f2f49b6d740 1 -- 172.21.15.28:0/4179186020 wait complete. 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.660+0000 7f2f49b6d740 20 client.17385 trim_cache size 0 max 16384 2024-05-09T08:44:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:44:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:44:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:44:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:44:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.660+0000 7f2f46e5d640 1 -- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f300a2fc0 msgr2=0x7f2f300a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.660+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f300a2fc0 0x7f2f300a5480 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c014ae0 tx=0x7f2f2c014570 comp rx=0 tx=0).stop 2024-05-09T08:44:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.660+0000 7f2f46e5d640 1 -- 172.21.15.28:0/174529282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f40102200 msgr2=0x7f2f40116320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.660+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f40102200 0x7f2f40116320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2f3402aa10 tx=0x7f2f34004610 comp rx=0 tx=0).stop 2024-05-09T08:44:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.661+0000 7f2f46e5d640 1 -- 172.21.15.28:0/174529282 shutdown_connections 2024-05-09T08:44:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.661+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2f300a2fc0 0x7f2f300a5480 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.661+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f40107850 0x7f2f40110ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.661+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f40102200 0x7f2f40116320 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.661+0000 7f2f46e5d640 1 --2- 172.21.15.28:0/174529282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f40101840 0x7f2f40115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.661+0000 7f2f46e5d640 1 -- 172.21.15.28:0/174529282 >> 172.21.15.28:0/174529282 conn(0x7f2f400fd2f0 msgr2=0x7f2f400fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.661+0000 7f2f46e5d640 1 -- 172.21.15.28:0/174529282 shutdown_connections 2024-05-09T08:44:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:44:52.661+0000 7f2f46e5d640 1 -- 172.21.15.28:0/174529282 wait complete. 2024-05-09T08:44:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:53 smithi165 ceph-mon[34540]: pgmap v596: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-05-09T08:44:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:53 smithi184 ceph-mon[34509]: pgmap v596: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-05-09T08:44:54.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:53 smithi028 ceph-mon[30119]: pgmap v596: 129 pgs: 69 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 38 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-05-09T08:44:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:55 smithi165 ceph-mon[34540]: pgmap v597: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 853 KiB/s wr, 166 op/s 2024-05-09T08:44:56.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:55 smithi165 ceph-mon[34540]: osdmap e131: 12 total, 12 up, 12 in 2024-05-09T08:44:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:55 smithi184 ceph-mon[34509]: pgmap v597: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 853 KiB/s wr, 166 op/s 2024-05-09T08:44:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:55 smithi184 ceph-mon[34509]: osdmap e131: 12 total, 12 up, 12 in 2024-05-09T08:44:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:55 smithi028 ceph-mon[30119]: pgmap v597: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 853 KiB/s wr, 166 op/s 2024-05-09T08:44:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:55 smithi028 ceph-mon[30119]: osdmap e131: 12 total, 12 up, 12 in 2024-05-09T08:44:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:57 smithi165 ceph-mon[34540]: pgmap v599: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 177 op/s 2024-05-09T08:44:58.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:57 smithi184 ceph-mon[34509]: pgmap v599: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 177 op/s 2024-05-09T08:44:58.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:57 smithi028 ceph-mon[30119]: pgmap v599: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 177 op/s 2024-05-09T08:45:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:44:59 smithi165 ceph-mon[34540]: pgmap v600: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 884 KiB/s wr, 179 op/s 2024-05-09T08:45:00.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:44:59 smithi184 ceph-mon[34509]: pgmap v600: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 884 KiB/s wr, 179 op/s 2024-05-09T08:45:00.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:44:59 smithi028 ceph-mon[30119]: pgmap v600: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 884 KiB/s wr, 179 op/s 2024-05-09T08:45:02.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:01 smithi165 ceph-mon[34540]: pgmap v601: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 203 op/s 2024-05-09T08:45:02.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:01 smithi184 ceph-mon[34509]: pgmap v601: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 203 op/s 2024-05-09T08:45:02.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:01 smithi028 ceph-mon[30119]: pgmap v601: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 203 op/s 2024-05-09T08:45:04.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:03 smithi165 ceph-mon[34540]: pgmap v602: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 252 op/s 2024-05-09T08:45:04.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:03 smithi184 ceph-mon[34509]: pgmap v602: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 252 op/s 2024-05-09T08:45:04.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:03 smithi028 ceph-mon[30119]: pgmap v602: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 252 op/s 2024-05-09T08:45:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.823+0000 7fa1f3215640 1 -- 172.21.15.28:0/10142666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec110020 msgr2=0x7fa1ec108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.823+0000 7fa1f3215640 1 --2- 172.21.15.28:0/10142666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec110020 0x7fa1ec108830 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa1e401ace0 tx=0x7fa1e403ba00 comp rx=0 tx=0).stop 2024-05-09T08:45:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.827+0000 7fa1f3215640 1 -- 172.21.15.28:0/10142666 shutdown_connections 2024-05-09T08:45:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.827+0000 7fa1f3215640 1 --2- 172.21.15.28:0/10142666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec11b010 0x7fa1ec11d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.827+0000 7fa1f3215640 1 --2- 172.21.15.28:0/10142666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ec110400 0x7fa1ec108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.827+0000 7fa1f3215640 1 --2- 172.21.15.28:0/10142666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec110020 0x7fa1ec108830 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.827+0000 7fa1f3215640 1 -- 172.21.15.28:0/10142666 >> 172.21.15.28:0/10142666 conn(0x7fa1ec06c900 msgr2=0x7fa1ec06cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.835+0000 7fa1f3215640 1 -- 172.21.15.28:0/10142666 shutdown_connections 2024-05-09T08:45:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.836+0000 7fa1f3215640 1 -- 172.21.15.28:0/10142666 wait complete. 2024-05-09T08:45:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.837+0000 7fa1f3215640 1 Processor -- start 2024-05-09T08:45:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.843+0000 7fa1f3215640 1 -- start start 2024-05-09T08:45:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.843+0000 7fa1f3215640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec110020 0x7fa1ec11a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.843+0000 7fa1f3215640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec11b010 0x7fa1ec1168a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.843+0000 7fa1f3215640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ec116de0 0x7fa1ec117290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.843+0000 7fa1f3215640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec113250 con 0x7fa1ec11b010 2024-05-09T08:45:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.843+0000 7fa1f3215640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec1130d0 con 0x7fa1ec110020 2024-05-09T08:45:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.843+0000 7fa1f3215640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec1133d0 con 0x7fa1ec116de0 2024-05-09T08:45:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.847+0000 7fa1ebfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec11b010 0x7fa1ec1168a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.847+0000 7fa1ebfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec11b010 0x7fa1ec1168a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37370/0 (socket says 172.21.15.28:37370) 2024-05-09T08:45:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.847+0000 7fa1ebfff640 1 -- 172.21.15.28:0/3410155545 learned_addr learned my addr 172.21.15.28:0/3410155545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.848+0000 7fa1f0fb1640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec110020 0x7fa1ec11a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.848+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ec116de0 0x7fa1ec117290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.851+0000 7fa1ebfff640 1 -- 172.21.15.28:0/3410155545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ec116de0 msgr2=0x7fa1ec117290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.851+0000 7fa1ebfff640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ec116de0 0x7fa1ec117290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.851+0000 7fa1ebfff640 1 -- 172.21.15.28:0/3410155545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec110020 msgr2=0x7fa1ec11a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.851+0000 7fa1ebfff640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec110020 0x7fa1ec11a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.851+0000 7fa1ebfff640 1 -- 172.21.15.28:0/3410155545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec1aae80 con 0x7fa1ec11b010 2024-05-09T08:45:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.854+0000 7fa1ebfff640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec11b010 0x7fa1ec1168a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000d7a0 tx=0x7fa1e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.860+0000 7fa1e9ffb640 1 -- 172.21.15.28:0/3410155545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0017070 con 0x7fa1ec11b010 2024-05-09T08:45:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.860+0000 7fa1e9ffb640 1 -- 172.21.15.28:0/3410155545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1e0009c30 con 0x7fa1ec11b010 2024-05-09T08:45:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.860+0000 7fa1e9ffb640 1 -- 172.21.15.28:0/3410155545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e000f620 con 0x7fa1ec11b010 2024-05-09T08:45:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.861+0000 7fa1f3215640 1 -- 172.21.15.28:0/3410155545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ec1ab170 con 0x7fa1ec11b010 2024-05-09T08:45:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.862+0000 7fa1f3215640 1 -- 172.21.15.28:0/3410155545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ec1ab670 con 0x7fa1ec11b010 2024-05-09T08:45:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.871+0000 7fa1e9ffb640 1 -- 172.21.15.28:0/3410155545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa1e001a070 con 0x7fa1ec11b010 2024-05-09T08:45:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.871+0000 7fa1f5fff740 1 Processor -- start 2024-05-09T08:45:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.873+0000 7fa1f5fff740 1 -- start start 2024-05-09T08:45:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.873+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c82830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.873+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c82d70 0x556142c8d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.873+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 0x556142c8fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.873+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556142be7340 con 0x556142c82d70 2024-05-09T08:45:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.873+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x556142b8f290 con 0x556142c8d630 2024-05-09T08:45:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.873+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556142be3300 con 0x556142c82450 2024-05-09T08:45:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.874+0000 7fa1ebfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 0x556142c8fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.874+0000 7fa1ebfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 0x556142c8fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:46764/0 (socket says 172.21.15.28:46764) 2024-05-09T08:45:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.874+0000 7fa1ebfff640 1 -- 172.21.15.28:0/1693732471 learned_addr learned my addr 172.21.15.28:0/1693732471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.874+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/1693732471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c82830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.874+0000 7fa1f17b2640 1 -- 172.21.15.28:0/1693732471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 msgr2=0x556142c8fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.874+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/1693732471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 0x556142c8fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.874+0000 7fa1f17b2640 1 -- 172.21.15.28:0/1693732471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c82d70 msgr2=0x556142c8d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.874+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/1693732471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c82d70 0x556142c8d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.874+0000 7fa1f17b2640 1 -- 172.21.15.28:0/1693732471 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556142bb38e0 con 0x556142c82450 2024-05-09T08:45:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.875+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/1693732471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c82830 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec10dad0 tx=0x7fa1dc00bb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9c427b193072024 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.875+0000 7fa1e9ffb640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1d40a2fd0 0x7fa1d40a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.876+0000 7fa1e9ffb640 1 -- 172.21.15.28:0/3410155545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9819+0+0 (secure 0 0 0) 0x7fa1e00cf040 con 0x7fa1ec11b010 2024-05-09T08:45:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.878+0000 7fa1d2ffd640 1 -- 172.21.15.28:0/1693732471 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc00d740 con 0x556142c82450 2024-05-09T08:45:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.878+0000 7fa1d2ffd640 1 -- 172.21.15.28:0/1693732471 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1dc019070 con 0x556142c82450 2024-05-09T08:45:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.878+0000 7fa1d2ffd640 1 -- 172.21.15.28:0/1693732471 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0123e0 con 0x556142c82450 2024-05-09T08:45:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.880+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1693732471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 msgr2=0x556142c82830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.880+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1693732471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c82830 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec10dad0 tx=0x7fa1dc00bb50 comp rx=0 tx=0).stop 2024-05-09T08:45:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.881+0000 7fa1f0fb1640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1d40a2fd0 0x7fa1d40a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.881+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1693732471 shutdown_connections 2024-05-09T08:45:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.881+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1693732471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 0x556142c8fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.881+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1693732471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c82d70 0x556142c8d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.881+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1693732471 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c82830 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.881+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1693732471 >> 172.21.15.28:0/1693732471 conn(0x556142c7e960 msgr2=0x556142c7ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.881+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1693732471 shutdown_connections 2024-05-09T08:45:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.885+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1693732471 wait complete. 2024-05-09T08:45:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.885+0000 7fa1f5fff740 1 -- start start 2024-05-09T08:45:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.885+0000 7fa1f5fff740 1 Processor -- start 2024-05-09T08:45:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.886+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c8a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.886+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c82d70 0x556142c8aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.886+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c8b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.886+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556142c90260 con 0x556142c8d630 2024-05-09T08:45:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.886+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x556142ba2790 con 0x556142c82d70 2024-05-09T08:45:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.886+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x556142c903e0 con 0x556142c82450 2024-05-09T08:45:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.886+0000 7fa1f5fff740 10 client.0 _ll_register_callbacks cb 0x556142bdda20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.887+0000 7fa1f17b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c8b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.887+0000 7fa1f17b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c8b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37374/0 (socket says 172.21.15.28:37374) 2024-05-09T08:45:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.887+0000 7fa1f17b2640 1 -- 172.21.15.28:0/3616616221 learned_addr learned my addr 172.21.15.28:0/3616616221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.887+0000 7fa1ebfff640 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c82d70 0x556142c8aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.888+0000 7fa1f17b2640 1 -- 172.21.15.28:0/3616616221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 msgr2=0x556142c8a590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.888+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c8a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.888+0000 7fa1f17b2640 1 -- 172.21.15.28:0/3616616221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c82d70 msgr2=0x556142c8aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.888+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c82d70 0x556142c8aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.888+0000 7fa1f17b2640 1 -- 172.21.15.28:0/3616616221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561428fb210 con 0x556142c8d630 2024-05-09T08:45:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.888+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c8b010 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc000c00 tx=0x7fa1dc014660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140b25c1ff8b864a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.888+0000 7fa1cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:45:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.889+0000 7fa1cbfff640 1 -- 172.21.15.28:0/3616616221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc021020 con 0x556142c8d630 2024-05-09T08:45:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.889+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3616616221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1dc019070 con 0x556142c8d630 2024-05-09T08:45:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.889+0000 7fa1cbfff640 1 -- 172.21.15.28:0/3616616221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9819+0+0 (secure 0 0 0) 0x7fa1dc00db50 con 0x556142c8d630 2024-05-09T08:45:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.889+0000 7fa1cbfff640 1 -- 172.21.15.28:0/3616616221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa1dc02c050 con 0x556142c8d630 2024-05-09T08:45:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.889+0000 7fa1f0fb1640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1d40a2fd0 0x7fa1d40a5490 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4000f80 tx=0x7fa1e401a8a0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.889+0000 7fa1f5fff740 10 client.17394 fetch_fsmap learned FSMap version 67 2024-05-09T08:45:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.889+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3616616221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x5561429c7620 con 0x556142c8d630 2024-05-09T08:45:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.891+0000 7fa1cbfff640 1 -- 172.21.15.28:0/3616616221 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1dc019070 con 0x556142c8d630 2024-05-09T08:45:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.891+0000 7fa1cbfff640 1 -- 172.21.15.28:0/3616616221 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fa1dc013070 con 0x556142c8d630 2024-05-09T08:45:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.891+0000 7fa1f5fff740 10 client.17394 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:45:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.891+0000 7fa1f5fff740 10 client.17394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:45:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.891+0000 7fa1f5fff740 20 client.17394 populate_metadata read hostname 'smithi028' 2024-05-09T08:45:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.891+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556142c9d7b0 0x556142c9fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.891+0000 7fa1f5fff740 4 client.17394 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.891+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3616616221 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556142bf4a70 con 0x556142c9d7b0 2024-05-09T08:45:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.893+0000 7fa1f0fb1640 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556142c9d7b0 0x556142c9fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.897+0000 7fa1cbfff640 10 client.17394 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.897+0000 7fa1f0fb1640 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556142c9d7b0 0x556142c9fc70 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de00aba4840a3f42 server_cookie=f22cd5d2726450e5 in_seq=0 out_seq=0 2024-05-09T08:45:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.899+0000 7fa1cbfff640 1 -- 172.21.15.28:0/3616616221 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556142bf4a70 con 0x556142c9d7b0 2024-05-09T08:45:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.899+0000 7fa1cbfff640 10 client.17394 handle_command_reply: tid=0 2024-05-09T08:45:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.899+0000 7fa1f5fff740 1 client.17394 shutdown 2024-05-09T08:45:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.900+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3616616221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 msgr2=0x556142c8b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.900+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c8b010 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc000c00 tx=0x7fa1dc014660 comp rx=0 tx=0).stop 2024-05-09T08:45:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.903+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3616616221 shutdown_connections 2024-05-09T08:45:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.903+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556142c9d7b0 0x556142c9fc70 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.903+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c8b010 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.903+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c82d70 0x556142c8aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.903+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3616616221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c8a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.903+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3616616221 >> 172.21.15.28:0/3616616221 conn(0x556142c7e960 msgr2=0x556142c846e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.904+0000 7fa1cbfff640 0 client.17394 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.910+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3616616221 shutdown_connections 2024-05-09T08:45:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.912+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3616616221 wait complete. 2024-05-09T08:45:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.914+0000 7fa1f5fff740 20 client.17394 trim_cache size 0 max 16384 2024-05-09T08:45:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.946+0000 7fa1f5fff740 1 Processor -- start 2024-05-09T08:45:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.946+0000 7fa1f5fff740 1 -- start start 2024-05-09T08:45:05.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.946+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c8ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.946+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c82d70 0x556142c99150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.947+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c7d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.947+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d80102c0 con 0x556142c8d630 2024-05-09T08:45:05.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.947+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d8012040 con 0x556142c82d70 2024-05-09T08:45:05.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.947+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d8010a50 con 0x556142c82450 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.948+0000 7fa1f17b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c7d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.948+0000 7fa1f17b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c7d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37382/0 (socket says 172.21.15.28:37382) 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.948+0000 7fa1f17b2640 1 -- 172.21.15.28:0/3463525959 learned_addr learned my addr 172.21.15.28:0/3463525959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.948+0000 7fa1f0fb1640 1 --2- 172.21.15.28:0/3463525959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c8ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.948+0000 7fa1f17b2640 1 -- 172.21.15.28:0/3463525959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 msgr2=0x556142c8ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.949+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/3463525959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c8ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.949+0000 7fa1f17b2640 1 -- 172.21.15.28:0/3463525959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c82d70 msgr2=0x556142c99150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.949+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/3463525959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c82d70 0x556142c99150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.949+0000 7fa1f17b2640 1 -- 172.21.15.28:0/3463525959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556142c8fd90 con 0x556142c8d630 2024-05-09T08:45:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.949+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/3463525959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c7d880 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00bb90 tx=0x7fa1dc01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53cfa3fbfaf0b769 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.949+0000 7fa1d27fc640 1 -- 172.21.15.28:0/3463525959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc021040 con 0x556142c8d630 2024-05-09T08:45:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.949+0000 7fa1d27fc640 1 -- 172.21.15.28:0/3463525959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1dc019040 con 0x556142c8d630 2024-05-09T08:45:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.950+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3463525959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 msgr2=0x556142c7d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.950+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3463525959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c7d880 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00bb90 tx=0x7fa1dc01c9c0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.951+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3463525959 shutdown_connections 2024-05-09T08:45:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.951+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3463525959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c8d630 0x556142c7d880 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.951+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3463525959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c82d70 0x556142c99150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.951+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/3463525959 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c8ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.951+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3463525959 >> 172.21.15.28:0/3463525959 conn(0x556142c7e960 msgr2=0x556142c7d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:05.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.957+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3463525959 shutdown_connections 2024-05-09T08:45:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.961+0000 7fa1f5fff740 1 -- 172.21.15.28:0/3463525959 wait complete. 2024-05-09T08:45:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.961+0000 7fa1f5fff740 1 -- start start 2024-05-09T08:45:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.962+0000 7fa1f5fff740 1 Processor -- start 2024-05-09T08:45:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.962+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c87240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.962+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 0x556142c87780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.962+0000 7fa1f5fff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c87cc0 0x556142d02df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.962+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d8011970 con 0x556142c87cc0 2024-05-09T08:45:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.962+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d80111e0 con 0x556142c8d630 2024-05-09T08:45:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.962+0000 7fa1f5fff740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d800e2a0 con 0x556142c82450 2024-05-09T08:45:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.962+0000 7fa1f5fff740 10 client.0 _ll_register_callbacks cb 0x7fa1e00d2940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:05.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.963+0000 7fa1f17b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c87cc0 0x556142d02df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.963+0000 7fa1f17b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c87cc0 0x556142d02df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37388/0 (socket says 172.21.15.28:37388) 2024-05-09T08:45:05.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.963+0000 7fa1f17b2640 1 -- 172.21.15.28:0/1294034857 learned_addr learned my addr 172.21.15.28:0/1294034857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:05.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.968+0000 7fa1ebfff640 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 0x556142c87780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.969+0000 7fa1f0fb1640 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c87240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.970+0000 7fa1f17b2640 1 -- 172.21.15.28:0/1294034857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 msgr2=0x556142c87240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.970+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c87240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.970+0000 7fa1f17b2640 1 -- 172.21.15.28:0/1294034857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 msgr2=0x556142c87780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.970+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 0x556142c87780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.970+0000 7fa1f17b2640 1 -- 172.21.15.28:0/1294034857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556142d03330 con 0x556142c87cc0 2024-05-09T08:45:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.970+0000 7fa1f17b2640 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c87cc0 0x556142d02df0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc008000 tx=0x7fa1dc00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2c534a276ff61c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.971+0000 7fa1cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:45:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.971+0000 7fa1cbfff640 1 -- 172.21.15.28:0/1294034857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc021040 con 0x556142c87cc0 2024-05-09T08:45:05.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.971+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1294034857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1dc019040 con 0x556142c87cc0 2024-05-09T08:45:05.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.971+0000 7fa1cbfff640 1 -- 172.21.15.28:0/1294034857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9819+0+0 (secure 0 0 0) 0x7fa1dc014660 con 0x556142c87cc0 2024-05-09T08:45:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.973+0000 7fa1cbfff640 1 -- 172.21.15.28:0/1294034857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa1dc014860 con 0x556142c87cc0 2024-05-09T08:45:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.973+0000 7fa1f5fff740 10 client.17400 fetch_fsmap learned FSMap version 67 2024-05-09T08:45:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.973+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1294034857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x556142d04460 con 0x556142c87cc0 2024-05-09T08:45:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.974+0000 7fa1cbfff640 1 -- 172.21.15.28:0/1294034857 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1dc019040 con 0x556142c87cc0 2024-05-09T08:45:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.974+0000 7fa1cbfff640 1 -- 172.21.15.28:0/1294034857 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fa1dc006020 con 0x556142c87cc0 2024-05-09T08:45:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.974+0000 7fa1f5fff740 10 client.17400 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:45:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.974+0000 7fa1f5fff740 10 client.17400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:45:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.974+0000 7fa1f5fff740 20 client.17400 populate_metadata read hostname 'smithi028' 2024-05-09T08:45:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.974+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556142d04b30 0x556142d06f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.974+0000 7fa1f5fff740 4 client.17400 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:45:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.974+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1294034857 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556142beedc0 con 0x556142d04b30 2024-05-09T08:45:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.975+0000 7fa1f0fb1640 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556142d04b30 0x556142d06f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.976+0000 7fa1f0fb1640 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556142d04b30 0x556142d06f50 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fd0f628005ffea0 server_cookie=3c13879a5c0a0315 in_seq=0 out_seq=0 2024-05-09T08:45:05.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:05.978+0000 7fa1cbfff640 10 client.17400 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:06.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:05 smithi165 ceph-mon[34540]: pgmap v603: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 260 op/s 2024-05-09T08:45:06.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:05 smithi184 ceph-mon[34509]: pgmap v603: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 260 op/s 2024-05-09T08:45:06.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:05 smithi028 ceph-mon[30119]: pgmap v603: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 260 op/s 2024-05-09T08:45:06.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1cbfff640 1 -- 172.21.15.28:0/1294034857 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556142beedc0 con 0x556142d04b30 2024-05-09T08:45:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1cbfff640 10 client.17400 handle_command_reply: tid=0 2024-05-09T08:45:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1f5fff740 1 client.17400 shutdown 2024-05-09T08:45:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1294034857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c87cc0 msgr2=0x556142d02df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c87cc0 0x556142d02df0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc008000 tx=0x7fa1dc00db80 comp rx=0 tx=0).stop 2024-05-09T08:45:06.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1294034857 shutdown_connections 2024-05-09T08:45:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x556142d04b30 0x556142d06f50 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556142c87cc0 0x556142d02df0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556142c8d630 0x556142c87780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1f5fff740 1 --2- 172.21.15.28:0/1294034857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x556142c82450 0x556142c87240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.229+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1294034857 >> 172.21.15.28:0/1294034857 conn(0x556142c7e960 msgr2=0x556142c7da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.230+0000 7fa1cbfff640 0 client.17400 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.230+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1294034857 shutdown_connections 2024-05-09T08:45:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.230+0000 7fa1f5fff740 1 -- 172.21.15.28:0/1294034857 wait complete. 2024-05-09T08:45:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.230+0000 7fa1f5fff740 20 client.17400 trim_cache size 0 max 16384 2024-05-09T08:45:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:45:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:45:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:45:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:45:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 -- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1d40a2fd0 msgr2=0x7fa1d40a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1d40a2fd0 0x7fa1d40a5490 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4000f80 tx=0x7fa1e401a8a0 comp rx=0 tx=0).stop 2024-05-09T08:45:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 -- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec11b010 msgr2=0x7fa1ec1168a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec11b010 0x7fa1ec1168a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000d7a0 tx=0x7fa1e000dc60 comp rx=0 tx=0).stop 2024-05-09T08:45:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 -- 172.21.15.28:0/3410155545 shutdown_connections 2024-05-09T08:45:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa1d40a2fd0 0x7fa1d40a5490 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa1ec116de0 0x7fa1ec117290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1ec11b010 0x7fa1ec1168a0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 --2- 172.21.15.28:0/3410155545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec110020 0x7fa1ec11a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 -- 172.21.15.28:0/3410155545 >> 172.21.15.28:0/3410155545 conn(0x7fa1ec06c900 msgr2=0x7fa1ec11b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 -- 172.21.15.28:0/3410155545 shutdown_connections 2024-05-09T08:45:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:06.231+0000 7fa1f3215640 1 -- 172.21.15.28:0/3410155545 wait complete. 2024-05-09T08:45:08.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:07 smithi165 ceph-mon[34540]: pgmap v604: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 278 op/s 2024-05-09T08:45:08.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:07 smithi184 ceph-mon[34509]: pgmap v604: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 278 op/s 2024-05-09T08:45:08.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:07 smithi028 ceph-mon[30119]: pgmap v604: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 278 op/s 2024-05-09T08:45:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:09 smithi028 ceph-mon[30119]: pgmap v605: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 250 op/s 2024-05-09T08:45:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:09 smithi028 ceph-mon[30119]: osdmap e132: 12 total, 12 up, 12 in 2024-05-09T08:45:10.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:09 smithi165 ceph-mon[34540]: pgmap v605: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 250 op/s 2024-05-09T08:45:10.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:09 smithi165 ceph-mon[34540]: osdmap e132: 12 total, 12 up, 12 in 2024-05-09T08:45:10.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:09 smithi184 ceph-mon[34509]: pgmap v605: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 250 op/s 2024-05-09T08:45:10.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:09 smithi184 ceph-mon[34509]: osdmap e132: 12 total, 12 up, 12 in 2024-05-09T08:45:12.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:11 smithi028 ceph-mon[30119]: pgmap v607: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 1.0 MiB/s wr, 264 op/s 2024-05-09T08:45:12.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:11 smithi165 ceph-mon[34540]: pgmap v607: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 1.0 MiB/s wr, 264 op/s 2024-05-09T08:45:12.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:11 smithi184 ceph-mon[34509]: pgmap v607: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 1.0 MiB/s wr, 264 op/s 2024-05-09T08:45:14.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:13 smithi028 ceph-mon[30119]: pgmap v608: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 245 op/s 2024-05-09T08:45:14.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:13 smithi165 ceph-mon[34540]: pgmap v608: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 245 op/s 2024-05-09T08:45:14.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:13 smithi184 ceph-mon[34509]: pgmap v608: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 245 op/s 2024-05-09T08:45:16.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:15 smithi028 ceph-mon[30119]: pgmap v609: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 301 op/s 2024-05-09T08:45:16.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:15 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:16.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:15 smithi165 ceph-mon[34540]: pgmap v609: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 301 op/s 2024-05-09T08:45:16.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:15 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:16.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:15 smithi184 ceph-mon[34509]: pgmap v609: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 301 op/s 2024-05-09T08:45:16.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:15 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:17.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:16 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:17.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:16 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:17.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:16 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:45:17.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:16 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:17.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:16 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:17.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:16 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:45:17.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:16 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:17.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:16 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:17.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:16 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:45:18.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:17 smithi028 ceph-mon[30119]: pgmap v610: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 282 op/s 2024-05-09T08:45:18.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:17 smithi165 ceph-mon[34540]: pgmap v610: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 282 op/s 2024-05-09T08:45:18.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:17 smithi184 ceph-mon[34509]: pgmap v610: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 282 op/s 2024-05-09T08:45:19.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:18 smithi028 ceph-mon[30119]: pgmap v611: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 281 op/s 2024-05-09T08:45:19.183 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:45:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:45:18.776+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:45:19.183 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:45:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:45:18.776+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:45:19.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:18 smithi165 ceph-mon[34540]: pgmap v611: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 281 op/s 2024-05-09T08:45:19.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:18 smithi184 ceph-mon[34509]: pgmap v611: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 281 op/s 2024-05-09T08:45:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.401+0000 7ff2170df640 1 -- 172.21.15.28:0/1242105956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff210110020 msgr2=0x7ff210108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.401+0000 7ff2170df640 1 --2- 172.21.15.28:0/1242105956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff210110020 0x7ff210108830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff20801a940 tx=0x7ff20803b520 comp rx=0 tx=0).stop 2024-05-09T08:45:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.406+0000 7ff2170df640 1 -- 172.21.15.28:0/1242105956 shutdown_connections 2024-05-09T08:45:19.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.406+0000 7ff2170df640 1 --2- 172.21.15.28:0/1242105956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff21011b010 0x7ff21011d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.406+0000 7ff2170df640 1 --2- 172.21.15.28:0/1242105956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff210110400 0x7ff210108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.406+0000 7ff2170df640 1 --2- 172.21.15.28:0/1242105956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff210110020 0x7ff210108830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.406+0000 7ff2170df640 1 -- 172.21.15.28:0/1242105956 >> 172.21.15.28:0/1242105956 conn(0x7ff21006c900 msgr2=0x7ff21006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.413+0000 7ff2170df640 1 -- 172.21.15.28:0/1242105956 shutdown_connections 2024-05-09T08:45:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.417+0000 7ff2170df640 1 -- 172.21.15.28:0/1242105956 wait complete. 2024-05-09T08:45:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.417+0000 7ff2170df640 1 Processor -- start 2024-05-09T08:45:19.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.419+0000 7ff2170df640 1 -- start start 2024-05-09T08:45:19.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.419+0000 7ff2170df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff210110020 0x7ff21011a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.419+0000 7ff2170df640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21011b010 0x7ff2101167e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.419+0000 7ff2170df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff210116d20 0x7ff2101171d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.419+0000 7ff2170df640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff210113250 con 0x7ff210110020 2024-05-09T08:45:19.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.419+0000 7ff2170df640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101130d0 con 0x7ff21011b010 2024-05-09T08:45:19.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.419+0000 7ff2170df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff2101133d0 con 0x7ff210116d20 2024-05-09T08:45:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.422+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21011b010 0x7ff2101167e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.424+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21011b010 0x7ff2101167e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60356/0 (socket says 172.21.15.28:60356) 2024-05-09T08:45:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.424+0000 7ff20ffff640 1 -- 172.21.15.28:0/3805536933 learned_addr learned my addr 172.21.15.28:0/3805536933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.424+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff210110020 0x7ff21011a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.425+0000 7ff214e7b640 1 -- 172.21.15.28:0/3805536933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff210116d20 msgr2=0x7ff2101171d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:45:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.425+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff210116d20 0x7ff2101171d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.425+0000 7ff214e7b640 1 -- 172.21.15.28:0/3805536933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21011b010 msgr2=0x7ff2101167e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.425+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21011b010 0x7ff2101167e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.425+0000 7ff214e7b640 1 -- 172.21.15.28:0/3805536933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2101aae00 con 0x7ff210110020 2024-05-09T08:45:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.426+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff210110020 0x7ff21011a750 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff20801a860 tx=0x7ff208005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.432+0000 7ff20dffb640 1 -- 172.21.15.28:0/3805536933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20804f070 con 0x7ff210110020 2024-05-09T08:45:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.432+0000 7ff2170df640 1 -- 172.21.15.28:0/3805536933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2101ab090 con 0x7ff210110020 2024-05-09T08:45:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.432+0000 7ff2170df640 1 -- 172.21.15.28:0/3805536933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2101ab520 con 0x7ff210110020 2024-05-09T08:45:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.432+0000 7ff20dffb640 1 -- 172.21.15.28:0/3805536933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff208004020 con 0x7ff210110020 2024-05-09T08:45:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.432+0000 7ff20dffb640 1 -- 172.21.15.28:0/3805536933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff208046070 con 0x7ff210110020 2024-05-09T08:45:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.440+0000 7ff20dffb640 1 -- 172.21.15.28:0/3805536933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff2080405e0 con 0x7ff210110020 2024-05-09T08:45:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.440+0000 7ff219ddf740 1 Processor -- start 2024-05-09T08:45:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.444+0000 7ff219ddf740 1 -- start start 2024-05-09T08:45:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4450 0x55f0e54d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4d70 0x55f0e54df0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 0x55f0e54e1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff219ddf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0e5439340 con 0x55f0e54d4450 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff219ddf740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f0e53e1290 con 0x55f0e54d4d70 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff219ddf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f0e5435300 con 0x55f0e54df630 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff20dffb640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff1fc0a2fc0 0x7ff1fc0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff214e7b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4d70 0x55f0e54df0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff214e7b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4d70 0x55f0e54df0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60370/0 (socket says 172.21.15.28:60370) 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.446+0000 7ff214e7b640 1 -- 172.21.15.28:0/3945530209 learned_addr learned my addr 172.21.15.28:0/3945530209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.447+0000 7ff214e7b640 1 -- 172.21.15.28:0/3945530209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 msgr2=0x55f0e54e1a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.447+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3945530209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 0x55f0e54e1a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.447+0000 7ff214e7b640 1 -- 172.21.15.28:0/3945530209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4450 msgr2=0x55f0e54d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.447+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3945530209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4450 0x55f0e54d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.447+0000 7ff214e7b640 1 -- 172.21.15.28:0/3945530209 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0e54058e0 con 0x55f0e54d4d70 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.447+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3945530209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4d70 0x55f0e54df0f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff2080fdd80 tx=0x7ff2080977e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=591cad39eef022c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.447+0000 7ff20dffb640 1 -- 172.21.15.28:0/3805536933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9879+0+0 (secure 0 0 0) 0x7ff208100c70 con 0x7ff210110020 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.447+0000 7ff206ffd640 1 -- 172.21.15.28:0/3945530209 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff208104050 con 0x55f0e54d4d70 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.447+0000 7ff206ffd640 1 -- 172.21.15.28:0/3945530209 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff20801a540 con 0x55f0e54d4d70 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.448+0000 7ff219ddf740 1 -- 172.21.15.28:0/3945530209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4d70 msgr2=0x55f0e54df0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.448+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3945530209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4d70 0x55f0e54df0f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff2080fdd80 tx=0x7ff2080977e0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.452+0000 7ff219ddf740 1 -- 172.21.15.28:0/3945530209 shutdown_connections 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.452+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3945530209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 0x55f0e54e1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.452+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3945530209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4d70 0x55f0e54df0f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.452+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3945530209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4450 0x55f0e54d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.452+0000 7ff219ddf740 1 -- 172.21.15.28:0/3945530209 >> 172.21.15.28:0/3945530209 conn(0x55f0e54d0960 msgr2=0x55f0e54d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.458+0000 7ff219ddf740 1 -- 172.21.15.28:0/3945530209 shutdown_connections 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.458+0000 7ff21567c640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff1fc0a2fc0 0x7ff1fc0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.462+0000 7ff21567c640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff1fc0a2fc0 0x7ff1fc0a5480 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff21010dad0 tx=0x7ff200001550 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.464+0000 7ff219ddf740 1 -- 172.21.15.28:0/3945530209 wait complete. 2024-05-09T08:45:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.464+0000 7ff219ddf740 1 -- start start 2024-05-09T08:45:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.465+0000 7ff219ddf740 1 Processor -- start 2024-05-09T08:45:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.465+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4450 0x55f0e54dc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.465+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 0x55f0e54dca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.465+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54dcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.465+0000 7ff219ddf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0e54e2020 con 0x55f0e54d4d70 2024-05-09T08:45:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.465+0000 7ff219ddf740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55f0e53f4790 con 0x55f0e54df630 2024-05-09T08:45:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.465+0000 7ff219ddf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f0e54e21a0 con 0x55f0e54d4450 2024-05-09T08:45:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.465+0000 7ff219ddf740 10 client.0 _ll_register_callbacks cb 0x55f0e542fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.466+0000 7ff21567c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54dcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.466+0000 7ff21567c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54dcf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:60372/0 (socket says 172.21.15.28:60372) 2024-05-09T08:45:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.466+0000 7ff21567c640 1 -- 172.21.15.28:0/3199800552 learned_addr learned my addr 172.21.15.28:0/3199800552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.469+0000 7ff20ffff640 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 0x55f0e54dca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.469+0000 7ff21567c640 1 -- 172.21.15.28:0/3199800552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4450 msgr2=0x55f0e54dc4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.469+0000 7ff21567c640 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4450 0x55f0e54dc4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.469+0000 7ff21567c640 1 -- 172.21.15.28:0/3199800552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 msgr2=0x55f0e54dca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.469+0000 7ff21567c640 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 0x55f0e54dca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.469+0000 7ff21567c640 1 -- 172.21.15.28:0/3199800552 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0e514d210 con 0x55f0e54df630 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.469+0000 7ff21567c640 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54dcf50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff200001c90 tx=0x7ff200006be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4f6ce391ae18e4a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.469+0000 7ff1effff640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.470+0000 7ff1effff640 1 -- 172.21.15.28:0/3199800552 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200012650 con 0x55f0e54df630 2024-05-09T08:45:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.470+0000 7ff219ddf740 1 -- 172.21.15.28:0/3199800552 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f0e5446a70 con 0x55f0e54df630 2024-05-09T08:45:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.470+0000 7ff1effff640 1 -- 172.21.15.28:0/3199800552 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9879+0+0 (secure 0 0 0) 0x7ff2000127f0 con 0x55f0e54df630 2024-05-09T08:45:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.470+0000 7ff1effff640 1 -- 172.21.15.28:0/3199800552 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff20002a050 con 0x55f0e54df630 2024-05-09T08:45:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.475+0000 7ff219ddf740 10 client.26843 fetch_fsmap learned FSMap version 67 2024-05-09T08:45:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.475+0000 7ff219ddf740 1 -- 172.21.15.28:0/3199800552 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55f0e5219620 con 0x55f0e54df630 2024-05-09T08:45:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.476+0000 7ff1effff640 1 -- 172.21.15.28:0/3199800552 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f0e5446a70 con 0x55f0e54df630 2024-05-09T08:45:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.477+0000 7ff1effff640 1 -- 172.21.15.28:0/3199800552 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7ff200021a20 con 0x55f0e54df630 2024-05-09T08:45:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.477+0000 7ff219ddf740 10 client.26843 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:45:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.477+0000 7ff219ddf740 10 client.26843 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:45:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.477+0000 7ff219ddf740 20 client.26843 populate_metadata read hostname 'smithi028' 2024-05-09T08:45:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.477+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f0e54ef620 0x55f0e54f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.477+0000 7ff219ddf740 4 client.26843 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.477+0000 7ff219ddf740 1 -- 172.21.15.28:0/3199800552 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0e5440dc0 con 0x55f0e54ef620 2024-05-09T08:45:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.481+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f0e54ef620 0x55f0e54f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.492+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f0e54ef620 0x55f0e54f1a80 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c85bb535d8401ff8 server_cookie=7937c78d153e9a57 in_seq=0 out_seq=0 2024-05-09T08:45:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.494+0000 7ff1effff640 10 client.26843 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.494+0000 7ff1effff640 1 -- 172.21.15.28:0/3199800552 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0e5440dc0 con 0x55f0e54ef620 2024-05-09T08:45:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.495+0000 7ff1effff640 10 client.26843 handle_command_reply: tid=0 2024-05-09T08:45:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.495+0000 7ff219ddf740 1 client.26843 shutdown 2024-05-09T08:45:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.502+0000 7ff219ddf740 1 -- 172.21.15.28:0/3199800552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 msgr2=0x55f0e54dcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.502+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54dcf50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff200001c90 tx=0x7ff200006be0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.505+0000 7ff219ddf740 1 -- 172.21.15.28:0/3199800552 shutdown_connections 2024-05-09T08:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.505+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f0e54ef620 0x55f0e54f1a80 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.505+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54dcf50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.505+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 0x55f0e54dca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.505+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3199800552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4450 0x55f0e54dc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.505+0000 7ff219ddf740 1 -- 172.21.15.28:0/3199800552 >> 172.21.15.28:0/3199800552 conn(0x55f0e54d0960 msgr2=0x55f0e54d66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.506+0000 7ff1effff640 0 client.26843 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.510+0000 7ff219ddf740 1 -- 172.21.15.28:0/3199800552 shutdown_connections 2024-05-09T08:45:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.512+0000 7ff219ddf740 1 -- 172.21.15.28:0/3199800552 wait complete. 2024-05-09T08:45:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.513+0000 7ff219ddf740 20 client.26843 trim_cache size 0 max 16384 2024-05-09T08:45:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.550+0000 7ff219ddf740 1 Processor -- start 2024-05-09T08:45:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.551+0000 7ff219ddf740 1 -- start start 2024-05-09T08:45:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.551+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4450 0x55f0e54dec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.551+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4d70 0x55f0e54db630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.551+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.552+0000 7ff219ddf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0032430 con 0x55f0e54d4450 2024-05-09T08:45:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.552+0000 7ff219ddf740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f00341b0 con 0x55f0e54df630 2024-05-09T08:45:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.552+0000 7ff219ddf740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0032bc0 con 0x55f0e54d4d70 2024-05-09T08:45:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.552+0000 7ff21567c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.552+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4d70 0x55f0e54db630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.552+0000 7ff20ffff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4d70 0x55f0e54db630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:39808/0 (socket says 172.21.15.28:39808) 2024-05-09T08:45:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.552+0000 7ff20ffff640 1 -- 172.21.15.28:0/1690046455 learned_addr learned my addr 172.21.15.28:0/1690046455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.554+0000 7ff21567c640 1 -- 172.21.15.28:0/1690046455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4d70 msgr2=0x55f0e54db630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.554+0000 7ff21567c640 1 --2- 172.21.15.28:0/1690046455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4d70 0x55f0e54db630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.554+0000 7ff21567c640 1 -- 172.21.15.28:0/1690046455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4450 msgr2=0x55f0e54dec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.554+0000 7ff21567c640 1 --2- 172.21.15.28:0/1690046455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4450 0x55f0e54dec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.554+0000 7ff21567c640 1 -- 172.21.15.28:0/1690046455 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0e54daa20 con 0x55f0e54df630 2024-05-09T08:45:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.555+0000 7ff21567c640 1 --2- 172.21.15.28:0/1690046455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54cf880 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff20000c520 tx=0x7ff200006b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b57ebfa0d2b58ee6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.561+0000 7ff2067fc640 1 -- 172.21.15.28:0/1690046455 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200014480 con 0x55f0e54df630 2024-05-09T08:45:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.561+0000 7ff2067fc640 1 -- 172.21.15.28:0/1690046455 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff200014e00 con 0x55f0e54df630 2024-05-09T08:45:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.561+0000 7ff2067fc640 1 -- 172.21.15.28:0/1690046455 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200021b70 con 0x55f0e54df630 2024-05-09T08:45:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.562+0000 7ff219ddf740 1 -- 172.21.15.28:0/1690046455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 msgr2=0x55f0e54cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.562+0000 7ff219ddf740 1 --2- 172.21.15.28:0/1690046455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54cf880 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff20000c520 tx=0x7ff200006b70 comp rx=0 tx=0).stop 2024-05-09T08:45:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.563+0000 7ff219ddf740 1 -- 172.21.15.28:0/1690046455 shutdown_connections 2024-05-09T08:45:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.563+0000 7ff219ddf740 1 --2- 172.21.15.28:0/1690046455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54df630 0x55f0e54cf880 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.563+0000 7ff219ddf740 1 --2- 172.21.15.28:0/1690046455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54d4d70 0x55f0e54db630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.563+0000 7ff219ddf740 1 --2- 172.21.15.28:0/1690046455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4450 0x55f0e54dec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.563+0000 7ff219ddf740 1 -- 172.21.15.28:0/1690046455 >> 172.21.15.28:0/1690046455 conn(0x55f0e54d0960 msgr2=0x55f0e54cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:19.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.564+0000 7ff219ddf740 1 -- 172.21.15.28:0/1690046455 shutdown_connections 2024-05-09T08:45:19.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.567+0000 7ff219ddf740 1 -- 172.21.15.28:0/1690046455 wait complete. 2024-05-09T08:45:19.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.567+0000 7ff219ddf740 1 -- start start 2024-05-09T08:45:19.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.568+0000 7ff219ddf740 1 Processor -- start 2024-05-09T08:45:19.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.568+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4450 0x55f0e54e1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.568+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 0x55f0e54e2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.569+0000 7ff219ddf740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 0x55f0e54e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.569+0000 7ff219ddf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0033ae0 con 0x55f0e54d4d70 2024-05-09T08:45:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.569+0000 7ff219ddf740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0033350 con 0x55f0e54d4450 2024-05-09T08:45:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.569+0000 7ff21567c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 0x55f0e54e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.569+0000 7ff21567c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 0x55f0e54e2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:39814/0 (socket says 172.21.15.28:39814) 2024-05-09T08:45:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.569+0000 7ff21567c640 1 -- 172.21.15.28:0/3481536554 learned_addr learned my addr 172.21.15.28:0/3481536554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.569+0000 7ff219ddf740 1 -- 172.21.15.28:0/3481536554 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0030430 con 0x55f0e54df630 2024-05-09T08:45:19.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.569+0000 7ff219ddf740 10 client.0 _ll_register_callbacks cb 0x7ff200004b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:19.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.570+0000 7ff20ffff640 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 0x55f0e54e2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.570+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4450 0x55f0e54e1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.571+0000 7ff21567c640 1 -- 172.21.15.28:0/3481536554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4450 msgr2=0x55f0e54e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:19.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.571+0000 7ff21567c640 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4450 0x55f0e54e1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.571+0000 7ff21567c640 1 -- 172.21.15.28:0/3481536554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 msgr2=0x55f0e54e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:19.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.571+0000 7ff21567c640 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 0x55f0e54e2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.571+0000 7ff21567c640 1 -- 172.21.15.28:0/3481536554 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0e5553da0 con 0x55f0e54df630 2024-05-09T08:45:19.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.575+0000 7ff21567c640 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 0x55f0e54e2a40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff200000f80 tx=0x7ff200004f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebd3acf57cb56380 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:19.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.575+0000 7ff1effff640 10 client.0 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:45:19.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.577+0000 7ff1effff640 1 -- 172.21.15.28:0/3481536554 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200014690 con 0x55f0e54df630 2024-05-09T08:45:19.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.577+0000 7ff219ddf740 1 -- 172.21.15.28:0/3481536554 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff200014e00 con 0x55f0e54df630 2024-05-09T08:45:19.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.577+0000 7ff1effff640 1 -- 172.21.15.28:0/3481536554 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9879+0+0 (secure 0 0 0) 0x7ff200034070 con 0x55f0e54df630 2024-05-09T08:45:19.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.578+0000 7ff1effff640 1 -- 172.21.15.28:0/3481536554 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff2000218f0 con 0x55f0e54df630 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.579+0000 7ff219ddf740 10 client.26683 fetch_fsmap learned FSMap version 67 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.579+0000 7ff219ddf740 1 -- 172.21.15.28:0/3481536554 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55f0e5555090 con 0x55f0e54df630 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.581+0000 7ff1effff640 1 -- 172.21.15.28:0/3481536554 <== mon.1 v2:172.21.15.184:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff200014e00 con 0x55f0e54df630 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.581+0000 7ff1effff640 1 -- 172.21.15.28:0/3481536554 <== mon.1 v2:172.21.15.184:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7ff200012bb0 con 0x55f0e54df630 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.581+0000 7ff219ddf740 10 client.26683 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.581+0000 7ff219ddf740 10 client.26683 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.581+0000 7ff219ddf740 20 client.26683 populate_metadata read hostname 'smithi028' 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.581+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f0e5555700 0x55f0e5557b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.581+0000 7ff219ddf740 4 client.26683 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:45:19.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.581+0000 7ff219ddf740 1 -- 172.21.15.28:0/3481536554 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f0e53be8e0 con 0x55f0e5555700 2024-05-09T08:45:19.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.582+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f0e5555700 0x55f0e5557b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:19.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.584+0000 7ff214e7b640 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f0e5555700 0x55f0e5557b20 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7b5b62936c10031 server_cookie=8d88d9baf0fbfdc7 in_seq=0 out_seq=0 2024-05-09T08:45:19.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.585+0000 7ff1effff640 10 client.26683 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.706+0000 7ff1effff640 1 -- 172.21.15.28:0/3481536554 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f0e53be8e0 con 0x55f0e5555700 2024-05-09T08:45:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff1effff640 10 client.26683 handle_command_reply: tid=0 2024-05-09T08:45:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 client.26683 shutdown 2024-05-09T08:45:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 -- 172.21.15.28:0/3481536554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 msgr2=0x55f0e54e2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:19.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 0x55f0e54e2a40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff200000f80 tx=0x7ff200004f40 comp rx=0 tx=0).stop 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 -- 172.21.15.28:0/3481536554 shutdown_connections 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55f0e5555700 0x55f0e5557b20 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f0e54df630 0x55f0e54e2a40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0e54d4d70 0x55f0e54e2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 --2- 172.21.15.28:0/3481536554 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0e54d4450 0x55f0e54e1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 -- 172.21.15.28:0/3481536554 >> 172.21.15.28:0/3481536554 conn(0x55f0e54d0960 msgr2=0x55f0e54cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff1effff640 0 client.26683 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 -- 172.21.15.28:0/3481536554 shutdown_connections 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.707+0000 7ff219ddf740 1 -- 172.21.15.28:0/3481536554 wait complete. 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.708+0000 7ff219ddf740 20 client.26683 trim_cache size 0 max 16384 2024-05-09T08:45:19.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:45:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:45:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:45:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:45:19.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.708+0000 7ff2170df640 1 -- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff1fc0a2fc0 msgr2=0x7ff1fc0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:19.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff1fc0a2fc0 0x7ff1fc0a5480 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff21010dad0 tx=0x7ff200001550 comp rx=0 tx=0).stop 2024-05-09T08:45:19.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 -- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff210110020 msgr2=0x7ff21011a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:19.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff210110020 0x7ff21011a750 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff20801a860 tx=0x7ff208005720 comp rx=0 tx=0).stop 2024-05-09T08:45:19.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 -- 172.21.15.28:0/3805536933 shutdown_connections 2024-05-09T08:45:19.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7ff1fc0a2fc0 0x7ff1fc0a5480 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff210116d20 0x7ff2101171d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21011b010 0x7ff2101167e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 --2- 172.21.15.28:0/3805536933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff210110020 0x7ff21011a750 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:19.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 -- 172.21.15.28:0/3805536933 >> 172.21.15.28:0/3805536933 conn(0x7ff21006c900 msgr2=0x7ff21011b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:19.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 -- 172.21.15.28:0/3805536933 shutdown_connections 2024-05-09T08:45:19.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:19.709+0000 7ff2170df640 1 -- 172.21.15.28:0/3805536933 wait complete. 2024-05-09T08:45:21.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:21 smithi165 ceph-mon[34540]: pgmap v612: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-05-09T08:45:21.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:21 smithi184 ceph-mon[34509]: pgmap v612: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-05-09T08:45:21.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:21 smithi028 ceph-mon[30119]: pgmap v612: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 267 op/s 2024-05-09T08:45:23.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:23 smithi165 ceph-mon[34540]: pgmap v613: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 265 op/s 2024-05-09T08:45:23.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:23 smithi184 ceph-mon[34509]: pgmap v613: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 265 op/s 2024-05-09T08:45:23.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:23 smithi028 ceph-mon[30119]: pgmap v613: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 265 op/s 2024-05-09T08:45:25.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:25 smithi165 ceph-mon[34540]: pgmap v614: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 247 op/s 2024-05-09T08:45:25.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:25 smithi184 ceph-mon[34509]: pgmap v614: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 247 op/s 2024-05-09T08:45:25.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:25 smithi028 ceph-mon[30119]: pgmap v614: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 247 op/s 2024-05-09T08:45:27.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:26 smithi165 ceph-mon[34540]: osdmap e133: 12 total, 12 up, 12 in 2024-05-09T08:45:27.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:26 smithi184 ceph-mon[34509]: osdmap e133: 12 total, 12 up, 12 in 2024-05-09T08:45:27.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:26 smithi028 ceph-mon[30119]: osdmap e133: 12 total, 12 up, 12 in 2024-05-09T08:45:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:27 smithi165 ceph-mon[34540]: pgmap v616: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 203 op/s 2024-05-09T08:45:28.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:27 smithi184 ceph-mon[34509]: pgmap v616: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 203 op/s 2024-05-09T08:45:28.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:27 smithi028 ceph-mon[30119]: pgmap v616: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 203 op/s 2024-05-09T08:45:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:29 smithi165 ceph-mon[34540]: pgmap v617: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 204 op/s 2024-05-09T08:45:30.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:29 smithi184 ceph-mon[34509]: pgmap v617: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 204 op/s 2024-05-09T08:45:30.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:29 smithi028 ceph-mon[30119]: pgmap v617: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 204 op/s 2024-05-09T08:45:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:31 smithi165 ceph-mon[34540]: pgmap v618: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 206 op/s 2024-05-09T08:45:32.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:31 smithi184 ceph-mon[34509]: pgmap v618: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 206 op/s 2024-05-09T08:45:32.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:31 smithi028 ceph-mon[30119]: pgmap v618: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 206 op/s 2024-05-09T08:45:32.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.868+0000 7fa65591b640 1 -- 172.21.15.28:0/3256175279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 msgr2=0x7fa650114ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:32.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.868+0000 7fa65591b640 1 --2- 172.21.15.28:0/3256175279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 0x7fa650114ce0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa648009ca0 tx=0x7fa64802a980 comp rx=0 tx=0).stop 2024-05-09T08:45:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.871+0000 7fa65591b640 1 -- 172.21.15.28:0/3256175279 shutdown_connections 2024-05-09T08:45:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.871+0000 7fa65591b640 1 --2- 172.21.15.28:0/3256175279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 0x7fa650114ce0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.871+0000 7fa65591b640 1 --2- 172.21.15.28:0/3256175279 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa650074450 0x7fa6501123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.871+0000 7fa65591b640 1 --2- 172.21.15.28:0/3256175279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa650073b30 0x7fa650073f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.871+0000 7fa65591b640 1 -- 172.21.15.28:0/3256175279 >> 172.21.15.28:0/3256175279 conn(0x7fa650078cc0 msgr2=0x7fa6500790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.877+0000 7fa65591b640 1 -- 172.21.15.28:0/3256175279 shutdown_connections 2024-05-09T08:45:32.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.882+0000 7fa65591b640 1 -- 172.21.15.28:0/3256175279 wait complete. 2024-05-09T08:45:32.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.882+0000 7fa65591b640 1 Processor -- start 2024-05-09T08:45:32.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.883+0000 7fa65591b640 1 -- start start 2024-05-09T08:45:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.883+0000 7fa65591b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa650073b30 0x7fa6500734f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.883+0000 7fa65591b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa650074450 0x7fa650070590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.884+0000 7fa65591b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 0x7fa650070b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.884+0000 7fa65591b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa650118340 con 0x7fa6501128f0 2024-05-09T08:45:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.884+0000 7fa65591b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa6501181c0 con 0x7fa650073b30 2024-05-09T08:45:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.884+0000 7fa65591b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6501184c0 con 0x7fa650074450 2024-05-09T08:45:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.886+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 0x7fa650070b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.886+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 0x7fa650070b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37560/0 (socket says 172.21.15.28:37560) 2024-05-09T08:45:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.886+0000 7fa64f7fe640 1 -- 172.21.15.28:0/1123262715 learned_addr learned my addr 172.21.15.28:0/1123262715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.889+0000 7fa64f7fe640 1 -- 172.21.15.28:0/1123262715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa650074450 msgr2=0x7fa650070590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:45:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.889+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa650074450 0x7fa650070590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.889+0000 7fa64f7fe640 1 -- 172.21.15.28:0/1123262715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa650073b30 msgr2=0x7fa6500734f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:45:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.889+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa650073b30 0x7fa6500734f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.889+0000 7fa64f7fe640 1 -- 172.21.15.28:0/1123262715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65010ebe0 con 0x7fa6501128f0 2024-05-09T08:45:32.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.894+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 0x7fa650070b60 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fa6480308b0 tx=0x7fa6480308e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.897+0000 7fa654919640 1 -- 172.21.15.28:0/1123262715 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648047070 con 0x7fa6501128f0 2024-05-09T08:45:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.897+0000 7fa65591b640 1 -- 172.21.15.28:0/1123262715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65010ee70 con 0x7fa6501128f0 2024-05-09T08:45:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.897+0000 7fa65591b640 1 -- 172.21.15.28:0/1123262715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa65010f360 con 0x7fa6501128f0 2024-05-09T08:45:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.897+0000 7fa654919640 1 -- 172.21.15.28:0/1123262715 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa648004040 con 0x7fa6501128f0 2024-05-09T08:45:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.897+0000 7fa654919640 1 -- 172.21.15.28:0/1123262715 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648034070 con 0x7fa6501128f0 2024-05-09T08:45:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.901+0000 7fa65862b740 1 Processor -- start 2024-05-09T08:45:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.902+0000 7fa654919640 1 -- 172.21.15.28:0/1123262715 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa64804a070 con 0x7fa6501128f0 2024-05-09T08:45:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.903+0000 7fa654919640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa63c0a2f50 0x7fa63c0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.904+0000 7fa654919640 1 -- 172.21.15.28:0/1123262715 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9803+0+0 (secure 0 0 0) 0x7fa6480f0450 con 0x7fa6501128f0 2024-05-09T08:45:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.904+0000 7fa65862b740 1 -- start start 2024-05-09T08:45:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.905+0000 7fa64e7fc640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa63c0a2f50 0x7fa63c0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.905+0000 7fa65862b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ec450 0x55bfef8ec830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.905+0000 7fa65862b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ecd70 0x55bfef8f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.905+0000 7fa65862b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.905+0000 7fa65862b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bfef851340 con 0x55bfef8ec450 2024-05-09T08:45:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.905+0000 7fa65862b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55bfef7f9290 con 0x55bfef8f7630 2024-05-09T08:45:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.906+0000 7fa65862b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bfef84d300 con 0x55bfef8ecd70 2024-05-09T08:45:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.909+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.909+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:45536/0 (socket says 172.21.15.28:45536) 2024-05-09T08:45:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.909+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3113171176 learned_addr learned my addr 172.21.15.28:0/3113171176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.909+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3113171176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ecd70 msgr2=0x55bfef8f70f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:45:32.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.910+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/3113171176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ecd70 0x55bfef8f70f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.910+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3113171176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ec450 msgr2=0x55bfef8ec830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:32.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.910+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/3113171176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ec450 0x55bfef8ec830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.910+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3113171176 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfef81d8e0 con 0x55bfef8f7630 2024-05-09T08:45:32.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.912+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/3113171176 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f9a20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa648030f50 tx=0x7fa6480bc370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f67926e83df207aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:32.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.913+0000 7fa645ffb640 1 -- 172.21.15.28:0/3113171176 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480c50e0 con 0x55bfef8f7630 2024-05-09T08:45:32.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.913+0000 7fa645ffb640 1 -- 172.21.15.28:0/3113171176 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6480ba510 con 0x55bfef8f7630 2024-05-09T08:45:32.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.913+0000 7fa645ffb640 1 -- 172.21.15.28:0/3113171176 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480b8ae0 con 0x55bfef8f7630 2024-05-09T08:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.913+0000 7fa64e7fc640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa63c0a2f50 0x7fa63c0a5410 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa650071d50 tx=0x7fa638006bd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.914+0000 7fa65862b740 1 -- 172.21.15.28:0/3113171176 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 msgr2=0x55bfef8f9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.914+0000 7fa65862b740 1 --2- 172.21.15.28:0/3113171176 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f9a20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa648030f50 tx=0x7fa6480bc370 comp rx=0 tx=0).stop 2024-05-09T08:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.915+0000 7fa65862b740 1 -- 172.21.15.28:0/3113171176 shutdown_connections 2024-05-09T08:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.915+0000 7fa65862b740 1 --2- 172.21.15.28:0/3113171176 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f9a20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.915+0000 7fa65862b740 1 --2- 172.21.15.28:0/3113171176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ecd70 0x55bfef8f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.915+0000 7fa65862b740 1 --2- 172.21.15.28:0/3113171176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ec450 0x55bfef8ec830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.915+0000 7fa65862b740 1 -- 172.21.15.28:0/3113171176 >> 172.21.15.28:0/3113171176 conn(0x55bfef8e8960 msgr2=0x55bfef8e8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:32.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.922+0000 7fa65862b740 1 -- 172.21.15.28:0/3113171176 shutdown_connections 2024-05-09T08:45:32.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.922+0000 7fa65862b740 1 -- 172.21.15.28:0/3113171176 wait complete. 2024-05-09T08:45:32.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.923+0000 7fa65862b740 1 -- start start 2024-05-09T08:45:32.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.923+0000 7fa65862b740 1 Processor -- start 2024-05-09T08:45:32.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa65862b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef8f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa65862b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa65862b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa65862b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bfef7f9290 con 0x55bfef8ecd70 2024-05-09T08:45:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa65862b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55bfef80c790 con 0x55bfef8f7630 2024-05-09T08:45:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa65862b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55bfef8fa020 con 0x55bfef8ec450 2024-05-09T08:45:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa65862b740 10 client.0 _ll_register_callbacks cb 0x55bfef847a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa64e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa64e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37582/0 (socket says 172.21.15.28:37582) 2024-05-09T08:45:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa64e7fc640 1 -- 172.21.15.28:0/334343157 learned_addr learned my addr 172.21.15.28:0/334343157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.924+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:32.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.925+0000 7fa64e7fc640 1 -- 172.21.15.28:0/334343157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 msgr2=0x55bfef8f45b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:32.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.925+0000 7fa64e7fc640 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef8f45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.925+0000 7fa64e7fc640 1 -- 172.21.15.28:0/334343157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 msgr2=0x55bfef8f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:32.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.925+0000 7fa64e7fc640 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.925+0000 7fa64e7fc640 1 -- 172.21.15.28:0/334343157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfef565210 con 0x55bfef8ecd70 2024-05-09T08:45:32.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.928+0000 7fa64e7fc640 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4af0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa638013f70 tx=0x7fa638004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d38c16d3d63b440 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.928+0000 7fa62effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.929+0000 7fa62effd640 1 -- 172.21.15.28:0/334343157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6380125a0 con 0x55bfef8ecd70 2024-05-09T08:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.929+0000 7fa62effd640 1 -- 172.21.15.28:0/334343157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9803+0+0 (secure 0 0 0) 0x7fa638012740 con 0x55bfef8ecd70 2024-05-09T08:45:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.929+0000 7fa65862b740 1 -- 172.21.15.28:0/334343157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6480ba510 con 0x55bfef8ecd70 2024-05-09T08:45:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.929+0000 7fa62effd640 1 -- 172.21.15.28:0/334343157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa638012980 con 0x55bfef8ecd70 2024-05-09T08:45:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.932+0000 7fa65862b740 10 client.17421 fetch_fsmap learned FSMap version 67 2024-05-09T08:45:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.932+0000 7fa65862b740 1 -- 172.21.15.28:0/334343157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55bfef631620 con 0x55bfef8ecd70 2024-05-09T08:45:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.933+0000 7fa62effd640 1 -- 172.21.15.28:0/334343157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6480ba510 con 0x55bfef8ecd70 2024-05-09T08:45:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.933+0000 7fa62effd640 1 -- 172.21.15.28:0/334343157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fa638012070 con 0x55bfef8ecd70 2024-05-09T08:45:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.933+0000 7fa65862b740 10 client.17421 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:45:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.933+0000 7fa65862b740 10 client.17421 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:45:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.933+0000 7fa65862b740 20 client.17421 populate_metadata read hostname 'smithi028' 2024-05-09T08:45:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.933+0000 7fa65862b740 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bfef9075b0 0x55bfef909a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.933+0000 7fa65862b740 4 client.17421 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.933+0000 7fa65862b740 1 -- 172.21.15.28:0/334343157 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfef85ea70 con 0x55bfef9075b0 2024-05-09T08:45:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.936+0000 7fa64effd640 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bfef9075b0 0x55bfef909a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.944+0000 7fa64effd640 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bfef9075b0 0x55bfef909a30 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c42cee39b2e7a2d6 server_cookie=603ed373f85feb8 in_seq=0 out_seq=0 2024-05-09T08:45:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.945+0000 7fa62effd640 10 client.17421 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.948+0000 7fa62effd640 1 -- 172.21.15.28:0/334343157 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfef85ea70 con 0x55bfef9075b0 2024-05-09T08:45:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.949+0000 7fa62effd640 10 client.17421 handle_command_reply: tid=0 2024-05-09T08:45:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.949+0000 7fa65862b740 1 client.17421 shutdown 2024-05-09T08:45:32.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.952+0000 7fa65862b740 1 -- 172.21.15.28:0/334343157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 msgr2=0x55bfef8f4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:32.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.952+0000 7fa65862b740 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4af0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa638013f70 tx=0x7fa638004700 comp rx=0 tx=0).stop 2024-05-09T08:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.957+0000 7fa65862b740 1 -- 172.21.15.28:0/334343157 shutdown_connections 2024-05-09T08:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.957+0000 7fa65862b740 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bfef9075b0 0x55bfef909a30 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.957+0000 7fa65862b740 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.957+0000 7fa65862b740 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4af0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.957+0000 7fa65862b740 1 --2- 172.21.15.28:0/334343157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef8f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.957+0000 7fa65862b740 1 -- 172.21.15.28:0/334343157 >> 172.21.15.28:0/334343157 conn(0x55bfef8e8960 msgr2=0x55bfef8ee6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.958+0000 7fa62effd640 0 client.17421 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.962+0000 7fa65862b740 1 -- 172.21.15.28:0/334343157 shutdown_connections 2024-05-09T08:45:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.966+0000 7fa65862b740 1 -- 172.21.15.28:0/334343157 wait complete. 2024-05-09T08:45:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:32.970+0000 7fa65862b740 20 client.17421 trim_cache size 0 max 16384 2024-05-09T08:45:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.003+0000 7fa65862b740 1 Processor -- start 2024-05-09T08:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.004+0000 7fa65862b740 1 -- start start 2024-05-09T08:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.004+0000 7fa65862b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef902b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.004+0000 7fa65862b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.004+0000 7fa65862b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef90f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.004+0000 7fa65862b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa630010390 con 0x55bfef8ecd70 2024-05-09T08:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.004+0000 7fa65862b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa630012110 con 0x55bfef8f7630 2024-05-09T08:45:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.004+0000 7fa65862b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa630010b20 con 0x55bfef8ec450 2024-05-09T08:45:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.004+0000 7fa64effd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef902b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.005+0000 7fa64e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.005+0000 7fa64e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37592/0 (socket says 172.21.15.28:37592) 2024-05-09T08:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.005+0000 7fa64e7fc640 1 -- 172.21.15.28:0/2221697503 learned_addr learned my addr 172.21.15.28:0/2221697503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.005+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/2221697503 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef90f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.005+0000 7fa64e7fc640 1 -- 172.21.15.28:0/2221697503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 msgr2=0x55bfef902b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.005+0000 7fa64e7fc640 1 --2- 172.21.15.28:0/2221697503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef902b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.005+0000 7fa64e7fc640 1 -- 172.21.15.28:0/2221697503 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 msgr2=0x55bfef90f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.005+0000 7fa64e7fc640 1 --2- 172.21.15.28:0/2221697503 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef90f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.005+0000 7fa64e7fc640 1 -- 172.21.15.28:0/2221697503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfef8f9d90 con 0x55bfef8ecd70 2024-05-09T08:45:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.006+0000 7fa64e7fc640 1 --2- 172.21.15.28:0/2221697503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4e70 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa638013f40 tx=0x7fa638009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50de8eab438fff6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.007+0000 7fa6457fa640 1 -- 172.21.15.28:0/2221697503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638010530 con 0x55bfef8ecd70 2024-05-09T08:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.007+0000 7fa6457fa640 1 -- 172.21.15.28:0/2221697503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa638022070 con 0x55bfef8ecd70 2024-05-09T08:45:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.007+0000 7fa6457fa640 1 -- 172.21.15.28:0/2221697503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638005bf0 con 0x55bfef8ecd70 2024-05-09T08:45:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.010+0000 7fa65862b740 1 -- 172.21.15.28:0/2221697503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 msgr2=0x55bfef8f4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.010+0000 7fa65862b740 1 --2- 172.21.15.28:0/2221697503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4e70 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa638013f40 tx=0x7fa638009510 comp rx=0 tx=0).stop 2024-05-09T08:45:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.016+0000 7fa65862b740 1 -- 172.21.15.28:0/2221697503 shutdown_connections 2024-05-09T08:45:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.016+0000 7fa65862b740 1 --2- 172.21.15.28:0/2221697503 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef90f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.016+0000 7fa65862b740 1 --2- 172.21.15.28:0/2221697503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f4e70 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.016+0000 7fa65862b740 1 --2- 172.21.15.28:0/2221697503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef902b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.016+0000 7fa65862b740 1 -- 172.21.15.28:0/2221697503 >> 172.21.15.28:0/2221697503 conn(0x55bfef8e8960 msgr2=0x55bfef8e7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.020+0000 7fa65862b740 1 -- 172.21.15.28:0/2221697503 shutdown_connections 2024-05-09T08:45:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.022+0000 7fa65862b740 1 -- 172.21.15.28:0/2221697503 wait complete. 2024-05-09T08:45:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.022+0000 7fa65862b740 1 -- start start 2024-05-09T08:45:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.022+0000 7fa65862b740 1 Processor -- start 2024-05-09T08:45:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.028+0000 7fa65862b740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef8ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.028+0000 7fa65862b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.028+0000 7fa65862b740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.028+0000 7fa65862b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa630011a40 con 0x55bfef8ecd70 2024-05-09T08:45:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.028+0000 7fa65862b740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa6300112b0 con 0x55bfef8f7630 2024-05-09T08:45:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.028+0000 7fa65862b740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa63000e370 con 0x55bfef8ec450 2024-05-09T08:45:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.028+0000 7fa65862b740 10 client.0 _ll_register_callbacks cb 0x7fa638013430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.030+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.030+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:45566/0 (socket says 172.21.15.28:45566) 2024-05-09T08:45:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.030+0000 7fa64f7fe640 1 -- 172.21.15.28:0/498609823 learned_addr learned my addr 172.21.15.28:0/498609823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.030+0000 7fa64e7fc640 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.030+0000 7fa64effd640 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef8ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.031+0000 7fa64f7fe640 1 -- 172.21.15.28:0/498609823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 msgr2=0x55bfef8ff2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.031+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef8ff2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.031+0000 7fa64f7fe640 1 -- 172.21.15.28:0/498609823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 msgr2=0x55bfef8f0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.031+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.031+0000 7fa64f7fe640 1 -- 172.21.15.28:0/498609823 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfef8f1a30 con 0x55bfef8f7630 2024-05-09T08:45:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.032+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f13c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa648006010 tx=0x7fa648009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc43921d1f507565 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.032+0000 7fa62effd640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-05-09T08:45:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.035+0000 7fa62effd640 1 -- 172.21.15.28:0/498609823 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648009890 con 0x55bfef8f7630 2024-05-09T08:45:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.035+0000 7fa62effd640 1 -- 172.21.15.28:0/498609823 <== mon.2 v2:172.21.15.165:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9803+0+0 (secure 0 0 0) 0x7fa648030d10 con 0x55bfef8f7630 2024-05-09T08:45:33.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.038+0000 7fa65862b740 1 -- 172.21.15.28:0/498609823 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa638022070 con 0x55bfef8f7630 2024-05-09T08:45:33.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.039+0000 7fa62effd640 1 -- 172.21.15.28:0/498609823 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7fa6480f0170 con 0x55bfef8f7630 2024-05-09T08:45:33.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.040+0000 7fa65862b740 10 client.26861 fetch_fsmap learned FSMap version 67 2024-05-09T08:45:33.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.040+0000 7fa65862b740 1 -- 172.21.15.28:0/498609823 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x55bfef96da50 con 0x55bfef8f7630 2024-05-09T08:45:33.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.042+0000 7fa62effd640 1 -- 172.21.15.28:0/498609823 <== mon.2 v2:172.21.15.165:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa638022070 con 0x55bfef8f7630 2024-05-09T08:45:33.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.042+0000 7fa62effd640 1 -- 172.21.15.28:0/498609823 <== mon.2 v2:172.21.15.165:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7fa648041030 con 0x55bfef8f7630 2024-05-09T08:45:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.043+0000 7fa65862b740 10 client.26861 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:45:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.043+0000 7fa65862b740 10 client.26861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:45:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.043+0000 7fa65862b740 20 client.26861 populate_metadata read hostname 'smithi028' 2024-05-09T08:45:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.043+0000 7fa65862b740 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bfef96df30 0x55bfef970350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.043+0000 7fa65862b740 4 client.26861 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:45:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.043+0000 7fa65862b740 1 -- 172.21.15.28:0/498609823 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bfef858dc0 con 0x55bfef96df30 2024-05-09T08:45:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.044+0000 7fa64effd640 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bfef96df30 0x55bfef970350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.053+0000 7fa64effd640 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bfef96df30 0x55bfef970350 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d10a4cfcb262188 server_cookie=63ed71aee3d0d6af in_seq=0 out_seq=0 2024-05-09T08:45:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.053+0000 7fa62effd640 10 client.26861 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:33.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.177+0000 7fa62effd640 1 -- 172.21.15.28:0/498609823 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bfef858dc0 con 0x55bfef96df30 2024-05-09T08:45:33.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.177+0000 7fa62effd640 10 client.26861 handle_command_reply: tid=0 2024-05-09T08:45:33.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.177+0000 7fa65862b740 1 client.26861 shutdown 2024-05-09T08:45:33.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 -- 172.21.15.28:0/498609823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 msgr2=0x55bfef8f13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:33.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f13c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa648006010 tx=0x7fa648009560 comp rx=0 tx=0).stop 2024-05-09T08:45:33.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 -- 172.21.15.28:0/498609823 shutdown_connections 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x55bfef96df30 0x55bfef970350 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55bfef8f7630 0x55bfef8f13c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfef8ecd70 0x55bfef8f0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 --2- 172.21.15.28:0/498609823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55bfef8ec450 0x55bfef8ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 -- 172.21.15.28:0/498609823 >> 172.21.15.28:0/498609823 conn(0x55bfef8e8960 msgr2=0x55bfef8ede50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa62effd640 0 client.26861 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 -- 172.21.15.28:0/498609823 shutdown_connections 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 1 -- 172.21.15.28:0/498609823 wait complete. 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.178+0000 7fa65862b740 20 client.26861 trim_cache size 0 max 16384 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:45:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:45:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:45:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:45:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 -- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa63c0a2f50 msgr2=0x7fa63c0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa63c0a2f50 0x7fa63c0a5410 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa650071d50 tx=0x7fa638006bd0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 -- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 msgr2=0x7fa650070b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 0x7fa650070b60 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fa6480308b0 tx=0x7fa6480308e0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 -- 172.21.15.28:0/1123262715 shutdown_connections 2024-05-09T08:45:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6501128f0 0x7fa650070b60 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa63c0a2f50 0x7fa63c0a5410 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa650074450 0x7fa650070590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 --2- 172.21.15.28:0/1123262715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa650073b30 0x7fa6500734f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 -- 172.21.15.28:0/1123262715 >> 172.21.15.28:0/1123262715 conn(0x7fa650078cc0 msgr2=0x7fa6500fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 -- 172.21.15.28:0/1123262715 shutdown_connections 2024-05-09T08:45:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:33.179+0000 7fa65591b640 1 -- 172.21.15.28:0/1123262715 wait complete. 2024-05-09T08:45:34.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:33 smithi165 ceph-mon[34540]: pgmap v619: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 999 KiB/s wr, 195 op/s 2024-05-09T08:45:34.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:33 smithi184 ceph-mon[34509]: pgmap v619: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 999 KiB/s wr, 195 op/s 2024-05-09T08:45:34.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:33 smithi028 ceph-mon[30119]: pgmap v619: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 999 KiB/s wr, 195 op/s 2024-05-09T08:45:36.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:35 smithi165 ceph-mon[34540]: pgmap v620: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 179 op/s 2024-05-09T08:45:36.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:35 smithi184 ceph-mon[34509]: pgmap v620: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 179 op/s 2024-05-09T08:45:36.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:35 smithi028 ceph-mon[30119]: pgmap v620: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 179 op/s 2024-05-09T08:45:38.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:37 smithi165 ceph-mon[34540]: pgmap v621: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-05-09T08:45:38.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:37 smithi184 ceph-mon[34509]: pgmap v621: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-05-09T08:45:38.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:37 smithi028 ceph-mon[30119]: pgmap v621: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 182 op/s 2024-05-09T08:45:40.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:39 smithi165 ceph-mon[34540]: pgmap v622: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-05-09T08:45:40.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:39 smithi184 ceph-mon[34509]: pgmap v622: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-05-09T08:45:40.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:39 smithi028 ceph-mon[30119]: pgmap v622: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-05-09T08:45:42.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:41 smithi165 ceph-mon[34540]: pgmap v623: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-05-09T08:45:42.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:41 smithi184 ceph-mon[34509]: pgmap v623: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-05-09T08:45:42.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:41 smithi028 ceph-mon[30119]: pgmap v623: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-05-09T08:45:44.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:43 smithi165 ceph-mon[34540]: pgmap v624: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-05-09T08:45:44.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:43 smithi184 ceph-mon[34509]: pgmap v624: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-05-09T08:45:44.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:43 smithi028 ceph-mon[30119]: pgmap v624: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-05-09T08:45:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:45 smithi165 ceph-mon[34540]: pgmap v625: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-05-09T08:45:46.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:45 smithi184 ceph-mon[34509]: pgmap v625: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-05-09T08:45:46.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:45 smithi028 ceph-mon[30119]: pgmap v625: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-05-09T08:45:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.364+0000 7f6602f55640 1 -- 172.21.15.28:0/4102953834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f409acf0 msgr2=0x7f65f4004740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.364+0000 7f6602f55640 1 --2- 172.21.15.28:0/4102953834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f409acf0 0x7f65f4004740 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009860 tx=0x7f65f002a380 comp rx=0 tx=0).stop 2024-05-09T08:45:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.365+0000 7f6602f55640 1 -- 172.21.15.28:0/4102953834 shutdown_connections 2024-05-09T08:45:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.365+0000 7f6602f55640 1 --2- 172.21.15.28:0/4102953834 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f40986e0 0x7f65f4098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.365+0000 7f6602f55640 1 --2- 172.21.15.28:0/4102953834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65f4004c80 0x7f65f4005100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.365+0000 7f6602f55640 1 --2- 172.21.15.28:0/4102953834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f409acf0 0x7f65f4004740 secure :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009860 tx=0x7f65f002a380 comp rx=0 tx=0).stop 2024-05-09T08:45:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.365+0000 7f6602f55640 1 -- 172.21.15.28:0/4102953834 >> 172.21.15.28:0/4102953834 conn(0x7f65f400a740 msgr2=0x7f65f400ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:46.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.368+0000 7f6602f55640 1 -- 172.21.15.28:0/4102953834 shutdown_connections 2024-05-09T08:45:46.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.369+0000 7f6602f55640 1 -- 172.21.15.28:0/4102953834 wait complete. 2024-05-09T08:45:46.369 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.370+0000 7f6602f55640 1 Processor -- start 2024-05-09T08:45:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.370+0000 7f6602f55640 1 -- start start 2024-05-09T08:45:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.370+0000 7f6602f55640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f4004c80 0x7f65f4149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.370+0000 7f6602f55640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65f40986e0 0x7f65f4149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.370+0000 7f6602f55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f4143af0 0x7f65f4143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.370+0000 7f6602f55640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65f40bd610 con 0x7f65f4143af0 2024-05-09T08:45:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.370+0000 7f6602f55640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f65f40bd490 con 0x7f65f40986e0 2024-05-09T08:45:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.370+0000 7f6602f55640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f65f40bd790 con 0x7f65f4004c80 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6601752640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65f40986e0 0x7f65f4149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6602754640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f4143af0 0x7f65f4143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6601f53640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f4004c80 0x7f65f4149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6602754640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f4143af0 0x7f65f4143fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55726/0 (socket says 172.21.15.28:55726) 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6602754640 1 -- 172.21.15.28:0/3241429721 learned_addr learned my addr 172.21.15.28:0/3241429721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6601752640 1 -- 172.21.15.28:0/3241429721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f4004c80 msgr2=0x7f65f4149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6601752640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f4004c80 0x7f65f4149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6601752640 1 -- 172.21.15.28:0/3241429721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f4143af0 msgr2=0x7f65f4143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6601752640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f4143af0 0x7f65f4143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.371+0000 7f6601752640 1 -- 172.21.15.28:0/3241429721 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f4144610 con 0x7f65f40986e0 2024-05-09T08:45:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.372+0000 7f6601752640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65f40986e0 0x7f65f4149e80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f65f800e790 tx=0x7f65f800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.374+0000 7f65eeffd640 1 -- 172.21.15.28:0/3241429721 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f800eea0 con 0x7f65f40986e0 2024-05-09T08:45:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.374+0000 7f65eeffd640 1 -- 172.21.15.28:0/3241429721 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f800ccc0 con 0x7f65f40986e0 2024-05-09T08:45:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.375+0000 7f65eeffd640 1 -- 172.21.15.28:0/3241429721 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8018780 con 0x7f65f40986e0 2024-05-09T08:45:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.375+0000 7f6602f55640 1 -- 172.21.15.28:0/3241429721 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f4144900 con 0x7f65f40986e0 2024-05-09T08:45:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.375+0000 7f6602f55640 1 -- 172.21.15.28:0/3241429721 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f409b770 con 0x7f65f40986e0 2024-05-09T08:45:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.383+0000 7f6606fa5740 1 Processor -- start 2024-05-09T08:45:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.385+0000 7f6606fa5740 1 -- start start 2024-05-09T08:45:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.386+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e445d0 0x563835e449b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.386+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 0x563835e4f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.388+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e51ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.388+0000 7f6606fa5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563835da9340 con 0x563835e4f7b0 2024-05-09T08:45:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.388+0000 7f6606fa5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x563835d51290 con 0x563835e445d0 2024-05-09T08:45:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.388+0000 7f6606fa5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563835da5300 con 0x563835e44ef0 2024-05-09T08:45:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.388+0000 7f6601f53640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 0x563835e4f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.388+0000 7f6601f53640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 0x563835e4f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:53034/0 (socket says 172.21.15.28:53034) 2024-05-09T08:45:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.388+0000 7f6601f53640 1 -- 172.21.15.28:0/2835783896 learned_addr learned my addr 172.21.15.28:0/2835783896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.388+0000 7f65eeffd640 1 -- 172.21.15.28:0/3241429721 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f65f801f020 con 0x7f65f40986e0 2024-05-09T08:45:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.389+0000 7f6601752640 1 --2- 172.21.15.28:0/2835783896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e51ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.392+0000 7f6601752640 1 -- 172.21.15.28:0/2835783896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 msgr2=0x563835e4f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.392+0000 7f6601752640 1 --2- 172.21.15.28:0/2835783896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 0x563835e4f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.392+0000 7f6601752640 1 -- 172.21.15.28:0/2835783896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e445d0 msgr2=0x563835e449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.392+0000 7f6601752640 1 --2- 172.21.15.28:0/2835783896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e445d0 0x563835e449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.392+0000 7f6601752640 1 -- 172.21.15.28:0/2835783896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563835d758e0 con 0x563835e4f7b0 2024-05-09T08:45:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.393+0000 7f65eeffd640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f65e80a2de0 0x7f65e80a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.393+0000 7f65eeffd640 1 -- 172.21.15.28:0/3241429721 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9803+0+0 (secure 0 0 0) 0x7f65f80cf800 con 0x7f65f40986e0 2024-05-09T08:45:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.393+0000 7f6601752640 1 --2- 172.21.15.28:0/2835783896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e51ba0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f65f80cc9f0 tx=0x7f65f80d5c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e574dfdfa25bd29b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.394+0000 7f65e7fff640 1 -- 172.21.15.28:0/2835783896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8097a60 con 0x563835e4f7b0 2024-05-09T08:45:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.395+0000 7f65e7fff640 1 -- 172.21.15.28:0/2835783896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f809bec0 con 0x563835e4f7b0 2024-05-09T08:45:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.395+0000 7f65e7fff640 1 -- 172.21.15.28:0/2835783896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f809b120 con 0x563835e4f7b0 2024-05-09T08:45:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.396+0000 7f6606fa5740 1 -- 172.21.15.28:0/2835783896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 msgr2=0x563835e51ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.396+0000 7f6606fa5740 1 --2- 172.21.15.28:0/2835783896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e51ba0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f65f80cc9f0 tx=0x7f65f80d5c50 comp rx=0 tx=0).stop 2024-05-09T08:45:46.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.397+0000 7f6606fa5740 1 -- 172.21.15.28:0/2835783896 shutdown_connections 2024-05-09T08:45:46.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.397+0000 7f6606fa5740 1 --2- 172.21.15.28:0/2835783896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e51ba0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.397+0000 7f6606fa5740 1 --2- 172.21.15.28:0/2835783896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 0x563835e4f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.397+0000 7f6606fa5740 1 --2- 172.21.15.28:0/2835783896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e445d0 0x563835e449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.397+0000 7f6606fa5740 1 -- 172.21.15.28:0/2835783896 >> 172.21.15.28:0/2835783896 conn(0x563835e40ae0 msgr2=0x563835e40ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.398+0000 7f6602754640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f65e80a2de0 0x7f65e80a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.398+0000 7f6606fa5740 1 -- 172.21.15.28:0/2835783896 shutdown_connections 2024-05-09T08:45:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.399+0000 7f6606fa5740 1 -- 172.21.15.28:0/2835783896 wait complete. 2024-05-09T08:45:46.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.399+0000 7f6606fa5740 1 -- start start 2024-05-09T08:45:46.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.399+0000 7f6606fa5740 1 Processor -- start 2024-05-09T08:45:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.400+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e4c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.400+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e44ef0 0x563835e4cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.401+0000 7f6601f53640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e4c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.401+0000 7f6601f53640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e4c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:53036/0 (socket says 172.21.15.28:53036) 2024-05-09T08:45:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.401+0000 7f6601f53640 1 -- 172.21.15.28:0/1791199121 learned_addr learned my addr 172.21.15.28:0/1791199121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.401+0000 7f6606fa5740 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 0x563835e4d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.401+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563835e52430 con 0x563835e44ef0 2024-05-09T08:45:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.401+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x563835d64790 con 0x563835e4f7b0 2024-05-09T08:45:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.401+0000 7f6602754640 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 0x563835e4d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.401+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x563835e525b0 con 0x563835e445d0 2024-05-09T08:45:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.401+0000 7f6606fa5740 10 client.0 _ll_register_callbacks cb 0x563835d9fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.402+0000 7f6601f53640 1 -- 172.21.15.28:0/1791199121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 msgr2=0x563835e4d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.402+0000 7f6601f53640 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 0x563835e4d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.402+0000 7f6601f53640 1 -- 172.21.15.28:0/1791199121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e44ef0 msgr2=0x563835e4cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.402+0000 7f6601f53640 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e44ef0 0x563835e4cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.402+0000 7f6601f53640 1 -- 172.21.15.28:0/1791199121 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563835abd210 con 0x563835e445d0 2024-05-09T08:45:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.402+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f809bec0 con 0x563835e445d0 2024-05-09T08:45:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.405+0000 7f6601f53640 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e4c690 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f65f002a970 tx=0x7f65f0030510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40dda7922cd36f56 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.406+0000 7f65e57fa640 10 client.26701 ms_handle_connect on v2:172.21.15.184:3300/0 2024-05-09T08:45:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.407+0000 7f65e57fa640 1 -- 172.21.15.28:0/1791199121 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0037040 con 0x563835e445d0 2024-05-09T08:45:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.407+0000 7f6602754640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f65e80a2de0 0x7f65e80a52a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f65fc050dd0 tx=0x7f65fc068020 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.407+0000 7f65e57fa640 1 -- 172.21.15.28:0/1791199121 <== mon.1 v2:172.21.15.184:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9803+0+0 (secure 0 0 0) 0x7f65f0005040 con 0x563835e445d0 2024-05-09T08:45:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.408+0000 7f65e57fa640 1 -- 172.21.15.28:0/1791199121 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f65f0005260 con 0x563835e445d0 2024-05-09T08:45:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.410+0000 7f6606fa5740 10 client.26701 fetch_fsmap learned FSMap version 67 2024-05-09T08:45:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.410+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({fsmap=67}) -- 0x563835b89620 con 0x563835e445d0 2024-05-09T08:45:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.413+0000 7f65e57fa640 1 -- 172.21.15.28:0/1791199121 <== mon.1 v2:172.21.15.184:3300/0 4 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f65f0005400 con 0x563835e445d0 2024-05-09T08:45:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.415+0000 7f6606fa5740 10 client.26701 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:45:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.415+0000 7f6606fa5740 10 client.26701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:45:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.415+0000 7f6606fa5740 20 client.26701 populate_metadata read hostname 'smithi028' 2024-05-09T08:45:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.415+0000 7f6606fa5740 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563835e5f4d0 0x563835e618f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.415+0000 7f6606fa5740 4 client.26701 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.415+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563835db6a70 con 0x563835e5f4d0 2024-05-09T08:45:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.417+0000 7f6601752640 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563835e5f4d0 0x563835e618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.427+0000 7f6601752640 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563835e5f4d0 0x563835e618f0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60938ff96d55b448 server_cookie=2db0c4b629c2b0c6 in_seq=0 out_seq=0 2024-05-09T08:45:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.429+0000 7f65e57fa640 10 client.26701 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.430+0000 7f65e57fa640 1 -- 172.21.15.28:0/1791199121 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563835db6a70 con 0x563835e5f4d0 2024-05-09T08:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.430+0000 7f65e57fa640 10 client.26701 handle_command_reply: tid=0 2024-05-09T08:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.431+0000 7f6606fa5740 1 client.26701 shutdown 2024-05-09T08:45:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.436+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 msgr2=0x563835e4c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.436+0000 7f6606fa5740 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e4c690 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f65f002a970 tx=0x7f65f0030510 comp rx=0 tx=0).stop 2024-05-09T08:45:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.436+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 shutdown_connections 2024-05-09T08:45:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.436+0000 7f6606fa5740 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563835e5f4d0 0x563835e618f0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.437+0000 7f6606fa5740 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 0x563835e4d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.437+0000 7f6606fa5740 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e44ef0 0x563835e4cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.437+0000 7f6606fa5740 1 --2- 172.21.15.28:0/1791199121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e4c690 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.437+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 >> 172.21.15.28:0/1791199121 conn(0x563835e40ae0 msgr2=0x563835e46810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.439+0000 7f65e57fa640 0 client.26701 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.441+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 shutdown_connections 2024-05-09T08:45:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.441+0000 7f6606fa5740 1 -- 172.21.15.28:0/1791199121 wait complete. 2024-05-09T08:45:46.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.441+0000 7f6606fa5740 20 client.26701 trim_cache size 0 max 16384 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.473+0000 7f6606fa5740 1 Processor -- start 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.473+0000 7f6606fa5740 1 -- start start 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.473+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e445d0 0x563835e5b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.474+0000 7f6601f53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e445d0 0x563835e5b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.474+0000 7f6601f53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e445d0 0x563835e5b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55762/0 (socket says 172.21.15.28:55762) 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.474+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 0x563835e671e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.474+0000 7f6601f53640 1 -- 172.21.15.28:0/3017286198 learned_addr learned my addr 172.21.15.28:0/3017286198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.474+0000 7f6606fa5740 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 0x563835e67720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.474+0000 7f6606fa5740 1 -- 172.21.15.28:0/3017286198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65f4163680 con 0x563835e445d0 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.474+0000 7f6606fa5740 1 -- 172.21.15.28:0/3017286198 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f65f4165400 con 0x563835e4f7b0 2024-05-09T08:45:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.474+0000 7f6606fa5740 1 -- 172.21.15.28:0/3017286198 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f65f4163e10 con 0x563835e44ef0 2024-05-09T08:45:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.475+0000 7f6602754640 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 0x563835e67720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.475+0000 7f6601752640 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 0x563835e671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.476+0000 7f6601f53640 1 -- 172.21.15.28:0/3017286198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 msgr2=0x563835e671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.476+0000 7f6601f53640 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 0x563835e671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.476+0000 7f6601f53640 1 -- 172.21.15.28:0/3017286198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 msgr2=0x563835e67720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.476+0000 7f6601f53640 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 0x563835e67720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.476+0000 7f6601f53640 1 -- 172.21.15.28:0/3017286198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563835e44d20 con 0x563835e445d0 2024-05-09T08:45:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.476+0000 7f6601f53640 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e445d0 0x563835e5b380 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f65f0030670 tx=0x7f65f002b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98ae98477a9bd6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.477+0000 7f65e77fe640 1 -- 172.21.15.28:0/3017286198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0037070 con 0x563835e445d0 2024-05-09T08:45:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.477+0000 7f65e77fe640 1 -- 172.21.15.28:0/3017286198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f809bec0 con 0x563835e445d0 2024-05-09T08:45:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.478+0000 7f6606fa5740 1 -- 172.21.15.28:0/3017286198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e445d0 msgr2=0x563835e5b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.478+0000 7f6606fa5740 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e445d0 0x563835e5b380 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f65f0030670 tx=0x7f65f002b000 comp rx=0 tx=0).stop 2024-05-09T08:45:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.479+0000 7f6606fa5740 1 -- 172.21.15.28:0/3017286198 shutdown_connections 2024-05-09T08:45:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.479+0000 7f6606fa5740 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e4f7b0 0x563835e67720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.479+0000 7f6606fa5740 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e44ef0 0x563835e671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.479+0000 7f6606fa5740 1 --2- 172.21.15.28:0/3017286198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e445d0 0x563835e5b380 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.479+0000 7f6606fa5740 1 -- 172.21.15.28:0/3017286198 >> 172.21.15.28:0/3017286198 conn(0x563835e40ae0 msgr2=0x563835e4ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.480+0000 7f6606fa5740 1 -- 172.21.15.28:0/3017286198 shutdown_connections 2024-05-09T08:45:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.480+0000 7f6606fa5740 1 -- 172.21.15.28:0/3017286198 wait complete. 2024-05-09T08:45:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.480+0000 7f6606fa5740 1 -- start start 2024-05-09T08:45:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.480+0000 7f6606fa5740 1 Processor -- start 2024-05-09T08:45:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.481+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e57610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.481+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e44ef0 0x563835e57b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.481+0000 7f6606fa5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e54aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.481+0000 7f6606fa5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65f4164d30 con 0x563835e4f7b0 2024-05-09T08:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.481+0000 7f6606fa5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f65f41645a0 con 0x563835e44ef0 2024-05-09T08:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.481+0000 7f6606fa5740 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f65f4095220 con 0x563835e445d0 2024-05-09T08:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.481+0000 7f6606fa5740 10 client.0 _ll_register_callbacks cb 0x7f65f8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.482+0000 7f6602754640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e54aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.482+0000 7f6602754640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e54aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55770/0 (socket says 172.21.15.28:55770) 2024-05-09T08:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.482+0000 7f6602754640 1 -- 172.21.15.28:0/12879612 learned_addr learned my addr 172.21.15.28:0/12879612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.482+0000 7f6601f53640 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e57610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.482+0000 7f6601752640 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e44ef0 0x563835e57b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.483+0000 7f6602754640 1 -- 172.21.15.28:0/12879612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 msgr2=0x563835e57610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.483+0000 7f6602754640 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e57610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.483+0000 7f6602754640 1 -- 172.21.15.28:0/12879612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e44ef0 msgr2=0x563835e57b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.483+0000 7f6602754640 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e44ef0 0x563835e57b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.483+0000 7f6602754640 1 -- 172.21.15.28:0/12879612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563835e55160 con 0x563835e4f7b0 2024-05-09T08:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.483+0000 7f6602754640 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e54aa0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0649a0 tx=0x7f65fc067b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed249e57c9a0512d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.483+0000 7f65e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-09T08:45:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.486+0000 7f65e57fa640 1 -- 172.21.15.28:0/12879612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc091060 con 0x563835e4f7b0 2024-05-09T08:45:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.486+0000 7f65e57fa640 1 -- 172.21.15.28:0/12879612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9803+0+0 (secure 0 0 0) 0x7f65fc06f3f0 con 0x563835e4f7b0 2024-05-09T08:45:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.488+0000 7f6606fa5740 1 -- 172.21.15.28:0/12879612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f809bec0 con 0x563835e4f7b0 2024-05-09T08:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.489+0000 7f65e57fa640 1 -- 172.21.15.28:0/12879612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f65fc072230 con 0x563835e4f7b0 2024-05-09T08:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.491+0000 7f6606fa5740 10 client.17442 fetch_fsmap learned FSMap version 67 2024-05-09T08:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.491+0000 7f6606fa5740 1 -- 172.21.15.28:0/12879612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=4+}) -- 0x563835ec5530 con 0x563835e4f7b0 2024-05-09T08:45:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.495+0000 7f65e57fa640 1 -- 172.21.15.28:0/12879612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f809bec0 con 0x563835e4f7b0 2024-05-09T08:45:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.495+0000 7f65e57fa640 1 -- 172.21.15.28:0/12879612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 67) ==== 8442+0+0 (secure 0 0 0) 0x7f65fc08b870 con 0x563835e4f7b0 2024-05-09T08:45:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.495+0000 7f6606fa5740 10 client.17442 fetch_fsmap finished waiting for FSMap version 67 2024-05-09T08:45:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.495+0000 7f6606fa5740 10 client.17442 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T08:45:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.495+0000 7f6606fa5740 20 client.17442 populate_metadata read hostname 'smithi028' 2024-05-09T08:45:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.495+0000 7f6606fa5740 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563835ec5c90 0x563835ec80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.495+0000 7f6606fa5740 4 client.17442 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:45:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.495+0000 7f6606fa5740 1 -- 172.21.15.28:0/12879612 --> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563835db0dc0 con 0x563835ec5c90 2024-05-09T08:45:46.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.497+0000 7f6601f53640 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563835ec5c90 0x563835ec80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:46.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.500+0000 7f6601f53640 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563835ec5c90 0x563835ec80d0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f24093d292533cf9 server_cookie=ebc15d9498cbce63 in_seq=0 out_seq=0 2024-05-09T08:45:46.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.500+0000 7f65e57fa640 10 client.17442 ms_handle_connect on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:46.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f65e57fa640 1 -- 172.21.15.28:0/12879612 <== mds.0 v2:172.21.15.165:6838/1723030158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563835db0dc0 con 0x563835ec5c90 2024-05-09T08:45:46.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f65e57fa640 10 client.17442 handle_command_reply: tid=0 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 client.17442 shutdown 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 -- 172.21.15.28:0/12879612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 msgr2=0x563835e54aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e54aa0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0649a0 tx=0x7f65fc067b00 comp rx=0 tx=0).stop 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 -- 172.21.15.28:0/12879612 shutdown_connections 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.165:6838/1723030158,v1:172.21.15.165:6839/1723030158] conn(0x563835ec5c90 0x563835ec80d0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563835e4f7b0 0x563835e54aa0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563835e44ef0 0x563835e57b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 --2- 172.21.15.28:0/12879612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x563835e445d0 0x563835e57610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 -- 172.21.15.28:0/12879612 >> 172.21.15.28:0/12879612 conn(0x563835e40ae0 msgr2=0x563835e5ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f65e57fa640 0 client.17442 ms_handle_reset on v2:172.21.15.165:6838/1723030158 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.693+0000 7f6606fa5740 1 -- 172.21.15.28:0/12879612 shutdown_connections 2024-05-09T08:45:46.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.694+0000 7f6606fa5740 1 -- 172.21.15.28:0/12879612 wait complete. 2024-05-09T08:45:46.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.694+0000 7f6606fa5740 20 client.17442 trim_cache size 0 max 16384 2024-05-09T08:45:46.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-05-09T08:45:46.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-05-09T08:45:46.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-05-09T08:45:46.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-05-09T08:45:46.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 -- 172.21.15.28:0/3241429721 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f65e80a2de0 msgr2=0x7f65e80a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f65e80a2de0 0x7f65e80a52a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f65fc050dd0 tx=0x7f65fc068020 comp rx=0 tx=0).stop 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 -- 172.21.15.28:0/3241429721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65f40986e0 msgr2=0x7f65f4149e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65f40986e0 0x7f65f4149e80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f65f800e790 tx=0x7f65f800ec50 comp rx=0 tx=0).stop 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 -- 172.21.15.28:0/3241429721 shutdown_connections 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f4143af0 0x7f65f4143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65f40986e0 0x7f65f4149e80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f65e80a2de0 0x7f65e80a52a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 --2- 172.21.15.28:0/3241429721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65f4004c80 0x7f65f4149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 -- 172.21.15.28:0/3241429721 >> 172.21.15.28:0/3241429721 conn(0x7f65f400a740 msgr2=0x7f65f40922e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.695+0000 7f6602f55640 1 -- 172.21.15.28:0/3241429721 shutdown_connections 2024-05-09T08:45:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-05-09T08:45:46.696+0000 7f6602f55640 1 -- 172.21.15.28:0/3241429721 wait complete. 2024-05-09T08:45:48.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:47 smithi165 ceph-mon[34540]: pgmap v626: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-05-09T08:45:48.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:47 smithi184 ceph-mon[34509]: pgmap v626: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-05-09T08:45:48.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:47 smithi028 ceph-mon[30119]: pgmap v626: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-05-09T08:45:50.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:49 smithi165 ceph-mon[34540]: pgmap v627: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 1018 KiB/s wr, 108 op/s 2024-05-09T08:45:50.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:49 smithi184 ceph-mon[34509]: pgmap v627: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 1018 KiB/s wr, 108 op/s 2024-05-09T08:45:50.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:49 smithi028 ceph-mon[30119]: pgmap v627: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 1018 KiB/s wr, 108 op/s 2024-05-09T08:45:52.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:51 smithi165 ceph-mon[34540]: pgmap v628: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 1.0 MiB/s wr, 115 op/s 2024-05-09T08:45:52.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:51 smithi184 ceph-mon[34509]: pgmap v628: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 1.0 MiB/s wr, 115 op/s 2024-05-09T08:45:52.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:51 smithi028 ceph-mon[30119]: pgmap v628: 129 pgs: 64 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 44 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 1.0 MiB/s wr, 115 op/s 2024-05-09T08:45:54.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:53 smithi165 ceph-mon[34540]: pgmap v629: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-05-09T08:45:54.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:53 smithi184 ceph-mon[34509]: pgmap v629: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-05-09T08:45:54.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:53 smithi028 ceph-mon[30119]: pgmap v629: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-05-09T08:45:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:55 smithi184 ceph-mon[34509]: pgmap v630: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-05-09T08:45:56.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:55 smithi184 ceph-mon[34509]: osdmap e134: 12 total, 12 up, 12 in 2024-05-09T08:45:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:55 smithi028 ceph-mon[30119]: pgmap v630: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-05-09T08:45:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:55 smithi028 ceph-mon[30119]: osdmap e134: 12 total, 12 up, 12 in 2024-05-09T08:45:56.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:55 smithi165 ceph-mon[34540]: pgmap v630: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-05-09T08:45:56.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:55 smithi165 ceph-mon[34540]: osdmap e134: 12 total, 12 up, 12 in 2024-05-09T08:45:58.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:57 smithi028 ceph-mon[30119]: pgmap v632: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-05-09T08:45:58.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:57 smithi165 ceph-mon[34540]: pgmap v632: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-05-09T08:45:58.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:57 smithi184 ceph-mon[34509]: pgmap v632: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-05-09T08:45:58.506 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-05-09T08:45:58.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-09T08:45:59.084 DEBUG:teuthology.parallel:result is None 2024-05-09T08:45:59.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T08:45:59.176 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T08:45:59.176 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-05-09T08:45:59.187 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:45:59.199 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-09T08:45:59.242 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-09T08:45:59.242 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:45:59.242 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:45:59.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-09T08:45:59.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-09T08:45:59.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-09T08:45:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.557+0000 7f13f19c1640 1 -- 172.21.15.28:0/1575574322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec1010d0 msgr2=0x7f13ec10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.557+0000 7f13f19c1640 1 --2- 172.21.15.28:0/1575574322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec1010d0 0x7f13ec10d5e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f13e401ace0 tx=0x7f13e403db80 comp rx=0 tx=0).stop 2024-05-09T08:45:59.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.562+0000 7f13f19c1640 1 -- 172.21.15.28:0/1575574322 shutdown_connections 2024-05-09T08:45:59.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.562+0000 7f13f19c1640 1 --2- 172.21.15.28:0/1575574322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec1010d0 0x7f13ec10d5e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.562+0000 7f13f19c1640 1 --2- 172.21.15.28:0/1575574322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13ec100710 0x7f13ec100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.562+0000 7f13f19c1640 1 --2- 172.21.15.28:0/1575574322 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec106720 0x7f13ec106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.562+0000 7f13f19c1640 1 -- 172.21.15.28:0/1575574322 >> 172.21.15.28:0/1575574322 conn(0x7f13ec0fc200 msgr2=0x7f13ec0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.564+0000 7f13f19c1640 1 -- 172.21.15.28:0/1575574322 shutdown_connections 2024-05-09T08:45:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.564+0000 7f13f19c1640 1 -- 172.21.15.28:0/1575574322 wait complete. 2024-05-09T08:45:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.565+0000 7f13f19c1640 1 Processor -- start 2024-05-09T08:45:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.565+0000 7f13f19c1640 1 -- start start 2024-05-09T08:45:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.566+0000 7f13f19c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec100710 0x7f13ec072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.566+0000 7f13f19c1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1010d0 0x7f13ec06e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.566+0000 7f13f19c1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13ec106720 0x7f13ec06e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.566+0000 7f13f19c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec06cac0 con 0x7f13ec100710 2024-05-09T08:45:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.566+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec100710 0x7f13ec072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec100710 0x7f13ec072ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56622/0 (socket says 172.21.15.28:56622) 2024-05-09T08:45:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13eaffd640 1 -- 172.21.15.28:0/4091756255 learned_addr learned my addr 172.21.15.28:0/4091756255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:45:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13ea7fc640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1010d0 0x7f13ec06e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec06c940 con 0x7f13ec106720 2024-05-09T08:45:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec06cc40 con 0x7f13ec1010d0 2024-05-09T08:45:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13eb7fe640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13ec106720 0x7f13ec06e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13eb7fe640 1 -- 172.21.15.28:0/4091756255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1010d0 msgr2=0x7f13ec06e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13eb7fe640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1010d0 0x7f13ec06e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13eb7fe640 1 -- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec100710 msgr2=0x7f13ec072ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13eb7fe640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec100710 0x7f13ec072ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.567+0000 7f13eb7fe640 1 -- 172.21.15.28:0/4091756255 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec06eaf0 con 0x7f13ec106720 2024-05-09T08:45:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.569+0000 7f13eb7fe640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13ec106720 0x7f13ec06e5b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f13e403e170 tx=0x7f13e4003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.571+0000 7f13eaffd640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec100710 0x7f13ec072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:45:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.571+0000 7f13ea7fc640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1010d0 0x7f13ec06e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:45:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.571+0000 7f13f09bf640 1 -- 172.21.15.28:0/4091756255 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4057070 con 0x7f13ec106720 2024-05-09T08:45:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.572+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ec06ed80 con 0x7f13ec106720 2024-05-09T08:45:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.572+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ec19e510 con 0x7f13ec106720 2024-05-09T08:45:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.572+0000 7f13f09bf640 1 -- 172.21.15.28:0/4091756255 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13e4040950 con 0x7f13ec106720 2024-05-09T08:45:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.572+0000 7f13f09bf640 1 -- 172.21.15.28:0/4091756255 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4004900 con 0x7f13ec106720 2024-05-09T08:45:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.573+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d8005290 con 0x7f13ec106720 2024-05-09T08:45:59.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.575+0000 7f13f09bf640 1 -- 172.21.15.28:0/4091756255 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f13e4004040 con 0x7f13ec106720 2024-05-09T08:45:59.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.576+0000 7f13f09bf640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13d40ab760 0x7f13d40adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.577+0000 7f13f09bf640 1 -- 172.21.15.28:0/4091756255 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9787+0+0 (secure 0 0 0) 0x7f13e4101640 con 0x7f13ec106720 2024-05-09T08:45:59.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.578+0000 7f13eaffd640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13d40ab760 0x7f13d40adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.579+0000 7f13eaffd640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13d40ab760 0x7f13d40adc20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f13e00058c0 tx=0x7f13e0005df0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.585+0000 7f13f09bf640 1 -- 172.21.15.28:0/4091756255 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f13e4045030 con 0x7f13ec106720 2024-05-09T08:45:59.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:45:59.994+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 --> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f13d8000bc0 con 0x7f13d40ab760 2024-05-09T08:46:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.054+0000 7f13f09bf640 1 -- 172.21.15.28:0/4091756255 <== mgr.15528 v2:172.21.15.28:6800/160415970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f13d8000bc0 con 0x7f13d40ab760 2024-05-09T08:46:00.054 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-09T00:00:00", "created": "2024-05-09T08:26:24", "first": "2024-05-09T08:27:00", "last": "2024-05-09T08:45:00", "last_pruned": "2024-05-09T08:45:01", "created_count": 19, "pruned_count": 13, "active": true} 2024-05-09T08:46:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13d40ab760 msgr2=0x7f13d40adc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13d40ab760 0x7f13d40adc20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f13e00058c0 tx=0x7f13e0005df0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13ec106720 msgr2=0x7f13ec06e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13ec106720 0x7f13ec06e5b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f13e403e170 tx=0x7f13e4003ba0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 shutdown_connections 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13ec106720 0x7f13ec06e5b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f13d40ab760 0x7f13d40adc20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13ec1010d0 0x7f13ec06e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 --2- 172.21.15.28:0/4091756255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13ec100710 0x7f13ec072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 >> 172.21.15.28:0/4091756255 conn(0x7f13ec0fc200 msgr2=0x7f13ec0fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 shutdown_connections 2024-05-09T08:46:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.058+0000 7f13f19c1640 1 -- 172.21.15.28:0/4091756255 wait complete. 2024-05-09T08:46:00.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-09T08:46:00.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:45:59 smithi028 ceph-mon[30119]: pgmap v633: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-09T08:46:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.283+0000 7fd24c607640 1 -- 172.21.15.28:0/1994822930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2441000f0 msgr2=0x7fd2441004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.283+0000 7fd24c607640 1 --2- 172.21.15.28:0/1994822930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2441000f0 0x7fd2441004d0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00adb0 tx=0x7fd23c02a6a0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.287+0000 7fd24c607640 1 -- 172.21.15.28:0/1994822930 shutdown_connections 2024-05-09T08:46:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.287+0000 7fd24c607640 1 --2- 172.21.15.28:0/1994822930 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2441051c0 0x7fd24410cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.287+0000 7fd24c607640 1 --2- 172.21.15.28:0/1994822930 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd244100aa0 0x7fd244104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.287+0000 7fd24c607640 1 --2- 172.21.15.28:0/1994822930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2441000f0 0x7fd2441004d0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.287+0000 7fd24c607640 1 -- 172.21.15.28:0/1994822930 >> 172.21.15.28:0/1994822930 conn(0x7fd244076cc0 msgr2=0x7fd2440fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.292+0000 7fd24c607640 1 -- 172.21.15.28:0/1994822930 shutdown_connections 2024-05-09T08:46:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.292+0000 7fd24c607640 1 -- 172.21.15.28:0/1994822930 wait complete. 2024-05-09T08:46:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.293+0000 7fd24c607640 1 Processor -- start 2024-05-09T08:46:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.293+0000 7fd24c607640 1 -- start start 2024-05-09T08:46:00.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd24c607640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2441000f0 0x7fd244195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:00.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd24c607640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2441051c0 0x7fd244196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:00.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd24c607640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24419a7a0 0x7fd24419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd249ba2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2441051c0 0x7fd244196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd249ba2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2441051c0 0x7fd244196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:34426/0 (socket says 172.21.15.28:34426) 2024-05-09T08:46:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd24a3a3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2441000f0 0x7fd244195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd24410fa40 con 0x7fd24419a7a0 2024-05-09T08:46:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd24410f8c0 con 0x7fd2441051c0 2024-05-09T08:46:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd24410fbc0 con 0x7fd2441000f0 2024-05-09T08:46:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.299+0000 7fd249ba2640 1 -- 172.21.15.28:0/2552578513 learned_addr learned my addr 172.21.15.28:0/2552578513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:46:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.300+0000 7fd24aba4640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24419a7a0 0x7fd24419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.300+0000 7fd249ba2640 1 -- 172.21.15.28:0/2552578513 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2441000f0 msgr2=0x7fd244195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.300+0000 7fd249ba2640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2441000f0 0x7fd244195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.300+0000 7fd249ba2640 1 -- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24419a7a0 msgr2=0x7fd24419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.300+0000 7fd249ba2640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24419a7a0 0x7fd24419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.300+0000 7fd249ba2640 1 -- 172.21.15.28:0/2552578513 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24419b290 con 0x7fd2441051c0 2024-05-09T08:46:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.301+0000 7fd249ba2640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2441051c0 0x7fd244196420 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd24000a8d0 tx=0x7fd24000ad90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.302+0000 7fd23b7fe640 1 -- 172.21.15.28:0/2552578513 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240019070 con 0x7fd2441051c0 2024-05-09T08:46:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.302+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd24419b580 con 0x7fd2441051c0 2024-05-09T08:46:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.303+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2441a2dc0 con 0x7fd2441051c0 2024-05-09T08:46:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.304+0000 7fd23b7fe640 1 -- 172.21.15.28:0/2552578513 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd240014410 con 0x7fd2441051c0 2024-05-09T08:46:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.304+0000 7fd23b7fe640 1 -- 172.21.15.28:0/2552578513 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2400135f0 con 0x7fd2441051c0 2024-05-09T08:46:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.307+0000 7fd23b7fe640 1 -- 172.21.15.28:0/2552578513 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd24001c070 con 0x7fd2441051c0 2024-05-09T08:46:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.308+0000 7fd23b7fe640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2300a2e70 0x7fd2300a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.309+0000 7fd23b7fe640 1 -- 172.21.15.28:0/2552578513 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9787+0+0 (secure 0 0 0) 0x7fd2400ce9d0 con 0x7fd2441051c0 2024-05-09T08:46:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.309+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd228005290 con 0x7fd2441051c0 2024-05-09T08:46:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.316+0000 7fd24a3a3640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2300a2e70 0x7fd2300a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:00.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.322+0000 7fd23b7fe640 1 -- 172.21.15.28:0/2552578513 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd24009aab0 con 0x7fd2441051c0 2024-05-09T08:46:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.327+0000 7fd24a3a3640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2300a2e70 0x7fd2300a5330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd23c000f80 tx=0x7fd23c030000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:00.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:45:59 smithi165 ceph-mon[34540]: pgmap v633: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-09T08:46:00.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:45:59 smithi184 ceph-mon[34509]: pgmap v633: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 121 op/s 2024-05-09T08:46:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:00.737+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 --> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fd228000bc0 con 0x7fd2300a2e70 2024-05-09T08:46:01.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.384+0000 7fd23b7fe640 1 -- 172.21.15.28:0/2552578513 <== mgr.15528 v2:172.21.15.28:6800/160415970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fd228000bc0 con 0x7fd2300a2e70 2024-05-09T08:46:01.385 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-05-09T08:46:01.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.387+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2300a2e70 msgr2=0x7fd2300a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:01.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.387+0000 7fd24c607640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2300a2e70 0x7fd2300a5330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd23c000f80 tx=0x7fd23c030000 comp rx=0 tx=0).stop 2024-05-09T08:46:01.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.387+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2441051c0 msgr2=0x7fd244196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:01.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.387+0000 7fd24c607640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2441051c0 0x7fd244196420 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd24000a8d0 tx=0x7fd24000ad90 comp rx=0 tx=0).stop 2024-05-09T08:46:01.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.388+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 shutdown_connections 2024-05-09T08:46:01.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.388+0000 7fd24c607640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24419a7a0 0x7fd24419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:01.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.388+0000 7fd24c607640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2441051c0 0x7fd244196420 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:01.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.388+0000 7fd24c607640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fd2300a2e70 0x7fd2300a5330 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:01.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.388+0000 7fd24c607640 1 --2- 172.21.15.28:0/2552578513 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2441000f0 0x7fd244195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:01.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.388+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 >> 172.21.15.28:0/2552578513 conn(0x7fd244076cc0 msgr2=0x7fd2441039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:01.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.388+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 shutdown_connections 2024-05-09T08:46:01.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.388+0000 7fd24c607640 1 -- 172.21.15.28:0/2552578513 wait complete. 2024-05-09T08:46:01.405 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-09T08:46:01.419 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-09T08:46:01.419 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-05-09T08:46:01.420 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-09T08:46:01.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:46:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.664+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/1172902621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a54100700 msgr2=0x7f5a54104270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.664+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/1172902621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a54100700 0x7f5a54104270 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c01acb0 tx=0x7f5a4c03b900 comp rx=0 tx=0).stop 2024-05-09T08:46:01.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.674+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/1172902621 shutdown_connections 2024-05-09T08:46:01.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.674+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/1172902621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a54100700 0x7f5a54104270 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:01.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.674+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/1172902621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a540ffd40 0x7f5a541001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:01.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.674+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/1172902621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a54106810 0x7f5a54106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:01.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.674+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/1172902621 >> 172.21.15.28:0/1172902621 conn(0x7f5a54076cd0 msgr2=0x7f5a540fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:01.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.677+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/1172902621 shutdown_connections 2024-05-09T08:46:01.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.677+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/1172902621 wait complete. 2024-05-09T08:46:01.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.678+0000 7f5a5b2ba640 1 Processor -- start 2024-05-09T08:46:01.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.678+0000 7f5a5b2ba640 1 -- start start 2024-05-09T08:46:01.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.679+0000 7f5a5b2ba640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a540ffd40 0x7f5a54072260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:01.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.679+0000 7f5a5b2ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a54100700 0x7f5a540727a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:01.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.679+0000 7f5a5b2ba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a54106810 0x7f5a54115570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.679+0000 7f5a5b2ba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54109a10 con 0x7f5a54100700 2024-05-09T08:46:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.679+0000 7f5a5b2ba640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54109890 con 0x7f5a540ffd40 2024-05-09T08:46:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.679+0000 7f5a5b2ba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54109b90 con 0x7f5a54106810 2024-05-09T08:46:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.679+0000 7f5a59056640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a540ffd40 0x7f5a54072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.681+0000 7f5a59857640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a54106810 0x7f5a54115570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.681+0000 7f5a59857640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a54106810 0x7f5a54115570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:41924/0 (socket says 172.21.15.28:41924) 2024-05-09T08:46:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.681+0000 7f5a59857640 1 -- 172.21.15.28:0/2717918520 learned_addr learned my addr 172.21.15.28:0/2717918520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:46:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.681+0000 7f5a59857640 1 -- 172.21.15.28:0/2717918520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a540ffd40 msgr2=0x7f5a54072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.681+0000 7f5a59857640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a540ffd40 0x7f5a54072260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.681+0000 7f5a59857640 1 -- 172.21.15.28:0/2717918520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a54100700 msgr2=0x7f5a540727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.681+0000 7f5a59857640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a54100700 0x7f5a540727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.681+0000 7f5a59857640 1 -- 172.21.15.28:0/2717918520 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5410d2a0 con 0x7f5a54106810 2024-05-09T08:46:01.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.682+0000 7f5a59857640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a54106810 0x7f5a54115570 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c03be70 tx=0x7f5a4c0406c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:01.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.682+0000 7f5a4a7fc640 1 -- 172.21.15.28:0/2717918520 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c045070 con 0x7f5a54106810 2024-05-09T08:46:01.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.682+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5410d530 con 0x7f5a54106810 2024-05-09T08:46:01.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.682+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a5410daf0 con 0x7f5a54106810 2024-05-09T08:46:01.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.682+0000 7f5a4a7fc640 1 -- 172.21.15.28:0/2717918520 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a4c003e50 con 0x7f5a54106810 2024-05-09T08:46:01.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.682+0000 7f5a4a7fc640 1 -- 172.21.15.28:0/2717918520 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c0042a0 con 0x7f5a54106810 2024-05-09T08:46:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.685+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a38005290 con 0x7f5a54106810 2024-05-09T08:46:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.686+0000 7f5a4a7fc640 1 -- 172.21.15.28:0/2717918520 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5a4c048020 con 0x7f5a54106810 2024-05-09T08:46:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.687+0000 7f5a4a7fc640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a400a2e40 0x7f5a400a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.687+0000 7f5a4a7fc640 1 -- 172.21.15.28:0/2717918520 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9787+0+0 (secure 0 0 0) 0x7f5a4c1013a0 con 0x7f5a54106810 2024-05-09T08:46:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.687+0000 7f5a59056640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a400a2e40 0x7f5a400a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.688+0000 7f5a59056640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a400a2e40 0x7f5a400a5300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5a54078250 tx=0x7f5a44008040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:01.696+0000 7f5a4a7fc640 1 -- 172.21.15.28:0/2717918520 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5a4c0cdb80 con 0x7f5a54106810 2024-05-09T08:46:02.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.064+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a38005850 con 0x7f5a54106810 2024-05-09T08:46:02.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.078+0000 7f5a4a7fc640 1 -- 172.21.15.28:0/2717918520 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+25770 (secure 0 0 0) 0x7f5a4c0cd480 con 0x7f5a54106810 2024-05-09T08:46:02.078 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:46:02.078 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":134,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:45:55.737972+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":132,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":132,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3330063293":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3745037139":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/2130261041":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/273816117":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":18,"length":4},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":27,"length":1},{"begin":29,"length":1},{"begin":31,"length":1},{"begin":33,"length":1}]},{"pool":4,"snaps":[{"begin":16,"length":1},{"begin":18,"length":4},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":27,"length":1},{"begin":29,"length":1},{"begin":31,"length":1},{"begin":33,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":14,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:46:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.081+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a400a2e40 msgr2=0x7f5a400a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.081+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a400a2e40 0x7f5a400a5300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5a54078250 tx=0x7f5a44008040 comp rx=0 tx=0).stop 2024-05-09T08:46:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.081+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a54106810 msgr2=0x7f5a54115570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.081+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a54106810 0x7f5a54115570 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c03be70 tx=0x7f5a4c0406c0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.082+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 shutdown_connections 2024-05-09T08:46:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.082+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5a400a2e40 0x7f5a400a5300 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.082+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a54106810 0x7f5a54115570 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.082+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a54100700 0x7f5a540727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.082+0000 7f5a5b2ba640 1 --2- 172.21.15.28:0/2717918520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a540ffd40 0x7f5a54072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.082+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 >> 172.21.15.28:0/2717918520 conn(0x7f5a54076cd0 msgr2=0x7f5a54101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.082+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 shutdown_connections 2024-05-09T08:46:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.082+0000 7f5a5b2ba640 1 -- 172.21.15.28:0/2717918520 wait complete. 2024-05-09T08:46:02.103 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 132, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 132, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:46:02.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:46:02.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:01 smithi028 ceph-mon[30119]: from='client.26879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:02.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:01 smithi028 ceph-mon[30119]: pgmap v634: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-05-09T08:46:02.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:01 smithi028 ceph-mon[30119]: from='client.26885 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:02.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.309+0000 7f07dfe2a640 1 -- 172.21.15.28:0/3032761605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d8105bc0 msgr2=0x7f07d8106050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:02.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.309+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/3032761605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d8105bc0 0x7f07d8106050 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009900 tx=0x7f07d402a560 comp rx=0 tx=0).stop 2024-05-09T08:46:02.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.311+0000 7f07dfe2a640 1 -- 172.21.15.28:0/3032761605 shutdown_connections 2024-05-09T08:46:02.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.311+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/3032761605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d8105bc0 0x7f07d8106050 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.311+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/3032761605 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07d81011c0 0x7f07d8105680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.311+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/3032761605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07d81008a0 0x7f07d8100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.311+0000 7f07dfe2a640 1 -- 172.21.15.28:0/3032761605 >> 172.21.15.28:0/3032761605 conn(0x7f07d8077830 msgr2=0x7f07d8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:02.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.315+0000 7f07dfe2a640 1 -- 172.21.15.28:0/3032761605 shutdown_connections 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.318+0000 7f07dfe2a640 1 -- 172.21.15.28:0/3032761605 wait complete. 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.318+0000 7f07dfe2a640 1 Processor -- start 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.319+0000 7f07dfe2a640 1 -- start start 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.319+0000 7f07dfe2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d81008a0 0x7f07d8072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.319+0000 7f07dfe2a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07d81011c0 0x7f07d80734e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.319+0000 7f07dfe2a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07d8105bc0 0x7f07d806c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.319+0000 7f07dfe2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07d8118250 con 0x7f07d81008a0 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.319+0000 7f07dfe2a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f07d81180d0 con 0x7f07d81011c0 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.319+0000 7f07dfe2a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f07d81183d0 con 0x7f07d8105bc0 2024-05-09T08:46:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.320+0000 7f07de3c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07d8105bc0 0x7f07d806c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.322+0000 7f07de3c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07d8105bc0 0x7f07d806c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:41936/0 (socket says 172.21.15.28:41936) 2024-05-09T08:46:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.322+0000 7f07de3c7640 1 -- 172.21.15.28:0/2524453898 learned_addr learned my addr 172.21.15.28:0/2524453898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:46:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.323+0000 7f07dd3c5640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07d81011c0 0x7f07d80734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.323+0000 7f07ddbc6640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d81008a0 0x7f07d8072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:02.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.327+0000 7f07de3c7640 1 -- 172.21.15.28:0/2524453898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07d81011c0 msgr2=0x7f07d80734e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:02.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.327+0000 7f07de3c7640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07d81011c0 0x7f07d80734e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.327+0000 7f07de3c7640 1 -- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d81008a0 msgr2=0x7f07d8072fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:02.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.327+0000 7f07de3c7640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d81008a0 0x7f07d8072fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.327+0000 7f07de3c7640 1 -- 172.21.15.28:0/2524453898 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d806cac0 con 0x7f07d8105bc0 2024-05-09T08:46:02.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.327+0000 7f07de3c7640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07d8105bc0 0x7f07d806c200 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f07d402aad0 tx=0x7f07d4003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:02.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.328+0000 7f07ceffd640 1 -- 172.21.15.28:0/2524453898 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d403f070 con 0x7f07d8105bc0 2024-05-09T08:46:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.328+0000 7f07ceffd640 1 -- 172.21.15.28:0/2524453898 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07d4031c90 con 0x7f07d8105bc0 2024-05-09T08:46:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.328+0000 7f07ceffd640 1 -- 172.21.15.28:0/2524453898 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4035030 con 0x7f07d8105bc0 2024-05-09T08:46:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.329+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d806cd50 con 0x7f07d8105bc0 2024-05-09T08:46:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.330+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d8070ca0 con 0x7f07d8105bc0 2024-05-09T08:46:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.332+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d80691c0 con 0x7f07d8105bc0 2024-05-09T08:46:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.339+0000 7f07ceffd640 1 -- 172.21.15.28:0/2524453898 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f07d4042070 con 0x7f07d8105bc0 2024-05-09T08:46:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.340+0000 7f07ceffd640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07c40a2dc0 0x7f07c40a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.340+0000 7f07ceffd640 1 -- 172.21.15.28:0/2524453898 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9787+0+0 (secure 0 0 0) 0x7f07d40f0030 con 0x7f07d8105bc0 2024-05-09T08:46:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.344+0000 7f07ddbc6640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07c40a2dc0 0x7f07c40a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.344+0000 7f07ddbc6640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07c40a2dc0 0x7f07c40a5280 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f07d0005fd0 tx=0x7f07d0009680 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:02.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:01 smithi165 ceph-mon[34540]: from='client.26879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:02.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:01 smithi165 ceph-mon[34540]: pgmap v634: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-05-09T08:46:02.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:01 smithi165 ceph-mon[34540]: from='client.26885 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:02.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:01 smithi184 ceph-mon[34509]: from='client.26879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:02.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:01 smithi184 ceph-mon[34509]: pgmap v634: 129 pgs: 58 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 51 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 131 op/s 2024-05-09T08:46:02.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:01 smithi184 ceph-mon[34509]: from='client.26885 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:46:02.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.356+0000 7f07ceffd640 1 -- 172.21.15.28:0/2524453898 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f07d4049020 con 0x7f07d8105bc0 2024-05-09T08:46:02.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.728+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f07d8070fc0 con 0x7f07d8105bc0 2024-05-09T08:46:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.730+0000 7f07ceffd640 1 -- 172.21.15.28:0/2524453898 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) ==== 93+0+10 (secure 0 0 0) 0x7f07d40bc700 con 0x7f07d8105bc0 2024-05-09T08:46:02.729 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:46:02.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07c40a2dc0 msgr2=0x7f07c40a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:02.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07c40a2dc0 0x7f07c40a5280 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f07d0005fd0 tx=0x7f07d0009680 comp rx=0 tx=0).stop 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07d8105bc0 msgr2=0x7f07d806c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07d8105bc0 0x7f07d806c200 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f07d402aad0 tx=0x7f07d4003b90 comp rx=0 tx=0).stop 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 shutdown_connections 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f07c40a2dc0 0x7f07c40a5280 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07d8105bc0 0x7f07d806c200 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07d81011c0 0x7f07d80734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 --2- 172.21.15.28:0/2524453898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07d81008a0 0x7f07d8072fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 >> 172.21.15.28:0/2524453898 conn(0x7f07d8077830 msgr2=0x7f07d80fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 shutdown_connections 2024-05-09T08:46:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.733+0000 7f07dfe2a640 1 -- 172.21.15.28:0/2524453898 wait complete. 2024-05-09T08:46:02.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:46:02.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.955+0000 7f57fddd6640 1 -- 172.21.15.28:0/404415871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57f00957e0 msgr2=0x7f57f00a1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:02.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.955+0000 7f57fddd6640 1 --2- 172.21.15.28:0/404415871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57f00957e0 0x7f57f00a1ab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f57e4010910 tx=0x7f57e402e310 comp rx=0 tx=0).stop 2024-05-09T08:46:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.958+0000 7f57fddd6640 1 -- 172.21.15.28:0/404415871 shutdown_connections 2024-05-09T08:46:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.958+0000 7f57fddd6640 1 --2- 172.21.15.28:0/404415871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57f00957e0 0x7f57f00a1ab0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.958+0000 7f57fddd6640 1 --2- 172.21.15.28:0/404415871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0094e40 0x7f57f00952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.958+0000 7f57fddd6640 1 --2- 172.21.15.28:0/404415871 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57f0099bb0 0x7f57f0099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.958+0000 7f57fddd6640 1 -- 172.21.15.28:0/404415871 >> 172.21.15.28:0/404415871 conn(0x7f57f000a730 msgr2=0x7f57f000ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:02.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.961+0000 7f57fddd6640 1 -- 172.21.15.28:0/404415871 shutdown_connections 2024-05-09T08:46:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.961+0000 7f57fddd6640 1 -- 172.21.15.28:0/404415871 wait complete. 2024-05-09T08:46:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.962+0000 7f57fddd6640 1 Processor -- start 2024-05-09T08:46:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.962+0000 7f57fddd6640 1 -- start start 2024-05-09T08:46:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.963+0000 7f57fddd6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57f0094e40 0x7f57f01420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.963+0000 7f57fddd6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57f00957e0 0x7f57f01425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.963+0000 7f57fddd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0099bb0 0x7f57f013b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.963+0000 7f57fddd6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57f00a3ef0 con 0x7f57f0099bb0 2024-05-09T08:46:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.963+0000 7f57fddd6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f57f00a3d70 con 0x7f57f0094e40 2024-05-09T08:46:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.963+0000 7f57fddd6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f57f00a4070 con 0x7f57f00957e0 2024-05-09T08:46:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.963+0000 7f57fd5d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0099bb0 0x7f57f013b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.963+0000 7f57fd5d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0099bb0 0x7f57f013b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56694/0 (socket says 172.21.15.28:56694) 2024-05-09T08:46:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.963+0000 7f57fd5d5640 1 -- 172.21.15.28:0/2043731795 learned_addr learned my addr 172.21.15.28:0/2043731795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:46:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.964+0000 7f57fd5d5640 1 -- 172.21.15.28:0/2043731795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57f00957e0 msgr2=0x7f57f01425f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:46:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.964+0000 7f57fd5d5640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57f00957e0 0x7f57f01425f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.964+0000 7f57fd5d5640 1 -- 172.21.15.28:0/2043731795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57f0094e40 msgr2=0x7f57f01420b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:46:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.964+0000 7f57fd5d5640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57f0094e40 0x7f57f01420b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.964+0000 7f57fd5d5640 1 -- 172.21.15.28:0/2043731795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f013bbd0 con 0x7f57f0099bb0 2024-05-09T08:46:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.964+0000 7f57fd5d5640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0099bb0 0x7f57f013b310 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f57e4039040 tx=0x7f57e40099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.967+0000 7f57f5ffb640 1 -- 172.21.15.28:0/2043731795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e403b070 con 0x7f57f0099bb0 2024-05-09T08:46:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.967+0000 7f57f5ffb640 1 -- 172.21.15.28:0/2043731795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57e4010270 con 0x7f57f0099bb0 2024-05-09T08:46:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.967+0000 7f57f5ffb640 1 -- 172.21.15.28:0/2043731795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e4040920 con 0x7f57f0099bb0 2024-05-09T08:46:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.968+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f013be60 con 0x7f57f0099bb0 2024-05-09T08:46:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.968+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f013fdb0 con 0x7f57f0099bb0 2024-05-09T08:46:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.973+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f00962e0 con 0x7f57f0099bb0 2024-05-09T08:46:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.981+0000 7f57f5ffb640 1 -- 172.21.15.28:0/2043731795 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f57e4040ac0 con 0x7f57f0099bb0 2024-05-09T08:46:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.982+0000 7f57f5ffb640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f57e80a2ed0 0x7f57e80a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.982+0000 7f57f5ffb640 1 -- 172.21.15.28:0/2043731795 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9787+0+0 (secure 0 0 0) 0x7f57e40ef760 con 0x7f57f0099bb0 2024-05-09T08:46:02.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.989+0000 7f57fcdd4640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f57e80a2ed0 0x7f57e80a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:02.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:02.989+0000 7f57f5ffb640 1 -- 172.21.15.28:0/2043731795 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f57e4045060 con 0x7f57f0099bb0 2024-05-09T08:46:03.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.003+0000 7f57fcdd4640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f57e80a2ed0 0x7f57e80a5390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00a860 tx=0x7f57ec005b00 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:03.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:02 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2717918520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:46:03.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:02 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2524453898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:46:03.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:02 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2717918520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:46:03.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:02 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2524453898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:46:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.354+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f57f0140860 con 0x7f57f0099bb0 2024-05-09T08:46:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.355+0000 7f57f5ffb640 1 -- 172.21.15.28:0/2043731795 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) ==== 104+0+11 (secure 0 0 0) 0x7f57e40353a0 con 0x7f57f0099bb0 2024-05-09T08:46:03.355 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:46:03.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:02 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2717918520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:46:03.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:02 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2524453898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:46:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.359+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f57e80a2ed0 msgr2=0x7f57e80a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.359+0000 7f57fddd6640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f57e80a2ed0 0x7f57e80a5390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00a860 tx=0x7f57ec005b00 comp rx=0 tx=0).stop 2024-05-09T08:46:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.359+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0099bb0 msgr2=0x7f57f013b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.359+0000 7f57fddd6640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0099bb0 0x7f57f013b310 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f57e4039040 tx=0x7f57e40099e0 comp rx=0 tx=0).stop 2024-05-09T08:46:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.359+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 shutdown_connections 2024-05-09T08:46:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.359+0000 7f57fddd6640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0099bb0 0x7f57f013b310 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.359+0000 7f57fddd6640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f57e80a2ed0 0x7f57e80a5390 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00a860 tx=0x7f57ec005b00 comp rx=0 tx=0).stop 2024-05-09T08:46:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.359+0000 7f57fddd6640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57f00957e0 0x7f57f01425f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.359+0000 7f57fddd6640 1 --2- 172.21.15.28:0/2043731795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57f0094e40 0x7f57f01420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.360+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 >> 172.21.15.28:0/2043731795 conn(0x7f57f000a730 msgr2=0x7f57f0092a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.360+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 shutdown_connections 2024-05-09T08:46:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.360+0000 7f57fddd6640 1 -- 172.21.15.28:0/2043731795 wait complete. 2024-05-09T08:46:03.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:46:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.581+0000 7f794d9dc640 1 -- 172.21.15.28:0/2789499068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 msgr2=0x7f7948101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.581+0000 7f794d9dc640 1 --2- 172.21.15.28:0/2789499068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 0x7f7948101d00 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f793c01a940 tx=0x7f793c03b490 comp rx=0 tx=0).stop 2024-05-09T08:46:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.585+0000 7f794d9dc640 1 -- 172.21.15.28:0/2789499068 shutdown_connections 2024-05-09T08:46:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.585+0000 7f794d9dc640 1 --2- 172.21.15.28:0/2789499068 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7948102240 0x7f794810e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.585+0000 7f794d9dc640 1 --2- 172.21.15.28:0/2789499068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 0x7f7948101d00 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.585+0000 7f794d9dc640 1 --2- 172.21.15.28:0/2789499068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7948107890 0x7f7948107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.585+0000 7f794d9dc640 1 -- 172.21.15.28:0/2789499068 >> 172.21.15.28:0/2789499068 conn(0x7f79480fd390 msgr2=0x7f79480ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.586+0000 7f794d9dc640 1 -- 172.21.15.28:0/2789499068 shutdown_connections 2024-05-09T08:46:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.588+0000 7f794d9dc640 1 -- 172.21.15.28:0/2789499068 wait complete. 2024-05-09T08:46:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.589+0000 7f794d9dc640 1 Processor -- start 2024-05-09T08:46:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.592+0000 7f794d9dc640 1 -- start start 2024-05-09T08:46:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.593+0000 7f794d9dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 0x7f7948073cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.593+0000 7f794d9dc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7948102240 0x7f794806ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.593+0000 7f794d9dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7948107890 0x7f794806e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.593+0000 7f794d9dc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f794806c6b0 con 0x7f7948101880 2024-05-09T08:46:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.593+0000 7f794d9dc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f794806c530 con 0x7f7948102240 2024-05-09T08:46:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.593+0000 7f794d9dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f794806c830 con 0x7f7948107890 2024-05-09T08:46:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.595+0000 7f7946ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 0x7f7948073cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.595+0000 7f7946ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 0x7f7948073cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56700/0 (socket says 172.21.15.28:56700) 2024-05-09T08:46:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.595+0000 7f7946ffd640 1 -- 172.21.15.28:0/1036873142 learned_addr learned my addr 172.21.15.28:0/1036873142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:46:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.595+0000 7f79467fc640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7948102240 0x7f794806ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.598+0000 7f79477fe640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7948107890 0x7f794806e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.600+0000 7f7946ffd640 1 -- 172.21.15.28:0/1036873142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7948107890 msgr2=0x7f794806e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.600+0000 7f7946ffd640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7948107890 0x7f794806e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.600+0000 7f7946ffd640 1 -- 172.21.15.28:0/1036873142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7948102240 msgr2=0x7f794806ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.600+0000 7f7946ffd640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7948102240 0x7f794806ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.600+0000 7f7946ffd640 1 -- 172.21.15.28:0/1036873142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794806e8c0 con 0x7f7948101880 2024-05-09T08:46:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.604+0000 7f7946ffd640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 0x7f7948073cb0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f794000db80 tx=0x7f794000b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.608+0000 7f794c9da640 1 -- 172.21.15.28:0/1036873142 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940017070 con 0x7f7948101880 2024-05-09T08:46:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.608+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794806ebb0 con 0x7f7948101880 2024-05-09T08:46:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.608+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79481ab390 con 0x7f7948101880 2024-05-09T08:46:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.608+0000 7f794c9da640 1 -- 172.21.15.28:0/1036873142 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7940003b70 con 0x7f7948101880 2024-05-09T08:46:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.608+0000 7f794c9da640 1 -- 172.21.15.28:0/1036873142 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79400106c0 con 0x7f7948101880 2024-05-09T08:46:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.613+0000 7f794c9da640 1 -- 172.21.15.28:0/1036873142 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7940010860 con 0x7f7948101880 2024-05-09T08:46:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.614+0000 7f794c9da640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79300a2f10 0x7f79300a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.615+0000 7f794c9da640 1 -- 172.21.15.28:0/1036873142 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9787+0+0 (secure 0 0 0) 0x7f79400cf5d0 con 0x7f7948101880 2024-05-09T08:46:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.616+0000 7f79467fc640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79300a2f10 0x7f79300a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:03.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.617+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7934005290 con 0x7f7948101880 2024-05-09T08:46:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.623+0000 7f79467fc640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79300a2f10 0x7f79300a53d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f793c000c00 tx=0x7f793c01a500 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:03.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.630+0000 7f794c9da640 1 -- 172.21.15.28:0/1036873142 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7940020070 con 0x7f7948101880 2024-05-09T08:46:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.998+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7934005850 con 0x7f7948101880 2024-05-09T08:46:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:03.999+0000 7f794c9da640 1 -- 172.21.15.28:0/1036873142 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) ==== 100+0+11 (secure 0 0 0) 0x7f794009bca0 con 0x7f7948101880 2024-05-09T08:46:03.999 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:46:04.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.003+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79300a2f10 msgr2=0x7f79300a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.003+0000 7f794d9dc640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79300a2f10 0x7f79300a53d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f793c000c00 tx=0x7f793c01a500 comp rx=0 tx=0).stop 2024-05-09T08:46:04.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.003+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 msgr2=0x7f7948073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.003+0000 7f794d9dc640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 0x7f7948073cb0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f794000db80 tx=0x7f794000b630 comp rx=0 tx=0).stop 2024-05-09T08:46:04.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.004+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 shutdown_connections 2024-05-09T08:46:04.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.004+0000 7f794d9dc640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f79300a2f10 0x7f79300a53d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.004+0000 7f794d9dc640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7948107890 0x7f794806e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.004+0000 7f794d9dc640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7948102240 0x7f794806ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.004+0000 7f794d9dc640 1 --2- 172.21.15.28:0/1036873142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7948101880 0x7f7948073cb0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.004+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 >> 172.21.15.28:0/1036873142 conn(0x7f79480fd390 msgr2=0x7f79480fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:04.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.004+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 shutdown_connections 2024-05-09T08:46:04.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.004+0000 7f794d9dc640 1 -- 172.21.15.28:0/1036873142 wait complete. 2024-05-09T08:46:04.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:46:04.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:03 smithi028 ceph-mon[30119]: pgmap v635: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 869 KiB/s wr, 148 op/s 2024-05-09T08:46:04.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:03 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2043731795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:46:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.210+0000 7faa7e76a640 1 -- 172.21.15.28:0/4240320320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa781010c0 msgr2=0x7faa7810d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.210+0000 7faa7e76a640 1 --2- 172.21.15.28:0/4240320320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa781010c0 0x7faa7810d5d0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7faa68009930 tx=0x7faa6802a440 comp rx=0 tx=0).stop 2024-05-09T08:46:04.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.217+0000 7faa7e76a640 1 -- 172.21.15.28:0/4240320320 shutdown_connections 2024-05-09T08:46:04.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.217+0000 7faa7e76a640 1 --2- 172.21.15.28:0/4240320320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa781010c0 0x7faa7810d5d0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.217+0000 7faa7e76a640 1 --2- 172.21.15.28:0/4240320320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa78100700 0x7faa78100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.217+0000 7faa7e76a640 1 --2- 172.21.15.28:0/4240320320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa78106710 0x7faa78106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.217+0000 7faa7e76a640 1 -- 172.21.15.28:0/4240320320 >> 172.21.15.28:0/4240320320 conn(0x7faa78077830 msgr2=0x7faa78077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:04.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.223+0000 7faa7e76a640 1 -- 172.21.15.28:0/4240320320 shutdown_connections 2024-05-09T08:46:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.227+0000 7faa7e76a640 1 -- 172.21.15.28:0/4240320320 wait complete. 2024-05-09T08:46:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.227+0000 7faa7e76a640 1 Processor -- start 2024-05-09T08:46:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.229+0000 7faa7e76a640 1 -- start start 2024-05-09T08:46:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.229+0000 7faa7e76a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa78100700 0x7faa78116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.229+0000 7faa7e76a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa781010c0 0x7faa781101c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.229+0000 7faa7e76a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa78106710 0x7faa78110700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.229+0000 7faa7e76a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa7810ebd0 con 0x7faa78100700 2024-05-09T08:46:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.229+0000 7faa7e76a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faa7810ea50 con 0x7faa781010c0 2024-05-09T08:46:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.229+0000 7faa7e76a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faa7810ed50 con 0x7faa78106710 2024-05-09T08:46:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.230+0000 7faa77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa78100700 0x7faa78116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.230+0000 7faa77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa78100700 0x7faa78116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56722/0 (socket says 172.21.15.28:56722) 2024-05-09T08:46:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.230+0000 7faa77fff640 1 -- 172.21.15.28:0/675403243 learned_addr learned my addr 172.21.15.28:0/675403243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:46:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.232+0000 7faa7cd07640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa78106710 0x7faa78110700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.232+0000 7faa77fff640 1 -- 172.21.15.28:0/675403243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa78106710 msgr2=0x7faa78110700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.232+0000 7faa77fff640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa78106710 0x7faa78110700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.232+0000 7faa77fff640 1 -- 172.21.15.28:0/675403243 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa781010c0 msgr2=0x7faa781101c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:46:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.232+0000 7faa77fff640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa781010c0 0x7faa781101c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.232+0000 7faa77fff640 1 -- 172.21.15.28:0/675403243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa78110e80 con 0x7faa78100700 2024-05-09T08:46:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.235+0000 7faa77fff640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa78100700 0x7faa78116150 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7faa7000cce0 tx=0x7faa7000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.236+0000 7faa757fa640 1 -- 172.21.15.28:0/675403243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70004b20 con 0x7faa78100700 2024-05-09T08:46:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.236+0000 7faa757fa640 1 -- 172.21.15.28:0/675403243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa70004cc0 con 0x7faa78100700 2024-05-09T08:46:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.236+0000 7faa757fa640 1 -- 172.21.15.28:0/675403243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7000f7d0 con 0x7faa78100700 2024-05-09T08:46:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.238+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa78071280 con 0x7faa78100700 2024-05-09T08:46:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.238+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa780716f0 con 0x7faa78100700 2024-05-09T08:46:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.239+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa78101e40 con 0x7faa78100700 2024-05-09T08:46:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.247+0000 7faa757fa640 1 -- 172.21.15.28:0/675403243 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7faa7000f970 con 0x7faa78100700 2024-05-09T08:46:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.248+0000 7faa757fa640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7faa600a2f00 0x7faa600a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.248+0000 7faa757fa640 1 -- 172.21.15.28:0/675403243 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9787+0+0 (secure 0 0 0) 0x7faa700d01e0 con 0x7faa78100700 2024-05-09T08:46:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.254+0000 7faa777fe640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7faa600a2f00 0x7faa600a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.256+0000 7faa757fa640 1 -- 172.21.15.28:0/675403243 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faa7001d020 con 0x7faa78100700 2024-05-09T08:46:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.272+0000 7faa777fe640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7faa600a2f00 0x7faa600a53c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faa78111980 tx=0x7faa6c005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:03 smithi165 ceph-mon[34540]: pgmap v635: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 869 KiB/s wr, 148 op/s 2024-05-09T08:46:04.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:03 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2043731795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:46:04.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:03 smithi184 ceph-mon[34509]: pgmap v635: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 869 KiB/s wr, 148 op/s 2024-05-09T08:46:04.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:03 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2043731795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:46:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.630+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7faa7802d1c0 con 0x7faa78100700 2024-05-09T08:46:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.639+0000 7faa757fa640 1 -- 172.21.15.28:0/675403243 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v134) ==== 103+0+11 (secure 0 0 0) 0x7faa70096e20 con 0x7faa78100700 2024-05-09T08:46:04.639 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:46:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7faa600a2f00 msgr2=0x7faa600a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7faa600a2f00 0x7faa600a53c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faa78111980 tx=0x7faa6c005f90 comp rx=0 tx=0).stop 2024-05-09T08:46:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa78100700 msgr2=0x7faa78116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa78100700 0x7faa78116150 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7faa7000cce0 tx=0x7faa7000a620 comp rx=0 tx=0).stop 2024-05-09T08:46:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 shutdown_connections 2024-05-09T08:46:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7faa600a2f00 0x7faa600a53c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faa78106710 0x7faa78110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa781010c0 0x7faa781101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 --2- 172.21.15.28:0/675403243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa78100700 0x7faa78116150 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 >> 172.21.15.28:0/675403243 conn(0x7faa78077830 msgr2=0x7faa78102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.643+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 shutdown_connections 2024-05-09T08:46:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.644+0000 7faa7e76a640 1 -- 172.21.15.28:0/675403243 wait complete. 2024-05-09T08:46:04.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-09T08:46:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.855+0000 7f2ca9529640 1 -- 172.21.15.28:0/2866399543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ca410e790 msgr2=0x7f2ca411d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.855+0000 7f2ca9529640 1 --2- 172.21.15.28:0/2866399543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ca410e790 0x7f2ca411d080 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009be0 tx=0x7f2c9c02a6e0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.856+0000 7f2ca9529640 1 -- 172.21.15.28:0/2866399543 shutdown_connections 2024-05-09T08:46:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.856+0000 7f2ca9529640 1 --2- 172.21.15.28:0/2866399543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ca410e790 0x7f2ca411d080 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.856+0000 7f2ca9529640 1 --2- 172.21.15.28:0/2866399543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca4110400 0x7f2ca410e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.856+0000 7f2ca9529640 1 --2- 172.21.15.28:0/2866399543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca410ff90 0x7f2ca410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.856+0000 7f2ca9529640 1 -- 172.21.15.28:0/2866399543 >> 172.21.15.28:0/2866399543 conn(0x7f2ca406c950 msgr2=0x7f2ca406cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:04.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.861+0000 7f2ca9529640 1 -- 172.21.15.28:0/2866399543 shutdown_connections 2024-05-09T08:46:04.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.862+0000 7f2ca9529640 1 -- 172.21.15.28:0/2866399543 wait complete. 2024-05-09T08:46:04.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.862+0000 7f2ca9529640 1 Processor -- start 2024-05-09T08:46:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.866+0000 7f2ca9529640 1 -- start start 2024-05-09T08:46:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.866+0000 7f2ca9529640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca410ff90 0x7f2ca411cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.866+0000 7f2ca9529640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4118e20 0x7f2ca4119280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.866+0000 7f2ca9529640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ca41197c0 0x7f2ca41aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.866+0000 7f2ca9529640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca41131c0 con 0x7f2ca4118e20 2024-05-09T08:46:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.866+0000 7f2ca9529640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca4113040 con 0x7f2ca410ff90 2024-05-09T08:46:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.866+0000 7f2ca9529640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca4113340 con 0x7f2ca41197c0 2024-05-09T08:46:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.868+0000 7f2ca37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ca41197c0 0x7f2ca41aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.868+0000 7f2ca27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4118e20 0x7f2ca4119280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.868+0000 7f2ca27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4118e20 0x7f2ca4119280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56320/0 (socket says 172.21.15.28:56320) 2024-05-09T08:46:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.868+0000 7f2ca27fc640 1 -- 172.21.15.28:0/1933428163 learned_addr learned my addr 172.21.15.28:0/1933428163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:46:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.868+0000 7f2ca2ffd640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca410ff90 0x7f2ca411cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.869+0000 7f2ca2ffd640 1 -- 172.21.15.28:0/1933428163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ca41197c0 msgr2=0x7f2ca41aadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.869+0000 7f2ca2ffd640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ca41197c0 0x7f2ca41aadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.869+0000 7f2ca2ffd640 1 -- 172.21.15.28:0/1933428163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4118e20 msgr2=0x7f2ca4119280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.869+0000 7f2ca2ffd640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4118e20 0x7f2ca4119280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.869+0000 7f2ca2ffd640 1 -- 172.21.15.28:0/1933428163 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca41ab2f0 con 0x7f2ca410ff90 2024-05-09T08:46:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.869+0000 7f2ca2ffd640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca410ff90 0x7f2ca411cd40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800b3f0 tx=0x7f2c9800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.870+0000 7f2c8bfff640 1 -- 172.21.15.28:0/1933428163 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98015020 con 0x7f2ca410ff90 2024-05-09T08:46:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.870+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca41ab4c0 con 0x7f2ca410ff90 2024-05-09T08:46:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.870+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca41aba00 con 0x7f2ca410ff90 2024-05-09T08:46:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.871+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca406dbe0 con 0x7f2ca410ff90 2024-05-09T08:46:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.872+0000 7f2c8bfff640 1 -- 172.21.15.28:0/1933428163 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c98011d10 con 0x7f2ca410ff90 2024-05-09T08:46:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.872+0000 7f2c8bfff640 1 -- 172.21.15.28:0/1933428163 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98010e40 con 0x7f2ca410ff90 2024-05-09T08:46:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.876+0000 7f2c8bfff640 1 -- 172.21.15.28:0/1933428163 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2c98018020 con 0x7f2ca410ff90 2024-05-09T08:46:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.876+0000 7f2c8bfff640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2c8c0a2e40 0x7f2c8c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.877+0000 7f2c8bfff640 1 -- 172.21.15.28:0/1933428163 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9787+0+0 (secure 0 0 0) 0x7f2c980cf310 con 0x7f2ca410ff90 2024-05-09T08:46:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.882+0000 7f2c8bfff640 1 -- 172.21.15.28:0/1933428163 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2c9801f020 con 0x7f2ca410ff90 2024-05-09T08:46:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.883+0000 7f2ca27fc640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2c8c0a2e40 0x7f2c8c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:04.884+0000 7f2ca27fc640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2c8c0a2e40 0x7f2c8c0a5300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2c940058c0 tx=0x7f2c9401a270 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1036873142' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:46:04.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/675403243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:46:05.252 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1036873142' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:46:05.252 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/675403243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:46:05.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1036873142' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:46:05.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/675403243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:46:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.350+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f2ca410d190 con 0x7f2ca410ff90 2024-05-09T08:46:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.375+0000 7f2c8bfff640 1 -- 172.21.15.28:0/1933428163 <== mon.2 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740897 (secure 0 0 0) 0x7f2c9809baf0 con 0x7f2ca410ff90 2024-05-09T08:46:05.386 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 25, 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 15528, 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-05-09T08:46:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 160415970 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 160415970 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/160415970", 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-05-09T08:26:17.965238+0000", 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540701547738038271, 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-05-09T08:46:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 15522, 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-05-09T08:46:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-05-09T08:46:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-05-09T08:46:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-05-09T08:46:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-05-09T08:46:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-05-09T08:46:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-05-09T08:46:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-05-09T08:46:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-05-09T08:46:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-05-09T08:46:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-05-09T08:46:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-05-09T08:46:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-05-09T08:46:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-05-09T08:46:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-05-09T08:46:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-05-09T08:46:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-05-09T08:46:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-05-09T08:46:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-05-09T08:46:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T08:46:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-05-09T08:46:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-05-09T08:46:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-05-09T08:46:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-05-09T08:46:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-05-09T08:46:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-05-09T08:46:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T08:46:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-05-09T08:46:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-05-09T08:46:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-05-09T08:46:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T08:46:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:46:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-05-09T08:46:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-05-09T08:46:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-09T08:46:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-09T08:46:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T08:46:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:46:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-05-09T08:46:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-05-09T08:46:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-05-09T08:46:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-05-09T08:46:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T08:46:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-05-09T08:46:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-05-09T08:46:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-05-09T08:46:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-05-09T08:46:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-05-09T08:46:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-05-09T08:46:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T08:46:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-05-09T08:46:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-05-09T08:46:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-05-09T08:46:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T08:46:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-05-09T08:46:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-05-09T08:46:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T08:46:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T08:46:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-05-09T08:46:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-05-09T08:46:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T08:46:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-05-09T08:46:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T08:46:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-05-09T08:46:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T08:46:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-05-09T08:46:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-05-09T08:46:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-05-09T08:46:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-05-09T08:46:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-05-09T08:46:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T08:46:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-05-09T08:46:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-05-09T08:46:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-05-09T08:46:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T08:46:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-05-09T08:46:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-05-09T08:46:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-05-09T08:46:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T08:46:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-05-09T08:46:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-05-09T08:46:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-05-09T08:46:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T08:46:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-05-09T08:46:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-05-09T08:46:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T08:46:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-05-09T08:46:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-05-09T08:46:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-05-09T08:46:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T08:46:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-05-09T08:46:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-05-09T08:46:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T08:46:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-05-09T08:46:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-05-09T08:46:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-05-09T08:46:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T08:46:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-05-09T08:46:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-05-09T08:46:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-05-09T08:46:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T08:46:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-05-09T08:46:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-05-09T08:46:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-05-09T08:46:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T08:46:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T08:46:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-05-09T08:46:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-05-09T08:46:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-05-09T08:46:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T08:46:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T08:46:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T08:46:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-05-09T08:46:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-05-09T08:46:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T08:46:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T08:46:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-05-09T08:46:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-05-09T08:46:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T08:46:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T08:46:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-05-09T08:46:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T08:46:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T08:46:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T08:46:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T08:46:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T08:46:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T08:46:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T08:46:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T08:46:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T08:46:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T08:46:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T08:46:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T08:46:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T08:46:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-09T08:46:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-09T08:46:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-09T08:46:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-05-09T08:46:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-05-09T08:46:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-05-09T08:46:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-05-09T08:46:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-09T08:46:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-05-09T08:46:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T08:46:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-05-09T08:46:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T08:46:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-09T08:46:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-05-09T08:46:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T08:46:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-05-09T08:46:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25081, 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.864 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-05-09T08:46:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-05-09T08:46:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-05-09T08:46:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-05-09T08:46:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-09T08:46:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T08:46:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-05-09T08:46:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T08:46:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T08:46:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-05-09T08:46:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-05-09T08:46:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-05-09T08:46:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-09T08:46:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-05-09T08:46:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-05-09T08:46:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T08:46:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-05-09T08:46:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-05-09T08:46:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-05-09T08:46:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-05-09T08:46:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-05-09T08:46:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-05-09T08:46:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-05-09T08:46:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T08:46:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T08:46:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-09T08:46:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-05-09T08:46:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-05-09T08:46:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-05-09T08:46:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-05-09T08:46:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:46:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T08:46:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T08:46:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T08:46:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-05-09T08:46:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T08:46:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-05-09T08:46:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-05-09T08:46:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-05-09T08:46:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-05-09T08:46:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-05-09T08:46:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-09T08:46:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T08:46:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-05-09T08:46:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-05-09T08:46:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-05-09T08:46:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-05-09T08:46:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-09T08:46:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T08:46:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T08:46:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T08:46:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-09T08:46:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-05-09T08:46:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "restful", 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_schedule" 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-05-09T08:46:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-05-09T08:46:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-05-09T08:46:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-05-09T08:46:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-05-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T08:46:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:46:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:46:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-05-09T08:46:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-05-09T08:46:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T08:46:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-05-09T08:46:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T08:46:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-05-09T08:46:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-05-09T08:46:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-05-09T08:46:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T08:46:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-05-09T08:46:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T08:46:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-05-09T08:46:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T08:46:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-05-09T08:46:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-05-09T08:46:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T08:46:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T08:46:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T08:46:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T08:46:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T08:46:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T08:46:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-05-09T08:46:06.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T08:46:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-05-09T08:46:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T08:46:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-09T08:46:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-09T08:46:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-05-09T08:46:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-05-09T08:46:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:46:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-05-09T08:46:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-05-09T08:46:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-05-09T08:46:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-05-09T08:46:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-05-09T08:46:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-05-09T08:46:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-05-09T08:46:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T08:46:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-09T08:46:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-05-09T08:46:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T08:46:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-05-09T08:46:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-05-09T08:46:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-09T08:46:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-05-09T08:46:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-05-09T08:46:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-05-09T08:46:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T08:46:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-05-09T08:46:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T08:46:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-05-09T08:46:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-05-09T08:46:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-09T08:46:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-05-09T08:46:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-09T08:46:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-09T08:46:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-09T08:46:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-09T08:46:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-05-09T08:46:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-05-09T08:46:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-05-09T08:46:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-09T08:46:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-09T08:46:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 94, 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2987503874 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-09T08:46:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 99368723 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-09T08:46:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3521326588 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2148779990 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 163390625 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1921411598 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-09T08:46:06.810 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:46:06.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:05 smithi184 ceph-mon[34509]: pgmap v636: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 907 KiB/s wr, 160 op/s 2024-05-09T08:46:06.812 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:05 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1933428163' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T08:46:06.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:05 smithi165 ceph-mon[34540]: pgmap v636: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 907 KiB/s wr, 160 op/s 2024-05-09T08:46:06.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:05 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1933428163' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T08:46:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2c8c0a2e40 msgr2=0x7f2c8c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2c8c0a2e40 0x7f2c8c0a5300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2c940058c0 tx=0x7f2c9401a270 comp rx=0 tx=0).stop 2024-05-09T08:46:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca410ff90 msgr2=0x7f2ca411cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca410ff90 0x7f2ca411cd40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800b3f0 tx=0x7f2c9800b8b0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 shutdown_connections 2024-05-09T08:46:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f2c8c0a2e40 0x7f2c8c0a5300 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ca41197c0 0x7f2ca41aadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4118e20 0x7f2ca4119280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 --2- 172.21.15.28:0/1933428163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca410ff90 0x7f2ca411cd40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.383+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 >> 172.21.15.28:0/1933428163 conn(0x7f2ca406c950 msgr2=0x7f2ca40710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:06.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.384+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 shutdown_connections 2024-05-09T08:46:06.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:46:05.384+0000 7f2ca9529640 1 -- 172.21.15.28:0/1933428163 wait complete. 2024-05-09T08:46:06.853 DEBUG:tasks.check_counter:Getting stats from a 2024-05-09T08:46:06.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-09T08:46:06.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:05 smithi028 ceph-mon[30119]: pgmap v636: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 907 KiB/s wr, 160 op/s 2024-05-09T08:46:06.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:05 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1933428163' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T08:46:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:46:08.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:07 smithi028 ceph-mon[30119]: pgmap v637: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 846 KiB/s wr, 159 op/s 2024-05-09T08:46:08.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:07 smithi165 ceph-mon[34540]: pgmap v637: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 846 KiB/s wr, 159 op/s 2024-05-09T08:46:08.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:07 smithi184 ceph-mon[34509]: pgmap v637: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 846 KiB/s wr, 159 op/s 2024-05-09T08:46:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:09 smithi028 ceph-mon[30119]: pgmap v638: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 469 KiB/s wr, 135 op/s 2024-05-09T08:46:10.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:09 smithi028 ceph-mon[30119]: osdmap e135: 12 total, 12 up, 12 in 2024-05-09T08:46:10.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:09 smithi165 ceph-mon[34540]: pgmap v638: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 469 KiB/s wr, 135 op/s 2024-05-09T08:46:10.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:09 smithi165 ceph-mon[34540]: osdmap e135: 12 total, 12 up, 12 in 2024-05-09T08:46:10.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:09 smithi184 ceph-mon[34509]: pgmap v638: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 469 KiB/s wr, 135 op/s 2024-05-09T08:46:10.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:09 smithi184 ceph-mon[34509]: osdmap e135: 12 total, 12 up, 12 in 2024-05-09T08:46:11.071 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2552, 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2504, 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 69540464, 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 896660, 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.390777105, 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.093368937, 2024-05-09T08:46:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.158467070, 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.131520658, 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2503, 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.276607131, 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000110510 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1789152, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 230768 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1355, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1626, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 143136357, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 1074068, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 9, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 7, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.296231901, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.047951469, 2024-05-09T08:46:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.178919779, 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.054226613, 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1623, 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.168549688, 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000103850 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 644064 2024-05-09T08:46:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 802, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 802, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 134448625, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 254213, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.164281516, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.022599562, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.111813199, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.024162021, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 801, 2024-05-09T08:46:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.060050707, 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000074969 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-05-09T08:46:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-05-09T08:46:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-05-09T08:46:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6928, 2024-05-09T08:46:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.205470950, 2024-05-09T08:46:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000029658 2024-05-09T08:46:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-05-09T08:46:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 39644, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 5768, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 33876, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-05-09T08:46:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 31403, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 1, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-05-09T08:46:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:46:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-05-09T08:46:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-05-09T08:46:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 96234, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 129, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 9, 2024-05-09T08:46:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 31881079, 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 348287721, 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 348279957, 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 96234 2024-05-09T08:46:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 12695, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 85182, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 72487, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 467, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 3000, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 2533, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 39644, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 141893, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 102249, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 145404, 2024-05-09T08:46:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 215420 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-05-09T08:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-05-09T08:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-05-09T08:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-05-09T08:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-05-09T08:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-05-09T08:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-05-09T08:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-05-09T08:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-05-09T08:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-05-09T08:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-05-09T08:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-05-09T08:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-05-09T08:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-05-09T08:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-05-09T08:46:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-05-09T08:46:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-05-09T08:46:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-05-09T08:46:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-05-09T08:46:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-05-09T08:46:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:46:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:46:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-05-09T08:46:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-05-09T08:46:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-05-09T08:46:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 2, 2024-05-09T08:46:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 1, 2024-05-09T08:46:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-05-09T08:46:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-05-09T08:46:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-05-09T08:46:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-05-09T08:46:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 3422, 2024-05-09T08:46:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-05-09T08:46:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-05-09T08:46:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-05-09T08:46:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 719785, 2024-05-09T08:46:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 13042, 2024-05-09T08:46:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:46:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-05-09T08:46:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-05-09T08:46:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-05-09T08:46:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 10112, 2024-05-09T08:46:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 203, 2024-05-09T08:46:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-05-09T08:46:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 58098306, 2024-05-09T08:46:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 547610, 2024-05-09T08:46:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-05-09T08:46:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-05-09T08:46:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-05-09T08:46:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-05-09T08:46:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-05-09T08:46:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 4244, 2024-05-09T08:46:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-05-09T08:46:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-05-09T08:46:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 4244, 2024-05-09T08:46:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-05-09T08:46:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4244, 2024-05-09T08:46:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 91.099405598, 2024-05-09T08:46:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.021465458 2024-05-09T08:46:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-05-09T08:46:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-05-09T08:46:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4244, 2024-05-09T08:46:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 4245 2024-05-09T08:46:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 4244, 2024-05-09T08:46:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 4244, 2024-05-09T08:46:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-05-09T08:46:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-05-09T08:46:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-05-09T08:46:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 2571, 2024-05-09T08:46:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-05-09T08:46:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 1671, 2024-05-09T08:46:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-05-09T08:46:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-05-09T08:46:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-05-09T08:46:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-05-09T08:46:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-05-09T08:46:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-05-09T08:46:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-05-09T08:46:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-05-09T08:46:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-05-09T08:46:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-05-09T08:46:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-05-09T08:46:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-05-09T08:46:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:46:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-05-09T08:46:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:46:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-05-09T08:46:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-05-09T08:46:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-05-09T08:46:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:46:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-05-09T08:46:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:46:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-05-09T08:46:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-05-09T08:46:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-05-09T08:46:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-05-09T08:46:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-05-09T08:46:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-05-09T08:46:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-05-09T08:46:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-05-09T08:46:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-05-09T08:46:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-05-09T08:46:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-05-09T08:46:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-05-09T08:46:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-05-09T08:46:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-05-09T08:46:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-05-09T08:46:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-05-09T08:46:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-05-09T08:46:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 135, 2024-05-09T08:46:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-05-09T08:46:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 46, 2024-05-09T08:46:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 11, 2024-05-09T08:46:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 11, 2024-05-09T08:46:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-05-09T08:46:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-05-09T08:46:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 4705, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 346675927, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 4705, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4705, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 346675927, 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 4244, 2024-05-09T08:46:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 344151792, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4244, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 344151792, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-09T08:46:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 4244, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 4244, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4244, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 4244, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-05-09T08:46:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x557d68edc6e0": { 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:11.140 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:46:11.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:10 smithi165 ceph-mon[34540]: pgmap v640: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 105 KiB/s wr, 124 op/s 2024-05-09T08:46:11.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:10 smithi184 ceph-mon[34509]: pgmap v640: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 105 KiB/s wr, 124 op/s 2024-05-09T08:46:11.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:10 smithi028 ceph-mon[30119]: pgmap v640: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 105 KiB/s wr, 124 op/s 2024-05-09T08:46:12.754 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-05-09T08:46:12.755 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-05-09T08:46:12.755 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-05-09T08:46:12.755 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-05-09T08:46:12.755 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-05-09T08:46:12.755 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:46:12.755 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:46:12.755 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-05-09T08:46:12.755 DEBUG:tasks.check_counter:Getting stats from d 2024-05-09T08:46:12.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.d perf dump 2024-05-09T08:46:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:46:13.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:13 smithi165 ceph-mon[34540]: pgmap v641: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 70 KiB/s wr, 102 op/s 2024-05-09T08:46:13.851 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:13 smithi028 ceph-mon[30119]: pgmap v641: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 70 KiB/s wr, 102 op/s 2024-05-09T08:46:13.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:13 smithi184 ceph-mon[34509]: pgmap v641: 129 pgs: 67 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 39 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 70 KiB/s wr, 102 op/s 2024-05-09T08:46:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:15 smithi028 ceph-mon[30119]: pgmap v642: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 91 op/s 2024-05-09T08:46:15.754 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:46:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-05-09T08:46:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-05-09T08:46:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-05-09T08:46:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-05-09T08:46:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-05-09T08:46:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-05-09T08:46:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.003167528, 2024-05-09T08:46:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T08:46:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000034711, 2024-05-09T08:46:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:46:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:46:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 118, 2024-05-09T08:46:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 117, 2024-05-09T08:46:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 174016, 2024-05-09T08:46:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 71088, 2024-05-09T08:46:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-05-09T08:46:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-05-09T08:46:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.025034808, 2024-05-09T08:46:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.007339929, 2024-05-09T08:46:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.011298397, 2024-05-09T08:46:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.005308705, 2024-05-09T08:46:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 116, 2024-05-09T08:46:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.010545897, 2024-05-09T08:46:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000090912 2024-05-09T08:46:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 174016, 2024-05-09T08:46:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 71088 2024-05-09T08:46:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-05-09T08:46:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 90, 2024-05-09T08:46:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2848, 2024-05-09T08:46:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 93440, 2024-05-09T08:46:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 3, 2024-05-09T08:46:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-05-09T08:46:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.007890690, 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.004861601, 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001947820, 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 88, 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.008120195, 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000092274 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2848, 2024-05-09T08:46:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 93440 2024-05-09T08:46:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-05-09T08:46:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-05-09T08:46:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-05-09T08:46:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-05-09T08:46:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-05-09T08:46:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148759, 2024-05-09T08:46:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 25, 2024-05-09T08:46:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:46:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-05-09T08:46:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-05-09T08:46:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-05-09T08:46:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-05-09T08:46:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-05-09T08:46:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-09T08:46:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 119, 2024-05-09T08:46:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 157370, 2024-05-09T08:46:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 119, 2024-05-09T08:46:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-09T08:46:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-09T08:46:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 119, 2024-05-09T08:46:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 157370, 2024-05-09T08:46:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:15.773 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:46:15.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:15 smithi165 ceph-mon[34540]: pgmap v642: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 91 op/s 2024-05-09T08:46:15.856 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:15 smithi184 ceph-mon[34509]: pgmap v642: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 91 op/s 2024-05-09T08:46:16.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:16 smithi028 ceph-mon[30119]: osdmap e136: 12 total, 12 up, 12 in 2024-05-09T08:46:17.087 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-05-09T08:46:17.087 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-05-09T08:46:17.088 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-05-09T08:46:17.088 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-05-09T08:46:17.088 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-05-09T08:46:17.088 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-05-09T08:46:17.088 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-05-09T08:46:17.088 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-05-09T08:46:17.088 DEBUG:tasks.check_counter:Getting stats from g 2024-05-09T08:46:17.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.g perf dump 2024-05-09T08:46:17.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:16 smithi165 ceph-mon[34540]: osdmap e136: 12 total, 12 up, 12 in 2024-05-09T08:46:17.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:16 smithi184 ceph-mon[34509]: osdmap e136: 12 total, 12 up, 12 in 2024-05-09T08:46:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:46:18.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:17 smithi028 ceph-mon[30119]: pgmap v644: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 87 op/s 2024-05-09T08:46:18.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:17 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:17 smithi165 ceph-mon[34540]: pgmap v644: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 87 op/s 2024-05-09T08:46:18.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:17 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:18.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:17 smithi184 ceph-mon[34509]: pgmap v644: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 87 op/s 2024-05-09T08:46:18.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:17 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:19.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:19.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:19.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:18 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:46:19.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:19.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:19.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:18 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:46:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:18 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' 2024-05-09T08:46:19.172 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:46:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:46:18.813+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:46:19.172 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:46:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:46:18.813+0000 7fc7929be700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:46:19.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:19 smithi028 ceph-mon[30119]: pgmap v645: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 KiB/s wr, 97 op/s 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2142, 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2139, 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 168466782, 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 678897, 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 8, 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.339502860, 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.068821224, 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.171622968, 2024-05-09T08:46:20.084 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.081178921, 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2138, 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.171372059, 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000080155 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.085 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1421, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1695, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 118809640, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 1100421, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 9, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 7, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.232795566, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.044948306, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.122055078, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.054281692, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1692, 2024-05-09T08:46:20.086 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.118057780, 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000069774 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 649440 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1101, 2024-05-09T08:46:20.087 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1053, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 165610705, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 436093, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.250816140, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.041734185, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.180297177, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.031571016, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1052, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.104996195, 2024-05-09T08:46:20.088 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000099806 2024-05-09T08:46:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.089 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1790416, 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 232880 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-05-09T08:46:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-05-09T08:46:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-05-09T08:46:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6849, 2024-05-09T08:46:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.204025765, 2024-05-09T08:46:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000029789 2024-05-09T08:46:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-05-09T08:46:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-05-09T08:46:20.091 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 87083, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 4485, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 82598, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 80675, 2024-05-09T08:46:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 3, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 1, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-05-09T08:46:20.094 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:46:20.095 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:46:20.097 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-05-09T08:46:20.098 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 121581, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 139, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 11, 2024-05-09T08:46:20.099 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-05-09T08:46:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-05-09T08:46:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 65261312, 2024-05-09T08:46:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 453965154, 2024-05-09T08:46:20.100 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 453955025, 2024-05-09T08:46:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-05-09T08:46:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.101 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 121581 2024-05-09T08:46:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-05-09T08:46:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 31380, 2024-05-09T08:46:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 66572, 2024-05-09T08:46:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 35192, 2024-05-09T08:46:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 1068, 2024-05-09T08:46:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 1499, 2024-05-09T08:46:20.103 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 431, 2024-05-09T08:46:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 87083, 2024-05-09T08:46:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 112556, 2024-05-09T08:46:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 25473, 2024-05-09T08:46:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-05-09T08:46:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-05-09T08:46:20.104 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-05-09T08:46:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 213992, 2024-05-09T08:46:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 215420 2024-05-09T08:46:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-05-09T08:46:20.105 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-05-09T08:46:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-05-09T08:46:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-05-09T08:46:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-05-09T08:46:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-05-09T08:46:20.106 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-05-09T08:46:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.107 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-05-09T08:46:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.109 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-05-09T08:46:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-05-09T08:46:20.110 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.111 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-05-09T08:46:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.114 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:19 smithi184 ceph-mon[34509]: pgmap v645: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 KiB/s wr, 97 op/s 2024-05-09T08:46:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:19 smithi165 ceph-mon[34540]: pgmap v645: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 KiB/s wr, 97 op/s 2024-05-09T08:46:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-05-09T08:46:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 2, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 3, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 1, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-05-09T08:46:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 3434, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 1423567, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 32083, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:46:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 10080, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 201, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 136208553, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 1351790, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 4190, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 4190, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4190, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 159.837446625, 2024-05-09T08:46:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.038147361 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4190, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 4191 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 4190, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 4190, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 2473, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-05-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 1715, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-05-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 136, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 47, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 11, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 11, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-05-09T08:46:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T08:46:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 4660, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 452441460, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 4660, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4660, 2024-05-09T08:46:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 452441460, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 4190, 2024-05-09T08:46:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 449929087, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4190, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 449929087, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-09T08:46:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 4190, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 4190, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4190, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 4190, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x55a7d40ec6e0": { 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:20.137 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:46:20.951 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-05-09T08:46:20.951 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-05-09T08:46:20.952 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-05-09T08:46:20.952 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-05-09T08:46:20.952 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-05-09T08:46:20.952 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:46:20.952 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:46:20.952 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-05-09T08:46:20.952 DEBUG:tasks.check_counter:Getting stats from j 2024-05-09T08:46:20.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.j perf dump 2024-05-09T08:46:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:46:21.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:21 smithi028 ceph-mon[30119]: pgmap v646: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 614 B/s wr, 93 op/s 2024-05-09T08:46:22.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:21 smithi165 ceph-mon[34540]: pgmap v646: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 614 B/s wr, 93 op/s 2024-05-09T08:46:22.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:21 smithi184 ceph-mon[34509]: pgmap v646: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 614 B/s wr, 93 op/s 2024-05-09T08:46:23.508 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001180828, 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T08:46:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000131644, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.510 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 381, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 369, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1490416, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 227424, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.064333334, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.022197892, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.039937599, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.008761507, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 369, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.026419606, 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000071597 2024-05-09T08:46:23.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1490416, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 227424 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 281, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 204960, 2024-05-09T08:46:23.512 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.020205814, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.015555071, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001844393, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 278, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.019498824, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000070139 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:23.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 204960 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:23.514 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:23.515 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:23.516 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148933, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 30, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:23.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:23.518 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 379, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1434135, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 379, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 379, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1434135, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:23.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-09T08:46:23.520 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-09T08:46:23.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:23 smithi028 ceph-mon[30119]: pgmap v647: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 716 B/s wr, 95 op/s 2024-05-09T08:46:24.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:23 smithi165 ceph-mon[34540]: pgmap v647: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 716 B/s wr, 95 op/s 2024-05-09T08:46:24.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:23 smithi184 ceph-mon[34509]: pgmap v647: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 716 B/s wr, 95 op/s 2024-05-09T08:46:24.328 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-05-09T08:46:24.328 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-05-09T08:46:24.328 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-05-09T08:46:24.328 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-05-09T08:46:24.328 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-05-09T08:46:24.328 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-05-09T08:46:24.329 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-05-09T08:46:24.329 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-05-09T08:46:24.329 DEBUG:tasks.check_counter:Getting stats from b 2024-05-09T08:46:24.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.b perf dump 2024-05-09T08:46:24.842 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:46:25.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:25 smithi165 ceph-mon[34540]: pgmap v648: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 614 B/s wr, 97 op/s 2024-05-09T08:46:26.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:25 smithi184 ceph-mon[34509]: pgmap v648: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 614 B/s wr, 97 op/s 2024-05-09T08:46:26.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:25 smithi028 ceph-mon[30119]: pgmap v648: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 614 B/s wr, 97 op/s 2024-05-09T08:46:28.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:27 smithi165 ceph-mon[34540]: pgmap v649: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 476 B/s wr, 102 op/s 2024-05-09T08:46:28.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:27 smithi184 ceph-mon[34509]: pgmap v649: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 476 B/s wr, 102 op/s 2024-05-09T08:46:28.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:27 smithi028 ceph-mon[30119]: pgmap v649: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 476 B/s wr, 102 op/s 2024-05-09T08:46:28.709 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-05-09T08:46:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 89661, 2024-05-09T08:46:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 91098, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 49212086, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 319184036, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 13, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 6, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 8.418371312, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 2.173678485, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 2.017075048, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 4.101930653, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 91094, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 326.605468197, 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.003585367 2024-05-09T08:46:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 39048, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.008173431, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000209 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 525808 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 293636, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 498477, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 104317429, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 460475662, 2024-05-09T08:46:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 35, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 5, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 19.965519151, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 7.758525134, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 118.755187836, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 3.756625390, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 498477, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 154.022298014, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000308985 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 402734, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.126612193, 2024-05-09T08:46:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000314 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 710480, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 369536 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 101177, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 106466, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 85564455, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 246898478, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 178, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 6, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 8.949425721, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 3.090629448, 2024-05-09T08:46:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 3.758345705, 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 2.748196170, 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 106465, 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 66.842430127, 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000627834 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 95538, 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.020990770, 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000219 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 1876368, 2024-05-09T08:46:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 252976 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "cct": { 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "total_workers": 1, 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "unhealthy_workers": 0 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-MDSRank": { 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 12931, 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 174.736090360, 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.013512960 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-PurgeQueue": { 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 41398, 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 11.943420998, 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000288502 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "mds": { 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "request": 131412, 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "reply": 122160, 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "reply_latency": { 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 122160, 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 2199.616993329, 2024-05-09T08:46:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.018006033 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_reply": 0, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "forward": 9242, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_complete": 393, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_keys": 1, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_commit": 8772, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_split": 145, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_merge": 1, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes": 35346, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_top": 24458, 2024-05-09T08:46:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_bottom": 10742, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pin_tail": 146, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pinned": 404, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_expired": 157763, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_with_caps": 2, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "caps": 3, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "subtrees": 6, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse": 200438, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_hit": 178032, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_forward": 9043, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_discover": 893, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_dir_fetch": 42, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_remote_ino": 0, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_lock": 96, 2024-05-09T08:46:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "load_cent": 24, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "q": 0, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "exported": 565, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "exported_inodes": 115043, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "imported": 460, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "imported_inodes": 108989, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_dir_fetch": 0, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_backtrace_fetch": 1568, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_peer_discover": 0, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rfiles": 4, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rbytes": 234, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rsnaps": 6, 2024-05-09T08:46:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_fetch": 1568, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_set_tag": 0, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_inodes": 58, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_base_inodes": 4, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dirfrag_rstats": 54, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_file_inodes": 1510, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_inode_file_caps": 175, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_revoke": 18575, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_grant": 55460, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_trunc": 7, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:46:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps": 64228, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps_dirty": 34498, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_cap_release": 1238, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "process_request_cap_release": 59573 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache": { 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update": 4084, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update_receipt": 8427, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_try_discover": 4662, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_send_discover": 5912, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_handle_discover": 8811, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays": 171, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_delayed": 0, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:46:28.721 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_created": 21720, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_enqueued": 21549, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_reintegrated": 0, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_migrated": 0, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_processing": 0, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_started": 0, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_completed": 0, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_enqueue_scrub": 4, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_exportdir": 574, 2024-05-09T08:46:28.722 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_flush": 0, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragmentdir": 148, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragstats": 0, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_inodestats": 0 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log": { 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "evlrg": 9, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "evadd": 157453, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "evex": 157331, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "evtrm": 157331, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "ev": 122, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "evexg": 0, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "evexd": 30, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "segadd": 288, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "segex": 286, 2024-05-09T08:46:28.723 INFO:teuthology.orchestra.run.smithi165.stdout: "segtrm": 286, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "seg": 2, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "segmjr": 1, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "segexg": 0, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "segexd": 1, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "expos": 540976467, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "wrpos": 541631049, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "rdpos": 0, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "jlat": { 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 4090, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 817.622707532, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.199907752 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "replayed": 0 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.724 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_mem": { 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "ino": 15332, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "ino+": 126023, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "ino-": 110691, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "dir": 941, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "dir+": 7657, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "dir-": 6716, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "dn": 35346, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "dn+": 223712, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "dn-": 188366, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "cap": 3, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "cap+": 77453, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "cap-": 77450, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "rss": 238116, 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: "heap": 231804 2024-05-09T08:46:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_server": { 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_client_request": 137016, 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_server_request": 9628, 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_request": 131412, 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_session": 485, 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_peer_request": 19660, 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "req_create_latency": { 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 31883, 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 179.262282180, 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.005622503 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getattr_latency": { 2024-05-09T08:46:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1056, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 57.203313104, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.054169804 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getfilelock_latency": { 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "req_link_latency": { 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookup_latency": { 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 29831, 2024-05-09T08:46:28.727 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 1569.160495262, 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.052601672 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookuphash_latency": { 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupino_latency": { 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1, 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000172313, 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000172313 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupname_latency": { 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.728 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.729 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.729 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupparent_latency": { 2024-05-09T08:46:28.729 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.729 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.729 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupsnap_latency": { 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lssnap_latency": { 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 20, 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.015452918, 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000772645 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.730 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mkdir_latency": { 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 247, 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 5.074163787, 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.020543173 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mknod_latency": { 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.731 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mksnap_latency": { 2024-05-09T08:46:28.732 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 20, 2024-05-09T08:46:28.732 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 5.471172044, 2024-05-09T08:46:28.732 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.273558602 2024-05-09T08:46:28.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.732 INFO:teuthology.orchestra.run.smithi165.stdout: "req_open_latency": { 2024-05-09T08:46:28.732 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 7535, 2024-05-09T08:46:28.733 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 47.571590109, 2024-05-09T08:46:28.733 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.006313416 2024-05-09T08:46:28.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.733 INFO:teuthology.orchestra.run.smithi165.stdout: "req_readdir_latency": { 2024-05-09T08:46:28.733 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1003, 2024-05-09T08:46:28.733 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 11.278926161, 2024-05-09T08:46:28.733 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.011245190 2024-05-09T08:46:28.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.734 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rename_latency": { 2024-05-09T08:46:28.734 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 31493, 2024-05-09T08:46:28.734 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 210.578151393, 2024-05-09T08:46:28.734 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.006686506 2024-05-09T08:46:28.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.734 INFO:teuthology.orchestra.run.smithi165.stdout: "req_renamesnap_latency": { 2024-05-09T08:46:28.734 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.735 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.735 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.735 INFO:teuthology.orchestra.run.smithi165.stdout: "req_snapdiff_latency": { 2024-05-09T08:46:28.735 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.735 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.735 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.735 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmdir_latency": { 2024-05-09T08:46:28.736 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 175, 2024-05-09T08:46:28.736 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 51.133925387, 2024-05-09T08:46:28.736 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.292193859 2024-05-09T08:46:28.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.736 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmsnap_latency": { 2024-05-09T08:46:28.736 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 14, 2024-05-09T08:46:28.737 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 2.685044652, 2024-05-09T08:46:28.737 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.191788903 2024-05-09T08:46:28.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.737 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmxattr_latency": { 2024-05-09T08:46:28.737 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.737 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.738 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.738 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setattr_latency": { 2024-05-09T08:46:28.738 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 9, 2024-05-09T08:46:28.738 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.215001723, 2024-05-09T08:46:28.738 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.023889080 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setdirlayout_latency": { 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setfilelock_latency": { 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setlayout_latency": { 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setxattr_latency": { 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 8, 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.020658563, 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.002582320 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "req_symlink_latency": { 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 3, 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.002117732, 2024-05-09T08:46:28.740 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000705910 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "req_unlink_latency": { 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 18862, 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 59.944526001, 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.003178057 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getvxattr_latency": { 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:28.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_sessions": { 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "session_count": 2, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "session_add": 6, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "session_remove": 4, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_open": 2, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_stale": 0, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "total_load": 270, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "average_load": 135, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "avg_session_uptime": 1189, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "mdthresh_evicted": 0 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "mempool": { 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_bytes": 1440, 2024-05-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_items": 1440, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:28.743 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_items": 0, 2024-05-09T08:46:28.744 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_bytes": 2044282, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_items": 15608, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_bytes": 88, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_items": 1, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bytes": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_items": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_bytes": 10080, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_items": 201, 2024-05-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_items": 0, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_bytes": 62983870, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_items": 624030, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_items": 0, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_items": 0 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter": { 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "op_active": 0, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "op_laggy": 0, 2024-05-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send": 173226, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send_bytes": 687663553, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "op_resend": 37, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "op_reply": 173226, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "op_latency": { 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 173226, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 271409.516794959, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 1.566794342 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "op_inflight": 0, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "oplen_avg": { 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 173226, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 360899 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "op": 173226, 2024-05-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi165.stdout: "op_r": 3390, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "op_w": 169836, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "op_rmw": 0, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "op_pg": 0, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_stat": 7319, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_create": 111341, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_read": 482, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_write": 4579, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writefull": 494, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writesame": 0, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_append": 0, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_zero": 0, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_truncate": 145, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_delete": 43866, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_mapext": 0, 2024-05-09T08:46:28.748 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_sparse_read": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_clonerange": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_getxattr": 1588, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_setxattr": 167815, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_rmxattr": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_call": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_watch": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_notify": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_other": 7, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_active": 0, 2024-05-09T08:46:28.749 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_send": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_resend": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_ping": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_active": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_send": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_resend": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_active": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_send": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_resend": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_active": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_send": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_resend": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "command_active": 0, 2024-05-09T08:46:28.750 INFO:teuthology.orchestra.run.smithi165.stdout: "command_send": 0, 2024-05-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi165.stdout: "command_resend": 0, 2024-05-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi165.stdout: "map_epoch": 136, 2024-05-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi165.stdout: "map_full": 0, 2024-05-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi165.stdout: "map_inc": 52, 2024-05-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_sessions": 12, 2024-05-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_open": 12, 2024-05-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_close": 0, 2024-05-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_laggy": 0, 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_wr": 17499, 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_rd": 1730, 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_del": 4034 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "oft": { 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_objs": 1, 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_updates": 0, 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_removes": 0 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "osd-slow-ops": { 2024-05-09T08:46:28.752 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_ops_count": 27 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "purge_queue": { 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops": 0, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops_high_water": 10, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing": 0, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_high_water": 6, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed_ops": 43091, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed": 21549, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_item_in_journal": 0 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 484470, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 198752432, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 484470, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 484470, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 198752432, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.754 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_bytes": { 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 173226, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 690695760, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 173226, 2024-05-09T08:46:28.755 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 690695760, 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_ops": { 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 1024, 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 173226, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 173226, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 173226, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 173226, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:46:28.757 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 21549, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 3070569, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 21549, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 485, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 3070569, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.758 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle-0x55f3de51a500": { 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 157453, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 537436745, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 157453, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 4090, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 537436745, 2024-05-09T08:46:28.759 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:28.760 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:28.760 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:28.760 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:28.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:28.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:28.760 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-05-09T08:46:29.717 INFO:tasks.check_counter:Daemon mds.b mds.exported=565 2024-05-09T08:46:29.718 INFO:tasks.check_counter:Daemon mds.b mds.imported=460 2024-05-09T08:46:29.718 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=4084 2024-05-09T08:46:29.718 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=8427 2024-05-09T08:46:29.718 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-05-09T08:46:29.718 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=20 2024-05-09T08:46:29.718 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=14 2024-05-09T08:46:29.718 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=145 2024-05-09T08:46:29.718 DEBUG:tasks.check_counter:Getting stats from e 2024-05-09T08:46:29.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.e perf dump 2024-05-09T08:46:30.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:29 smithi165 ceph-mon[34540]: pgmap v650: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 90 op/s 2024-05-09T08:46:30.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:29 smithi184 ceph-mon[34509]: pgmap v650: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 90 op/s 2024-05-09T08:46:30.167 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:46:30.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:29 smithi028 ceph-mon[30119]: pgmap v650: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 90 op/s 2024-05-09T08:46:32.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:31 smithi165 ceph-mon[34540]: pgmap v651: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 89 op/s 2024-05-09T08:46:32.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:31 smithi184 ceph-mon[34509]: pgmap v651: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 89 op/s 2024-05-09T08:46:32.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:31 smithi028 ceph-mon[30119]: pgmap v651: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 89 op/s 2024-05-09T08:46:33.506 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-05-09T08:46:33.506 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:33.506 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 426, 2024-05-09T08:46:33.506 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 428, 2024-05-09T08:46:33.506 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 17458902, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 135483, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 8, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 4, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.053254691, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.008055465, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.023916714, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.017460537, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 426, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.058771649, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000137961 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:33.507 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 1263, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 1260, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 21515122, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 464495, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 5, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 4, 2024-05-09T08:46:33.508 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.152808430, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.037676229, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.056634011, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.054703096, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1260, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.109822028, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000087160 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 239856, 2024-05-09T08:46:33.509 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 144208 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 72, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 244, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 8287993, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 359877, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 7, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 5, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.020474193, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.008387252, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.007345690, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.002807311, 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:33.510 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 241, 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.032215753, 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000133675 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 337616 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "cct": { 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "total_workers": 1, 2024-05-09T08:46:33.511 INFO:teuthology.orchestra.run.smithi165.stdout: "unhealthy_workers": 0 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-MDSRank": { 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 2533, 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.194013590, 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000076594 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-PurgeQueue": { 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.512 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "mds": { 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "request": 0, 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "reply": 0, 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "reply_latency": { 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_reply": 0, 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "forward": 0, 2024-05-09T08:46:33.513 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_complete": 0, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_keys": 0, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_commit": 0, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_split": 0, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_merge": 0, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes": 19114, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_top": 13141, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_bottom": 5973, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pin_tail": 0, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pinned": 342, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_expired": 0, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_with_caps": 0, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "caps": 0, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "subtrees": 2, 2024-05-09T08:46:33.514 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_hit": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_forward": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_discover": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_remote_ino": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_lock": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "load_cent": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "q": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "exported": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "exported_inodes": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "imported": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "imported_inodes": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_dir_fetch": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_backtrace_fetch": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_peer_discover": 0, 2024-05-09T08:46:33.515 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rfiles": 4, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rbytes": 234, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rsnaps": 6, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_set_tag": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_file_inodes": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_grant": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:46:33.516 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps_dirty": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_cap_release": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "process_request_cap_release": 0 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache": { 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update_receipt": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_try_discover": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_send_discover": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_handle_discover": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_delayed": 0, 2024-05-09T08:46:33.517 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_created": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_enqueued": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_reintegrated": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_migrated": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_processing": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_started": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_completed": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_exportdir": 0, 2024-05-09T08:46:33.518 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_flush": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragstats": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_inodestats": 0 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log": { 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "evlrg": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "evadd": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "evex": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "evtrm": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "ev": 11292, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "evexg": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "evexd": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "segadd": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "segex": 0, 2024-05-09T08:46:33.519 INFO:teuthology.orchestra.run.smithi165.stdout: "segtrm": 0, 2024-05-09T08:46:33.520 INFO:teuthology.orchestra.run.smithi165.stdout: "seg": 2, 2024-05-09T08:46:33.520 INFO:teuthology.orchestra.run.smithi165.stdout: "segmjr": 30, 2024-05-09T08:46:33.520 INFO:teuthology.orchestra.run.smithi165.stdout: "segexg": 0, 2024-05-09T08:46:33.520 INFO:teuthology.orchestra.run.smithi165.stdout: "segexd": 0, 2024-05-09T08:46:33.520 INFO:teuthology.orchestra.run.smithi165.stdout: "expos": 540976467, 2024-05-09T08:46:33.520 INFO:teuthology.orchestra.run.smithi165.stdout: "wrpos": 541631049, 2024-05-09T08:46:33.520 INFO:teuthology.orchestra.run.smithi165.stdout: "rdpos": 541627971, 2024-05-09T08:46:33.520 INFO:teuthology.orchestra.run.smithi165.stdout: "jlat": { 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "replayed": 11292 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_mem": { 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "ino": 6751, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "ino+": 12362, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "ino-": 5611, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "dir": 339, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "dir+": 373, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "dir-": 34, 2024-05-09T08:46:33.521 INFO:teuthology.orchestra.run.smithi165.stdout: "dn": 19114, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "dn+": 23012, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "dn-": 3898, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "cap": 0, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "cap+": 0, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "cap-": 0, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "rss": 87444, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "heap": 215420 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_server": { 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_client_request": 0, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_server_request": 0, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_request": 0, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_session": 0, 2024-05-09T08:46:33.522 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_peer_request": 0, 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "req_create_latency": { 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getattr_latency": { 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getfilelock_latency": { 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.523 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "req_link_latency": { 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookup_latency": { 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookuphash_latency": { 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.524 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupino_latency": { 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupname_latency": { 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupparent_latency": { 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.525 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupsnap_latency": { 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lssnap_latency": { 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.527 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mkdir_latency": { 2024-05-09T08:46:33.527 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.527 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.527 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.527 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mknod_latency": { 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mksnap_latency": { 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "req_open_latency": { 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.528 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: "req_readdir_latency": { 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rename_latency": { 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.529 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: "req_renamesnap_latency": { 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: "req_snapdiff_latency": { 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.530 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmdir_latency": { 2024-05-09T08:46:33.531 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.531 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.531 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.531 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmsnap_latency": { 2024-05-09T08:46:33.531 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.531 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmxattr_latency": { 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setattr_latency": { 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setdirlayout_latency": { 2024-05-09T08:46:33.532 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setfilelock_latency": { 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setlayout_latency": { 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.533 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setxattr_latency": { 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "req_symlink_latency": { 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "req_unlink_latency": { 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getvxattr_latency": { 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_sessions": { 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "session_count": 2, 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "session_add": 2, 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "session_remove": 0, 2024-05-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_open": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_stale": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "total_load": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "average_load": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "avg_session_uptime": 2546, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "mdthresh_evicted": 0 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "mempool": { 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:33.537 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_items": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:33.538 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_bytes": 485225, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_items": 6954, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_items": 0, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bytes": 0, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_items": 0, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:33.539 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_bytes": 10080, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_items": 201, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_items": 0, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_bytes": 29551944, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_items": 233717, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_items": 0, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:33.540 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_items": 0 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter": { 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "op_active": 0, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "op_laggy": 0, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send": 1503, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send_bytes": 26, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "op_resend": 4, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "op_reply": 1503, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "op_latency": { 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1503, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 6.134913476, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.004081778 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.541 INFO:teuthology.orchestra.run.smithi165.stdout: "op_inflight": 0, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "oplen_avg": { 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1503, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 1504 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "op": 1503, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "op_r": 1503, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "op_w": 0, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "op_rmw": 0, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "op_pg": 0, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_stat": 875, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_create": 0, 2024-05-09T08:46:33.542 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_read": 625, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_write": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writefull": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writesame": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_append": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_zero": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_truncate": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_delete": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_mapext": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_sparse_read": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_clonerange": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_getxattr": 2, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_setxattr": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:46:33.543 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_rmxattr": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_call": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_watch": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_notify": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_other": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_active": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_send": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_resend": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_ping": 0, 2024-05-09T08:46:33.544 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_active": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_send": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_resend": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_active": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_send": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_resend": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_active": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_send": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_resend": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "command_active": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "command_send": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "command_resend": 0, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "map_epoch": 136, 2024-05-09T08:46:33.545 INFO:teuthology.orchestra.run.smithi165.stdout: "map_full": 0, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "map_inc": 19, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_sessions": 10, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_open": 10, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_close": 0, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_laggy": 0, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_wr": 0, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_rd": 2, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_del": 0 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "oft": { 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_objs": 0, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:46:33.546 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_updates": 0, 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_removes": 0 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "osd-slow-ops": { 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_ops_count": 0 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "purge_queue": { 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops": 0, 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing": 0, 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_high_water": 0, 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed_ops": 0, 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed": 0, 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_item_in_journal": 0 2024-05-09T08:46:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 1757, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 47087971, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 1757, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 1757, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 47087971, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.548 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_bytes": { 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 1503, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 46598730, 2024-05-09T08:46:33.549 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 1503, 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 46598730, 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_ops": { 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 1024, 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:33.550 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 1503, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 1503, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 1503, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 1503, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:46:33.551 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.552 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle-0x5636c93006e0": { 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 0, 2024-05-09T08:46:33.553 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:33.554 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:33.554 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:33.554 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:33.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:33.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:33.554 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-05-09T08:46:34.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:33 smithi165 ceph-mon[34540]: pgmap v652: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 89 op/s 2024-05-09T08:46:34.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:33 smithi184 ceph-mon[34509]: pgmap v652: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 89 op/s 2024-05-09T08:46:34.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:33 smithi028 ceph-mon[30119]: pgmap v652: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 89 op/s 2024-05-09T08:46:34.745 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-05-09T08:46:34.745 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-05-09T08:46:34.745 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-05-09T08:46:34.745 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-05-09T08:46:34.745 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-05-09T08:46:34.745 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:46:34.745 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:46:34.745 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-05-09T08:46:34.745 DEBUG:tasks.check_counter:Getting stats from h 2024-05-09T08:46:34.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.h perf dump 2024-05-09T08:46:35.243 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:46:36.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:35 smithi165 ceph-mon[34540]: pgmap v653: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 94 op/s 2024-05-09T08:46:36.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:35 smithi184 ceph-mon[34509]: pgmap v653: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 94 op/s 2024-05-09T08:46:36.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:35 smithi028 ceph-mon[30119]: pgmap v653: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 94 op/s 2024-05-09T08:46:37.452 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-05-09T08:46:37.452 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:37.452 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 58566, 2024-05-09T08:46:37.452 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 49556, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 77759732, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 153987689, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 14, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 4, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 4.445756753, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 1.835343325, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 3.465345288, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.715112608, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 49560, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 9.070080261, 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000183012 2024-05-09T08:46:37.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 84192, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.016585726, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000196 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 1814672, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 359744 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 34030, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 36701, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 16956863, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 134275556, 2024-05-09T08:46:37.454 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 15, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 7, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 3.123360053, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 1.040671972, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 1.264048549, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.920027986, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 36698, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 3.974811442, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000108311 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 44405, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.008505993, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000191 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.455 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 656096 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 235844, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 352624, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 83867600, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 288496267, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 16, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 7, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 11.262860457, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 5.224675365, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 79.564092142, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.509429293, 2024-05-09T08:46:37.456 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 352618, 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 63.394608931, 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000179782 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 291979, 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.098777019, 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000338 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "cct": { 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "total_workers": 1, 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: "unhealthy_workers": 0 2024-05-09T08:46:37.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-MDSRank": { 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 7724, 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 77.248639795, 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.010001118 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-PurgeQueue": { 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 22531, 2024-05-09T08:46:37.458 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 4.353930031, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000193241 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "mds": { 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "request": 94465, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "reply": 91392, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "reply_latency": { 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 91392, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 2446.157339865, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.026765552 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_reply": 0, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "forward": 3072, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_complete": 165, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_keys": 0, 2024-05-09T08:46:37.459 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_commit": 732, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_split": 81, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_merge": 0, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes": 21344, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_top": 417, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_bottom": 16908, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pin_tail": 4019, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pinned": 20745, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_expired": 111071, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_with_caps": 1, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "caps": 1, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "subtrees": 6, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse": 173771, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_hit": 139707, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_forward": 2629, 2024-05-09T08:46:37.460 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_discover": 1560, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_dir_fetch": 42, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_remote_ino": 0, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_lock": 717, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "load_cent": 0, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "q": 0, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "exported": 352, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "exported_inodes": 91499, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "imported": 621, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "imported_inodes": 88635, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_dir_fetch": 0, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_backtrace_fetch": 347, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_peer_discover": 0, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rfiles": 27663, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rbytes": 621812736, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rsnaps": 6, 2024-05-09T08:46:37.461 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_fetch": 347, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_set_tag": 0, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_file_inodes": 347, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_inode_file_caps": 199, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_revoke": 12098, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_grant": 27168, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps": 49189, 2024-05-09T08:46:37.462 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps_dirty": 17871, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_cap_release": 938, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "process_request_cap_release": 30813 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache": { 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update": 3643, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update_receipt": 9594, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_try_discover": 5375, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_send_discover": 7162, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_handle_discover": 5068, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays": 71, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_delayed": 0, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_created": 11309, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_enqueued": 11238, 2024-05-09T08:46:37.463 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_reintegrated": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_migrated": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_processing": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_started": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_completed": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_exportdir": 353, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_flush": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragmentdir": 84, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragstats": 0, 2024-05-09T08:46:37.464 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_inodestats": 0 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log": { 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "evlrg": 4, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "evadd": 96234, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "evex": 12934, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "evtrm": 12291, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "ev": 83943, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "evexg": 0, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "evexd": 643, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "segadd": 146, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "segex": 18, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "segtrm": 17, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "seg": 129, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "segmjr": 7, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "segexg": 0, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "segexd": 1, 2024-05-09T08:46:37.465 INFO:teuthology.orchestra.run.smithi165.stdout: "expos": 31881079, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "wrpos": 348287721, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "rdpos": 0, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "jlat": { 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 3337, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 438.279021410, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.131339233 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "replayed": 0 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_mem": { 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "ino": 11354, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "ino+": 87023, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "ino-": 75669, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "dir": 514, 2024-05-09T08:46:37.466 INFO:teuthology.orchestra.run.smithi165.stdout: "dir+": 7031, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "dir-": 6517, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "dn": 21344, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "dn+": 147654, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "dn-": 126310, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "cap": 1, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "cap+": 56317, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "cap-": 56316, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "rss": 201688, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "heap": 223612 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_server": { 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_client_request": 105260, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_server_request": 4790, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_request": 94465, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_session": 400, 2024-05-09T08:46:37.467 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_peer_request": 12128, 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: "req_create_latency": { 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 14351, 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 87.033503432, 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.006064629 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getattr_latency": { 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 309, 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 5.754864435, 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.018624156 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.469 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getfilelock_latency": { 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "req_link_latency": { 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookup_latency": { 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 47511, 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 1805.294354500, 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.037997397 2024-05-09T08:46:37.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookuphash_latency": { 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupino_latency": { 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000304256, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000304256 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupname_latency": { 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupparent_latency": { 2024-05-09T08:46:37.471 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.472 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.472 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.472 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupsnap_latency": { 2024-05-09T08:46:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lssnap_latency": { 2024-05-09T08:46:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mkdir_latency": { 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 90, 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 13.115956727, 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.145732852 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mknod_latency": { 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mksnap_latency": { 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.474 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "req_open_latency": { 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 4974, 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 51.497638683, 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.010353365 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "req_readdir_latency": { 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 419, 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 3.938004898, 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.009398579 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rename_latency": { 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 14350, 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 425.007335955, 2024-05-09T08:46:37.475 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.029617235 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "req_renamesnap_latency": { 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "req_snapdiff_latency": { 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmdir_latency": { 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 71, 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 27.143840749, 2024-05-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.382307616 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmsnap_latency": { 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmxattr_latency": { 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setattr_latency": { 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.477 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setdirlayout_latency": { 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setfilelock_latency": { 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setlayout_latency": { 2024-05-09T08:46:37.478 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setxattr_latency": { 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "req_symlink_latency": { 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "req_unlink_latency": { 2024-05-09T08:46:37.479 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 9316, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 27.371536230, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.002938121 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getvxattr_latency": { 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_sessions": { 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "session_count": 1, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "session_add": 2, 2024-05-09T08:46:37.480 INFO:teuthology.orchestra.run.smithi165.stdout: "session_remove": 1, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_open": 1, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_stale": 0, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "total_load": 190, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "average_load": 190, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "avg_session_uptime": 1218, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "mdthresh_evicted": 0 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "mempool": { 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:37.481 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:37.482 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_items": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_bytes": 1298974, 2024-05-09T08:46:37.483 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_items": 11392, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_bytes": 176, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_items": 2, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bytes": 0, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_items": 0, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_bytes": 10080, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_items": 201, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:37.484 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_items": 0, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_bytes": 42885914, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_items": 426069, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_items": 0, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_items": 0 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter": { 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "op_active": 0, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "op_laggy": 0, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send": 29398, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send_bytes": 362341115, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "op_resend": 33, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "op_reply": 29398, 2024-05-09T08:46:37.485 INFO:teuthology.orchestra.run.smithi165.stdout: "op_latency": { 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 29398, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 1726.738569778, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.058736600 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "op_inflight": 0, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "oplen_avg": { 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 29398, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 31179 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "op": 29398, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "op_r": 1384, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "op_w": 28014, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "op_rmw": 0, 2024-05-09T08:46:37.486 INFO:teuthology.orchestra.run.smithi165.stdout: "op_pg": 0, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_stat": 0, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_create": 20, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_read": 500, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_write": 3845, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writefull": 279, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writesame": 0, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_append": 0, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_zero": 0, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_truncate": 81, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_delete": 22576, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_mapext": 0, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_sparse_read": 0, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_clonerange": 0, 2024-05-09T08:46:37.487 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_getxattr": 349, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_setxattr": 40, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_rmxattr": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_call": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_watch": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_notify": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_other": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_active": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_send": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_resend": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_ping": 0, 2024-05-09T08:46:37.488 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_active": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_send": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_resend": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_active": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_send": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_resend": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_active": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_send": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_resend": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "command_active": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "command_send": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "command_resend": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "map_epoch": 136, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "map_full": 0, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "map_inc": 47, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_sessions": 12, 2024-05-09T08:46:37.489 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_open": 12, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_close": 0, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_laggy": 0, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_wr": 2334, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_rd": 702, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_del": 453 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "oft": { 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_objs": 1, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_updates": 0, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_removes": 0 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "osd-slow-ops": { 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_ops_count": 26 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.490 INFO:teuthology.orchestra.run.smithi165.stdout: "purge_queue": { 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops": 0, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops_high_water": 10, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing": 0, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_high_water": 5, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed_ops": 22476, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed": 11238, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_item_in_journal": 0 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 328436, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 152282606, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:37.491 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 328436, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 328436, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 152282606, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_bytes": { 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:37.492 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 29398, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 363991079, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 29398, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 363991079, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_ops": { 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 1024, 2024-05-09T08:46:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 29398, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 29398, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 29398, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 29398, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.494 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 11238, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 1665254, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 11238, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 505, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 1665254, 2024-05-09T08:46:37.495 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle-0x55c59816e500": { 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 96234, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 344093417, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 96234, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:37.496 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:37.497 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 3337, 2024-05-09T08:46:37.497 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 344093417, 2024-05-09T08:46:37.497 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:37.497 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:37.497 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:37.497 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:37.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:37.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:37.497 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-05-09T08:46:38.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:37 smithi165 ceph-mon[34540]: pgmap v654: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 85 B/s wr, 95 op/s 2024-05-09T08:46:38.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:37 smithi184 ceph-mon[34509]: pgmap v654: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 85 B/s wr, 95 op/s 2024-05-09T08:46:38.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:37 smithi028 ceph-mon[30119]: pgmap v654: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 85 B/s wr, 95 op/s 2024-05-09T08:46:38.257 INFO:tasks.check_counter:Daemon mds.h mds.exported=352 2024-05-09T08:46:38.257 INFO:tasks.check_counter:Daemon mds.h mds.imported=621 2024-05-09T08:46:38.257 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=3643 2024-05-09T08:46:38.257 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=9594 2024-05-09T08:46:38.257 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-05-09T08:46:38.257 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:46:38.257 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:46:38.257 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=81 2024-05-09T08:46:38.257 DEBUG:tasks.check_counter:Getting stats from k 2024-05-09T08:46:38.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.k perf dump 2024-05-09T08:46:38.727 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:46:40.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:39 smithi184 ceph-mon[34509]: pgmap v655: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 92 op/s 2024-05-09T08:46:40.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:39 smithi028 ceph-mon[30119]: pgmap v655: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 92 op/s 2024-05-09T08:46:40.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:39 smithi165 ceph-mon[34540]: pgmap v655: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 92 op/s 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 1516, 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 1521, 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 148699894, 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 482142, 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 8, 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 4, 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.208587973, 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.035549939, 2024-05-09T08:46:41.719 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.115804343, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.046642547, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1519, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.075433639, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000049660 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.720 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 1269, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 1549, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 151324511, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 1057465, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 9, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 7, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.213344725, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.034749047, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.129781857, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.038145076, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1547, 2024-05-09T08:46:41.721 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.112780713, 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000072902 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 656144 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 2194, 2024-05-09T08:46:41.722 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 2146, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 90440028, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 787198, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 5, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 4, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.285050616, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.066459662, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.142992301, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.071564643, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 2145, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.151765375, 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000070753 2024-05-09T08:46:41.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 1793408, 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 239920 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "cct": { 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "total_workers": 1, 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "unhealthy_workers": 0 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-MDSRank": { 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-05-09T08:46:41.724 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 7329, 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.199671505, 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000027244 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-PurgeQueue": { 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.725 INFO:teuthology.orchestra.run.smithi165.stdout: "mds": { 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "request": 0, 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "reply": 0, 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "reply_latency": { 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_reply": 0, 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "forward": 0, 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_complete": 0, 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_keys": 0, 2024-05-09T08:46:41.726 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_commit": 0, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_split": 0, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_merge": 0, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes": 73052, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_top": 10180, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_bottom": 62872, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pin_tail": 0, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pinned": 58508, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_expired": 2, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_with_caps": 0, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "caps": 0, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "subtrees": 1, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse": 0, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_hit": 0, 2024-05-09T08:46:41.727 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_forward": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_discover": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_remote_ino": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_lock": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "load_cent": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "q": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "exported": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "exported_inodes": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "imported": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "imported_inodes": 0, 2024-05-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_dir_fetch": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_backtrace_fetch": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_peer_discover": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rfiles": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rbytes": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rsnaps": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_set_tag": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_file_inodes": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_grant": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps_dirty": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_cap_release": 0, 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: "process_request_cap_release": 0 2024-05-09T08:46:41.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache": { 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update_receipt": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_try_discover": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_send_discover": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_handle_discover": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_delayed": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_created": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_enqueued": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_reintegrated": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_migrated": 0, 2024-05-09T08:46:41.731 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_processing": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_started": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_completed": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_exportdir": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_flush": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragstats": 0, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_inodestats": 0 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.732 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log": { 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "evlrg": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "evadd": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "evex": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "evtrm": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "ev": 105001, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "evexg": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "evexd": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "segadd": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "segex": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "segtrm": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "seg": 134, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "segmjr": 10, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "segexg": 0, 2024-05-09T08:46:41.733 INFO:teuthology.orchestra.run.smithi165.stdout: "segexd": 0, 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "expos": 33156160, 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "wrpos": 391544766, 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "rdpos": 391535110, 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "jlat": { 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "replayed": 105001 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_mem": { 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "ino": 25435, 2024-05-09T08:46:41.734 INFO:teuthology.orchestra.run.smithi165.stdout: "ino+": 78389, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "ino-": 52954, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "dir": 943, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "dir+": 2122, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "dir-": 1179, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "dn": 73052, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "dn+": 127453, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "dn-": 54401, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "cap": 0, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "cap+": 0, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "cap-": 0, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "rss": 184272, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "heap": 215420 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_server": { 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_client_request": 0, 2024-05-09T08:46:41.735 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_server_request": 0, 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_request": 0, 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_session": 0, 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_peer_request": 0, 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "req_create_latency": { 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getattr_latency": { 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getfilelock_latency": { 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "req_link_latency": { 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookup_latency": { 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.737 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookuphash_latency": { 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupino_latency": { 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupname_latency": { 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.738 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupparent_latency": { 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupsnap_latency": { 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lssnap_latency": { 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.739 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.740 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mkdir_latency": { 2024-05-09T08:46:41.740 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.740 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.740 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.740 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mknod_latency": { 2024-05-09T08:46:41.740 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.740 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mksnap_latency": { 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "req_open_latency": { 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.741 INFO:teuthology.orchestra.run.smithi165.stdout: "req_readdir_latency": { 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rename_latency": { 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "req_renamesnap_latency": { 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "req_snapdiff_latency": { 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmdir_latency": { 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmsnap_latency": { 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmxattr_latency": { 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setattr_latency": { 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setdirlayout_latency": { 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.744 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setfilelock_latency": { 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setlayout_latency": { 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setxattr_latency": { 2024-05-09T08:46:41.745 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "req_symlink_latency": { 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "req_unlink_latency": { 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:46:41.746 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getvxattr_latency": { 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_sessions": { 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "session_count": 1, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "session_add": 4, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "session_remove": 3, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_open": 3, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_stale": 0, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "total_load": 0, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "average_load": 0, 2024-05-09T08:46:41.747 INFO:teuthology.orchestra.run.smithi165.stdout: "avg_session_uptime": 5221, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "mdthresh_evicted": 0 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "mempool": { 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:41.748 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:41.749 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_items": 0, 2024-05-09T08:46:41.750 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_bytes": 1200903, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_items": 26026, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_items": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bytes": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_items": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:41.751 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_bytes": 10080, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_items": 201, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_items": 0, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_bytes": 111836152, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_items": 1001810, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_items": 0, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_items": 0 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter": { 2024-05-09T08:46:41.752 INFO:teuthology.orchestra.run.smithi165.stdout: "op_active": 0, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "op_laggy": 0, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send": 4498, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send_bytes": 21, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "op_resend": 3, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "op_reply": 4498, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "op_latency": { 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 4498, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 115.515303279, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.025681481 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "op_inflight": 0, 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "oplen_avg": { 2024-05-09T08:46:41.753 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 4498, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 4499 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "op": 4498, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "op_r": 4498, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "op_w": 0, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "op_rmw": 0, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "op_pg": 0, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_stat": 2708, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_create": 0, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_read": 1788, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_write": 0, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writefull": 0, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writesame": 0, 2024-05-09T08:46:41.754 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_append": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_zero": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_truncate": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_delete": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_mapext": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_sparse_read": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_clonerange": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_getxattr": 1, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_setxattr": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_rmxattr": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_call": 0, 2024-05-09T08:46:41.755 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_watch": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_notify": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_other": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_active": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_send": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_resend": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_ping": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_active": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_send": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_resend": 0, 2024-05-09T08:46:41.756 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_active": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_send": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_resend": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_active": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_send": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_resend": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "command_active": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "command_send": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "command_resend": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "map_epoch": 136, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "map_full": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "map_inc": 47, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_sessions": 11, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_open": 11, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_close": 0, 2024-05-09T08:46:41.757 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_laggy": 0, 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_wr": 0, 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_rd": 2, 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_del": 0 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "oft": { 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_objs": 0, 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_updates": 0, 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_removes": 0 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "osd-slow-ops": { 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_ops_count": 0 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.758 INFO:teuthology.orchestra.run.smithi165.stdout: "purge_queue": { 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops": 0, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing": 0, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_high_water": 0, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed_ops": 0, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed": 0, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_item_in_journal": 0 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:41.759 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 4975, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 389989844, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 4975, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 4975, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 389989844, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_bytes": { 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 4498, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 387419567, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 4498, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 387419567, 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:41.761 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_ops": { 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 1024, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:41.762 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 4498, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 4498, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 4498, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 4498, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:41.763 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 0, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 0, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle-0x56349ec526e0": { 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-05-09T08:46:41.764 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:41.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-05-09T08:46:41.766 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-05-09T08:46:42.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:41 smithi165 ceph-mon[34540]: pgmap v656: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 103 op/s 2024-05-09T08:46:42.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:41 smithi028 ceph-mon[30119]: pgmap v656: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 103 op/s 2024-05-09T08:46:42.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:41 smithi184 ceph-mon[34509]: pgmap v656: 129 pgs: 65 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 43 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 103 op/s 2024-05-09T08:46:42.638 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-05-09T08:46:42.638 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-05-09T08:46:42.639 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-05-09T08:46:42.639 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-05-09T08:46:42.639 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-05-09T08:46:42.639 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:46:42.639 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:46:42.639 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-05-09T08:46:42.639 DEBUG:tasks.check_counter:Getting stats from c 2024-05-09T08:46:42.639 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.c perf dump 2024-05-09T08:46:43.083 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:46:44.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:43 smithi028 ceph-mon[30119]: pgmap v657: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 98 op/s 2024-05-09T08:46:44.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:43 smithi165 ceph-mon[34540]: pgmap v657: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 98 op/s 2024-05-09T08:46:44.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:43 smithi184 ceph-mon[34509]: pgmap v657: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 98 op/s 2024-05-09T08:46:46.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:45 smithi165 ceph-mon[34540]: pgmap v658: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 101 op/s 2024-05-09T08:46:46.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:45 smithi165 ceph-mon[34540]: osdmap e137: 12 total, 12 up, 12 in 2024-05-09T08:46:46.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:45 smithi028 ceph-mon[30119]: pgmap v658: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 101 op/s 2024-05-09T08:46:46.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:45 smithi028 ceph-mon[30119]: osdmap e137: 12 total, 12 up, 12 in 2024-05-09T08:46:46.243 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 213687, 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 318942, 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 74945364, 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 348411482, 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 22, 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 5, 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 15.670783066, 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 6.332024187, 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 62.885147873, 2024-05-09T08:46:46.244 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 1.506877633, 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 318936, 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 68.982423953, 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000216289 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 258919, 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.086592904, 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000334 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.245 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 2512, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 2416 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 40292, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 45151, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 26375725, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 168660235, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 15, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 8, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 4.372705237, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 1.381957858, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 1.393225560, 2024-05-09T08:46:46.246 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 1.605623105, 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 45146, 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 8.649222492, 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000191583 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 35497, 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.007282591, 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000205 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 2304, 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 131680 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:46.247 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 49841, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 56201, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 48487404, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 88913331, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 10, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 5, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 4.458381175, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 2.009946730, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 2.335337285, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.363251077, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 56207, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 12.697283381, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000225902 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.248 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 86525, 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.026675629, 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000308 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 1820288, 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 862144 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "cct": { 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "total_workers": 1, 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "unhealthy_workers": 0 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-MDSRank": { 2024-05-09T08:46:46.249 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 7562, 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 87.427245734, 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.011561391 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-PurgeQueue": { 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 41777, 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 7.018558767, 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000168000 2024-05-09T08:46:46.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "mds": { 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "request": 104583, 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "reply": 100444, 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "reply_latency": { 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 100444, 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 1672.062459121, 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.016646713 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_reply": 0, 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "forward": 4116, 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_complete": 339, 2024-05-09T08:46:46.251 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_keys": 0, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_commit": 677, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_split": 75, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_merge": 0, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes": 41866, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_top": 3105, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_bottom": 35176, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pin_tail": 3585, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pinned": 37427, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_expired": 63518, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_with_caps": 0, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "caps": 0, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "subtrees": 5, 2024-05-09T08:46:46.252 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse": 185753, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_hit": 159857, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_forward": 3872, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_discover": 1034, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_dir_fetch": 2, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_remote_ino": 0, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_lock": 312, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "load_cent": 0, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "q": 0, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "exported": 523, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "exported_inodes": 50865, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "imported": 415, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "imported_inodes": 57938, 2024-05-09T08:46:46.253 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_dir_fetch": 0, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_backtrace_fetch": 373, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_peer_discover": 0, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rfiles": 27663, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rbytes": 621812736, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rsnaps": 6, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_fetch": 373, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_set_tag": 0, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_file_inodes": 373, 2024-05-09T08:46:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_inode_file_caps": 245, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_revoke": 10711, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_grant": 33982, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps": 33005, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps_dirty": 17675, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_cap_release": 640, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "process_request_cap_release": 41852 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_cache": { 2024-05-09T08:46:46.255 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update": 5023, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update_receipt": 9620, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_try_discover": 5530, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_send_discover": 6846, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_handle_discover": 6213, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays": 141, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_delayed": 0, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_created": 21559, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_enqueued": 21418, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_reintegrated": 0, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_migrated": 0, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_processing": 0, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:46:46.256 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_started": 0, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_completed": 0, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_exportdir": 526, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_flush": 0, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragmentdir": 75, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragstats": 0, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_inodestats": 0 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_log": { 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "evlrg": 3, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "evadd": 105001, 2024-05-09T08:46:46.257 INFO:teuthology.orchestra.run.smithi184.stdout: "evex": 16602, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "evtrm": 12291, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "ev": 92710, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "evexg": 0, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "evexd": 4311, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "segadd": 150, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "segex": 22, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "segtrm": 16, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "seg": 134, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "segmjr": 8, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "segexg": 0, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "segexd": 6, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "expos": 33156160, 2024-05-09T08:46:46.258 INFO:teuthology.orchestra.run.smithi184.stdout: "wrpos": 391544766, 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: "rdpos": 0, 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: "jlat": { 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 2683, 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 354.212693227, 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.132021130 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: "replayed": 0 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_mem": { 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ino": 23724, 2024-05-09T08:46:46.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ino+": 80702, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "ino-": 56978, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "dir": 979, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "dir+": 6076, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "dir-": 5097, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "dn": 41866, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "dn+": 131909, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "dn-": 90043, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "cap": 0, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "cap+": 43651, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "cap-": 43651, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "rss": 204284, 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: "heap": 223612 2024-05-09T08:46:46.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_server": { 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_client_request": 116977, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_server_request": 2869, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_request": 104583, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_session": 218, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_peer_request": 12560, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "req_create_latency": { 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 11437, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 67.839426391, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.005931575 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getattr_latency": { 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 526, 2024-05-09T08:46:46.261 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 10.109406272, 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.019219403 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getfilelock_latency": { 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "req_link_latency": { 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.262 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookup_latency": { 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 52011, 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 1222.920240411, 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.023512723 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookuphash_latency": { 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupino_latency": { 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupname_latency": { 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupparent_latency": { 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.264 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupsnap_latency": { 2024-05-09T08:46:46.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:45 smithi184 ceph-mon[34509]: pgmap v658: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 101 op/s 2024-05-09T08:46:46.265 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:45 smithi184 ceph-mon[34509]: osdmap e137: 12 total, 12 up, 12 in 2024-05-09T08:46:46.265 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.265 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.265 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lssnap_latency": { 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mkdir_latency": { 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 64, 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 2.058474505, 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.032163664 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mknod_latency": { 2024-05-09T08:46:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mksnap_latency": { 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "req_open_latency": { 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 4047, 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 10.459616378, 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.002584535 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.267 INFO:teuthology.orchestra.run.smithi184.stdout: "req_readdir_latency": { 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1017, 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 9.429796950, 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.009272170 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rename_latency": { 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 11448, 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 240.969252555, 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.021049026 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "req_renamesnap_latency": { 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "req_snapdiff_latency": { 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmdir_latency": { 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 141, 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 37.648529170, 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.267010845 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmsnap_latency": { 2024-05-09T08:46:46.269 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmxattr_latency": { 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setattr_latency": { 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.270 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setdirlayout_latency": { 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setfilelock_latency": { 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setlayout_latency": { 2024-05-09T08:46:46.271 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setxattr_latency": { 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "req_symlink_latency": { 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.272 INFO:teuthology.orchestra.run.smithi184.stdout: "req_unlink_latency": { 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 19753, 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 70.627716489, 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.003575543 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getvxattr_latency": { 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:46.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_sessions": { 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "session_count": 1, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "session_add": 4, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "session_remove": 3, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_open": 1, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_stale": 0, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "total_load": 4411, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "average_load": 4411, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "avg_session_uptime": 1145, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "mdthresh_evicted": 0 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "mempool": { 2024-05-09T08:46:46.274 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:46.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:46.276 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:46.277 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_items": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_bytes": 2340448, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_items": 23821, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_bytes": 704, 2024-05-09T08:46:46.278 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_items": 8, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_bytes": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_items": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_bytes": 10080, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_items": 201, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_items": 0, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_bytes": 87083225, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_items": 868553, 2024-05-09T08:46:46.279 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_items": 0, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_items": 0 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "objecter": { 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "op_active": 0, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "op_laggy": 0, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send": 49687, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send_bytes": 404369208, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "op_resend": 28, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "op_reply": 49687, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "op_latency": { 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 49687, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 2871.883175200, 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.057799488 2024-05-09T08:46:46.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "op_inflight": 0, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "oplen_avg": { 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 49687, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 51702 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "op": 49687, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "op_r": 1947, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "op_w": 47740, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "op_rmw": 0, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "op_pg": 0, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_stat": 0, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_create": 21, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_read": 505, 2024-05-09T08:46:46.281 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_write": 3193, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writefull": 247, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writesame": 0, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_append": 0, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_zero": 0, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_truncate": 75, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_delete": 42946, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_mapext": 0, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_sparse_read": 0, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_clonerange": 0, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_getxattr": 376, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_setxattr": 42, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:46:46.282 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_rmxattr": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_call": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_watch": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_notify": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_other": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_active": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_send": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_resend": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_ping": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_active": 0, 2024-05-09T08:46:46.283 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_send": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_resend": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_active": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_send": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_resend": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_active": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_send": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_resend": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "command_active": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "command_send": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "command_resend": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "map_epoch": 137, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "map_full": 0, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "map_inc": 48, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_sessions": 12, 2024-05-09T08:46:46.284 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_open": 12, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_close": 0, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_laggy": 0, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_wr": 2420, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_rd": 1408, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_del": 469 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "oft": { 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_objs": 1, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_updates": 0, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_removes": 0 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "osd-slow-ops": { 2024-05-09T08:46:46.285 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_ops_count": 914 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "purge_queue": { 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops": 0, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops_high_water": 10, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing": 0, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_high_water": 5, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed_ops": 42836, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed": 21418, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_item_in_journal": 0 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-05-09T08:46:46.286 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 303816, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 125295892, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 303816, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 303816, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 125295892, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_bytes": { 2024-05-09T08:46:46.287 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 49687, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 407611721, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 49687, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 407611721, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.288 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_ops": { 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 1024, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 49687, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 49687, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 49687, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 49687, 2024-05-09T08:46:46.289 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 21418, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 3272586, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 21418, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:46:46.290 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 508, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 3272586, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle-0x5619ea262500": { 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 105001, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 387350462, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 105001, 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 2683, 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 387350462, 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:46.292 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-05-09T08:46:47.211 INFO:tasks.check_counter:Daemon mds.c mds.exported=523 2024-05-09T08:46:47.211 INFO:tasks.check_counter:Daemon mds.c mds.imported=415 2024-05-09T08:46:47.211 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=5023 2024-05-09T08:46:47.211 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=9620 2024-05-09T08:46:47.211 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-05-09T08:46:47.211 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:46:47.211 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:46:47.211 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=75 2024-05-09T08:46:47.211 DEBUG:tasks.check_counter:Getting stats from f 2024-05-09T08:46:47.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.f perf dump 2024-05-09T08:46:47.688 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:46:48.108 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:47 smithi184 ceph-mon[34509]: pgmap v660: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 90 op/s 2024-05-09T08:46:48.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:47 smithi028 ceph-mon[30119]: pgmap v660: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 90 op/s 2024-05-09T08:46:48.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:47 smithi165 ceph-mon[34540]: pgmap v660: 129 pgs: 61 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 48 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 90 op/s 2024-05-09T08:46:50.163 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:49 smithi184 ceph-mon[34509]: pgmap v661: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 89 op/s 2024-05-09T08:46:50.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:49 smithi028 ceph-mon[30119]: pgmap v661: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 89 op/s 2024-05-09T08:46:50.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:49 smithi165 ceph-mon[34540]: pgmap v661: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 89 op/s 2024-05-09T08:46:50.982 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 41000, 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 38856, 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 20579282, 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 161861785, 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 20, 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 5, 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 3.897250553, 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 1.094394881, 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 1.474172841, 2024-05-09T08:46:50.983 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 1.466942637, 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 38854, 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 8.499058982, 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000218743 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 44790, 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.009796782, 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000218 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 530224 2024-05-09T08:46:50.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 75201, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 70579, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 46744068, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 190625690, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 9, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 6, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 6.886633658, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 2.353814148, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 3.186261072, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 1.734784614, 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:50.985 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 70579, 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 15.794673572, 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000223787 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 99795, 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.020707006, 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000207 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 1809024, 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 380960 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:50.986 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 215149, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 314443, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 73958434, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 317995164, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 17, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 8, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 15.405067076, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 6.336662378, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 135.107924961, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 1.313722838, 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:50.987 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 314440, 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 69.242144185, 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000220207 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 267982, 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.093518916, 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000348 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 6464, 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 156528 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "cct": { 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "total_workers": 1, 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: "unhealthy_workers": 0 2024-05-09T08:46:50.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-MDSRank": { 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 8301, 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 109.016379287, 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.013132921 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-PurgeQueue": { 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 41559, 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 6.740609299, 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000162193 2024-05-09T08:46:50.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "mds": { 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "request": 100806, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "reply": 97665, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "reply_latency": { 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 97665, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 1227.846099530, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.012572017 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_reply": 0, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "forward": 3137, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_complete": 191, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_keys": 0, 2024-05-09T08:46:50.990 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_commit": 935, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_split": 86, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_merge": 0, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes": 53807, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_top": 2370, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_bottom": 50461, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pin_tail": 976, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pinned": 50418, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_expired": 31976, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_with_caps": 1, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "caps": 1, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "subtrees": 6, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse": 167832, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_hit": 148873, 2024-05-09T08:46:50.991 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_forward": 3022, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_discover": 1142, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_dir_fetch": 29, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_remote_ino": 0, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_lock": 418, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "load_cent": 0, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "q": 0, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "exported": 413, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "exported_inodes": 21730, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "imported": 698, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "imported_inodes": 34215, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_dir_fetch": 0, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_backtrace_fetch": 432, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_peer_discover": 0, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rfiles": 27663, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rbytes": 621812736, 2024-05-09T08:46:50.992 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rsnaps": 6, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_fetch": 432, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_set_tag": 0, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_file_inodes": 432, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_inode_file_caps": 276, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_revoke": 14685, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_grant": 40513, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:46:50.993 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps": 47685, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps_dirty": 22487, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_cap_release": 752, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "process_request_cap_release": 48561 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_cache": { 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update": 4250, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update_receipt": 8361, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_try_discover": 4794, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_send_discover": 6023, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_handle_discover": 8465, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays": 155, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_delayed": 0, 2024-05-09T08:46:50.994 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_created": 21303, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_enqueued": 21149, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_reintegrated": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_migrated": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_processing": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_started": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_completed": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:46:50.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_exportdir": 429, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_flush": 0, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragmentdir": 86, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragstats": 0, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_inodestats": 0 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_log": { 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "evlrg": 5, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "evadd": 121581, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "evex": 33382, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "evtrm": 24581, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ev": 97000, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "evexg": 0, 2024-05-09T08:46:50.996 INFO:teuthology.orchestra.run.smithi184.stdout: "evexd": 8801, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "segadd": 170, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "segex": 42, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "segtrm": 31, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "seg": 139, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "segmjr": 8, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "segexg": 0, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "segexd": 11, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "expos": 65261312, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "wrpos": 453965154, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "rdpos": 0, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "jlat": { 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 3993, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 993.942929470, 2024-05-09T08:46:50.997 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.248921344 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "replayed": 0 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_mem": { 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "ino": 29904, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "ino+": 68260, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "ino-": 38356, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "dir": 1114, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "dir+": 4922, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "dir-": 3808, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "dn": 53807, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "dn+": 117801, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "dn-": 63994, 2024-05-09T08:46:50.998 INFO:teuthology.orchestra.run.smithi184.stdout: "cap": 1, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "cap+": 36564, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "cap-": 36563, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "rss": 233248, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "heap": 223612 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_server": { 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_client_request": 108415, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_server_request": 11466, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_request": 100806, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_session": 518, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_peer_request": 24046, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "req_create_latency": { 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 17866, 2024-05-09T08:46:50.999 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 76.777277298, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.004297396 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getattr_latency": { 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 103, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 29.845137188, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.289758613 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getfilelock_latency": { 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "req_link_latency": { 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookup_latency": { 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 36410, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 666.172060571, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.018296403 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookuphash_latency": { 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupino_latency": { 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.001 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupname_latency": { 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupparent_latency": { 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.002 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupsnap_latency": { 2024-05-09T08:46:51.003 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.003 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.003 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.003 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lssnap_latency": { 2024-05-09T08:46:51.003 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.003 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.003 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mkdir_latency": { 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 124, 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 28.802659463, 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.232279511 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mknod_latency": { 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mksnap_latency": { 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.004 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "req_open_latency": { 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 4787, 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 28.838664888, 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.006024371 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "req_readdir_latency": { 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 978, 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 6.314189727, 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.006456226 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rename_latency": { 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 17848, 2024-05-09T08:46:51.005 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 267.995783078, 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.015015451 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "req_renamesnap_latency": { 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "req_snapdiff_latency": { 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmdir_latency": { 2024-05-09T08:46:51.006 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 158, 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 52.786907878, 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.334094353 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmsnap_latency": { 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmxattr_latency": { 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setattr_latency": { 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setdirlayout_latency": { 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setfilelock_latency": { 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.008 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setlayout_latency": { 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setxattr_latency": { 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "req_symlink_latency": { 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.009 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "req_unlink_latency": { 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 19391, 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 70.313419439, 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.003626085 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getvxattr_latency": { 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:51.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_sessions": { 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "session_count": 2, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "session_add": 3, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "session_remove": 1, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_open": 2, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_stale": 0, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "total_load": 1185, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "average_load": 592, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "avg_session_uptime": 1187, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "mdthresh_evicted": 0 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "mempool": { 2024-05-09T08:46:51.011 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:51.012 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:51.013 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_items": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_bytes": 2642575, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_items": 28401, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_bytes": 264, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_items": 3, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_bytes": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_items": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:51.014 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_bytes": 10080, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_items": 201, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_items": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_bytes": 110203794, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_items": 1145233, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_items": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_items": 0 2024-05-09T08:46:51.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "objecter": { 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "op_active": 0, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "op_laggy": 0, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send": 52436, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send_bytes": 471882992, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "op_resend": 28, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "op_reply": 52436, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "op_latency": { 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 52436, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 3325.885277877, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.063427516 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "op_inflight": 0, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "oplen_avg": { 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 52436, 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 57640 2024-05-09T08:46:51.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "op": 52436, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "op_r": 1635, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "op_w": 50801, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "op_rmw": 0, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "op_pg": 0, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_stat": 152, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_create": 1823, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_read": 609, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_write": 4602, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writefull": 305, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writesame": 0, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_append": 0, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_zero": 1, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_truncate": 86, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_delete": 42433, 2024-05-09T08:46:51.017 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_mapext": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_sparse_read": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_clonerange": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_getxattr": 432, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_setxattr": 2755, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_rmxattr": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_call": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_watch": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_notify": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls": 0, 2024-05-09T08:46:51.018 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_other": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_active": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_send": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_resend": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_ping": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_active": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_send": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_resend": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_active": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_send": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_resend": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_active": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_send": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_resend": 0, 2024-05-09T08:46:51.019 INFO:teuthology.orchestra.run.smithi184.stdout: "command_active": 0, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "command_send": 0, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "command_resend": 0, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "map_epoch": 137, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "map_full": 0, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "map_inc": 48, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_sessions": 12, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_open": 12, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_close": 0, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_laggy": 0, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_wr": 3001, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_rd": 785, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_del": 656 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "oft": { 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_objs": 1, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:46:51.020 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_updates": 0, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_removes": 0 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "osd-slow-ops": { 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_ops_count": 25 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "purge_queue": { 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops": 0, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops_high_water": 17, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing": 0, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_high_water": 6, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed_ops": 42301, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed": 21149, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_item_in_journal": 0 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:51.021 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:51.022 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-05-09T08:46:51.022 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:51.022 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 331346, 2024-05-09T08:46:51.022 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 114326537, 2024-05-09T08:46:51.022 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:51.022 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 331346, 2024-05-09T08:46:51.022 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:46:51.022 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:46:51.022 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 331346, 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 114326537, 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_bytes": { 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:46:51.023 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 52436, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 475054885, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 52436, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 475054885, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_ops": { 2024-05-09T08:46:51.024 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 1024, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 52436, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 52436, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 52436, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 52436, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 21149, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 3188593, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 21149, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 609, 2024-05-09T08:46:51.026 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 3188593, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle-0x555da1dba500": { 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 121581, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 449770850, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 121581, 2024-05-09T08:46:51.027 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 3993, 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 449770850, 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:51.028 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-05-09T08:46:51.854 INFO:tasks.check_counter:Daemon mds.f mds.exported=413 2024-05-09T08:46:51.854 INFO:tasks.check_counter:Daemon mds.f mds.imported=698 2024-05-09T08:46:51.854 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=4250 2024-05-09T08:46:51.854 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=8361 2024-05-09T08:46:51.854 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-05-09T08:46:51.854 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:46:51.854 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:46:51.854 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=86 2024-05-09T08:46:51.854 DEBUG:tasks.check_counter:Getting stats from i 2024-05-09T08:46:51.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.i perf dump 2024-05-09T08:46:52.106 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:51 smithi184 ceph-mon[34509]: pgmap v662: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 88 op/s 2024-05-09T08:46:52.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:51 smithi028 ceph-mon[30119]: pgmap v662: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 88 op/s 2024-05-09T08:46:52.348 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:46:52.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:51 smithi165 ceph-mon[34540]: pgmap v662: 129 pgs: 54 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 56 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 88 op/s 2024-05-09T08:46:54.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:53 smithi028 ceph-mon[30119]: pgmap v663: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 94 op/s 2024-05-09T08:46:54.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:53 smithi165 ceph-mon[34540]: pgmap v663: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 94 op/s 2024-05-09T08:46:54.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:53 smithi184 ceph-mon[34509]: pgmap v663: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 94 op/s 2024-05-09T08:46:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:55 smithi028 ceph-mon[30119]: pgmap v664: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 101 op/s 2024-05-09T08:46:56.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:55 smithi165 ceph-mon[34540]: pgmap v664: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 101 op/s 2024-05-09T08:46:56.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:55 smithi184 ceph-mon[34509]: pgmap v664: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 101 op/s 2024-05-09T08:46:57.109 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-05-09T08:46:57.109 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:46:57.109 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 63579, 2024-05-09T08:46:57.109 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 65602, 2024-05-09T08:46:57.109 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 39321328, 2024-05-09T08:46:57.109 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 164198917, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 10, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 4, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 5.496678979, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 2.032352424, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 2.897552932, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.895761439, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 65602, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 17.029100303, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000259582 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.110 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 105521, 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.025463597, 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000241 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 29199, 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 29604, 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 15984031, 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 142557818, 2024-05-09T08:46:57.111 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 13, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 7, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 3.794499486, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 1.451029464, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 1.206795569, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 1.144184882, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 29603, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 7.582836297, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000256150 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 32822, 2024-05-09T08:46:57.112 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.009063437, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000276 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 1822160, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 872304 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 193326, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 281848, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 64267109, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 243675574, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 21, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 10, 2024-05-09T08:46:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 12.333143020, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 4.800271861, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 32.522240160, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 1.386847850, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 281844, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 68.558833861, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000243250 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 218004, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.080750974, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000370 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 4112, 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 131520 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "cct": { 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "total_workers": 1, 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "unhealthy_workers": 0 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-MDSRank": { 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 6480, 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 100.496809860, 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.015508766 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:57.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-PurgeQueue": { 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 30581, 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 4.646605621, 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000151944 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "mds": { 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "request": 92927, 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "reply": 87244, 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "reply_latency": { 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 87244, 2024-05-09T08:46:57.116 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 1386.486517744, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.015892055 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_reply": 0, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "forward": 5681, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_complete": 114, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_keys": 0, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_commit": 916, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_split": 94, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_merge": 0, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes": 27229, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_top": 2006, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_bottom": 14065, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pin_tail": 11158, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pinned": 24361, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_expired": 51595, 2024-05-09T08:46:57.117 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_with_caps": 0, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "caps": 0, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "subtrees": 5, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse": 159123, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_hit": 128600, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_forward": 5599, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_discover": 1060, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_dir_fetch": 14, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_remote_ino": 0, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_lock": 340, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "load_cent": 0, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "q": 0, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "exported": 697, 2024-05-09T08:46:57.118 INFO:teuthology.orchestra.run.smithi184.stdout: "exported_inodes": 36380, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "imported": 356, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "imported_inodes": 25740, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_dir_fetch": 0, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_backtrace_fetch": 268, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_peer_discover": 0, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rfiles": 27663, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rbytes": 621812736, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rsnaps": 6, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_fetch": 268, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_set_tag": 0, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:46:57.119 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_file_inodes": 268, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_inode_file_caps": 134, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_revoke": 14279, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_grant": 33201, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps": 45266, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps_dirty": 22282, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_cap_release": 412, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "process_request_cap_release": 41114 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_cache": { 2024-05-09T08:46:57.120 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update": 3010, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update_receipt": 9996, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_try_discover": 5627, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_send_discover": 7080, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_handle_discover": 4895, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays": 86, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_delayed": 0, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_created": 16230, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_enqueued": 16144, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_reintegrated": 0, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_migrated": 0, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_processing": 0, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:46:57.121 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_started": 0, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_completed": 0, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_exportdir": 699, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_flush": 0, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragmentdir": 94, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragstats": 0, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_inodestats": 0 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_log": { 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "evlrg": 1, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "evadd": 108416, 2024-05-09T08:46:57.122 INFO:teuthology.orchestra.run.smithi184.stdout: "evex": 20260, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "evtrm": 12291, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "ev": 96125, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "evexg": 0, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "evexd": 7969, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "segadd": 154, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "segex": 26, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "segtrm": 17, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "seg": 137, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "segmjr": 8, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "segexg": 0, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "segexd": 9, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "expos": 34402387, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "wrpos": 360841350, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "rdpos": 0, 2024-05-09T08:46:57.123 INFO:teuthology.orchestra.run.smithi184.stdout: "jlat": { 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 2762, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 469.148008079, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.169858076 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "replayed": 0 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_mem": { 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "ino": 12983, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "ino+": 51035, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "ino-": 38052, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "dir": 500, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "dir+": 5793, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "dir-": 5293, 2024-05-09T08:46:57.124 INFO:teuthology.orchestra.run.smithi184.stdout: "dn": 27229, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "dn+": 97110, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "dn-": 69881, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "cap": 0, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "cap+": 31414, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "cap-": 31414, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "rss": 173136, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "heap": 223612 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_server": { 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_client_request": 103143, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_server_request": 1706, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_request": 92927, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_session": 420, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_peer_request": 9920, 2024-05-09T08:46:57.125 INFO:teuthology.orchestra.run.smithi184.stdout: "req_create_latency": { 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 15955, 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 63.352592757, 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.003970704 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getattr_latency": { 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 476, 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 36.780872153, 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.077270739 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getfilelock_latency": { 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "req_link_latency": { 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookup_latency": { 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 34124, 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 980.779255846, 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.028741626 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookuphash_latency": { 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupino_latency": { 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupname_latency": { 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupparent_latency": { 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.128 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupsnap_latency": { 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lssnap_latency": { 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.129 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mkdir_latency": { 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 112, 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 20.911021197, 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.186705546 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mknod_latency": { 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mksnap_latency": { 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "req_open_latency": { 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 5047, 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 19.526302206, 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.003868892 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "req_readdir_latency": { 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 660, 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 4.995779366, 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.007569362 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rename_latency": { 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 15974, 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 171.224049397, 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.010718921 2024-05-09T08:46:57.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "req_renamesnap_latency": { 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "req_snapdiff_latency": { 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmdir_latency": { 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 86, 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 25.560759856, 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.297218137 2024-05-09T08:46:57.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmsnap_latency": { 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmxattr_latency": { 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setattr_latency": { 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.133 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setdirlayout_latency": { 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setfilelock_latency": { 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setlayout_latency": { 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.134 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setxattr_latency": { 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "req_symlink_latency": { 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "req_unlink_latency": { 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 14810, 2024-05-09T08:46:57.135 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 63.355884966, 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.004277912 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getvxattr_latency": { 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_sessions": { 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "session_count": 1, 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "session_add": 2, 2024-05-09T08:46:57.136 INFO:teuthology.orchestra.run.smithi184.stdout: "session_remove": 1, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_open": 1, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_stale": 0, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "total_load": 11709, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "average_load": 11709, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "avg_session_uptime": 1155, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "mdthresh_evicted": 0 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "mempool": { 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_items": 0, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:46:57.137 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_items": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_items": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:46:57.138 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_items": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_items": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_bytes": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_items": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:46:57.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_bytes": 1596961, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_items": 11900, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_bytes": 1056, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_items": 12, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_bytes": 0, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_items": 0, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_items": 0, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_items": 0, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_bytes": 10080, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_items": 201, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:46:57.140 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_bytes": 0, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_items": 0, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_bytes": 50540917, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_items": 554986, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_bytes": 0, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_items": 0, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_bytes": 0, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_items": 0 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "objecter": { 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "op_active": 9, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "op_laggy": 0, 2024-05-09T08:46:57.141 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send": 38315, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send_bytes": 378237535, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "op_resend": 33, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "op_reply": 38306, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "op_latency": { 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 38306, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 2955.913705471, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.077165814 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "op_inflight": 9, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "oplen_avg": { 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 38315, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 42598 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.142 INFO:teuthology.orchestra.run.smithi184.stdout: "op": 38315, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "op_r": 1082, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "op_w": 37233, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "op_rmw": 0, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "op_pg": 0, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_stat": 66, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_create": 1462, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_read": 428, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_write": 3191, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writefull": 241, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writesame": 0, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_append": 0, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_zero": 0, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_truncate": 94, 2024-05-09T08:46:57.143 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_delete": 30760, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_mapext": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_sparse_read": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_clonerange": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_getxattr": 275, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_setxattr": 2207, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_rmxattr": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_call": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_watch": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_notify": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:46:57.144 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_other": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_active": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_send": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_resend": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_ping": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_active": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_send": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_resend": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_active": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_send": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_resend": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_active": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_send": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_resend": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "command_active": 0, 2024-05-09T08:46:57.145 INFO:teuthology.orchestra.run.smithi184.stdout: "command_send": 0, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "command_resend": 0, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "map_epoch": 137, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "map_full": 0, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "map_inc": 48, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_sessions": 12, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_open": 12, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_close": 0, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_laggy": 0, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_wr": 2869, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_rd": 500, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_del": 505 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.146 INFO:teuthology.orchestra.run.smithi184.stdout: "oft": { 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_objs": 1, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_updates": 0, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_removes": 0 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "osd-slow-ops": { 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_ops_count": 8 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "purge_queue": { 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops": 10, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops_high_water": 10, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing": 5, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_high_water": 5, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed_ops": 30646, 2024-05-09T08:46:57.147 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed": 15323, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_item_in_journal": 1276 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 286100, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 96585263, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 286100, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 286100, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 96585263, 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_bytes": { 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 38315, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 380692188, 2024-05-09T08:46:57.149 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 38306, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 380692188, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_ops": { 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 9, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 1024, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:57.150 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 38315, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 38315, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 38306, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 38306, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 16144, 2024-05-09T08:46:57.151 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 2465800, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 16144, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 429, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 2465800, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle-0x55c79d48c500": { 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:46:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 108416, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 356647046, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 108416, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 2762, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 356647046, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:46:57.153 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:46:57.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:57.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:46:57.154 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-05-09T08:46:57.973 INFO:tasks.check_counter:Daemon mds.i mds.exported=697 2024-05-09T08:46:57.973 INFO:tasks.check_counter:Daemon mds.i mds.imported=356 2024-05-09T08:46:57.973 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=3010 2024-05-09T08:46:57.973 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=9996 2024-05-09T08:46:57.973 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-05-09T08:46:57.974 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:46:57.974 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:46:57.974 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=94 2024-05-09T08:46:57.974 DEBUG:tasks.check_counter:Getting stats from l 2024-05-09T08:46:57.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 -- ceph daemon mds.l perf dump 2024-05-09T08:46:58.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:57 smithi028 ceph-mon[30119]: pgmap v665: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 0 B/s wr, 101 op/s 2024-05-09T08:46:58.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:57 smithi165 ceph-mon[34540]: pgmap v665: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 0 B/s wr, 101 op/s 2024-05-09T08:46:58.361 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:57 smithi184 ceph-mon[34509]: pgmap v665: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 0 B/s wr, 101 op/s 2024-05-09T08:46:58.415 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:47:00.173 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:46:59 smithi184 ceph-mon[34509]: pgmap v666: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 97 op/s 2024-05-09T08:47:00.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:46:59 smithi028 ceph-mon[30119]: pgmap v666: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 97 op/s 2024-05-09T08:47:00.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:46:59 smithi165 ceph-mon[34540]: pgmap v666: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 97 op/s 2024-05-09T08:47:00.547 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-05-09T08:47:00.547 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:47:00.547 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 3126, 2024-05-09T08:47:00.547 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 3128, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 139351648, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 994075, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 8, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 4, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.378189637, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.076071739, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.158618799, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.123960870, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 3126, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.211992560, 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000067815 2024-05-09T08:47:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 1232, 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 1182, 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 98612719, 2024-05-09T08:47:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 481988, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 6, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 5, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.196288517, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.038014195, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.128085071, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.035110011, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1182, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.086611292, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000073275 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 1798240, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 243744 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_messages": 854, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages": 1141, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_bytes": 121909286, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_bytes": 933658, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_created_connections": 8, 2024-05-09T08:47:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_active_connections": 6, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_total_time": 0.158667798, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_send_time": 0.030251682, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_recv_time": 0.095001943, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_running_fast_dispatch_time": 0.025670794, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 1138, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.069076651, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000060700 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "msgr_send_encrypted_bytes": 663040 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "cct": { 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "total_workers": 1, 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "unhealthy_workers": 0 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-MDSRank": { 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 7739, 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.213749600, 2024-05-09T08:47:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000027619 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "finisher-PurgeQueue": { 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "queue_len": 0, 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "complete_latency": { 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "mds": { 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "request": 0, 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "reply": 0, 2024-05-09T08:47:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "reply_latency": { 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_reply": 0, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "forward": 0, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_complete": 0, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_fetch_keys": 0, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_commit": 0, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_split": 0, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_merge": 0, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes": 50647, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_top": 1864, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_bottom": 48783, 2024-05-09T08:47:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pin_tail": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_pinned": 47983, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_expired": 4, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "inodes_with_caps": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "caps": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "subtrees": 1, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_hit": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_forward": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_discover": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_remote_ino": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "traverse_lock": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "load_cent": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "q": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "exported": 0, 2024-05-09T08:47:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "exported_inodes": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "imported": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "imported_inodes": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_dir_fetch": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_backtrace_fetch": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "openino_peer_discover": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rfiles": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rbytes": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "root_rsnaps": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_set_tag": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:47:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "scrub_file_inodes": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_grant": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_caps_dirty": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_cap_release": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "process_request_cap_release": 0 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_cache": { 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_update_receipt": 0, 2024-05-09T08:47:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_try_discover": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_send_discover": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "dir_handle_discover": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_delayed": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_created": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_enqueued": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_reintegrated": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "strays_migrated": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_processing": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_started": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "recovery_completed": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:47:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_exportdir": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_flush": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_fragstats": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "ireq_inodestats": 0 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_log": { 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "evlrg": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "evadd": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "evex": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "evtrm": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "ev": 108416, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "evexg": 0, 2024-05-09T08:47:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "evexd": 0, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "segadd": 0, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "segex": 0, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "segtrm": 0, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "seg": 137, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "segmjr": 10, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "segexg": 0, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "segexd": 0, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "expos": 34402387, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "wrpos": 360841350, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "rdpos": 360823609, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "jlat": { 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "replayed": 108416 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_mem": { 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "ino": 13391, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "ino+": 50167, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "ino-": 36776, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "dir": 489, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "dir+": 1456, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "dir-": 967, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "dn": 50647, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "dn+": 92571, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "dn-": 41924, 2024-05-09T08:47:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "cap": 0, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "cap+": 0, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "cap-": 0, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "rss": 140084, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "heap": 215420 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_server": { 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_client_request": 0, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "dispatch_server_request": 0, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_request": 0, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_client_session": 0, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "handle_peer_request": 0, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "req_create_latency": { 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getattr_latency": { 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getfilelock_latency": { 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "req_link_latency": { 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookup_latency": { 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookuphash_latency": { 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupino_latency": { 2024-05-09T08:47:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupname_latency": { 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupparent_latency": { 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lookupsnap_latency": { 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "req_lssnap_latency": { 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mkdir_latency": { 2024-05-09T08:47:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mknod_latency": { 2024-05-09T08:47:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "req_mksnap_latency": { 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "req_open_latency": { 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "req_readdir_latency": { 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rename_latency": { 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "req_renamesnap_latency": { 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "req_snapdiff_latency": { 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmdir_latency": { 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmsnap_latency": { 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "req_rmxattr_latency": { 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setattr_latency": { 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setdirlayout_latency": { 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setfilelock_latency": { 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setlayout_latency": { 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "req_setxattr_latency": { 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "req_symlink_latency": { 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "req_unlink_latency": { 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "req_getvxattr_latency": { 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:47:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_sessions": { 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "session_count": 1, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "session_add": 2, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "session_remove": 1, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_open": 2, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "sessions_stale": 0, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "total_load": 0, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "average_load": 0, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "avg_session_uptime": 3919, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "mdthresh_evicted": 0 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "mempool": { 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "bloom_filter_items": 0, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:47:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_extent_items": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_blob_items": 0, 2024-05-09T08:47:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_txc_items": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluestore_writing_items": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_bytes": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_items": 0, 2024-05-09T08:47:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_bytes": 746822, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_anon_items": 13774, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "buffer_meta_items": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_bytes": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_items": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_mapbl_items": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_pglog_items": 0, 2024-05-09T08:47:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_bytes": 10080, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_items": 201, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_bytes": 0, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "pgmap_items": 0, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_bytes": 67734079, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_co_items": 713052, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_bytes": 0, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_1_items": 0, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_bytes": 0, 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "unittest_2_items": 0 2024-05-09T08:47:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "objecter": { 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "op_active": 0, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "op_laggy": 0, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send": 4726, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "op_send_bytes": 21, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "op_resend": 3, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "op_reply": 4726, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "op_latency": { 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 4726, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 71.291641100, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.015084985 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "op_inflight": 0, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "oplen_avg": { 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 4726, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 4727 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "op": 4726, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "op_r": 4726, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "op_w": 0, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "op_rmw": 0, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "op_pg": 0, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_stat": 2854, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_create": 0, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_read": 1870, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_write": 0, 2024-05-09T08:47:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writefull": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_writesame": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_append": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_zero": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_truncate": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_delete": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_mapext": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_sparse_read": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_clonerange": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_getxattr": 1, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_setxattr": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_rmxattr": 0, 2024-05-09T08:47:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_call": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_watch": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_notify": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "osdop_other": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_active": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_send": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_resend": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "linger_ping": 0, 2024-05-09T08:47:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_active": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_send": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "poolop_resend": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_active": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_send": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "poolstat_resend": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_active": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_send": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "statfs_resend": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "command_active": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "command_send": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "command_resend": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "map_epoch": 137, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "map_full": 0, 2024-05-09T08:47:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "map_inc": 48, 2024-05-09T08:47:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_sessions": 11, 2024-05-09T08:47:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_open": 11, 2024-05-09T08:47:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_session_close": 0, 2024-05-09T08:47:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_laggy": 0, 2024-05-09T08:47:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_wr": 0, 2024-05-09T08:47:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_rd": 2, 2024-05-09T08:47:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_del": 0 2024-05-09T08:47:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "oft": { 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_objs": 0, 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_updates": 0, 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "omap_total_removes": 0 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "osd-slow-ops": { 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "slow_ops_count": 0 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "purge_queue": { 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops": 0, 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing": 0, 2024-05-09T08:47:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executing_high_water": 0, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed_ops": 0, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_executed": 0, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pq_item_in_journal": 0 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 5208, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 359377792, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 5208, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 5208, 2024-05-09T08:47:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 359377792, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_bytes": { 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 104857600, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 4726, 2024-05-09T08:47:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 356763387, 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 4726, 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 356763387, 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-objecter_ops": { 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 1024, 2024-05-09T08:47:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 4726, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 4726, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 4726, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 4726, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 0, 2024-05-09T08:47:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "throttle-write_buf_throttle-0x560d48fb26e0": { 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 0, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "max": 3758096384, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "get_started": 0, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "get": 0, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "get_sum": 0, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_fail": 0, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "get_or_fail_success": 0, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "take": 0, 2024-05-09T08:47:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "take_sum": 0, 2024-05-09T08:47:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "put": 0, 2024-05-09T08:47:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "put_sum": 0, 2024-05-09T08:47:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "wait": { 2024-05-09T08:47:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "avgcount": 0, 2024-05-09T08:47:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "sum": 0.000000000, 2024-05-09T08:47:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "avgtime": 0.000000000 2024-05-09T08:47:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:47:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-05-09T08:47:00.593 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-05-09T08:47:02.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:01 smithi028 ceph-mon[30119]: pgmap v667: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 107 op/s 2024-05-09T08:47:02.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:01 smithi165 ceph-mon[34540]: pgmap v667: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 107 op/s 2024-05-09T08:47:02.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:01 smithi184 ceph-mon[34509]: pgmap v667: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 107 op/s 2024-05-09T08:47:03.249 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-05-09T08:47:03.250 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-05-09T08:47:03.250 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-05-09T08:47:03.250 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-05-09T08:47:03.250 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-05-09T08:47:03.250 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:47:03.250 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:47:03.250 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-05-09T08:47:03.253 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-09T08:47:03.265 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-09T08:47:03.265 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:47:03.265 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:47:03.313 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:47:03.313 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:47:03.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:47:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.636+0000 7fa0c1934640 1 -- 172.21.15.28:0/1294603524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 msgr2=0x7fa0bc1151d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.636+0000 7fa0c1934640 1 --2- 172.21.15.28:0/1294603524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 0x7fa0bc1151d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4005fd0 tx=0x7fa0b402e720 comp rx=0 tx=0).stop 2024-05-09T08:47:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.639+0000 7fa0c1934640 1 -- 172.21.15.28:0/1294603524 shutdown_connections 2024-05-09T08:47:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.639+0000 7fa0c1934640 1 --2- 172.21.15.28:0/1294603524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 0x7fa0bc1151d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.639+0000 7fa0c1934640 1 --2- 172.21.15.28:0/1294603524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc109450 0x7fa0bc10cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.639+0000 7fa0c1934640 1 --2- 172.21.15.28:0/1294603524 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0bc108aa0 0x7fa0bc108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.639+0000 7fa0c1934640 1 -- 172.21.15.28:0/1294603524 >> 172.21.15.28:0/1294603524 conn(0x7fa0bc06c950 msgr2=0x7fa0bc06cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.640+0000 7fa0c1934640 1 -- 172.21.15.28:0/1294603524 shutdown_connections 2024-05-09T08:47:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.640+0000 7fa0c1934640 1 -- 172.21.15.28:0/1294603524 wait complete. 2024-05-09T08:47:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.640+0000 7fa0c1934640 1 Processor -- start 2024-05-09T08:47:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.645+0000 7fa0c1934640 1 -- start start 2024-05-09T08:47:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.645+0000 7fa0c1934640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc108aa0 0x7fa0bc1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.645+0000 7fa0c1934640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 0x7fa0bc1a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.645+0000 7fa0c1934640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0bc19e670 0x7fa0bc19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.645+0000 7fa0c1934640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc1182a0 con 0x7fa0bc108aa0 2024-05-09T08:47:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.645+0000 7fa0c1934640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc118120 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.645+0000 7fa0c1934640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc118420 con 0x7fa0bc19e670 2024-05-09T08:47:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.646+0000 7fa0ba7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 0x7fa0bc1a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.646+0000 7fa0ba7fc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 0x7fa0bc1a59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:57674/0 (socket says 172.21.15.28:57674) 2024-05-09T08:47:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.646+0000 7fa0ba7fc640 1 -- 172.21.15.28:0/2911203548 learned_addr learned my addr 172.21.15.28:0/2911203548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:03.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.648+0000 7fa0ba7fc640 1 -- 172.21.15.28:0/2911203548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0bc19e670 msgr2=0x7fa0bc19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:03.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.648+0000 7fa0ba7fc640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0bc19e670 0x7fa0bc19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.648+0000 7fa0ba7fc640 1 -- 172.21.15.28:0/2911203548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc108aa0 msgr2=0x7fa0bc1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:03.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.648+0000 7fa0ba7fc640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc108aa0 0x7fa0bc1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.648+0000 7fa0ba7fc640 1 -- 172.21.15.28:0/2911203548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0bc1a2fb0 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.650+0000 7fa0ba7fc640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 0x7fa0bc1a59d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000d770 tx=0x7fa0b000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:03.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.654+0000 7fa0c0932640 1 -- 172.21.15.28:0/2911203548 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0017070 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.654+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0bc1a32a0 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.654+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0bc1a37f0 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.654+0000 7fa0c0932640 1 -- 172.21.15.28:0/2911203548 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0b0009c30 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.654+0000 7fa0c0932640 1 -- 172.21.15.28:0/2911203548 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b000f620 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.659+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0a80052c0 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.661+0000 7fa0c0932640 1 -- 172.21.15.28:0/2911203548 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa0b001a070 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.662+0000 7fa0c0932640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa0a40a2f10 0x7fa0a40a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.662+0000 7fa0c0932640 1 -- 172.21.15.28:0/2911203548 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9771+0+0 (secure 0 0 0) 0x7fa0b00cf880 con 0x7fa0bc10d5c0 2024-05-09T08:47:03.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.664+0000 7fa0baffd640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa0a40a2f10 0x7fa0a40a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.668+0000 7fa0baffd640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa0a40a2f10 0x7fa0a40a53d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00b9a0 tx=0x7fa0ac005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:03.669+0000 7fa0c0932640 1 -- 172.21.15.28:0/2911203548 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa0b0009de0 con 0x7fa0bc10d5c0 2024-05-09T08:47:04.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.045+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0a8005880 con 0x7fa0bc10d5c0 2024-05-09T08:47:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.047+0000 7fa0c0932640 1 -- 172.21.15.28:0/2911203548 <== mon.2 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) ==== 74+0+25746 (secure 0 0 0) 0x7fa0b00ac4b0 con 0x7fa0bc10d5c0 2024-05-09T08:47:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:47:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":137,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:46:45.765685+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":35,"snap_epoch":135,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":35,"snap_epoch":135,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3330063293":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3745037139":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/2130261041":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/273816117":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":19,"length":5},{"begin":25,"length":1},{"begin":27,"length":1},{"begin":29,"length":1},{"begin":31,"length":1},{"begin":33,"length":1},{"begin":35,"length":1}]},{"pool":4,"snaps":[{"begin":18,"length":6},{"begin":25,"length":1},{"begin":27,"length":1},{"begin":29,"length":1},{"begin":31,"length":1},{"begin":33,"length":1},{"begin":35,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":16,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa0a40a2f10 msgr2=0x7fa0a40a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa0a40a2f10 0x7fa0a40a53d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00b9a0 tx=0x7fa0ac005f90 comp rx=0 tx=0).stop 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 msgr2=0x7fa0bc1a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 0x7fa0bc1a59d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000d770 tx=0x7fa0b000dc30 comp rx=0 tx=0).stop 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 shutdown_connections 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fa0a40a2f10 0x7fa0a40a53d0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00b9a0 tx=0x7fa0ac005f90 comp rx=0 tx=0).stop 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0bc19e670 0x7fa0bc19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0bc10d5c0 0x7fa0bc1a59d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 --2- 172.21.15.28:0/2911203548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0bc108aa0 0x7fa0bc1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.051+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 >> 172.21.15.28:0/2911203548 conn(0x7fa0bc06c950 msgr2=0x7fa0bc0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:04.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.052+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 shutdown_connections 2024-05-09T08:47:04.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.053+0000 7fa0c1934640 1 -- 172.21.15.28:0/2911203548 wait complete. 2024-05-09T08:47:04.079 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:19:12.083519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:23:49.496601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:23:50.303180+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 35, 'snap_epoch': 135, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:23:54.086488+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 35, 'snap_epoch': 135, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:47:04.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:47:04.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:03 smithi028 ceph-mon[30119]: pgmap v668: 129 pgs: 54 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 59 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 170 B/s wr, 99 op/s 2024-05-09T08:47:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.286+0000 7eff25989640 1 -- 172.21.15.28:0/2527293431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2010e0b0 msgr2=0x7eff2010e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.286+0000 7eff25989640 1 --2- 172.21.15.28:0/2527293431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2010e0b0 0x7eff2010e530 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7eff08009890 tx=0x7eff0802a370 comp rx=0 tx=0).stop 2024-05-09T08:47:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.289+0000 7eff25989640 1 -- 172.21.15.28:0/2527293431 shutdown_connections 2024-05-09T08:47:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.289+0000 7eff25989640 1 --2- 172.21.15.28:0/2527293431 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff2011b010 0x7eff2011d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.289+0000 7eff25989640 1 --2- 172.21.15.28:0/2527293431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2010e0b0 0x7eff2010e530 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.289+0000 7eff25989640 1 --2- 172.21.15.28:0/2527293431 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff20110080 0x7eff2010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.289+0000 7eff25989640 1 -- 172.21.15.28:0/2527293431 >> 172.21.15.28:0/2527293431 conn(0x7eff2006c940 msgr2=0x7eff2006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.295+0000 7eff25989640 1 -- 172.21.15.28:0/2527293431 shutdown_connections 2024-05-09T08:47:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.295+0000 7eff25989640 1 -- 172.21.15.28:0/2527293431 wait complete. 2024-05-09T08:47:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.297+0000 7eff25989640 1 Processor -- start 2024-05-09T08:47:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.299+0000 7eff25989640 1 -- start start 2024-05-09T08:47:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.300+0000 7eff25989640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff2010e0b0 0x7eff201a6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.300+0000 7eff25989640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff20110080 0x7eff201a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.300+0000 7eff25989640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff2011b010 0x7eff201ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.300+0000 7eff25989640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff201209e0 con 0x7eff20110080 2024-05-09T08:47:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.300+0000 7eff25989640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7eff20120860 con 0x7eff2010e0b0 2024-05-09T08:47:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.300+0000 7eff25989640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7eff20120b60 con 0x7eff2011b010 2024-05-09T08:47:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.302+0000 7eff1f7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff2011b010 0x7eff201ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.302+0000 7eff1e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff20110080 0x7eff201a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.302+0000 7eff1e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff20110080 0x7eff201a74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52524/0 (socket says 172.21.15.28:52524) 2024-05-09T08:47:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.302+0000 7eff1e7fc640 1 -- 172.21.15.28:0/2030080739 learned_addr learned my addr 172.21.15.28:0/2030080739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.305+0000 7eff1e7fc640 1 -- 172.21.15.28:0/2030080739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff2011b010 msgr2=0x7eff201ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.305+0000 7eff1e7fc640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff2011b010 0x7eff201ab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.305+0000 7eff1e7fc640 1 -- 172.21.15.28:0/2030080739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff2010e0b0 msgr2=0x7eff201a6f80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:47:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.305+0000 7eff1e7fc640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff2010e0b0 0x7eff201a6f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.305+0000 7eff1e7fc640 1 -- 172.21.15.28:0/2030080739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff201abf70 con 0x7eff20110080 2024-05-09T08:47:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.305+0000 7eff1e7fc640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff20110080 0x7eff201a74c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7eff08035040 tx=0x7eff08033d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.310+0000 7eff24987640 1 -- 172.21.15.28:0/2030080739 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08037040 con 0x7eff20110080 2024-05-09T08:47:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.310+0000 7eff24987640 1 -- 172.21.15.28:0/2030080739 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff08040070 con 0x7eff20110080 2024-05-09T08:47:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.310+0000 7eff24987640 1 -- 172.21.15.28:0/2030080739 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08003b50 con 0x7eff20110080 2024-05-09T08:47:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.311+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff201ac200 con 0x7eff20110080 2024-05-09T08:47:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.311+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff201ac5e0 con 0x7eff20110080 2024-05-09T08:47:04.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.314+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff20068650 con 0x7eff20110080 2024-05-09T08:47:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.318+0000 7eff24987640 1 -- 172.21.15.28:0/2030080739 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7eff08003cf0 con 0x7eff20110080 2024-05-09T08:47:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.319+0000 7eff24987640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7eff0c0a2f00 0x7eff0c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.319+0000 7eff24987640 1 -- 172.21.15.28:0/2030080739 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9771+0+0 (secure 0 0 0) 0x7eff080ef920 con 0x7eff20110080 2024-05-09T08:47:04.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.327+0000 7eff24987640 1 -- 172.21.15.28:0/2030080739 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7eff080bc110 con 0x7eff20110080 2024-05-09T08:47:04.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.336+0000 7eff1effd640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7eff0c0a2f00 0x7eff0c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.346+0000 7eff1effd640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7eff0c0a2f00 0x7eff0c0a53c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7eff1800a890 tx=0x7eff18005b80 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:04.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:03 smithi165 ceph-mon[34540]: pgmap v668: 129 pgs: 54 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 59 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 170 B/s wr, 99 op/s 2024-05-09T08:47:04.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:03 smithi184 ceph-mon[34509]: pgmap v668: 129 pgs: 54 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 59 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 170 B/s wr, 99 op/s 2024-05-09T08:47:04.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.712+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff201b3d00 con 0x7eff20110080 2024-05-09T08:47:04.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.713+0000 7eff24987640 1 -- 172.21.15.28:0/2030080739 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v137) ==== 93+0+10 (secure 0 0 0) 0x7eff080bba10 con 0x7eff20110080 2024-05-09T08:47:04.712 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-09T08:47:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7eff0c0a2f00 msgr2=0x7eff0c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7eff0c0a2f00 0x7eff0c0a53c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7eff1800a890 tx=0x7eff18005b80 comp rx=0 tx=0).stop 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff20110080 msgr2=0x7eff201a74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff20110080 0x7eff201a74c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7eff08035040 tx=0x7eff08033d10 comp rx=0 tx=0).stop 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 shutdown_connections 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7eff0c0a2f00 0x7eff0c0a53c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff2011b010 0x7eff201ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff20110080 0x7eff201a74c0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 --2- 172.21.15.28:0/2030080739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff2010e0b0 0x7eff201a6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.717+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 >> 172.21.15.28:0/2030080739 conn(0x7eff2006c940 msgr2=0x7eff20119ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.718+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 shutdown_connections 2024-05-09T08:47:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.718+0000 7eff25989640 1 -- 172.21.15.28:0/2030080739 wait complete. 2024-05-09T08:47:04.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:47:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.934+0000 7f1108b34640 1 -- 172.21.15.28:0/2809884447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11041010c0 msgr2=0x7f110410d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.934+0000 7f1108b34640 1 --2- 172.21.15.28:0/2809884447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11041010c0 0x7f110410d5d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009890 tx=0x7f10f402a3b0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.940+0000 7f1108b34640 1 -- 172.21.15.28:0/2809884447 shutdown_connections 2024-05-09T08:47:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.941+0000 7f1108b34640 1 --2- 172.21.15.28:0/2809884447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11041010c0 0x7f110410d5d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.941+0000 7f1108b34640 1 --2- 172.21.15.28:0/2809884447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1104100700 0x7f1104100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.941+0000 7f1108b34640 1 --2- 172.21.15.28:0/2809884447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1104106710 0x7f1104106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.941+0000 7f1108b34640 1 -- 172.21.15.28:0/2809884447 >> 172.21.15.28:0/2809884447 conn(0x7f1104077830 msgr2=0x7f1104077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.948+0000 7f1108b34640 1 -- 172.21.15.28:0/2809884447 shutdown_connections 2024-05-09T08:47:04.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.950+0000 7f1108b34640 1 -- 172.21.15.28:0/2809884447 wait complete. 2024-05-09T08:47:04.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.951+0000 7f1108b34640 1 Processor -- start 2024-05-09T08:47:04.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.952+0000 7f1108b34640 1 -- start start 2024-05-09T08:47:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1108b34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1104100700 0x7f1104195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1108b34640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11041010c0 0x7f11041962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1108b34640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1104106710 0x7f11041967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1108b34640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f110410fa10 con 0x7f1104100700 2024-05-09T08:47:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1108b34640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f110410f890 con 0x7f11041010c0 2024-05-09T08:47:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1108b34640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f110410fb90 con 0x7f1104106710 2024-05-09T08:47:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f110259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1104100700 0x7f1104195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1102d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1104106710 0x7f11041967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1102d9d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1104106710 0x7f11041967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48208/0 (socket says 172.21.15.28:48208) 2024-05-09T08:47:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f110259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1104100700 0x7f1104195d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34454/0 (socket says 172.21.15.28:34454) 2024-05-09T08:47:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1102d9d640 1 -- 172.21.15.28:0/800186421 learned_addr learned my addr 172.21.15.28:0/800186421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1102d9d640 1 -- 172.21.15.28:0/800186421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11041010c0 msgr2=0x7f11041962b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:47:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1102d9d640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11041010c0 0x7f11041962b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1102d9d640 1 -- 172.21.15.28:0/800186421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1104100700 msgr2=0x7f1104195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.953+0000 7f1102d9d640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1104100700 0x7f1104195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:04.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.954+0000 7f1102d9d640 1 -- 172.21.15.28:0/800186421 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11041aa9f0 con 0x7f1104106710 2024-05-09T08:47:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.962+0000 7f1102d9d640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1104106710 0x7f11041967f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f10f402a920 tx=0x7f10f4033d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.964+0000 7f10f37fe640 1 -- 172.21.15.28:0/800186421 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4037040 con 0x7f1104106710 2024-05-09T08:47:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.964+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11041aac80 con 0x7f1104106710 2024-05-09T08:47:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.965+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11041ab170 con 0x7f1104106710 2024-05-09T08:47:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.965+0000 7f10f37fe640 1 -- 172.21.15.28:0/800186421 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f4040070 con 0x7f1104106710 2024-05-09T08:47:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.965+0000 7f10f37fe640 1 -- 172.21.15.28:0/800186421 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f40053d0 con 0x7f1104106710 2024-05-09T08:47:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.966+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1104101e40 con 0x7f1104106710 2024-05-09T08:47:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.971+0000 7f10f37fe640 1 -- 172.21.15.28:0/800186421 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f10f40304f0 con 0x7f1104106710 2024-05-09T08:47:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.972+0000 7f10f37fe640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f10ec0a2f00 0x7f10ec0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:04.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.972+0000 7f10f37fe640 1 -- 172.21.15.28:0/800186421 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9771+0+0 (secure 0 0 0) 0x7f10f40ef220 con 0x7f1104106710 2024-05-09T08:47:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.973+0000 7f110259c640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f10ec0a2f00 0x7f10ec0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.976+0000 7f10f37fe640 1 -- 172.21.15.28:0/800186421 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f10f4049020 con 0x7f1104106710 2024-05-09T08:47:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:04.977+0000 7f110259c640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f10ec0a2f00 0x7f10ec0a53c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f10f8005fd0 tx=0x7f10f8009260 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:05.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2911203548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:05.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:04 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2030080739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:47:05.251 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2911203548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:05.251 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:04 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2030080739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:47:05.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2911203548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:05.257 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:04 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2030080739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:47:05.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.338+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f110402d1c0 con 0x7f1104106710 2024-05-09T08:47:05.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.339+0000 7f10f37fe640 1 -- 172.21.15.28:0/800186421 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v137) ==== 104+0+11 (secure 0 0 0) 0x7f10f40bba90 con 0x7f1104106710 2024-05-09T08:47:05.338 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:47:05.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f10ec0a2f00 msgr2=0x7f10ec0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:05.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f10ec0a2f00 0x7f10ec0a53c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f10f8005fd0 tx=0x7f10f8009260 comp rx=0 tx=0).stop 2024-05-09T08:47:05.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1104106710 msgr2=0x7f11041967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:05.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1104106710 0x7f11041967f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f10f402a920 tx=0x7f10f4033d10 comp rx=0 tx=0).stop 2024-05-09T08:47:05.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 shutdown_connections 2024-05-09T08:47:05.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f10ec0a2f00 0x7f10ec0a53c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1104106710 0x7f11041967f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11041010c0 0x7f11041962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 --2- 172.21.15.28:0/800186421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1104100700 0x7f1104195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.343+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 >> 172.21.15.28:0/800186421 conn(0x7f1104077830 msgr2=0x7f1104102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.344+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 shutdown_connections 2024-05-09T08:47:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.344+0000 7f1108b34640 1 -- 172.21.15.28:0/800186421 wait complete. 2024-05-09T08:47:05.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:47:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.563+0000 7fc3e39dc640 1 -- 172.21.15.28:0/2482118015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc0fffb0 msgr2=0x7fc3dc100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.563+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/2482118015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc0fffb0 0x7fc3dc100390 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009860 tx=0x7fc3cc02a340 comp rx=0 tx=0).stop 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.564+0000 7fc3e39dc640 1 -- 172.21.15.28:0/2482118015 shutdown_connections 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.564+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/2482118015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc106bc0 0x7fc3dc10e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.564+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/2482118015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3dc100960 0x7fc3dc106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.564+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/2482118015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc0fffb0 0x7fc3dc100390 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.564+0000 7fc3e39dc640 1 -- 172.21.15.28:0/2482118015 >> 172.21.15.28:0/2482118015 conn(0x7fc3dc076cd0 msgr2=0x7fc3dc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.565+0000 7fc3e39dc640 1 -- 172.21.15.28:0/2482118015 shutdown_connections 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.565+0000 7fc3e39dc640 1 -- 172.21.15.28:0/2482118015 wait complete. 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.566+0000 7fc3e39dc640 1 Processor -- start 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.566+0000 7fc3e39dc640 1 -- start start 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.566+0000 7fc3e39dc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3dc0fffb0 0x7fc3dc072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.566+0000 7fc3e39dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc106bc0 0x7fc3dc0734f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.566+0000 7fc3e39dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc10da30 0x7fc3dc10deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.566+0000 7fc3e39dc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc111c70 con 0x7fc3dc106bc0 2024-05-09T08:47:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.566+0000 7fc3e39dc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc111af0 con 0x7fc3dc0fffb0 2024-05-09T08:47:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.566+0000 7fc3e39dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc111df0 con 0x7fc3dc10da30 2024-05-09T08:47:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1f79640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc10da30 0x7fc3dc10deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1f79640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc10da30 0x7fc3dc10deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48218/0 (socket says 172.21.15.28:48218) 2024-05-09T08:47:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1f79640 1 -- 172.21.15.28:0/918586192 learned_addr learned my addr 172.21.15.28:0/918586192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1778640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3dc0fffb0 0x7fc3dc072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e0f77640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc106bc0 0x7fc3dc0734f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1f79640 1 -- 172.21.15.28:0/918586192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3dc0fffb0 msgr2=0x7fc3dc072fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1f79640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3dc0fffb0 0x7fc3dc072fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1f79640 1 -- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc106bc0 msgr2=0x7fc3dc0734f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1f79640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc106bc0 0x7fc3dc0734f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1f79640 1 -- 172.21.15.28:0/918586192 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc10e740 con 0x7fc3dc10da30 2024-05-09T08:47:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e0f77640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc106bc0 0x7fc3dc0734f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:47:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.567+0000 7fc3e1f79640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc10da30 0x7fc3dc10deb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400efc0 tx=0x7fc3d400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.570+0000 7fc3d27fc640 1 -- 172.21.15.28:0/918586192 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4003970 con 0x7fc3dc10da30 2024-05-09T08:47:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.571+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3dc06e9e0 con 0x7fc3dc10da30 2024-05-09T08:47:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.571+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3dc06ef30 con 0x7fc3dc10da30 2024-05-09T08:47:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.571+0000 7fc3d27fc640 1 -- 172.21.15.28:0/918586192 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3d4004020 con 0x7fc3dc10da30 2024-05-09T08:47:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.572+0000 7fc3d27fc640 1 -- 172.21.15.28:0/918586192 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d400b9f0 con 0x7fc3dc10da30 2024-05-09T08:47:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.574+0000 7fc3d27fc640 1 -- 172.21.15.28:0/918586192 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc3d400bb90 con 0x7fc3dc10da30 2024-05-09T08:47:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.576+0000 7fc3d27fc640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc3c80a2f00 0x7fc3c80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.576+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c0005290 con 0x7fc3dc10da30 2024-05-09T08:47:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.576+0000 7fc3d27fc640 1 -- 172.21.15.28:0/918586192 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9771+0+0 (secure 0 0 0) 0x7fc3d40cf940 con 0x7fc3dc10da30 2024-05-09T08:47:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.576+0000 7fc3e1778640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc3c80a2f00 0x7fc3c80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.578+0000 7fc3e1778640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc3c80a2f00 0x7fc3c80a53c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc000c00 tx=0x7fc3cc009560 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.582+0000 7fc3d27fc640 1 -- 172.21.15.28:0/918586192 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc3d409c020 con 0x7fc3dc10da30 2024-05-09T08:47:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.957+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc3c0005850 con 0x7fc3dc10da30 2024-05-09T08:47:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.958+0000 7fc3d27fc640 1 -- 172.21.15.28:0/918586192 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v138) ==== 100+0+11 (secure 0 0 0) 0x7fc3d409b920 con 0x7fc3dc10da30 2024-05-09T08:47:05.957 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-09T08:47:05.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.963+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc3c80a2f00 msgr2=0x7fc3c80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:05.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.963+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc3c80a2f00 0x7fc3c80a53c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc000c00 tx=0x7fc3cc009560 comp rx=0 tx=0).stop 2024-05-09T08:47:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.963+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc10da30 msgr2=0x7fc3dc10deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.963+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc10da30 0x7fc3dc10deb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400efc0 tx=0x7fc3d400c800 comp rx=0 tx=0).stop 2024-05-09T08:47:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.964+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 shutdown_connections 2024-05-09T08:47:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.964+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7fc3c80a2f00 0x7fc3c80a53c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.964+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3dc10da30 0x7fc3dc10deb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.964+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc106bc0 0x7fc3dc0734f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.964+0000 7fc3e39dc640 1 --2- 172.21.15.28:0/918586192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3dc0fffb0 0x7fc3dc072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.964+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 >> 172.21.15.28:0/918586192 conn(0x7fc3dc076cd0 msgr2=0x7fc3dc104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.964+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 shutdown_connections 2024-05-09T08:47:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:05.964+0000 7fc3e39dc640 1 -- 172.21.15.28:0/918586192 wait complete. 2024-05-09T08:47:05.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:47:06.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.176+0000 7f5ed9e15640 1 -- 172.21.15.28:0/2533754099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ed4106710 msgr2=0x7f5ed4106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.176+0000 7f5ed9e15640 1 --2- 172.21.15.28:0/2533754099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ed4106710 0x7f5ed4106af0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc01acb0 tx=0x7f5ecc03b920 comp rx=0 tx=0).stop 2024-05-09T08:47:06.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.177+0000 7f5ed9e15640 1 -- 172.21.15.28:0/2533754099 shutdown_connections 2024-05-09T08:47:06.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.177+0000 7f5ed9e15640 1 --2- 172.21.15.28:0/2533754099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed41010c0 0x7f5ed410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.177+0000 7f5ed9e15640 1 --2- 172.21.15.28:0/2533754099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ed4100700 0x7f5ed4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.177+0000 7f5ed9e15640 1 --2- 172.21.15.28:0/2533754099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ed4106710 0x7f5ed4106af0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.177+0000 7f5ed9e15640 1 -- 172.21.15.28:0/2533754099 >> 172.21.15.28:0/2533754099 conn(0x7f5ed4077830 msgr2=0x7f5ed4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.178+0000 7f5ed9e15640 1 -- 172.21.15.28:0/2533754099 shutdown_connections 2024-05-09T08:47:06.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.181+0000 7f5ed9e15640 1 -- 172.21.15.28:0/2533754099 wait complete. 2024-05-09T08:47:06.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.181+0000 7f5ed9e15640 1 Processor -- start 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed9e15640 1 -- start start 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed9e15640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4100700 0x7f5ed410d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed9e15640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ed41010c0 0x7f5ed41081f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed9e15640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ed4106710 0x7f5ed4108730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed9e15640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed406dca0 con 0x7f5ed4100700 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed9e15640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed406db20 con 0x7f5ed4106710 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed9e15640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed406de20 con 0x7f5ed41010c0 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ed41010c0 0x7f5ed41081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed2ffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ed41010c0 0x7f5ed41081f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48238/0 (socket says 172.21.15.28:48238) 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.182+0000 7f5ed2ffd640 1 -- 172.21.15.28:0/104461902 learned_addr learned my addr 172.21.15.28:0/104461902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.183+0000 7f5ed2ffd640 1 -- 172.21.15.28:0/104461902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ed4106710 msgr2=0x7f5ed4108730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:47:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.183+0000 7f5ed2ffd640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ed4106710 0x7f5ed4108730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.183+0000 7f5ed2ffd640 1 -- 172.21.15.28:0/104461902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4100700 msgr2=0x7f5ed410d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.183+0000 7f5ed2ffd640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4100700 0x7f5ed410d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.183+0000 7f5ed2ffd640 1 -- 172.21.15.28:0/104461902 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed4108d20 con 0x7f5ed41010c0 2024-05-09T08:47:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.183+0000 7f5ed2ffd640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ed41010c0 0x7f5ed41081f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800b480 tx=0x7f5ec800b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.188+0000 7f5ed0ff9640 1 -- 172.21.15.28:0/104461902 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8014070 con 0x7f5ed41010c0 2024-05-09T08:47:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.188+0000 7f5ed0ff9640 1 -- 172.21.15.28:0/104461902 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ec8010d50 con 0x7f5ed41010c0 2024-05-09T08:47:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.188+0000 7f5ed0ff9640 1 -- 172.21.15.28:0/104461902 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8017360 con 0x7f5ed41010c0 2024-05-09T08:47:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.189+0000 7f5ed9e15640 1 -- 172.21.15.28:0/104461902 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed4109010 con 0x7f5ed41010c0 2024-05-09T08:47:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.189+0000 7f5ed9e15640 1 -- 172.21.15.28:0/104461902 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed419a1e0 con 0x7f5ed41010c0 2024-05-09T08:47:06.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.191+0000 7f5eba7fc640 1 -- 172.21.15.28:0/104461902 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed4101e40 con 0x7f5ed41010c0 2024-05-09T08:47:06.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.192+0000 7f5ed0ff9640 1 -- 172.21.15.28:0/104461902 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5ec8017070 con 0x7f5ed41010c0 2024-05-09T08:47:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.193+0000 7f5ed0ff9640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ebc0a2e80 0x7f5ebc0a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.194+0000 7f5ed0ff9640 1 -- 172.21.15.28:0/104461902 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9771+0+0 (secure 0 0 0) 0x7f5ec80cefa0 con 0x7f5ed41010c0 2024-05-09T08:47:06.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.195+0000 7f5ed37fe640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ebc0a2e80 0x7f5ebc0a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:06.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.198+0000 7f5ed37fe640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ebc0a2e80 0x7f5ebc0a5340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc01a630 tx=0x7f5ecc03be20 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:06.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.200+0000 7f5ed0ff9640 1 -- 172.21.15.28:0/104461902 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5ec800f3c0 con 0x7f5ed41010c0 2024-05-09T08:47:06.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:05 smithi184 ceph-mon[34509]: pgmap v669: 129 pgs: 54 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 59 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 126 op/s 2024-05-09T08:47:06.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:05 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/800186421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:47:06.267 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:05 smithi184 ceph-mon[34509]: osdmap e138: 12 total, 12 up, 12 in 2024-05-09T08:47:06.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:05 smithi165 ceph-mon[34540]: pgmap v669: 129 pgs: 54 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 59 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 126 op/s 2024-05-09T08:47:06.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:05 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/800186421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:47:06.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:05 smithi165 ceph-mon[34540]: osdmap e138: 12 total, 12 up, 12 in 2024-05-09T08:47:06.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:05 smithi028 ceph-mon[30119]: pgmap v669: 129 pgs: 54 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 59 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 126 op/s 2024-05-09T08:47:06.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:05 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/800186421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:47:06.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:05 smithi028 ceph-mon[30119]: osdmap e138: 12 total, 12 up, 12 in 2024-05-09T08:47:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.569+0000 7f5eba7fc640 1 -- 172.21.15.28:0/104461902 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5ed4109720 con 0x7f5ed41010c0 2024-05-09T08:47:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.570+0000 7f5ed0ff9640 1 -- 172.21.15.28:0/104461902 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v138) ==== 103+0+11 (secure 0 0 0) 0x7f5ec809b790 con 0x7f5ed41010c0 2024-05-09T08:47:06.569 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-09T08:47:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.573+0000 7f5eba7fc640 1 -- 172.21.15.28:0/104461902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ebc0a2e80 msgr2=0x7f5ebc0a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.573+0000 7f5eba7fc640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ebc0a2e80 0x7f5ebc0a5340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc01a630 tx=0x7f5ecc03be20 comp rx=0 tx=0).stop 2024-05-09T08:47:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.573+0000 7f5eba7fc640 1 -- 172.21.15.28:0/104461902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ed41010c0 msgr2=0x7f5ed41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.573+0000 7f5eba7fc640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ed41010c0 0x7f5ed41081f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800b480 tx=0x7f5ec800b940 comp rx=0 tx=0).stop 2024-05-09T08:47:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.574+0000 7f5eba7fc640 1 -- 172.21.15.28:0/104461902 shutdown_connections 2024-05-09T08:47:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.574+0000 7f5eba7fc640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ed4106710 0x7f5ed4108730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.574+0000 7f5eba7fc640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ebc0a2e80 0x7f5ebc0a5340 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.574+0000 7f5eba7fc640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ed41010c0 0x7f5ed41081f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.574+0000 7f5eba7fc640 1 --2- 172.21.15.28:0/104461902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4100700 0x7f5ed410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.574+0000 7f5eba7fc640 1 -- 172.21.15.28:0/104461902 >> 172.21.15.28:0/104461902 conn(0x7f5ed4077830 msgr2=0x7f5ed4102200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.578+0000 7f5eba7fc640 1 -- 172.21.15.28:0/104461902 shutdown_connections 2024-05-09T08:47:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.578+0000 7f5eba7fc640 1 -- 172.21.15.28:0/104461902 wait complete. 2024-05-09T08:47:06.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T08:47:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.796+0000 7f045bc3f640 1 -- 172.21.15.28:0/4228971814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f045411b010 msgr2=0x7f045411d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.796+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4228971814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f045411b010 0x7f045411d400 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f0450009930 tx=0x7f045002a440 comp rx=0 tx=0).stop 2024-05-09T08:47:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.801+0000 7f045bc3f640 1 -- 172.21.15.28:0/4228971814 shutdown_connections 2024-05-09T08:47:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.801+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4228971814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f045411b010 0x7f045411d400 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.801+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4228971814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f045410e0b0 0x7f045410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.801+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4228971814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0454110080 0x7f045410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.801+0000 7f045bc3f640 1 -- 172.21.15.28:0/4228971814 >> 172.21.15.28:0/4228971814 conn(0x7f045406c940 msgr2=0x7f045406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.803+0000 7f045bc3f640 1 -- 172.21.15.28:0/4228971814 shutdown_connections 2024-05-09T08:47:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.803+0000 7f045bc3f640 1 -- 172.21.15.28:0/4228971814 wait complete. 2024-05-09T08:47:06.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.805+0000 7f045bc3f640 1 Processor -- start 2024-05-09T08:47:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.806+0000 7f045bc3f640 1 -- start start 2024-05-09T08:47:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.807+0000 7f045bc3f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f045410e0b0 0x7f045411a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.807+0000 7f045bc3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0454110080 0x7f0454113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.807+0000 7f045bc3f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f045411b010 0x7f0454113ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.807+0000 7f045bc3f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04541209f0 con 0x7f0454110080 2024-05-09T08:47:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.807+0000 7f045bc3f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0454120870 con 0x7f045410e0b0 2024-05-09T08:47:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.807+0000 7f045bc3f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0454120b70 con 0x7f045411b010 2024-05-09T08:47:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.808+0000 7f045a1dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f045411b010 0x7f0454113ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.808+0000 7f045a1dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f045411b010 0x7f0454113ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48254/0 (socket says 172.21.15.28:48254) 2024-05-09T08:47:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.808+0000 7f045a1dc640 1 -- 172.21.15.28:0/4255236646 learned_addr learned my addr 172.21.15.28:0/4255236646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.808+0000 7f04591da640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0454110080 0x7f0454113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.808+0000 7f045a1dc640 1 -- 172.21.15.28:0/4255236646 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f045410e0b0 msgr2=0x7f045411a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.808+0000 7f045a1dc640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f045410e0b0 0x7f045411a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.808+0000 7f045a1dc640 1 -- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0454110080 msgr2=0x7f0454113760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.808+0000 7f045a1dc640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0454110080 0x7f0454113760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.808+0000 7f045a1dc640 1 -- 172.21.15.28:0/4255236646 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04541142b0 con 0x7f045411b010 2024-05-09T08:47:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.809+0000 7f045a1dc640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f045411b010 0x7f0454113ca0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f045002a9b0 tx=0x7f0450003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.810+0000 7f044affd640 1 -- 172.21.15.28:0/4255236646 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045003f070 con 0x7f045411b010 2024-05-09T08:47:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.810+0000 7f044affd640 1 -- 172.21.15.28:0/4255236646 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0450031c90 con 0x7f045411b010 2024-05-09T08:47:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.810+0000 7f044affd640 1 -- 172.21.15.28:0/4255236646 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450035030 con 0x7f045411b010 2024-05-09T08:47:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.811+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0454114540 con 0x7f045411b010 2024-05-09T08:47:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.811+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0454118400 con 0x7f045411b010 2024-05-09T08:47:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.814+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f045406dbe0 con 0x7f045411b010 2024-05-09T08:47:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.816+0000 7f044affd640 1 -- 172.21.15.28:0/4255236646 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0450042070 con 0x7f045411b010 2024-05-09T08:47:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.817+0000 7f044affd640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04400a2f00 0x7f04400a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.817+0000 7f044affd640 1 -- 172.21.15.28:0/4255236646 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9831+0+0 (secure 0 0 0) 0x7f04500f06d0 con 0x7f045411b010 2024-05-09T08:47:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.817+0000 7f04599db640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04400a2f00 0x7f04400a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.818+0000 7f04599db640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04400a2f00 0x7f04400a53c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f044c00ab90 tx=0x7f044c009250 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:06.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:06.824+0000 7f044affd640 1 -- 172.21.15.28:0/4255236646 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0450049030 con 0x7f045411b010 2024-05-09T08:47:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.210+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f04541186f0 con 0x7f045411b010 2024-05-09T08:47:07.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.214+0000 7f044affd640 1 -- 172.21.15.28:0/4255236646 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v139) ==== 81+0+941 (secure 0 0 0) 0x7f04500bcdf0 con 0x7f045411b010 2024-05-09T08:47:07.213 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3330063293 2024-05-10T08:26:17.964999+0000 2024-05-09T08:47:07.213 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3745037139 2024-05-10T08:26:17.964999+0000 2024-05-09T08:47:07.213 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2130261041 2024-05-10T08:26:17.964999+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2931568341 2024-05-10T08:16:49.552676+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3510522803 2024-05-10T08:16:49.552676+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/273816117 2024-05-10T08:26:17.964999+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/934935086 2024-05-10T08:16:49.552676+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1194745694 2024-05-10T08:16:22.859488+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/953899001 2024-05-10T08:26:17.964999+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3920156841 2024-05-10T08:16:22.859488+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/1179097296 2024-05-10T08:16:22.859488+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/1179097296 2024-05-10T08:16:22.859488+0000 2024-05-09T08:47:07.214 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/953899001 2024-05-10T08:26:17.964999+0000 2024-05-09T08:47:07.215 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2931568341 2024-05-10T08:16:49.552676+0000 2024-05-09T08:47:07.215 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/466716681 2024-05-10T08:16:49.552676+0000 2024-05-09T08:47:07.215 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2416478420 2024-05-10T08:16:22.859488+0000 2024-05-09T08:47:07.215 INFO:teuthology.orchestra.run.smithi028.stderr:listed 16 entries 2024-05-09T08:47:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.219+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04400a2f00 msgr2=0x7f04400a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.219+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04400a2f00 0x7f04400a53c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f044c00ab90 tx=0x7f044c009250 comp rx=0 tx=0).stop 2024-05-09T08:47:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.219+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f045411b010 msgr2=0x7f0454113ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.219+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f045411b010 0x7f0454113ca0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f045002a9b0 tx=0x7f0450003b90 comp rx=0 tx=0).stop 2024-05-09T08:47:07.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.220+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 shutdown_connections 2024-05-09T08:47:07.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.220+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f04400a2f00 0x7f04400a53c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:07.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.220+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f045411b010 0x7f0454113ca0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:07.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.220+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0454110080 0x7f0454113760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.220+0000 7f045bc3f640 1 --2- 172.21.15.28:0/4255236646 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f045410e0b0 0x7f045411a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.220+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 >> 172.21.15.28:0/4255236646 conn(0x7f045406c940 msgr2=0x7f045411bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.223+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 shutdown_connections 2024-05-09T08:47:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:07.223+0000 7f045bc3f640 1 -- 172.21.15.28:0/4255236646 wait complete. 2024-05-09T08:47:07.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/918586192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:47:07.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:06 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/104461902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:47:07.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:06 smithi028 ceph-mon[30119]: osdmap e139: 12 total, 12 up, 12 in 2024-05-09T08:47:07.240 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T08:47:07.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:47:07.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-09T08:47:07.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:06 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/918586192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:47:07.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:06 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/104461902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:47:07.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:06 smithi165 ceph-mon[34540]: osdmap e139: 12 total, 12 up, 12 in 2024-05-09T08:47:07.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:06 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/918586192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:47:07.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:06 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/104461902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:47:07.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:06 smithi184 ceph-mon[34509]: osdmap e139: 12 total, 12 up, 12 in 2024-05-09T08:47:07.507 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T08:47:07.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:47:07.507 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T08:47:07.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:47:07.550 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-05-09T08:47:07.668 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T08:47:07.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:47:07.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T08:47:07.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:47:07.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-05-09T08:47:07.837 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-09T08:47:07.837 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-09T08:47:07.837 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:47:07.889 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057104,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1419 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/05b14529aa821067819b7b858702ed1cabda4f960539bbf8aef9bb31e506f93a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/05b14529aa821067819b7b858702ed1cabda4f960539bbf8aef9bb31e506f93a/diff,workdir=/var/lib/containers/storage/overlay/05b14529aa821067819b7b858702ed1cabda4f960539bbf8aef9bb31e506f93a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9e29e043cb0f99851477bef0e8243fff06c0780f471adffdda849438320684ae/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/9e29e043cb0f99851477bef0e8243fff06c0780f471adffdda849438320684ae/diff,workdir=/var/lib/containers/storage/overlay/9e29e043cb0f99851477bef0e8243fff06c0780f471adffdda849438320684ae/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/e85dfc3bf834ebb66efc0c712a04e99ce596d4bf22a90a4bfec01ec6d384689a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/e85dfc3bf834ebb66efc0c712a04e99ce596d4bf22a90a4bfec01ec6d384689a/diff,workdir=/var/lib/containers/storage/overlay/e85dfc3bf834ebb66efc0c712a04e99ce596d4bf22a90a4bfec01ec6d384689a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b686dfedf8ba946ae6ee1dbd06370cb9c7234ae18e9f56e6daec8e4663c40556/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/b686dfedf8ba946ae6ee1dbd06370cb9c7234ae18e9f56e6daec8e4663c40556/diff,workdir=/var/lib/containers/storage/overlay/b686dfedf8ba946ae6ee1dbd06370cb9c7234ae18e9f56e6daec8e4663c40556/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b90ab13cc012be3841d18882691d4c956c3cae5a3d826c9feb12aaad89bb74bb/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/b90ab13cc012be3841d18882691d4c956c3cae5a3d826c9feb12aaad89bb74bb/diff,workdir=/var/lib/containers/storage/overlay/b90ab13cc012be3841d18882691d4c956c3cae5a3d826c9feb12aaad89bb74bb/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e0fa84bf13c97723fed68a6002b28740f58f75b17ede8c6fbe775d0466cc7f6a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/e0fa84bf13c97723fed68a6002b28740f58f75b17ede8c6fbe775d0466cc7f6a/diff,workdir=/var/lib/containers/storage/overlay/e0fa84bf13c97723fed68a6002b28740f58f75b17ede8c6fbe775d0466cc7f6a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/13122165aec782bb6f91f2bcc16b2288f927757583d37dd6a187c3948b3539fc/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/13122165aec782bb6f91f2bcc16b2288f927757583d37dd6a187c3948b3539fc/diff,workdir=/var/lib/containers/storage/overlay/13122165aec782bb6f91f2bcc16b2288f927757583d37dd6a187c3948b3539fc/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/7b9cd4b08b919ec11955417b98a460d3256d0701b85b85e7a712210d7046a9a0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/7b9cd4b08b919ec11955417b98a460d3256d0701b85b85e7a712210d7046a9a0/diff,workdir=/var/lib/containers/storage/overlay/7b9cd4b08b919ec11955417b98a460d3256d0701b85b85e7a712210d7046a9a0/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6242d42840ec4f81eb664998050edd1802ab005e4edcf00509ee37b7d3f09a4e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/6242d42840ec4f81eb664998050edd1802ab005e4edcf00509ee37b7d3f09a4e/diff,workdir=/var/lib/containers/storage/overlay/6242d42840ec4f81eb664998050edd1802ab005e4edcf00509ee37b7d3f09a4e/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 overlay /var/lib/containers/storage/overlay/6a84ebbfca5c2dfd5244acbd343715436d43bea3a7518da2ea1310cef7e4ddd8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4GGHWHXGN2QPHJFWWAR6KJFIKE:/var/lib/containers/storage/overlay/l/ZRQOUGIOW4NO3FFA5AAZKGYFBY,upperdir=/var/lib/containers/storage/overlay/6a84ebbfca5c2dfd5244acbd343715436d43bea3a7518da2ea1310cef7e4ddd8/diff,workdir=/var/lib/containers/storage/overlay/6a84ebbfca5c2dfd5244acbd343715436d43bea3a7518da2ea1310cef7e4ddd8/work,metacopy=on,volatile 0 0 2024-05-09T08:47:07.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:47:07.889 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-05-09T08:47:07.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:47:07.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-05-09T08:47:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-05-09T08:47:08.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:07 smithi028 ceph-mon[30119]: pgmap v671: 129 pgs: 55 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 59 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 127 op/s 2024-05-09T08:47:08.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:07 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/4255236646' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:47:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:47:08.057 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-09T08:47:08.068 INFO:tasks.cephadm:Teardown begin 2024-05-09T08:47:08.068 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T08:47:08.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.311+0000 7f5cea74f640 1 -- 172.21.15.28:0/2459585747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ce4110400 msgr2=0x7f5ce410e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.311+0000 7f5cea74f640 1 --2- 172.21.15.28:0/2459585747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ce4110400 0x7f5ce410e0b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009ca0 tx=0x7f5cdc02a8f0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.316+0000 7f5cea74f640 1 -- 172.21.15.28:0/2459585747 shutdown_connections 2024-05-09T08:47:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.316+0000 7f5cea74f640 1 --2- 172.21.15.28:0/2459585747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce410e790 0x7f5ce411d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.316+0000 7f5cea74f640 1 --2- 172.21.15.28:0/2459585747 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ce4110400 0x7f5ce410e0b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.316+0000 7f5cea74f640 1 --2- 172.21.15.28:0/2459585747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce410ff90 0x7f5ce410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.316+0000 7f5cea74f640 1 -- 172.21.15.28:0/2459585747 >> 172.21.15.28:0/2459585747 conn(0x7f5ce406c950 msgr2=0x7f5ce406cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.324+0000 7f5cea74f640 1 -- 172.21.15.28:0/2459585747 shutdown_connections 2024-05-09T08:47:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.325+0000 7f5cea74f640 1 -- 172.21.15.28:0/2459585747 wait complete. 2024-05-09T08:47:08.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.325+0000 7f5cea74f640 1 Processor -- start 2024-05-09T08:47:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.328+0000 7f5cea74f640 1 -- start start 2024-05-09T08:47:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.328+0000 7f5cea74f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ce410ff90 0x7f5ce411c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.328+0000 7f5cea74f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce411cbd0 0x7f5ce4115710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.328+0000 7f5cea74f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce4115d60 0x7f5ce41161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.328+0000 7f5cea74f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4120a70 con 0x7f5ce4115d60 2024-05-09T08:47:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.328+0000 7f5cea74f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41208f0 con 0x7f5ce411cbd0 2024-05-09T08:47:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.328+0000 7f5cea74f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4120bf0 con 0x7f5ce410ff90 2024-05-09T08:47:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.329+0000 7f5ce37fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce411cbd0 0x7f5ce4115710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:08.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.330+0000 7f5ce8cec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce4115d60 0x7f5ce41161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:08.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.330+0000 7f5ce8cec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce4115d60 0x7f5ce41161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34554/0 (socket says 172.21.15.28:34554) 2024-05-09T08:47:08.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.330+0000 7f5ce8cec640 1 -- 172.21.15.28:0/1109364154 learned_addr learned my addr 172.21.15.28:0/1109364154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.333+0000 7f5ce8cec640 1 -- 172.21.15.28:0/1109364154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ce410ff90 msgr2=0x7f5ce411c690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:47:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.333+0000 7f5ce8cec640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ce410ff90 0x7f5ce411c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.333+0000 7f5ce8cec640 1 -- 172.21.15.28:0/1109364154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce411cbd0 msgr2=0x7f5ce4115710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.333+0000 7f5ce8cec640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce411cbd0 0x7f5ce4115710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.333+0000 7f5ce8cec640 1 -- 172.21.15.28:0/1109364154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce411a530 con 0x7f5ce4115d60 2024-05-09T08:47:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.335+0000 7f5ce8cec640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce4115d60 0x7f5ce41161e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009fd0 tx=0x7f5cd800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.339+0000 7f5ce17fa640 1 -- 172.21.15.28:0/1109364154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8011830 con 0x7f5ce4115d60 2024-05-09T08:47:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.339+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce411a820 con 0x7f5ce4115d60 2024-05-09T08:47:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.339+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce411ad70 con 0x7f5ce4115d60 2024-05-09T08:47:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.339+0000 7f5ce17fa640 1 -- 172.21.15.28:0/1109364154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd8016070 con 0x7f5ce4115d60 2024-05-09T08:47:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.340+0000 7f5ce17fa640 1 -- 172.21.15.28:0/1109364154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd800f2e0 con 0x7f5ce4115d60 2024-05-09T08:47:08.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.344+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cd0005290 con 0x7f5ce4115d60 2024-05-09T08:47:08.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.345+0000 7f5ce17fa640 1 -- 172.21.15.28:0/1109364154 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5cd800f480 con 0x7f5ce4115d60 2024-05-09T08:47:08.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.346+0000 7f5ce17fa640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ccc0a2f00 0x7f5ccc0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:08.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.346+0000 7f5ce17fa640 1 -- 172.21.15.28:0/1109364154 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9831+0+0 (secure 0 0 0) 0x7f5cd80cedf0 con 0x7f5ce4115d60 2024-05-09T08:47:08.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.347+0000 7f5ce3fff640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ccc0a2f00 0x7f5ccc0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:08.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.348+0000 7f5ce3fff640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ccc0a2f00 0x7f5ccc0a53c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400a890 tx=0x7f5cd4005b80 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:08.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:07 smithi165 ceph-mon[34540]: pgmap v671: 129 pgs: 55 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 59 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 127 op/s 2024-05-09T08:47:08.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:07 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/4255236646' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:47:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.354+0000 7f5ce17fa640 1 -- 172.21.15.28:0/1109364154 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5cd809c700 con 0x7f5ce4115d60 2024-05-09T08:47:08.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:07 smithi184 ceph-mon[34509]: pgmap v671: 129 pgs: 55 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 59 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 127 op/s 2024-05-09T08:47:08.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:07 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/4255236646' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:47:08.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.800+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5cd0005850 con 0x7f5ce4115d60 2024-05-09T08:47:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.805+0000 7f5ce17fa640 1 -- 172.21.15.28:0/1109364154 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+11968 (secure 0 0 0) 0x7f5cd809c000 con 0x7f5ce4115d60 2024-05-09T08:47:08.803 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:47:08.804 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":17043,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1569049047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1569049047},{"type":"v1","addr":"172.21.15.28:6837","nonce":1569049047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:39:11.397249+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":72,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":70,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":71,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":69,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26171":{"gid":26171,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2426983483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2426983483},{"type":"v1","addr":"172.21.15.165:6835","nonce":2426983483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24481,24499,24476,24512,24463]},"id":1}]} 2024-05-09T08:47:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 67 2024-05-09T08:47:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.808+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ccc0a2f00 msgr2=0x7f5ccc0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.808+0000 7f5cea74f640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ccc0a2f00 0x7f5ccc0a53c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400a890 tx=0x7f5cd4005b80 comp rx=0 tx=0).stop 2024-05-09T08:47:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.808+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce4115d60 msgr2=0x7f5ce41161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.808+0000 7f5cea74f640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce4115d60 0x7f5ce41161e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009fd0 tx=0x7f5cd800b8e0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.809+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 shutdown_connections 2024-05-09T08:47:08.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.809+0000 7f5cea74f640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce4115d60 0x7f5ce41161e0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.809+0000 7f5cea74f640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce411cbd0 0x7f5ce4115710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.809+0000 7f5cea74f640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f5ccc0a2f00 0x7f5ccc0a53c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.809+0000 7f5cea74f640 1 --2- 172.21.15.28:0/1109364154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ce410ff90 0x7f5ce411c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:08.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.809+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 >> 172.21.15.28:0/1109364154 conn(0x7f5ce406c950 msgr2=0x7f5ce4070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:08.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.812+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 shutdown_connections 2024-05-09T08:47:08.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:08.812+0000 7f5cea74f640 1 -- 172.21.15.28:0/1109364154 wait complete. 2024-05-09T08:47:08.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.026+0000 7f482eac1640 1 -- 172.21.15.28:0/324150459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828110400 msgr2=0x7f482810e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.026+0000 7f482eac1640 1 --2- 172.21.15.28:0/324150459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828110400 0x7f482810e0b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f481c009900 tx=0x7f481c02a440 comp rx=0 tx=0).stop 2024-05-09T08:47:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.038+0000 7f482eac1640 1 -- 172.21.15.28:0/324150459 shutdown_connections 2024-05-09T08:47:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.038+0000 7f482eac1640 1 --2- 172.21.15.28:0/324150459 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f482810e790 0x7f482811d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.038+0000 7f482eac1640 1 --2- 172.21.15.28:0/324150459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828110400 0x7f482810e0b0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.038+0000 7f482eac1640 1 --2- 172.21.15.28:0/324150459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f482810ff90 0x7f482810db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.038+0000 7f482eac1640 1 -- 172.21.15.28:0/324150459 >> 172.21.15.28:0/324150459 conn(0x7f482806c950 msgr2=0x7f482806cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.046+0000 7f482eac1640 1 -- 172.21.15.28:0/324150459 shutdown_connections 2024-05-09T08:47:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.048+0000 7f482eac1640 1 -- 172.21.15.28:0/324150459 wait complete. 2024-05-09T08:47:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.049+0000 7f482eac1640 1 Processor -- start 2024-05-09T08:47:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.053+0000 7f482eac1640 1 -- start start 2024-05-09T08:47:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.054+0000 7f482eac1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f482810ff90 0x7f482811a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.054+0000 7f482eac1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48281138a0 0x7f4828113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.054+0000 7f482eac1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828114240 0x7f48281182f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.054+0000 7f482eac1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4828120b30 con 0x7f4828114240 2024-05-09T08:47:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.054+0000 7f482eac1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f48281209b0 con 0x7f48281138a0 2024-05-09T08:47:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.054+0000 7f482eac1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4828120cb0 con 0x7f482810ff90 2024-05-09T08:47:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.060+0000 7f482c85d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f482810ff90 0x7f482811a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.060+0000 7f482c85d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f482810ff90 0x7f482811a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48274/0 (socket says 172.21.15.28:48274) 2024-05-09T08:47:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.060+0000 7f482c85d640 1 -- 172.21.15.28:0/3223699948 learned_addr learned my addr 172.21.15.28:0/3223699948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.060+0000 7f482d05e640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828114240 0x7f48281182f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.062+0000 7f4827fff640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48281138a0 0x7f4828113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.066+0000 7f482d05e640 1 -- 172.21.15.28:0/3223699948 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f482810ff90 msgr2=0x7f482811a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.066+0000 7f482d05e640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f482810ff90 0x7f482811a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.066+0000 7f482d05e640 1 -- 172.21.15.28:0/3223699948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48281138a0 msgr2=0x7f4828113d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.066+0000 7f482d05e640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48281138a0 0x7f4828113d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.066+0000 7f482d05e640 1 -- 172.21.15.28:0/3223699948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4828118830 con 0x7f4828114240 2024-05-09T08:47:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.073+0000 7f482d05e640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828114240 0x7f48281182f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f482000be00 tx=0x7f482000bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.077+0000 7f4825ffb640 1 -- 172.21.15.28:0/3223699948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820019070 con 0x7f4828114240 2024-05-09T08:47:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.077+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4828118ac0 con 0x7f4828114240 2024-05-09T08:47:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.077+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4828119010 con 0x7f4828114240 2024-05-09T08:47:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.077+0000 7f4825ffb640 1 -- 172.21.15.28:0/3223699948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4820014920 con 0x7f4828114240 2024-05-09T08:47:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.077+0000 7f4825ffb640 1 -- 172.21.15.28:0/3223699948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820012920 con 0x7f4828114240 2024-05-09T08:47:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.079+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4814005290 con 0x7f4828114240 2024-05-09T08:47:09.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.088+0000 7f4825ffb640 1 -- 172.21.15.28:0/3223699948 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4820014ad0 con 0x7f4828114240 2024-05-09T08:47:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.089+0000 7f4825ffb640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f48100a2f00 0x7f48100a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.089+0000 7f4825ffb640 1 -- 172.21.15.28:0/3223699948 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9831+0+0 (secure 0 0 0) 0x7f48200cf770 con 0x7f4828114240 2024-05-09T08:47:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.093+0000 7f482c85d640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f48100a2f00 0x7f48100a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:09.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.099+0000 7f4825ffb640 1 -- 172.21.15.28:0/3223699948 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f482009b950 con 0x7f4828114240 2024-05-09T08:47:09.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.103+0000 7f482c85d640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f48100a2f00 0x7f48100a53c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4818009a30 tx=0x7f4818009190 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:09.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:08 smithi028 ceph-mon[30119]: pgmap v673: 129 pgs: 62 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 125 op/s 2024-05-09T08:47:09.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:08 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1109364154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:09.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:08 smithi165 ceph-mon[34540]: pgmap v673: 129 pgs: 62 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 125 op/s 2024-05-09T08:47:09.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:08 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1109364154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:09.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:08 smithi184 ceph-mon[34509]: pgmap v673: 129 pgs: 62 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 125 op/s 2024-05-09T08:47:09.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:08 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1109364154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.550+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4814005580 con 0x7f4828114240 2024-05-09T08:47:09.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.552+0000 7f4825ffb640 1 -- 172.21.15.28:0/3223699948 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+11968 (secure 0 0 0) 0x7f482009fc50 con 0x7f4828114240 2024-05-09T08:47:09.551 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:47:09.551 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/523365778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":523365778},{"type":"v1","addr":"172.21.15.28:6835","nonce":523365778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":17043,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1569049047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1569049047},{"type":"v1","addr":"172.21.15.28:6837","nonce":1569049047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:23:50.918821+0000","modified":"2024-05-09T08:39:11.397249+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24499,"mds_2":24481,"mds_3":24476,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/612632862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":612632862},{"type":"v1","addr":"172.21.15.28:6839","nonce":612632862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/9520094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":9520094},{"type":"v1","addr":"172.21.15.28:6841","nonce":9520094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6833/3903056410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3903056410},{"type":"v1","addr":"172.21.15.165:6833","nonce":3903056410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.184:6833/1853687224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1853687224},{"type":"v1","addr":"172.21.15.184:6833","nonce":1853687224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":72,"addr":"172.21.15.184:6835/4284085086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4284085086},{"type":"v1","addr":"172.21.15.184:6835","nonce":4284085086}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":70,"addr":"172.21.15.165:6837/1615628763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1615628763},{"type":"v1","addr":"172.21.15.165:6837","nonce":1615628763}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":71,"addr":"172.21.15.184:6837/3795789755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":3795789755},{"type":"v1","addr":"172.21.15.184:6837","nonce":3795789755}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":69,"addr":"172.21.15.184:6839/224925653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6838","nonce":224925653},{"type":"v1","addr":"172.21.15.184:6839","nonce":224925653}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6839/1723030158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1723030158},{"type":"v1","addr":"172.21.15.165:6839","nonce":1723030158}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26171":{"gid":26171,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2426983483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2426983483},{"type":"v1","addr":"172.21.15.165:6835","nonce":2426983483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24481,24499,24476,24512,24463]},"id":1}]} 2024-05-09T08:47:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 67 2024-05-09T08:47:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.556+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f48100a2f00 msgr2=0x7f48100a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.556+0000 7f482eac1640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f48100a2f00 0x7f48100a53c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4818009a30 tx=0x7f4818009190 comp rx=0 tx=0).stop 2024-05-09T08:47:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.556+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828114240 msgr2=0x7f48281182f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.556+0000 7f482eac1640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828114240 0x7f48281182f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f482000be00 tx=0x7f482000bef0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.556+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 shutdown_connections 2024-05-09T08:47:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.556+0000 7f482eac1640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4828114240 0x7f48281182f0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.556+0000 7f482eac1640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48281138a0 0x7f4828113d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.556+0000 7f482eac1640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f48100a2f00 0x7f48100a53c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.556+0000 7f482eac1640 1 --2- 172.21.15.28:0/3223699948 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f482810ff90 0x7f482811a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.557+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 >> 172.21.15.28:0/3223699948 conn(0x7f482806c950 msgr2=0x7f482810cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.557+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 shutdown_connections 2024-05-09T08:47:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.557+0000 7f482eac1640 1 -- 172.21.15.28:0/3223699948 wait complete. 2024-05-09T08:47:09.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:47:09.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.809+0000 7f089f542640 1 -- 172.21.15.28:0/4066808523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f089810ff90 msgr2=0x7f089810db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:09.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.809+0000 7f089f542640 1 --2- 172.21.15.28:0/4066808523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f089810ff90 0x7f089810db70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0890009ca0 tx=0x7f089002a8f0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.815+0000 7f089f542640 1 -- 172.21.15.28:0/4066808523 shutdown_connections 2024-05-09T08:47:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.815+0000 7f089f542640 1 --2- 172.21.15.28:0/4066808523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f089810e790 0x7f089811d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.815+0000 7f089f542640 1 --2- 172.21.15.28:0/4066808523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0898110400 0x7f089810e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.815+0000 7f089f542640 1 --2- 172.21.15.28:0/4066808523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f089810ff90 0x7f089810db70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.815+0000 7f089f542640 1 -- 172.21.15.28:0/4066808523 >> 172.21.15.28:0/4066808523 conn(0x7f089806c950 msgr2=0x7f089806cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.817+0000 7f089f542640 1 -- 172.21.15.28:0/4066808523 shutdown_connections 2024-05-09T08:47:09.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.821+0000 7f089f542640 1 -- 172.21.15.28:0/4066808523 wait complete. 2024-05-09T08:47:09.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.821+0000 7f089f542640 1 Processor -- start 2024-05-09T08:47:09.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.823+0000 7f089f542640 1 -- start start 2024-05-09T08:47:09.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.823+0000 7f089f542640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f089810ff90 0x7f08981a6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:09.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.823+0000 7f089f542640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08981a7470 0x7f08981ab7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:09.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.823+0000 7f089f542640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08981abd30 0x7f08981ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:09.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.823+0000 7f089f542640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0898120a30 con 0x7f089810ff90 2024-05-09T08:47:09.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.823+0000 7f089f542640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f08981208b0 con 0x7f08981a7470 2024-05-09T08:47:09.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.823+0000 7f089f542640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0898120bb0 con 0x7f08981abd30 2024-05-09T08:47:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.826+0000 7f089cadd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08981a7470 0x7f08981ab7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.826+0000 7f089dadf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08981abd30 0x7f08981ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.826+0000 7f089cadd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08981a7470 0x7f08981ab7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:41814/0 (socket says 172.21.15.28:41814) 2024-05-09T08:47:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.826+0000 7f089cadd640 1 -- 172.21.15.28:0/2228411449 learned_addr learned my addr 172.21.15.28:0/2228411449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.830+0000 7f089d2de640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f089810ff90 0x7f08981a6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.833+0000 7f089dadf640 1 -- 172.21.15.28:0/2228411449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08981a7470 msgr2=0x7f08981ab7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.833+0000 7f089dadf640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08981a7470 0x7f08981ab7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.833+0000 7f089dadf640 1 -- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f089810ff90 msgr2=0x7f08981a6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:09.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.833+0000 7f089dadf640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f089810ff90 0x7f08981a6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:09.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.833+0000 7f089dadf640 1 -- 172.21.15.28:0/2228411449 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08981b3aa0 con 0x7f08981abd30 2024-05-09T08:47:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.837+0000 7f089dadf640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08981abd30 0x7f08981ac1e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f089400e970 tx=0x7f089400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.841+0000 7f088e7fc640 1 -- 172.21.15.28:0/2228411449 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089400ccc0 con 0x7f08981abd30 2024-05-09T08:47:09.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.841+0000 7f088e7fc640 1 -- 172.21.15.28:0/2228411449 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0894005020 con 0x7f08981abd30 2024-05-09T08:47:09.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.841+0000 7f088e7fc640 1 -- 172.21.15.28:0/2228411449 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894004a00 con 0x7f08981abd30 2024-05-09T08:47:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.842+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08981b3cd0 con 0x7f08981abd30 2024-05-09T08:47:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.844+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08981b41a0 con 0x7f08981abd30 2024-05-09T08:47:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.847+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0898068650 con 0x7f08981abd30 2024-05-09T08:47:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.853+0000 7f088e7fc640 1 -- 172.21.15.28:0/2228411449 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f08940051d0 con 0x7f08981abd30 2024-05-09T08:47:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.854+0000 7f088e7fc640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f08840a2f00 0x7f08840a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.854+0000 7f088e7fc640 1 -- 172.21.15.28:0/2228411449 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9831+0+0 (secure 0 0 0) 0x7f08940cf8e0 con 0x7f08981abd30 2024-05-09T08:47:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.859+0000 7f089d2de640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f08840a2f00 0x7f08840a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.863+0000 7f088e7fc640 1 -- 172.21.15.28:0/2228411449 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f089409c000 con 0x7f08981abd30 2024-05-09T08:47:09.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:09.878+0000 7f089d2de640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f08840a2f00 0x7f08840a53c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0890006010 tx=0x7f0890000f40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:10.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.237+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0898110400 con 0x7f08981abd30 2024-05-09T08:47:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.240+0000 7f088e7fc640 1 -- 172.21.15.28:0/2228411449 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+25839 (secure 0 0 0) 0x7f089409b900 con 0x7f08981abd30 2024-05-09T08:47:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-09T08:47:10.240 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":139,"fsid":"3891b212-0ddc-11ef-bc98-c7b262605968","created":"2024-05-09T08:15:46.667230+0000","modified":"2024-05-09T08:47:06.777457+0000","last_up_change":"2024-05-09T08:23:00.845181+0000","last_in_change":"2024-05-09T08:22:40.845026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:19:12.083519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:23:49.496601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:23:50.303180+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":37,"snap_epoch":139,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:23:54.086488+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":37,"snap_epoch":139,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb0b1568-830d-4e49-afa7-96ea4d64ce19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6803","nonce":1756971731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6805","nonce":1756971731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6809","nonce":1756971731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1756971731},{"type":"v1","addr":"172.21.15.28:6807","nonce":1756971731}]},"public_addr":"172.21.15.28:6803/1756971731","cluster_addr":"172.21.15.28:6805/1756971731","heartbeat_back_addr":"172.21.15.28:6809/1756971731","heartbeat_front_addr":"172.21.15.28:6807/1756971731","state":["exists","up"]},{"osd":1,"uuid":"1a57c6ac-92fa-4d2c-9ce8-ec3906cab422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6801","nonce":3539015661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6803","nonce":3539015661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6807","nonce":3539015661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3539015661},{"type":"v1","addr":"172.21.15.165:6805","nonce":3539015661}]},"public_addr":"172.21.15.165:6801/3539015661","cluster_addr":"172.21.15.165:6803/3539015661","heartbeat_back_addr":"172.21.15.165:6807/3539015661","heartbeat_front_addr":"172.21.15.165:6805/3539015661","state":["exists","up"]},{"osd":2,"uuid":"e9ab7ba3-b176-4560-a3d4-80a438a4efda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6801","nonce":3751267842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6803","nonce":3751267842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6807","nonce":3751267842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3751267842},{"type":"v1","addr":"172.21.15.184:6805","nonce":3751267842}]},"public_addr":"172.21.15.184:6801/3751267842","cluster_addr":"172.21.15.184:6803/3751267842","heartbeat_back_addr":"172.21.15.184:6807/3751267842","heartbeat_front_addr":"172.21.15.184:6805/3751267842","state":["exists","up"]},{"osd":3,"uuid":"00f5fefc-7634-4f7a-b543-8b3d86be1a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6811","nonce":3526420147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6813","nonce":3526420147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6817","nonce":3526420147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3526420147},{"type":"v1","addr":"172.21.15.28:6815","nonce":3526420147}]},"public_addr":"172.21.15.28:6811/3526420147","cluster_addr":"172.21.15.28:6813/3526420147","heartbeat_back_addr":"172.21.15.28:6817/3526420147","heartbeat_front_addr":"172.21.15.28:6815/3526420147","state":["exists","up"]},{"osd":4,"uuid":"cac8fdba-d8a4-453d-9091-121d932d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6809","nonce":3527398709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6811","nonce":3527398709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6815","nonce":3527398709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3527398709},{"type":"v1","addr":"172.21.15.165:6813","nonce":3527398709}]},"public_addr":"172.21.15.165:6809/3527398709","cluster_addr":"172.21.15.165:6811/3527398709","heartbeat_back_addr":"172.21.15.165:6815/3527398709","heartbeat_front_addr":"172.21.15.165:6813/3527398709","state":["exists","up"]},{"osd":5,"uuid":"072f3379-6c55-40cd-9898-639846b9369f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6809","nonce":2066099940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6811","nonce":2066099940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6815","nonce":2066099940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2066099940},{"type":"v1","addr":"172.21.15.184:6813","nonce":2066099940}]},"public_addr":"172.21.15.184:6809/2066099940","cluster_addr":"172.21.15.184:6811/2066099940","heartbeat_back_addr":"172.21.15.184:6815/2066099940","heartbeat_front_addr":"172.21.15.184:6813/2066099940","state":["exists","up"]},{"osd":6,"uuid":"d2cb1de2-c3d0-4e1c-81c2-1d6f2479dd19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6819","nonce":899794462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6821","nonce":899794462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6825","nonce":899794462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":899794462},{"type":"v1","addr":"172.21.15.28:6823","nonce":899794462}]},"public_addr":"172.21.15.28:6819/899794462","cluster_addr":"172.21.15.28:6821/899794462","heartbeat_back_addr":"172.21.15.28:6825/899794462","heartbeat_front_addr":"172.21.15.28:6823/899794462","state":["exists","up"]},{"osd":7,"uuid":"d1148aa0-4209-45a5-bcc5-3a3ccd5fbd1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6817","nonce":1526551047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6819","nonce":1526551047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6823","nonce":1526551047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1526551047},{"type":"v1","addr":"172.21.15.165:6821","nonce":1526551047}]},"public_addr":"172.21.15.165:6817/1526551047","cluster_addr":"172.21.15.165:6819/1526551047","heartbeat_back_addr":"172.21.15.165:6823/1526551047","heartbeat_front_addr":"172.21.15.165:6821/1526551047","state":["exists","up"]},{"osd":8,"uuid":"12434642-450b-4e7b-a86a-4f32e76d2286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6817","nonce":2287150908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6819","nonce":2287150908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6823","nonce":2287150908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2287150908},{"type":"v1","addr":"172.21.15.184:6821","nonce":2287150908}]},"public_addr":"172.21.15.184:6817/2287150908","cluster_addr":"172.21.15.184:6819/2287150908","heartbeat_back_addr":"172.21.15.184:6823/2287150908","heartbeat_front_addr":"172.21.15.184:6821/2287150908","state":["exists","up"]},{"osd":9,"uuid":"00768b2e-17b7-4ba7-b857-f83ea092f65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6827","nonce":712618237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6829","nonce":712618237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6833","nonce":712618237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":712618237},{"type":"v1","addr":"172.21.15.28:6831","nonce":712618237}]},"public_addr":"172.21.15.28:6827/712618237","cluster_addr":"172.21.15.28:6829/712618237","heartbeat_back_addr":"172.21.15.28:6833/712618237","heartbeat_front_addr":"172.21.15.28:6831/712618237","state":["exists","up"]},{"osd":10,"uuid":"111970a1-1719-4b83-bedf-de211b2088c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6825","nonce":3106810724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6827","nonce":3106810724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6831","nonce":3106810724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3106810724},{"type":"v1","addr":"172.21.15.165:6829","nonce":3106810724}]},"public_addr":"172.21.15.165:6825/3106810724","cluster_addr":"172.21.15.165:6827/3106810724","heartbeat_back_addr":"172.21.15.165:6831/3106810724","heartbeat_front_addr":"172.21.15.165:6829/3106810724","state":["exists","up"]},{"osd":11,"uuid":"c95c2ce7-e296-4064-b913-311b7b9dcbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6825","nonce":1865031750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6827","nonce":1865031750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6831","nonce":1865031750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":1865031750},{"type":"v1","addr":"172.21.15.184:6829","nonce":1865031750}]},"public_addr":"172.21.15.184:6825/1865031750","cluster_addr":"172.21.15.184:6827/1865031750","heartbeat_back_addr":"172.21.15.184:6831/1865031750","heartbeat_front_addr":"172.21.15.184:6829/1865031750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:23.104900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:18:46.459464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:09.782879+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:19:35.295429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:00.175523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:24.592837+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:20:50.737136+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:15.862463+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:21:40.999673+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:07.923444+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:33.378222+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:22:59.063608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3330063293":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3745037139":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/2130261041":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6800/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/3510522803":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/273816117":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/934935086":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/1194745694":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:0/3920156841":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6801/1179097296":"2024-05-10T08:16:22.859488+0000","172.21.15.28:6800/953899001":"2024-05-10T08:26:17.964999+0000","172.21.15.28:6801/2931568341":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/466716681":"2024-05-10T08:16:49.552676+0000","172.21.15.28:0/2416478420":"2024-05-10T08:16:22.859488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":20,"length":6},{"begin":27,"length":1},{"begin":29,"length":1},{"begin":31,"length":1},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":1}]},{"pool":4,"snaps":[{"begin":18,"length":8},{"begin":27,"length":1},{"begin":29,"length":1},{"begin":31,"length":1},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":37,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":24,"length":1},{"begin":37,"length":1}]},{"pool":4,"snaps":[{"begin":24,"length":1},{"begin":37,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:47:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f08840a2f00 msgr2=0x7f08840a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f08840a2f00 0x7f08840a53c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0890006010 tx=0x7f0890000f40 comp rx=0 tx=0).stop 2024-05-09T08:47:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08981abd30 msgr2=0x7f08981ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08981abd30 0x7f08981ac1e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f089400e970 tx=0x7f089400ee30 comp rx=0 tx=0).stop 2024-05-09T08:47:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 shutdown_connections 2024-05-09T08:47:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f08840a2f00 0x7f08840a53c0 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0890006010 tx=0x7f0890000f40 comp rx=0 tx=0).stop 2024-05-09T08:47:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08981abd30 0x7f08981ac1e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08981a7470 0x7f08981ab7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 --2- 172.21.15.28:0/2228411449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f089810ff90 0x7f08981a6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.245+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 >> 172.21.15.28:0/2228411449 conn(0x7f089806c950 msgr2=0x7f0898119b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.246+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 shutdown_connections 2024-05-09T08:47:10.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.247+0000 7f089f542640 1 -- 172.21.15.28:0/2228411449 wait complete. 2024-05-09T08:47:10.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T08:47:10.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:09 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3223699948' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:10.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:09 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3223699948' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:10.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:09 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3223699948' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.475+0000 7f273f170640 1 -- 172.21.15.28:0/3013565037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2738110080 msgr2=0x7f273810db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.475+0000 7f273f170640 1 --2- 172.21.15.28:0/3013565037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2738110080 0x7f273810db70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2730009930 tx=0x7f273002a480 comp rx=0 tx=0).stop 2024-05-09T08:47:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.482+0000 7f273f170640 1 -- 172.21.15.28:0/3013565037 shutdown_connections 2024-05-09T08:47:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.482+0000 7f273f170640 1 --2- 172.21.15.28:0/3013565037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f273811b010 0x7f273811d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.482+0000 7f273f170640 1 --2- 172.21.15.28:0/3013565037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f273810e0b0 0x7f273810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.482+0000 7f273f170640 1 --2- 172.21.15.28:0/3013565037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2738110080 0x7f273810db70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.482+0000 7f273f170640 1 -- 172.21.15.28:0/3013565037 >> 172.21.15.28:0/3013565037 conn(0x7f273806c940 msgr2=0x7f273806cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:10.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.486+0000 7f273f170640 1 -- 172.21.15.28:0/3013565037 shutdown_connections 2024-05-09T08:47:10.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.490+0000 7f273f170640 1 -- 172.21.15.28:0/3013565037 wait complete. 2024-05-09T08:47:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.491+0000 7f273f170640 1 Processor -- start 2024-05-09T08:47:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.491+0000 7f273f170640 1 -- start start 2024-05-09T08:47:10.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.492+0000 7f273f170640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f273810e0b0 0x7f273811a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.492+0000 7f273f170640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2738110080 0x7f2738114870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.492+0000 7f273f170640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f273811b010 0x7f2738114db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.492+0000 7f273f170640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2738111240 con 0x7f273811b010 2024-05-09T08:47:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.492+0000 7f273f170640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f27381110c0 con 0x7f2738110080 2024-05-09T08:47:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.492+0000 7f273f170640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27381113c0 con 0x7f273810e0b0 2024-05-09T08:47:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.493+0000 7f273cf0c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f273810e0b0 0x7f273811a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.494+0000 7f273cf0c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f273810e0b0 0x7f273811a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48320/0 (socket says 172.21.15.28:48320) 2024-05-09T08:47:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.494+0000 7f273cf0c640 1 -- 172.21.15.28:0/3539884295 learned_addr learned my addr 172.21.15.28:0/3539884295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.495+0000 7f273d70d640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f273811b010 0x7f2738114db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.497+0000 7f273cf0c640 1 -- 172.21.15.28:0/3539884295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2738110080 msgr2=0x7f2738114870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:47:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.497+0000 7f273cf0c640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2738110080 0x7f2738114870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.497+0000 7f273cf0c640 1 -- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f273811b010 msgr2=0x7f2738114db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.497+0000 7f273cf0c640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f273811b010 0x7f2738114db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.497+0000 7f273cf0c640 1 -- 172.21.15.28:0/3539884295 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2738115580 con 0x7f273810e0b0 2024-05-09T08:47:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.497+0000 7f273d70d640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f273811b010 0x7f2738114db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:47:10.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.500+0000 7f273cf0c640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f273810e0b0 0x7f273811a800 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2730006010 tx=0x7f2730004b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:10.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.503+0000 7f2735ffb640 1 -- 172.21.15.28:0/3539884295 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730040070 con 0x7f273810e0b0 2024-05-09T08:47:10.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.503+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27381bbdb0 con 0x7f273810e0b0 2024-05-09T08:47:10.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.503+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27381bc210 con 0x7f273810e0b0 2024-05-09T08:47:10.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.503+0000 7f2735ffb640 1 -- 172.21.15.28:0/3539884295 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27300314d0 con 0x7f273810e0b0 2024-05-09T08:47:10.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.504+0000 7f2735ffb640 1 -- 172.21.15.28:0/3539884295 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730035020 con 0x7f273810e0b0 2024-05-09T08:47:10.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.507+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2718005290 con 0x7f273810e0b0 2024-05-09T08:47:10.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.509+0000 7f2735ffb640 1 -- 172.21.15.28:0/3539884295 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2730043070 con 0x7f273810e0b0 2024-05-09T08:47:10.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.510+0000 7f2735ffb640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27240a2d30 0x7f27240a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:10.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.511+0000 7f2735ffb640 1 -- 172.21.15.28:0/3539884295 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9831+0+0 (secure 0 0 0) 0x7f27300f0b20 con 0x7f273810e0b0 2024-05-09T08:47:10.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.516+0000 7f2737fff640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27240a2d30 0x7f27240a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:10.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.516+0000 7f2735ffb640 1 -- 172.21.15.28:0/3539884295 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f27300bd360 con 0x7f273810e0b0 2024-05-09T08:47:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.533+0000 7f2737fff640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27240a2d30 0x7f27240a51f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2738116030 tx=0x7f2720006bd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:10.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.983+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2718005850 con 0x7f273810e0b0 2024-05-09T08:47:10.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.995+0000 7f2735ffb640 1 -- 172.21.15.28:0/3539884295 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) ==== 161+0+0 (secure 0 0 0) 0x7f27300bcc60 con 0x7f273810e0b0 2024-05-09T08:47:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T08:47:11.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27240a2d30 msgr2=0x7f27240a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:11.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27240a2d30 0x7f27240a51f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2738116030 tx=0x7f2720006bd0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f273810e0b0 msgr2=0x7f273811a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:11.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f273810e0b0 0x7f273811a800 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2730006010 tx=0x7f2730004b80 comp rx=0 tx=0).stop 2024-05-09T08:47:11.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 shutdown_connections 2024-05-09T08:47:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f273811b010 0x7f2738114db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2738110080 0x7f2738114870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f27240a2d30 0x7f27240a51f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 --2- 172.21.15.28:0/3539884295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f273810e0b0 0x7f273811a800 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:10.998+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 >> 172.21.15.28:0/3539884295 conn(0x7f273806c940 msgr2=0x7f273811c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.000+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 shutdown_connections 2024-05-09T08:47:11.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.001+0000 7f273f170640 1 -- 172.21.15.28:0/3539884295 wait complete. 2024-05-09T08:47:11.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T08:47:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.208+0000 7f84e8940640 1 -- 172.21.15.28:0/4179316042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84dc0b7a50 msgr2=0x7f84dc0b9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.208+0000 7f84e8940640 1 --2- 172.21.15.28:0/4179316042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84dc0b7a50 0x7f84dc0b9e40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f84e4067530 tx=0x7f84e408dd70 comp rx=0 tx=0).stop 2024-05-09T08:47:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.211+0000 7f84e8940640 1 -- 172.21.15.28:0/4179316042 shutdown_connections 2024-05-09T08:47:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.211+0000 7f84e8940640 1 --2- 172.21.15.28:0/4179316042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84dc0b7a50 0x7f84dc0b9e40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.211+0000 7f84e8940640 1 --2- 172.21.15.28:0/4179316042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84dc0a56e0 0x7f84dc0b7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.211+0000 7f84e8940640 1 --2- 172.21.15.28:0/4179316042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84dc0a4dc0 0x7f84dc0a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.211+0000 7f84e8940640 1 -- 172.21.15.28:0/4179316042 >> 172.21.15.28:0/4179316042 conn(0x7f84dc01a620 msgr2=0x7f84dc01aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:11.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.215+0000 7f84e8940640 1 -- 172.21.15.28:0/4179316042 shutdown_connections 2024-05-09T08:47:11.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.215+0000 7f84e8940640 1 -- 172.21.15.28:0/4179316042 wait complete. 2024-05-09T08:47:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.216+0000 7f84e8940640 1 Processor -- start 2024-05-09T08:47:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e8940640 1 -- start start 2024-05-09T08:47:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e8940640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84dc0a4dc0 0x7f84dc0b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e8940640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84dc0a56e0 0x7f84dc0b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e8940640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84dc0b7a50 0x7f84dc0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e8940640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc0ad720 con 0x7f84dc0a4dc0 2024-05-09T08:47:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e8940640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc0ad5a0 con 0x7f84dc0b7a50 2024-05-09T08:47:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e8940640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc0ad8a0 con 0x7f84dc0a56e0 2024-05-09T08:47:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e3fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84dc0b7a50 0x7f84dc0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:11.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84dc0a4dc0 0x7f84dc0b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e3fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84dc0b7a50 0x7f84dc0b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.28:41878/0 (socket says 172.21.15.28:41878) 2024-05-09T08:47:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e3fff640 1 -- 172.21.15.28:0/913622110 learned_addr learned my addr 172.21.15.28:0/913622110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.223+0000 7f84e2ffd640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84dc0a56e0 0x7f84dc0b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.224+0000 7f84e3fff640 1 -- 172.21.15.28:0/913622110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84dc0a56e0 msgr2=0x7f84dc0b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.224+0000 7f84e3fff640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84dc0a56e0 0x7f84dc0b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.224+0000 7f84e3fff640 1 -- 172.21.15.28:0/913622110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84dc0a4dc0 msgr2=0x7f84dc0b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.224+0000 7f84e3fff640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84dc0a4dc0 0x7f84dc0b71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.224+0000 7f84e3fff640 1 -- 172.21.15.28:0/913622110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84dc0b1f10 con 0x7f84dc0b7a50 2024-05-09T08:47:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.224+0000 7f84e3fff640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84dc0b7a50 0x7f84dc0b17c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f84e4050dd0 tx=0x7f84e4097a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.226+0000 7f84e0ff9640 1 -- 172.21.15.28:0/913622110 <== mon.2 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4097c90 con 0x7f84dc0b7a50 2024-05-09T08:47:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.226+0000 7f84e0ff9640 1 -- 172.21.15.28:0/913622110 <== mon.2 v2:172.21.15.165:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84e408ee50 con 0x7f84dc0b7a50 2024-05-09T08:47:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.227+0000 7f84e0ff9640 1 -- 172.21.15.28:0/913622110 <== mon.2 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4096630 con 0x7f84dc0b7a50 2024-05-09T08:47:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.227+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84dc1589a0 con 0x7f84dc0b7a50 2024-05-09T08:47:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.227+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84dc158d50 con 0x7f84dc0b7a50 2024-05-09T08:47:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.229+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84dc0a6210 con 0x7f84dc0b7a50 2024-05-09T08:47:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.234+0000 7f84e0ff9640 1 -- 172.21.15.28:0/913622110 <== mon.2 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f84e40967d0 con 0x7f84dc0b7a50 2024-05-09T08:47:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.235+0000 7f84e0ff9640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84d40a2c60 0x7f84d40a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.235+0000 7f84e0ff9640 1 -- 172.21.15.28:0/913622110 <== mon.2 v2:172.21.15.165:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 10173+0+0 (secure 0 0 0) 0x7f84e4152ca0 con 0x7f84dc0b7a50 2024-05-09T08:47:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.236+0000 7f84e37fe640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84d40a2c60 0x7f84d40a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.237+0000 7f84e37fe640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84d40a2c60 0x7f84d40a5120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84dc018900 tx=0x7f84d8005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:11.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.242+0000 7f84e0ff9640 1 -- 172.21.15.28:0/913622110 <== mon.2 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84e40a6020 con 0x7f84dc0b7a50 2024-05-09T08:47:11.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:10 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2228411449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:11.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:10 smithi165 ceph-mon[34540]: pgmap v674: 129 pgs: 67 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 43 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 68 KiB/s wr, 124 op/s 2024-05-09T08:47:11.351 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b[64615]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.351 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b[64615]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.351 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b[64615]: starting mds.b at 2024-05-09T08:47:11.352 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[69112]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.352 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[69112]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.352 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[69112]: starting mds.e at 2024-05-09T08:47:11.352 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k[63660]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.352 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k[63660]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.352 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mds[63664]: -- 172.21.15.165:0/1220535134 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d28e60ba40 con 0x55d28e5df000 2024-05-09T08:47:11.353 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k[63660]: starting mds.k at 2024-05-09T08:47:11.353 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.353 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.353 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:47:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: starting mds.h at 2024-05-09T08:47:11.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:10 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2228411449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:11.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:10 smithi184 ceph-mon[34509]: pgmap v674: 129 pgs: 67 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 43 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 68 KiB/s wr, 124 op/s 2024-05-09T08:47:11.357 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.357 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.357 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mds[65075]: -- 172.21.15.184:0/3941100057 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56015a99fa40 con 0x56015a972c00 2024-05-09T08:47:11.357 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: starting mds.c at 2024-05-09T08:47:11.357 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.358 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.358 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: starting mds.f at 2024-05-09T08:47:11.358 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.358 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.359 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mds[65397]: -- 172.21.15.184:0/4033747705 <== mon.1 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563ecdacfa40 con 0x563ecdaa3000 2024-05-09T08:47:11.359 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: starting mds.i at 2024-05-09T08:47:11.359 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l[64458]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.359 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l[64458]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.359 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mds[64462]: -- 172.21.15.184:0/2622024866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56426856ba40 con 0x56426853e800 2024-05-09T08:47:11.359 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:47:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l[64458]: starting mds.l at 2024-05-09T08:47:11.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:10 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2228411449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:47:11.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:10 smithi028 ceph-mon[30119]: pgmap v674: 129 pgs: 67 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 43 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 68 KiB/s wr, 124 op/s 2024-05-09T08:47:11.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:47:10.985+0000 7f8910535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T08:47:11.433 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a[69053]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.433 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a[69053]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.433 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mds[69057]: -- 172.21.15.28:0/1950098156 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55efa4125a40 con 0x55efa40f9000 2024-05-09T08:47:11.433 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a[69053]: starting mds.a at 2024-05-09T08:47:11.433 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:47:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g[68693]: ignoring --setuser ceph since I am not root 2024-05-09T08:47:11.433 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:47:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g[68693]: ignoring --setgroup ceph since I am not root 2024-05-09T08:47:11.433 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:47:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g[68693]: starting mds.g at 2024-05-09T08:47:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:11.677+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f84dc0a7e10 con 0x7f84dc0b7a50 2024-05-09T08:47:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.002+0000 7f84e0ff9640 1 -- 172.21.15.28:0/913622110 <== mon.2 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) ==== 104+0+0 (secure 0 0 0) 0x7f84e411edb0 con 0x7f84dc0b7a50 2024-05-09T08:47:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84d40a2c60 msgr2=0x7f84d40a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84d40a2c60 0x7f84d40a5120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84dc018900 tx=0x7f84d8005f90 comp rx=0 tx=0).stop 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84dc0b7a50 msgr2=0x7f84dc0b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84dc0b7a50 0x7f84dc0b17c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f84e4050dd0 tx=0x7f84e4097a10 comp rx=0 tx=0).stop 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 shutdown_connections 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84dc0b7a50 0x7f84dc0b17c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f84d40a2c60 0x7f84d40a5120 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f84dc0a56e0 0x7f84dc0b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 --2- 172.21.15.28:0/913622110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84dc0a4dc0 0x7f84dc0b71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 >> 172.21.15.28:0/913622110 conn(0x7f84dc01a620 msgr2=0x7f84dc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.006+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 shutdown_connections 2024-05-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.007+0000 7f84e8940640 1 -- 172.21.15.28:0/913622110 wait complete. 2024-05-09T08:47:12.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T08:47:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.244+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2310319291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fe0101800 msgr2=0x7f9fe0101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.244+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2310319291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fe0101800 0x7f9fe0101c80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0009900 tx=0x7f9fd002a3f0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.248+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2310319291 shutdown_connections 2024-05-09T08:47:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.248+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2310319291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fe01021c0 0x7f9fe010e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.248+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2310319291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fe0101800 0x7f9fe0101c80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.248+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2310319291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fe0107810 0x7f9fe0107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.248+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2310319291 >> 172.21.15.28:0/2310319291 conn(0x7f9fe0078c80 msgr2=0x7f9fe0079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.255+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2310319291 shutdown_connections 2024-05-09T08:47:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.255+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2310319291 wait complete. 2024-05-09T08:47:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.256+0000 7f9fe5a38640 1 Processor -- start 2024-05-09T08:47:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.256+0000 7f9fe5a38640 1 -- start start 2024-05-09T08:47:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fe5a38640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fe0101800 0x7f9fe019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fe5a38640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fe01021c0 0x7f9fe0195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fe5a38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fe0107810 0x7f9fe0196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fe5a38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe010fa70 con 0x7f9fe0107810 2024-05-09T08:47:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fe5a38640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe010f8f0 con 0x7f9fe01021c0 2024-05-09T08:47:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fe5a38640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe010fbf0 con 0x7f9fe0101800 2024-05-09T08:47:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fe0107810 0x7f9fe0196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fdeffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fe0101800 0x7f9fe019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fdeffd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fe0101800 0x7f9fe019ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48372/0 (socket says 172.21.15.28:48372) 2024-05-09T08:47:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.257+0000 7f9fdeffd640 1 -- 172.21.15.28:0/2889808318 learned_addr learned my addr 172.21.15.28:0/2889808318 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.258+0000 7f9fdf7fe640 1 -- 172.21.15.28:0/2889808318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fe0101800 msgr2=0x7f9fe019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.258+0000 7f9fdf7fe640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fe0101800 0x7f9fe019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.258+0000 7f9fdf7fe640 1 -- 172.21.15.28:0/2889808318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fe01021c0 msgr2=0x7f9fe0195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.258+0000 7f9fdf7fe640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fe01021c0 0x7f9fe0195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.258+0000 7f9fdf7fe640 1 -- 172.21.15.28:0/2889808318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe0196850 con 0x7f9fe0107810 2024-05-09T08:47:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.259+0000 7f9fdf7fe640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fe0107810 0x7f9fe0196290 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400c710 tx=0x7f9fd400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.263+0000 7f9fe4a36640 1 -- 172.21.15.28:0/2889808318 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4004220 con 0x7f9fe0107810 2024-05-09T08:47:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.263+0000 7f9fe4a36640 1 -- 172.21.15.28:0/2889808318 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd400ce20 con 0x7f9fe0107810 2024-05-09T08:47:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.263+0000 7f9fe4a36640 1 -- 172.21.15.28:0/2889808318 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd400f450 con 0x7f9fe0107810 2024-05-09T08:47:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.264+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe0196b40 con 0x7f9fe0107810 2024-05-09T08:47:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.264+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe019ab20 con 0x7f9fe0107810 2024-05-09T08:47:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.268+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe019b600 con 0x7f9fe0107810 2024-05-09T08:47:12.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.272+0000 7f9fe4a36640 1 -- 172.21.15.28:0/2889808318 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9fd40079b0 con 0x7f9fe0107810 2024-05-09T08:47:12.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.273+0000 7f9fe4a36640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9fc80a2f00 0x7f9fc80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:12.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.273+0000 7f9fe4a36640 1 -- 172.21.15.28:0/2889808318 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 10173+0+0 (secure 0 0 0) 0x7f9fd40cf770 con 0x7f9fe0107810 2024-05-09T08:47:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.277+0000 7f9fdeffd640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9fc80a2f00 0x7f9fc80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.280+0000 7f9fdeffd640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9fc80a2f00 0x7f9fc80a53c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9fd800b320 tx=0x7f9fd8011040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.282+0000 7f9fe4a36640 1 -- 172.21.15.28:0/2889808318 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9fd4014030 con 0x7f9fe0107810 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/3539884295' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: Health check cleared: MDS_DAMAGE (was: 4 MDSs report damaged metadata) 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: osdmap e140: 12 total, 12 up, 12 in 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/913622110' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:47:12.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:11 smithi165 ceph-mon[34540]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:47:12.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/3539884295' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T08:47:12.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T08:47:12.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T08:47:12.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T08:47:12.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: Health check cleared: MDS_DAMAGE (was: 4 MDSs report damaged metadata) 2024-05-09T08:47:12.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T08:47:12.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: osdmap e140: 12 total, 12 up, 12 in 2024-05-09T08:47:12.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-09T08:47:12.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/913622110' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:47:12.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:11 smithi184 ceph-mon[34509]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/3539884295' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: Health check cleared: MDS_DAMAGE (was: 4 MDSs report damaged metadata) 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: osdmap e140: 12 total, 12 up, 12 in 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/913622110' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:47:12.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:11 smithi028 ceph-mon[30119]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:47:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:12.645+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9fe01972b0 con 0x7f9fe0107810 2024-05-09T08:47:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.008+0000 7f9fe4a36640 1 -- 172.21.15.28:0/2889808318 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v141) ==== 181+0+0 (secure 0 0 0) 0x7f9fe01972b0 con 0x7f9fe0107810 2024-05-09T08:47:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-05-09T08:47:13.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9fc80a2f00 msgr2=0x7f9fc80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:13.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9fc80a2f00 0x7f9fc80a53c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9fd800b320 tx=0x7f9fd8011040 comp rx=0 tx=0).stop 2024-05-09T08:47:13.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fe0107810 msgr2=0x7f9fe0196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:13.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fe0107810 0x7f9fe0196290 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400c710 tx=0x7f9fd400cbd0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 shutdown_connections 2024-05-09T08:47:13.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fe0107810 0x7f9fe0196290 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fe01021c0 0x7f9fe0195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f9fc80a2f00 0x7f9fc80a53c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 --2- 172.21.15.28:0/2889808318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9fe0101800 0x7f9fe019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 >> 172.21.15.28:0/2889808318 conn(0x7f9fe0078c80 msgr2=0x7f9fe00ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.012+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 shutdown_connections 2024-05-09T08:47:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.013+0000 7f9fe5a38640 1 -- 172.21.15.28:0/2889808318 wait complete. 2024-05-09T08:47:13.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T08:47:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.234+0000 7f009a4a5640 1 -- 172.21.15.28:0/2810561518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0094073b00 msgr2=0x7f0094073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.234+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2810561518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0094073b00 0x7f0094073ee0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f0088009930 tx=0x7f008802a4d0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.235+0000 7f009a4a5640 1 -- 172.21.15.28:0/2810561518 shutdown_connections 2024-05-09T08:47:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.235+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2810561518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00941128e0 0x7f0094114cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.235+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2810561518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0094074420 0x7f00941123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.235+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2810561518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0094073b00 0x7f0094073ee0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.235+0000 7f009a4a5640 1 -- 172.21.15.28:0/2810561518 >> 172.21.15.28:0/2810561518 conn(0x7f0094078c80 msgr2=0x7f0094079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.235+0000 7f009a4a5640 1 -- 172.21.15.28:0/2810561518 shutdown_connections 2024-05-09T08:47:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.235+0000 7f009a4a5640 1 -- 172.21.15.28:0/2810561518 wait complete. 2024-05-09T08:47:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.236+0000 7f009a4a5640 1 Processor -- start 2024-05-09T08:47:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.236+0000 7f009a4a5640 1 -- start start 2024-05-09T08:47:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.237+0000 7f009a4a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0094073b00 0x7f0094073300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.237+0000 7f009a4a5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0094074420 0x7f009406d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.237+0000 7f009a4a5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00941128e0 0x7f009406d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.237+0000 7f009a4a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f009406bca0 con 0x7f0094073b00 2024-05-09T08:47:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.237+0000 7f009a4a5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f009406bb20 con 0x7f0094074420 2024-05-09T08:47:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.237+0000 7f009a4a5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f009406be20 con 0x7f00941128e0 2024-05-09T08:47:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.237+0000 7f0093fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0094073b00 0x7f0094073300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.238+0000 7f0098a42640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00941128e0 0x7f009406d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.238+0000 7f0098a42640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00941128e0 0x7f009406d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48390/0 (socket says 172.21.15.28:48390) 2024-05-09T08:47:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.238+0000 7f0098a42640 1 -- 172.21.15.28:0/2197459571 learned_addr learned my addr 172.21.15.28:0/2197459571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.238+0000 7f0093fff640 1 -- 172.21.15.28:0/2197459571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00941128e0 msgr2=0x7f009406d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.238+0000 7f0093fff640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00941128e0 0x7f009406d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.238+0000 7f0093fff640 1 -- 172.21.15.28:0/2197459571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0094074420 msgr2=0x7f009406d370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:47:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.238+0000 7f0093fff640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0094074420 0x7f009406d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.238+0000 7f0093fff640 1 -- 172.21.15.28:0/2197459571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009406e0f0 con 0x7f0094073b00 2024-05-09T08:47:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.239+0000 7f0093fff640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0094073b00 0x7f0094073300 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f0088009850 tx=0x7f0088003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.239+0000 7f00917fa640 1 -- 172.21.15.28:0/2197459571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008803f070 con 0x7f0094073b00 2024-05-09T08:47:13.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.239+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0094105b70 con 0x7f0094073b00 2024-05-09T08:47:13.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.239+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0094106080 con 0x7f0094073b00 2024-05-09T08:47:13.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.239+0000 7f00917fa640 1 -- 172.21.15.28:0/2197459571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0088031c90 con 0x7f0094073b00 2024-05-09T08:47:13.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.239+0000 7f00917fa640 1 -- 172.21.15.28:0/2197459571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088035030 con 0x7f0094073b00 2024-05-09T08:47:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.244+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0080005290 con 0x7f0094073b00 2024-05-09T08:47:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.244+0000 7f00917fa640 1 -- 172.21.15.28:0/2197459571 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0088042070 con 0x7f0094073b00 2024-05-09T08:47:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.246+0000 7f00917fa640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f007c0a2e40 0x7f007c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.246+0000 7f00917fa640 1 -- 172.21.15.28:0/2197459571 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 9590+0+0 (secure 0 0 0) 0x7f00880effe0 con 0x7f0094073b00 2024-05-09T08:47:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.247+0000 7f00937fe640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f007c0a2e40 0x7f007c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.248+0000 7f00937fe640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f007c0a2e40 0x7f007c0a5300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f009406eb30 tx=0x7f008400b370 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:13.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.256+0000 7f00917fa640 1 -- 172.21.15.28:0/2197459571 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f00880bc770 con 0x7f0094073b00 2024-05-09T08:47:13.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T08:47:13.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T08:47:13.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: Cluster is now healthy 2024-05-09T08:47:13.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:47:13.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6832/3908721259,v1:172.21.15.184:6834/3908721259] up:boot 2024-05-09T08:47:13.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6832/1195645009,v1:172.21.15.165:6833/1195645009] up:boot 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6835/2273292858,v1:172.21.15.165:6837/2273292858] up:boot 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6833/729340862,v1:172.21.15.184:6835/729340862] up:boot 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6834/3092671304,v1:172.21.15.165:6836/3092671304] up:boot 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.165:6838/1843967762,v1:172.21.15.165:6839/1843967762] up:boot 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6836/2197272803,v1:172.21.15.184:6837/2197272803] up:boot 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.184:6838/1518921987,v1:172.21.15.184:6839/1518921987] up:boot 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.28:6838/1234845725,v1:172.21.15.28:6839/1234845725] up:boot 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: mds.? [v2:172.21.15.28:6840/1128104836,v1:172.21.15.28:6841/1128104836] up:boot 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: fsmap 12 up:standby 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:47:13.352 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:47:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:47:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:47:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:47:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:47:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:47:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:47:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:47:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:47:13.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: pgmap v676: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.9 KiB/s wr, 68 op/s 2024-05-09T08:47:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:12 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2889808318' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:47:13.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T08:47:13.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T08:47:13.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: Cluster is now healthy 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6832/3908721259,v1:172.21.15.184:6834/3908721259] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6832/1195645009,v1:172.21.15.165:6833/1195645009] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6835/2273292858,v1:172.21.15.165:6837/2273292858] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6833/729340862,v1:172.21.15.184:6835/729340862] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6834/3092671304,v1:172.21.15.165:6836/3092671304] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.165:6838/1843967762,v1:172.21.15.165:6839/1843967762] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6836/2197272803,v1:172.21.15.184:6837/2197272803] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.184:6838/1518921987,v1:172.21.15.184:6839/1518921987] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.28:6838/1234845725,v1:172.21.15.28:6839/1234845725] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: mds.? [v2:172.21.15.28:6840/1128104836,v1:172.21.15.28:6841/1128104836] up:boot 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: fsmap 12 up:standby 2024-05-09T08:47:13.357 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: pgmap v676: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.9 KiB/s wr, 68 op/s 2024-05-09T08:47:13.358 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:12 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2889808318' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:47:13.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T08:47:13.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: Cluster is now healthy 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6832/3908721259,v1:172.21.15.184:6834/3908721259] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6832/1195645009,v1:172.21.15.165:6833/1195645009] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6835/2273292858,v1:172.21.15.165:6837/2273292858] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6833/729340862,v1:172.21.15.184:6835/729340862] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6834/3092671304,v1:172.21.15.165:6836/3092671304] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.165:6838/1843967762,v1:172.21.15.165:6839/1843967762] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6836/2197272803,v1:172.21.15.184:6837/2197272803] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.184:6838/1518921987,v1:172.21.15.184:6839/1518921987] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.28:6838/1234845725,v1:172.21.15.28:6839/1234845725] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: mds.? [v2:172.21.15.28:6840/1128104836,v1:172.21.15.28:6841/1128104836] up:boot 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: fsmap 12 up:standby 2024-05-09T08:47:13.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='mgr.15528 172.21.15.28:0/1858044748' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:47:13.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: pgmap v676: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 2.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.9 KiB/s wr, 68 op/s 2024-05-09T08:47:13.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:12 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2889808318' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:47:13.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:13.623+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0080005580 con 0x7f0094073b00 2024-05-09T08:47:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.022+0000 7f00917fa640 1 -- 172.21.15.28:0/2197459571 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v142) ==== 169+0+0 (secure 0 0 0) 0x7f00880bc070 con 0x7f0094073b00 2024-05-09T08:47:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-05-09T08:47:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f007c0a2e40 msgr2=0x7f007c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f007c0a2e40 0x7f007c0a5300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f009406eb30 tx=0x7f008400b370 comp rx=0 tx=0).stop 2024-05-09T08:47:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0094073b00 msgr2=0x7f0094073300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0094073b00 0x7f0094073300 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f0088009850 tx=0x7f0088003b90 comp rx=0 tx=0).stop 2024-05-09T08:47:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 shutdown_connections 2024-05-09T08:47:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f007c0a2e40 0x7f007c0a5300 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00941128e0 0x7f009406d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0094074420 0x7f009406d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 --2- 172.21.15.28:0/2197459571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0094073b00 0x7f0094073300 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:14.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 >> 172.21.15.28:0/2197459571 conn(0x7f0094078c80 msgr2=0x7f00940fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:14.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 shutdown_connections 2024-05-09T08:47:14.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.027+0000 7f009a4a5640 1 -- 172.21.15.28:0/2197459571 wait complete. 2024-05-09T08:47:14.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-05-09T08:47:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.257+0000 7f15b8a02640 1 -- 172.21.15.28:0/226865029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac09acf0 msgr2=0x7f15ac004740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.257+0000 7f15b8a02640 1 --2- 172.21.15.28:0/226865029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac09acf0 0x7f15ac004740 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009900 tx=0x7f15a802a440 comp rx=0 tx=0).stop 2024-05-09T08:47:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.259+0000 7f15b8a02640 1 -- 172.21.15.28:0/226865029 shutdown_connections 2024-05-09T08:47:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.259+0000 7f15b8a02640 1 --2- 172.21.15.28:0/226865029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15ac0986e0 0x7f15ac098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.259+0000 7f15b8a02640 1 --2- 172.21.15.28:0/226865029 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac004c80 0x7f15ac005100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.259+0000 7f15b8a02640 1 --2- 172.21.15.28:0/226865029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac09acf0 0x7f15ac004740 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.259+0000 7f15b8a02640 1 -- 172.21.15.28:0/226865029 >> 172.21.15.28:0/226865029 conn(0x7f15ac00a740 msgr2=0x7f15ac00ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.265+0000 7f15b8a02640 1 -- 172.21.15.28:0/226865029 shutdown_connections 2024-05-09T08:47:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.265+0000 7f15b8a02640 1 -- 172.21.15.28:0/226865029 wait complete. 2024-05-09T08:47:14.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.266+0000 7f15b8a02640 1 Processor -- start 2024-05-09T08:47:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.266+0000 7f15b8a02640 1 -- start start 2024-05-09T08:47:14.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.266+0000 7f15b8a02640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac004c80 0x7f15ac0a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.266+0000 7f15b8a02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15ac0986e0 0x7f15ac0a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.267+0000 7f15b8a02640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac09acf0 0x7f15ac0a28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.267+0000 7f15b37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac004c80 0x7f15ac0a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.267+0000 7f15b37fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac004c80 0x7f15ac0a92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.28:48418/0 (socket says 172.21.15.28:48418) 2024-05-09T08:47:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.267+0000 7f15b8a02640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac0ac740 con 0x7f15ac0986e0 2024-05-09T08:47:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.267+0000 7f15b8a02640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac0ac5c0 con 0x7f15ac09acf0 2024-05-09T08:47:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.267+0000 7f15b8a02640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac0ac8c0 con 0x7f15ac004c80 2024-05-09T08:47:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.267+0000 7f15b37fe640 1 -- 172.21.15.28:0/1763341598 learned_addr learned my addr 172.21.15.28:0/1763341598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-09T08:47:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.267+0000 7f15b2ffd640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15ac0986e0 0x7f15ac0a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.267+0000 7f15b3fff640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac09acf0 0x7f15ac0a28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.268+0000 7f15b2ffd640 1 -- 172.21.15.28:0/1763341598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac004c80 msgr2=0x7f15ac0a92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.268+0000 7f15b2ffd640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac004c80 0x7f15ac0a92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.268+0000 7f15b2ffd640 1 -- 172.21.15.28:0/1763341598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac09acf0 msgr2=0x7f15ac0a28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.268+0000 7f15b2ffd640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac09acf0 0x7f15ac0a28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.268+0000 7f15b2ffd640 1 -- 172.21.15.28:0/1763341598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ac0a2eb0 con 0x7f15ac0986e0 2024-05-09T08:47:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.268+0000 7f15b37fe640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac004c80 0x7f15ac0a92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.268+0000 7f15b2ffd640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15ac0986e0 0x7f15ac0a2360 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009fd0 tx=0x7f15a000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.269+0000 7f15b0ff9640 1 -- 172.21.15.28:0/1763341598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a0015020 con 0x7f15ac0986e0 2024-05-09T08:47:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.269+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15ac0a31a0 con 0x7f15ac0986e0 2024-05-09T08:47:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.269+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15ac0a7120 con 0x7f15ac0986e0 2024-05-09T08:47:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.269+0000 7f15b0ff9640 1 -- 172.21.15.28:0/1763341598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a0011d10 con 0x7f15ac0986e0 2024-05-09T08:47:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.269+0000 7f15b0ff9640 1 -- 172.21.15.28:0/1763341598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a00193b0 con 0x7f15ac0986e0 2024-05-09T08:47:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.272+0000 7f15b0ff9640 1 -- 172.21.15.28:0/1763341598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f15a0018020 con 0x7f15ac0986e0 2024-05-09T08:47:14.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.273+0000 7f15b0ff9640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f15a40a2dc0 0x7f15a40a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:14.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.274+0000 7f15b0ff9640 1 -- 172.21.15.28:0/1763341598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 9002+0+0 (secure 0 0 0) 0x7f15a00cee30 con 0x7f15ac0986e0 2024-05-09T08:47:14.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.274+0000 7f15b37fe640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f15a40a2dc0 0x7f15a40a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.275+0000 7f15b37fe640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f15a40a2dc0 0x7f15a40a5280 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f15a8031550 tx=0x7f15a8009580 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.275+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15ac095220 con 0x7f15ac0986e0 2024-05-09T08:47:14.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.285+0000 7f15b0ff9640 1 -- 172.21.15.28:0/1763341598 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f15a001f020 con 0x7f15ac0986e0 2024-05-09T08:47:14.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:14 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2889808318' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:47:14.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:14 smithi165 ceph-mon[34540]: osdmap e141: 12 total, 12 up, 12 in 2024-05-09T08:47:14.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:14 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2197459571' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:47:14.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:14 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2889808318' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:47:14.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:14 smithi184 ceph-mon[34509]: osdmap e141: 12 total, 12 up, 12 in 2024-05-09T08:47:14.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:14 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2197459571' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:47:14.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:14 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2889808318' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:47:14.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:14 smithi028 ceph-mon[30119]: osdmap e141: 12 total, 12 up, 12 in 2024-05-09T08:47:14.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:14 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2197459571' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:47:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:14.653+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f15ac0a7430 con 0x7f15ac0986e0 2024-05-09T08:47:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.040+0000 7f15b0ff9640 1 -- 172.21.15.28:0/1763341598 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v143) ==== 178+0+0 (secure 0 0 0) 0x7f15a009b960 con 0x7f15ac0986e0 2024-05-09T08:47:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' removed 2024-05-09T08:47:15.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f15a40a2dc0 msgr2=0x7f15a40a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:15.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f15a40a2dc0 0x7f15a40a5280 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f15a8031550 tx=0x7f15a8009580 comp rx=0 tx=0).stop 2024-05-09T08:47:15.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15ac0986e0 msgr2=0x7f15ac0a2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:15.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15ac0986e0 0x7f15ac0a2360 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009fd0 tx=0x7f15a000b870 comp rx=0 tx=0).stop 2024-05-09T08:47:15.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 shutdown_connections 2024-05-09T08:47:15.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac09acf0 0x7f15ac0a28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:15.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15ac0986e0 0x7f15ac0a2360 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:15.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.28:6800/160415970,v1:172.21.15.28:6801/160415970] conn(0x7f15a40a2dc0 0x7f15a40a5280 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:15.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 --2- 172.21.15.28:0/1763341598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15ac004c80 0x7f15ac0a92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:15.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.044+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 >> 172.21.15.28:0/1763341598 conn(0x7f15ac00a740 msgr2=0x7f15ac092270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:15.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.045+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 shutdown_connections 2024-05-09T08:47:15.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-09T08:47:15.045+0000 7f15b8a02640 1 -- 172.21.15.28:0/1763341598 wait complete. 2024-05-09T08:47:15.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:47:15.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:47:15.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:47:15.228 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-09T08:47:15.228 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T08:47:15.252 DEBUG:teuthology.orchestra.run.smithi165:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T08:47:15.273 DEBUG:teuthology.orchestra.run.smithi184:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T08:47:15.295 INFO:tasks.cephadm:Stopping all daemons... 2024-05-09T08:47:15.296 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-09T08:47:15.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.a 2024-05-09T08:47:15.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:15 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/2197459571' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:47:15.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:15 smithi028 ceph-mon[30119]: osdmap e142: 12 total, 12 up, 12 in 2024-05-09T08:47:15.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:15 smithi028 ceph-mon[30119]: pgmap v679: 33 pgs: 16 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 1 active+clean; 652 MiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:15.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:15 smithi028 ceph-mon[30119]: from='client.? 172.21.15.28:0/1763341598' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:47:15.350 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:15 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/2197459571' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:47:15.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:15 smithi165 ceph-mon[34540]: osdmap e142: 12 total, 12 up, 12 in 2024-05-09T08:47:15.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:15 smithi165 ceph-mon[34540]: pgmap v679: 33 pgs: 16 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 1 active+clean; 652 MiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:15.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:15 smithi165 ceph-mon[34540]: from='client.? 172.21.15.28:0/1763341598' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:47:15.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:15 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/2197459571' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:47:15.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:15 smithi184 ceph-mon[34509]: osdmap e142: 12 total, 12 up, 12 in 2024-05-09T08:47:15.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:15 smithi184 ceph-mon[34509]: pgmap v679: 33 pgs: 16 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 1 active+clean; 652 MiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:47:15.356 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:15 smithi184 ceph-mon[34509]: from='client.? 172.21.15.28:0/1763341598' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:15 smithi028 systemd[1]: Stopping Ceph mon.a for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:16.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:47:15.823+0000 7f8915d40700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:47:16.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a[30115]: 2024-05-09T08:47:15.823+0000 7f8915d40700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-09T08:47:16.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:15 smithi028 podman[99279]: 2024-05-09 08:47:15.928548349 +0000 UTC m=+0.254748429 container died ba577546d1881252c883862af96b2db20369410d2ca7bc7bf4a21d62ca6b236f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, ceph=True) 2024-05-09T08:47:16.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:16 smithi028 podman[99279]: 2024-05-09 08:47:16.65524195 +0000 UTC m=+0.981442026 container cleanup ba577546d1881252c883862af96b2db20369410d2ca7bc7bf4a21d62ca6b236f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-96805dc) 2024-05-09T08:47:16.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:16 smithi028 bash[99279]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a 2024-05-09T08:47:17.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 09 08:47:16 smithi028 podman[99306]: 2024-05-09 08:47:16.940378815 +0000 UTC m=+1.000834943 container remove ba577546d1881252c883862af96b2db20369410d2ca7bc7bf4a21d62ca6b236f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-a, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-96805dc) 2024-05-09T08:47:17.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.a.service' 2024-05-09T08:47:17.505 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:47:17.505 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-09T08:47:17.506 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-09T08:47:17.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.b 2024-05-09T08:47:17.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:17 smithi165 systemd[1]: Stopping Ceph mon.b for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:18.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:17 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-b[34536]: 2024-05-09T08:47:17.948+0000 7f5b28581700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:47:18.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:17 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-b[34536]: 2024-05-09T08:47:17.948+0000 7f5b28581700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-09T08:47:18.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:18 smithi165 podman[72855]: 2024-05-09 08:47:18.021308556 +0000 UTC m=+0.203083411 container died 4d34e1c042f3847783adf5d15846397712f65ea4fccfcc7d51668a881b5280e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:47:18.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:18 smithi165 podman[72855]: 2024-05-09 08:47:18.462669011 +0000 UTC m=+0.644443890 container cleanup 4d34e1c042f3847783adf5d15846397712f65ea4fccfcc7d51668a881b5280e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc) 2024-05-09T08:47:18.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:May 09 08:47:18 smithi165 bash[72855]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-b 2024-05-09T08:47:18.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.b.service' 2024-05-09T08:47:18.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:47:18.882 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-09T08:47:18.882 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-09T08:47:18.883 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.c 2024-05-09T08:47:19.258 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:18 smithi184 systemd[1]: Stopping Ceph mon.c for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:19.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:19 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-c[34505]: 2024-05-09T08:47:19.257+0000 7f53deeed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:47:19.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:19 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-c[34505]: 2024-05-09T08:47:19.257+0000 7f53deeed700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-09T08:47:19.606 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:19 smithi184 podman[72742]: 2024-05-09 08:47:19.339897984 +0000 UTC m=+0.215880811 container died 0923518f48c71bcc5aa1dbfee2d24b3da45526a7292eaf2424c9b884827aaf10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-09T08:47:19.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:19 smithi184 podman[72742]: 2024-05-09 08:47:19.709042465 +0000 UTC m=+0.585025293 container cleanup 0923518f48c71bcc5aa1dbfee2d24b3da45526a7292eaf2424c9b884827aaf10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-c, RELEASE=squid-96805dc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507) 2024-05-09T08:47:19.993 INFO:journalctl@ceph.mon.c.smithi184.stdout:May 09 08:47:19 smithi184 bash[72742]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mon-c 2024-05-09T08:47:20.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mon.c.service' 2024-05-09T08:47:20.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:47:20.153 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-09T08:47:20.153 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-09T08:47:20.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.x 2024-05-09T08:47:20.432 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:47:20 smithi028 systemd[1]: Stopping Ceph mgr.x for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:20.872 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:47:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x[30349]: 2024-05-09T08:47:20.465+0000 7fc7f7bae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:47:20.872 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:47:20 smithi028 podman[99730]: 2024-05-09 08:47:20.496736138 +0000 UTC m=+0.136384854 container died d33853de27af83dfa677d3a8b81389f9a0167c9512d9fea242855c47ab5d6ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-05-09T08:47:23.432 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:47:22 smithi028 podman[99730]: 2024-05-09 08:47:22.992313335 +0000 UTC m=+2.631962047 container cleanup d33853de27af83dfa677d3a8b81389f9a0167c9512d9fea242855c47ab5d6ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-09T08:47:23.432 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 09 08:47:22 smithi028 bash[99730]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-x 2024-05-09T08:47:25.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.x.service' 2024-05-09T08:47:25.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:47:25.767 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-09T08:47:25.767 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-09T08:47:25.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.y 2024-05-09T08:47:26.095 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:47:25 smithi165 systemd[1]: Stopping Ceph mgr.y for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:26.350 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:47:26 smithi165 podman[72976]: 2024-05-09 08:47:26.09615021 +0000 UTC m=+0.136566014 container died 7f78607ac683604e10947988d9d02102fdb6b0ec84038cb8e7be8776ab4cdf42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_CLEAN=True) 2024-05-09T08:47:27.850 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:47:27 smithi165 podman[72976]: 2024-05-09 08:47:27.653617887 +0000 UTC m=+1.694033705 container cleanup 7f78607ac683604e10947988d9d02102fdb6b0ec84038cb8e7be8776ab4cdf42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux ) 2024-05-09T08:47:27.851 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:47:27 smithi165 bash[72976]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y 2024-05-09T08:47:28.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.y.service' 2024-05-09T08:47:28.144 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:47:27 smithi165 podman[72990]: 2024-05-09 08:47:27.887353347 +0000 UTC m=+1.779887299 container remove 7f78607ac683604e10947988d9d02102fdb6b0ec84038cb8e7be8776ab4cdf42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:47:28.145 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:47:27 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-05-09T08:47:28.145 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:47:28 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-05-09T08:47:28.145 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:47:28 smithi165 systemd[1]: Stopped Ceph mgr.y for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:47:28.145 INFO:journalctl@ceph.mgr.y.smithi165.stdout:May 09 08:47:28 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.y.service: Consumed 23.830s CPU time. 2024-05-09T08:47:28.160 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:47:28.160 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-09T08:47:28.160 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-09T08:47:28.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.z 2024-05-09T08:47:28.499 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:47:28 smithi184 systemd[1]: Stopping Ceph mgr.z for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:28.805 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:47:28 smithi184 podman[72865]: 2024-05-09 08:47:28.500886815 +0000 UTC m=+0.139000308 container died 4a3ae0841ccb7dd0005aca57ccddd7d69bb7a5ede942e27995840bf116131609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:47:29.106 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:47:28 smithi184 podman[72865]: 2024-05-09 08:47:28.88913309 +0000 UTC m=+0.527246582 container cleanup 4a3ae0841ccb7dd0005aca57ccddd7d69bb7a5ede942e27995840bf116131609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-09T08:47:29.106 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:47:28 smithi184 bash[72865]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z 2024-05-09T08:47:29.381 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:47:29 smithi184 podman[72879]: 2024-05-09 08:47:29.124180865 +0000 UTC m=+0.575204806 container remove 4a3ae0841ccb7dd0005aca57ccddd7d69bb7a5ede942e27995840bf116131609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mgr-z, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-09T08:47:29.381 INFO:journalctl@ceph.mgr.z.smithi184.stdout:May 09 08:47:29 smithi184 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-05-09T08:47:29.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mgr.z.service' 2024-05-09T08:47:29.419 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:47:29.420 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-09T08:47:29.420 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-09T08:47:29.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.0 2024-05-09T08:47:29.714 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:29 smithi028 systemd[1]: Stopping Ceph osd.0 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:30.182 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:29 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0[42216]: 2024-05-09T08:47:29.714+0000 7f735b7f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:47:30.182 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:29 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0[42216]: 2024-05-09T08:47:29.714+0000 7f735b7f6700 -1 osd.0 143 *** Got signal Terminated *** 2024-05-09T08:47:30.182 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:29 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0[42216]: 2024-05-09T08:47:29.714+0000 7f735b7f6700 -1 osd.0 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:47:35.671 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:35 smithi028 podman[99851]: 2024-05-09 08:47:35.416144167 +0000 UTC m=+5.806221309 container died 9f2df7f66d8b4906858051ec054e6d481522867cb4cef0a9a5ae1c5d086a95a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507) 2024-05-09T08:47:35.931 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:35 smithi028 podman[99851]: 2024-05-09 08:47:35.756706163 +0000 UTC m=+6.146783309 container cleanup 9f2df7f66d8b4906858051ec054e6d481522867cb4cef0a9a5ae1c5d086a95a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-09T08:47:35.932 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:35 smithi028 bash[99851]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0 2024-05-09T08:47:36.405 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:35 smithi028 podman[99884]: 2024-05-09 08:47:35.974743454 +0000 UTC m=+0.542617682 container remove 9f2df7f66d8b4906858051ec054e6d481522867cb4cef0a9a5ae1c5d086a95a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-09T08:47:36.668 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:36 smithi028 podman[99973]: 2024-05-09 08:47:36.310067565 +0000 UTC m=+0.019986634 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:36.669 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:36 smithi028 podman[99973]: 2024-05-09 08:47:36.417465602 +0000 UTC m=+0.127384716 container create 396940846ec03330d8c460c74ee74caa6bf2493a3302537ee8ca231496c72ee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, ceph=True, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-09T08:47:36.932 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:36 smithi028 podman[99973]: 2024-05-09 08:47:36.670352963 +0000 UTC m=+0.380272037 container init 396940846ec03330d8c460c74ee74caa6bf2493a3302537ee8ca231496c72ee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc) 2024-05-09T08:47:36.932 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:36 smithi028 podman[99973]: 2024-05-09 08:47:36.673873396 +0000 UTC m=+0.383792464 container start 396940846ec03330d8c460c74ee74caa6bf2493a3302537ee8ca231496c72ee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-09T08:47:36.932 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:36 smithi028 podman[99973]: 2024-05-09 08:47:36.77880562 +0000 UTC m=+0.488724707 container attach 396940846ec03330d8c460c74ee74caa6bf2493a3302537ee8ca231496c72ee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-96805dc) 2024-05-09T08:47:37.432 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 09 08:47:37 smithi028 podman[99973]: 2024-05-09 08:47:37.17003251 +0000 UTC m=+0.879951581 container died 396940846ec03330d8c460c74ee74caa6bf2493a3302537ee8ca231496c72ee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-0-deactivate, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:47:37.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.0.service' 2024-05-09T08:47:37.869 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:47:37.869 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-09T08:47:37.869 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-09T08:47:37.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.1 2024-05-09T08:47:38.180 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:37 smithi165 systemd[1]: Stopping Ceph osd.1 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:38.600 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:38 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1[39124]: 2024-05-09T08:47:38.180+0000 7f9942d2b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:47:38.601 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:38 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1[39124]: 2024-05-09T08:47:38.180+0000 7f9942d2b700 -1 osd.1 143 *** Got signal Terminated *** 2024-05-09T08:47:38.601 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:38 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1[39124]: 2024-05-09T08:47:38.180+0000 7f9942d2b700 -1 osd.1 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:47:44.006 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:43 smithi165 podman[73103]: 2024-05-09 08:47:43.618897934 +0000 UTC m=+5.552836959 container died 20f69da742ed63b8649ed34ec1810b2a1db04ded489e36993624c1bd859b09f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:47:44.350 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:44 smithi165 podman[73103]: 2024-05-09 08:47:44.09821387 +0000 UTC m=+6.032152903 container cleanup 20f69da742ed63b8649ed34ec1810b2a1db04ded489e36993624c1bd859b09f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:47:44.350 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:44 smithi165 bash[73103]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1 2024-05-09T08:47:44.722 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:44 smithi165 podman[73132]: 2024-05-09 08:47:44.400086189 +0000 UTC m=+0.777367954 container remove 20f69da742ed63b8649ed34ec1810b2a1db04ded489e36993624c1bd859b09f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.license=GPLv2) 2024-05-09T08:47:45.004 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:44 smithi165 podman[73222]: 2024-05-09 08:47:44.626403585 +0000 UTC m=+0.018112515 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:45.004 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:44 smithi165 podman[73222]: 2024-05-09 08:47:44.767855082 +0000 UTC m=+0.159563999 container create 065db1329370645f2179c9246fa76e825ce7e0190bd7fd824519905295d3c299 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True) 2024-05-09T08:47:45.351 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:45 smithi165 podman[73222]: 2024-05-09 08:47:45.004025994 +0000 UTC m=+0.395734906 container init 065db1329370645f2179c9246fa76e825ce7e0190bd7fd824519905295d3c299 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:47:45.351 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:45 smithi165 podman[73222]: 2024-05-09 08:47:45.007432838 +0000 UTC m=+0.399141752 container start 065db1329370645f2179c9246fa76e825ce7e0190bd7fd824519905295d3c299 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1-deactivate, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:47:45.351 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:45 smithi165 podman[73222]: 2024-05-09 08:47:45.120775673 +0000 UTC m=+0.512484594 container attach 065db1329370645f2179c9246fa76e825ce7e0190bd7fd824519905295d3c299 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-09T08:47:45.851 INFO:journalctl@ceph.osd.1.smithi165.stdout:May 09 08:47:45 smithi165 podman[73222]: 2024-05-09 08:47:45.564056151 +0000 UTC m=+0.955765072 container died 065db1329370645f2179c9246fa76e825ce7e0190bd7fd824519905295d3c299 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-1-deactivate, RELEASE=squid-96805dc, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-09T08:47:46.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.1.service' 2024-05-09T08:47:46.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:47:46.297 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-09T08:47:46.297 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-09T08:47:46.297 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.2 2024-05-09T08:47:46.607 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:46 smithi184 systemd[1]: Stopping Ceph osd.2 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:47.106 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:46 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2[39523]: 2024-05-09T08:47:46.617+0000 7f843991e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:47:47.107 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:46 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2[39523]: 2024-05-09T08:47:46.617+0000 7f843991e700 -1 osd.2 143 *** Got signal Terminated *** 2024-05-09T08:47:47.107 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:46 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2[39523]: 2024-05-09T08:47:46.617+0000 7f843991e700 -1 osd.2 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:47:52.401 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:52 smithi184 podman[72992]: 2024-05-09 08:47:52.117853472 +0000 UTC m=+5.624088037 container died 9ee669cc621eb72a0302f2ddccaf5f1805152fc614022ad1f88b86162c815603 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-09T08:47:52.784 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:52 smithi184 podman[72992]: 2024-05-09 08:47:52.492800842 +0000 UTC m=+5.999035406 container cleanup 9ee669cc621eb72a0302f2ddccaf5f1805152fc614022ad1f88b86162c815603 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:47:52.784 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:52 smithi184 bash[72992]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2 2024-05-09T08:47:53.106 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:52 smithi184 podman[73022]: 2024-05-09 08:47:52.785822069 +0000 UTC m=+0.668855592 container remove 9ee669cc621eb72a0302f2ddccaf5f1805152fc614022ad1f88b86162c815603 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5) 2024-05-09T08:47:53.514 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:53 smithi184 podman[73112]: 2024-05-09 08:47:53.024756636 +0000 UTC m=+0.018421478 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:53.515 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:53 smithi184 podman[73112]: 2024-05-09 08:47:53.153776932 +0000 UTC m=+0.147441766 container create 27392cf158788bb2b8266053bf0ec6abce81248047651a7329580dfa9e73c726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-09T08:47:53.515 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:53 smithi184 podman[73112]: 2024-05-09 08:47:53.365496521 +0000 UTC m=+0.359161367 container init 27392cf158788bb2b8266053bf0ec6abce81248047651a7329580dfa9e73c726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240507, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_BRANCH=HEAD) 2024-05-09T08:47:53.515 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:53 smithi184 podman[73112]: 2024-05-09 08:47:53.369038143 +0000 UTC m=+0.362702977 container start 27392cf158788bb2b8266053bf0ec6abce81248047651a7329580dfa9e73c726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-09T08:47:53.515 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:53 smithi184 podman[73112]: 2024-05-09 08:47:53.515915753 +0000 UTC m=+0.509580601 container attach 27392cf158788bb2b8266053bf0ec6abce81248047651a7329580dfa9e73c726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-09T08:47:54.356 INFO:journalctl@ceph.osd.2.smithi184.stdout:May 09 08:47:54 smithi184 podman[73112]: 2024-05-09 08:47:54.096838645 +0000 UTC m=+1.090503489 container died 27392cf158788bb2b8266053bf0ec6abce81248047651a7329580dfa9e73c726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-09T08:47:54.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.2.service' 2024-05-09T08:47:54.771 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:47:54.772 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-09T08:47:54.772 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-09T08:47:54.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.3 2024-05-09T08:47:55.093 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:47:54 smithi028 systemd[1]: Stopping Ceph osd.3 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:47:55.432 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:47:55 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:47:55.093+0000 7f00ccf30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:47:55.432 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:47:55 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:47:55.093+0000 7f00ccf30700 -1 osd.3 143 *** Got signal Terminated *** 2024-05-09T08:47:55.432 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:47:55 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:47:55.093+0000 7f00ccf30700 -1 osd.3 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:47:56.432 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:47:56 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:47:56.048+0000 7f00c551c700 -1 osd.3 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.474523+0000 front 2024-05-09T08:47:33.474476+0000 (oldest deadline 2024-05-09T08:47:55.774596+0000) 2024-05-09T08:47:57.106 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:47:56 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:47:56.842+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:47:57.432 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:47:57 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:47:57.034+0000 7f00c551c700 -1 osd.3 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.474523+0000 front 2024-05-09T08:47:33.474476+0000 (oldest deadline 2024-05-09T08:47:55.774596+0000) 2024-05-09T08:47:57.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:47:57 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:47:57.222+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:47:58.106 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:47:57 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:47:57.823+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:47:58.432 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:47:58 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:47:58.026+0000 7f00c551c700 -1 osd.3 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.474523+0000 front 2024-05-09T08:47:33.474476+0000 (oldest deadline 2024-05-09T08:47:55.774596+0000) 2024-05-09T08:47:58.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:47:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:47:58.269+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:47:59.056 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:47:58 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:47:58.718+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:47:59.262 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:47:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:47:58.864+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:47:59.262 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:47:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:47:58.896+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:47:59.432 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:47:59 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:47:59.056+0000 7f00c551c700 -1 osd.3 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.474523+0000 front 2024-05-09T08:47:33.474476+0000 (oldest deadline 2024-05-09T08:47:55.774596+0000) 2024-05-09T08:47:59.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:47:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:47:59.262+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:00.041 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:47:59 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:47:59.686+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:00.258 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:47:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:47:59.892+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:00.259 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:47:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:47:59.924+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:00.432 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3[48679]: 2024-05-09T08:48:00.041+0000 7f00c551c700 -1 osd.3 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.474523+0000 front 2024-05-09T08:47:33.474476+0000 (oldest deadline 2024-05-09T08:47:55.774596+0000) 2024-05-09T08:48:00.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:00.259+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:00.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:00.726+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:00.932 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:00 smithi028 podman[100078]: 2024-05-09 08:48:00.713574326 +0000 UTC m=+5.745827083 container died 7ed65ca5eaa558a94b4e47d02ea4d7f6c1d858f6ec3996c4e2e6cd029a1a2297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2) 2024-05-09T08:48:00.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:00 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:00.655+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:01.279 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:00.887+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:01.279 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:00.883+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:01.350 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:00 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:00.976+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:01.403 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:01 smithi028 podman[100078]: 2024-05-09 08:48:01.129731114 +0000 UTC m=+6.161983876 container cleanup 7ed65ca5eaa558a94b4e47d02ea4d7f6c1d858f6ec3996c4e2e6cd029a1a2297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507) 2024-05-09T08:48:01.403 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:01 smithi028 bash[100078]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3 2024-05-09T08:48:01.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:01.279+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:01.681 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:01 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:01.634+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:01.682 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:01 smithi028 podman[100096]: 2024-05-09 08:48:01.404187643 +0000 UTC m=+0.691229440 container remove 7ed65ca5eaa558a94b4e47d02ea4d7f6c1d858f6ec3996c4e2e6cd029a1a2297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-09T08:48:02.037 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:01 smithi028 podman[100182]: 2024-05-09 08:48:01.804005418 +0000 UTC m=+0.018273618 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:02.037 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:01 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:01.758+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:02.106 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:01.938+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:02.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:01.848+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:02.329 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:02 smithi028 podman[100182]: 2024-05-09 08:48:02.038609424 +0000 UTC m=+0.252877606 container create e00b338c80f2aa03022957b4dc55ad28273aa5748e1c22fdfaedc845108bc4b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-09T08:48:02.350 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:01 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:01.953+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:02.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:02.320+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:02.610 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:02 smithi028 podman[100182]: 2024-05-09 08:48:02.331077078 +0000 UTC m=+0.545345259 container init e00b338c80f2aa03022957b4dc55ad28273aa5748e1c22fdfaedc845108bc4b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-09T08:48:02.610 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:02 smithi028 podman[100182]: 2024-05-09 08:48:02.335749941 +0000 UTC m=+0.550018130 container start e00b338c80f2aa03022957b4dc55ad28273aa5748e1c22fdfaedc845108bc4b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3-deactivate, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-05-09T08:48:02.610 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:02 smithi028 podman[100182]: 2024-05-09 08:48:02.433204892 +0000 UTC m=+0.647473087 container attach e00b338c80f2aa03022957b4dc55ad28273aa5748e1c22fdfaedc845108bc4b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-09T08:48:02.931 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:02 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:02.791+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:02.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:02 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:02.609+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:02.932 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:02 smithi028 podman[100182]: 2024-05-09 08:48:02.663098451 +0000 UTC m=+0.877366632 container died e00b338c80f2aa03022957b4dc55ad28273aa5748e1c22fdfaedc845108bc4b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-09T08:48:03.106 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:02.950+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:03.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:02.820+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:03.290 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 09 08:48:03 smithi028 podman[100217]: 2024-05-09 08:48:03.2786062 +0000 UTC m=+0.610057013 container remove e00b338c80f2aa03022957b4dc55ad28273aa5748e1c22fdfaedc845108bc4b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:48:03.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.3.service' 2024-05-09T08:48:03.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:48:03.337 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-09T08:48:03.337 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-09T08:48:03.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.4 2024-05-09T08:48:03.350 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:02 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:02.938+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:03.580 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:03 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:03.581+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:03.710 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:03 smithi165 systemd[1]: Stopping Ceph osd.4 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:48:03.836 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:03.367+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:03.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:03 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:03.816+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:03.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:03 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:03.816+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:04.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:03 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:03.901+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:04.101 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:03 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4[45640]: 2024-05-09T08:48:03.710+0000 7fd2633d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:48:04.101 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:03 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4[45640]: 2024-05-09T08:48:03.710+0000 7fd2633d1700 -1 osd.4 143 *** Got signal Terminated *** 2024-05-09T08:48:04.101 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:03 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4[45640]: 2024-05-09T08:48:03.710+0000 7fd2633d1700 -1 osd.4 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:48:04.106 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:03.911+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:04.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:03.836+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:04.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:04.370+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:04.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:04 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:04.609+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:04.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:04 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:04.781+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:04.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:04 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:04.781+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:05.100 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4[45640]: 2024-05-09T08:48:04.834+0000 7fd25f9c5700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.662209+0000 front 2024-05-09T08:47:41.662234+0000 (oldest deadline 2024-05-09T08:48:03.962395+0000) 2024-05-09T08:48:05.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:04 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:04.867+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:05.321 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:04.864+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:05.321 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:04.943+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:05.321 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:04.943+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:05.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:05.321+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:05.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:05 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:05.659+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:05.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:05 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:05.745+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:05.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:05 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:05.745+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:06.100 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:05 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4[45640]: 2024-05-09T08:48:05.804+0000 7fd25f9c5700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.662209+0000 front 2024-05-09T08:47:41.662234+0000 (oldest deadline 2024-05-09T08:48:03.962395+0000) 2024-05-09T08:48:06.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:05 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:05.865+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:06.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:05 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:05.865+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:06.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:05.901+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:06.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:05.971+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:06.357 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:05.971+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:06.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:06 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:06.363+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:06.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:06 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:06.363+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:06.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:06 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:06.636+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:06.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:06 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:06.636+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:06.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:06 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:06.748+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:06.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:06 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:06.748+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:07.100 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:06 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4[45640]: 2024-05-09T08:48:06.779+0000 7fd25f9c5700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.662209+0000 front 2024-05-09T08:47:41.662234+0000 (oldest deadline 2024-05-09T08:48:03.962395+0000) 2024-05-09T08:48:07.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:06 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:06.894+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:07.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:06 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:06.894+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:07.352 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:06 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:06.863+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:07.352 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:06 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:06.969+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:07.352 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:06 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:06.969+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:07.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:07 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:07.352+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:07.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:07 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:07.352+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:07.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:07 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:07.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:07.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:07 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:07.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:07.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:07 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:07.597+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:07.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:07 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:07.597+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:08.100 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:07 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4[45640]: 2024-05-09T08:48:07.816+0000 7fd25f9c5700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.662209+0000 front 2024-05-09T08:47:41.662234+0000 (oldest deadline 2024-05-09T08:48:03.962395+0000) 2024-05-09T08:48:08.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:07 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:07.890+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:08.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:07 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:07.890+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:08.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:07 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:07.877+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:08.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:07 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:07.923+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:08.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:07 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:07.923+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:08.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:08 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:08.383+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:08.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:08 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:08.383+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:08.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:08 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:08.605+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:08.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:08 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:08.605+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:08.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:08 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:08.726+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:08.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:08 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:08.726+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:09.341 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:09 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:09.044+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:09.341 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:08 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:08.936+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:09.341 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:08 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:08.936+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:09.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:08 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:08.901+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:09.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:08 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:08.927+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:09.357 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:08 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:08.927+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:09.600 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:09 smithi165 podman[73330]: 2024-05-09 08:48:09.342688279 +0000 UTC m=+5.812907383 container died 888c840bf9cb58784f593a19deb3a5b0e3dfa64476b3026454ef7fdc6ef66e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.build-date=20240507) 2024-05-09T08:48:09.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:09 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:09.392+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:09.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:09 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:09.392+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:09.913 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:09 smithi165 podman[73330]: 2024-05-09 08:48:09.710548803 +0000 UTC m=+6.180767894 container cleanup 888c840bf9cb58784f593a19deb3a5b0e3dfa64476b3026454ef7fdc6ef66e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-09T08:48:09.913 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:09 smithi165 bash[73330]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4 2024-05-09T08:48:09.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:09.761+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:09.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:09.761+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:09.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:09.600+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:09.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:09.600+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:10.341 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:10 smithi165 podman[73347]: 2024-05-09 08:48:10.03732719 +0000 UTC m=+0.692005145 container remove 888c840bf9cb58784f593a19deb3a5b0e3dfa64476b3026454ef7fdc6ef66e1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:48:10.341 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:10.045+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:10.342 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:09 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:09.913+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:10.342 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:09 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:09.913+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:10.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:09 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:09.950+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:10.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:09 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:09.897+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:10.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:09 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:09.897+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:10.600 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:10 smithi165 podman[73438]: 2024-05-09 08:48:10.245669787 +0000 UTC m=+0.020374429 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:10.601 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:10 smithi165 podman[73438]: 2024-05-09 08:48:10.363329398 +0000 UTC m=+0.138034040 container create da9bac06cd85fd362af669c148ce8a12aa3179deda705d6380195a721c269ff9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4-deactivate, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-09T08:48:10.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:10.403+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:10.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:10.403+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:10.891 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:10 smithi165 podman[73438]: 2024-05-09 08:48:10.641332713 +0000 UTC m=+0.416037360 container init da9bac06cd85fd362af669c148ce8a12aa3179deda705d6380195a721c269ff9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-09T08:48:10.891 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:10 smithi165 podman[73438]: 2024-05-09 08:48:10.645167459 +0000 UTC m=+0.419872113 container start da9bac06cd85fd362af669c148ce8a12aa3179deda705d6380195a721c269ff9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:48:10.891 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:10 smithi165 podman[73438]: 2024-05-09 08:48:10.757840202 +0000 UTC m=+0.532544866 container attach da9bac06cd85fd362af669c148ce8a12aa3179deda705d6380195a721c269ff9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T08:48:10.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:10.624+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:10.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:10.624+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:10.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:10.806+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:10.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:10 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:10.806+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:11.106 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:10.856+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:11.106 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:10.856+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:11.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:10 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:10.995+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:11.218 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:11.035+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:11.218 INFO:journalctl@ceph.osd.4.smithi165.stdout:May 09 08:48:10 smithi165 podman[73438]: 2024-05-09 08:48:10.930569006 +0000 UTC m=+0.705273648 container died da9bac06cd85fd362af669c148ce8a12aa3179deda705d6380195a721c269ff9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507) 2024-05-09T08:48:11.219 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:10.891+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:11.219 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:10 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:10.892+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:11.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.4.service' 2024-05-09T08:48:11.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:48:11.639 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-09T08:48:11.639 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-09T08:48:11.640 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.5 2024-05-09T08:48:11.665 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:11.395+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:11.665 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:11.395+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:11.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:11.659+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:11.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:11.659+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:11.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:11.822+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:11.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:11 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:11.822+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:11.962 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:11 smithi184 systemd[1]: Stopping Ceph osd.5 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:48:11.962 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:11.855+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:11.962 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:11.855+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:12.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:12.057+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:12.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:11.907+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:12.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:11 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:11.907+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:12.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:12.042+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:12.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:11.962+0000 7ff8d488e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:48:12.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:11.962+0000 7ff8d488e700 -1 osd.5 143 *** Got signal Terminated *** 2024-05-09T08:48:12.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:11 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:11.962+0000 7ff8d488e700 -1 osd.5 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:48:12.849 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:12.369+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:12.850 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:12.369+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:12.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:12 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:12.791+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:12.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:12 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:12.791+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:12.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:12 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:12.678+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:12.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:12 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:12.678+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:13.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:13.089+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:13.106 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:12.850+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:13.107 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:12 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:12.850+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:13.350 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:13.028+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:13.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:13.028+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:13.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:12.881+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:13.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:12 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:12.881+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:13.768 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:13.639+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:13.769 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:13.639+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:13.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:13.418+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:13.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:13.418+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:14.182 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:13.769+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:14.182 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:13.769+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:14.182 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:13 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:13.769+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:14.350 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:13.987+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:14.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:13.987+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:14.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:13.854+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:14.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:13 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:13.854+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:14.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:14 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:14.059+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:14.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:13.865+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:14.356 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:13 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:13.865+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:14.843 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:14 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:14.449+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:14.843 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:14 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:14.449+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:14.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:14 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:14.749+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:14.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:14 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:14.749+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:14.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:14 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:14.749+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:14.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:14 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:14.651+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:14.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:14 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:14.651+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:14.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:14 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:14.651+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:15.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:15.089+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:15.107 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:14 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:14.843+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:15.107 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:14 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:14.843+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:15.350 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:15.007+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:15.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:15.007+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:15.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:14 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:14.895+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:15.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:14 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:14.895+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:15.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:14 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:14.895+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:15.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:15.436+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:15.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:15.436+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:15.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:15.755+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:15.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:15.755+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:15.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:15.755+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:15.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:15.636+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:15.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:15.636+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:15.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:15.636+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:16.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:15.892+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:16.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:15.892+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:16.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:15.892+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:16.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:15.962+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:16.352 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:15 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:15.962+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:16.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:16.091+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:16.357 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:15.889+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:16.357 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:15.889+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:16.357 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:15 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:15.889+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.971657+0000 front 2024-05-09T08:47:50.971702+0000 (oldest deadline 2024-05-09T08:48:15.671860+0000) 2024-05-09T08:48:16.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:16.387+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:16.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:16.387+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:16.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:16.387+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:16.934 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:16.595+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:16.934 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:16.595+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:16.934 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:16.595+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:16.934 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:16.737+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:16.934 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:16.737+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:16.934 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:16 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:16.737+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:17.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:16.959+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:17.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:16.959+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:17.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:16.891+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:17.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:16.891+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:17.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:16 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:16.891+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:17.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:17 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:17.128+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:17.357 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:16.910+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.867937+0000 front 2024-05-09T08:47:31.868004+0000 (oldest deadline 2024-05-09T08:47:55.968003+0000) 2024-05-09T08:48:17.357 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:16.910+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.969848+0000 front 2024-05-09T08:47:42.969749+0000 (oldest deadline 2024-05-09T08:48:04.069917+0000) 2024-05-09T08:48:17.357 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:16 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5[46495]: 2024-05-09T08:48:16.910+0000 7ff8cce7a700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.971657+0000 front 2024-05-09T08:47:50.971702+0000 (oldest deadline 2024-05-09T08:48:15.671860+0000) 2024-05-09T08:48:17.637 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:17 smithi184 podman[73219]: 2024-05-09 08:48:17.601749474 +0000 UTC m=+5.761778818 container died f71598b9e4eed38feeda8dbfa31dfe3231ad7dd25f9c301263e05aca9a575ef0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T08:48:17.638 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:17 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:17.428+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:17.638 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:17 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:17.428+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:17.638 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:17 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:17.428+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:17.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:17.589+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:17.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:17.589+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:17.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:17.589+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:17.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:17.695+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:17.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:17.695+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:17.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:17 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:17.695+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:18.288 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:18 smithi184 podman[73219]: 2024-05-09 08:48:18.02205812 +0000 UTC m=+6.182087472 container cleanup f71598b9e4eed38feeda8dbfa31dfe3231ad7dd25f9c301263e05aca9a575ef0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-09T08:48:18.289 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:18 smithi184 bash[73219]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5 2024-05-09T08:48:18.289 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:18 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:18.132+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:18.289 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:18 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:18.132+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:18.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:17 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:17.996+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:18.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:17 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:17.996+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:18.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:17 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:17.885+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:18.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:17 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:17.885+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:18.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:17 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:17.885+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:18.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:18 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:18.393+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:18.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:18 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:18.393+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:18.606 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:18 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:18.393+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:18.607 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:18 smithi184 podman[73238]: 2024-05-09 08:48:18.290156551 +0000 UTC m=+0.678277236 container remove f71598b9e4eed38feeda8dbfa31dfe3231ad7dd25f9c301263e05aca9a575ef0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:48:18.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:18.585+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:18.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:18.585+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:18.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:18.585+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:18.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:18.722+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:18.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:18.722+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:18.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:18 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:18.722+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:19.001 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:18 smithi184 podman[73341]: 2024-05-09 08:48:18.652623302 +0000 UTC m=+0.018694011 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:19.001 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:18 smithi184 podman[73341]: 2024-05-09 08:48:18.774730133 +0000 UTC m=+0.140800826 container create 91e8948b8e5f8e5e51ae52af41044ae85c47e8cc4a96d75099376424be99f7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-09T08:48:19.001 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:19 smithi184 podman[73341]: 2024-05-09 08:48:19.002250503 +0000 UTC m=+0.368321194 container init 91e8948b8e5f8e5e51ae52af41044ae85c47e8cc4a96d75099376424be99f7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:48:19.300 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:19 smithi184 podman[73341]: 2024-05-09 08:48:19.005640555 +0000 UTC m=+0.371711255 container start 91e8948b8e5f8e5e51ae52af41044ae85c47e8cc4a96d75099376424be99f7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-09T08:48:19.301 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:19 smithi184 podman[73341]: 2024-05-09 08:48:19.110712928 +0000 UTC m=+0.476783630 container attach 91e8948b8e5f8e5e51ae52af41044ae85c47e8cc4a96d75099376424be99f7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5-deactivate, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:48:19.301 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:19 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:19.143+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:19.301 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:19 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:19.143+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:19.350 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:18 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:18.999+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:19.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:18 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:18.999+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:19.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:18 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:18.873+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:19.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:18 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:18.873+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:19.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:18 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:18.873+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:19.604 INFO:journalctl@ceph.osd.5.smithi184.stdout:May 09 08:48:19 smithi184 podman[73341]: 2024-05-09 08:48:19.30239274 +0000 UTC m=+0.668463440 container died 91e8948b8e5f8e5e51ae52af41044ae85c47e8cc4a96d75099376424be99f7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:48:19.604 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:19 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:19.424+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:19.604 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:19 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:19.424+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:19.605 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:19 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:19.424+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:19.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.5.service' 2024-05-09T08:48:19.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:19 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:19.617+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:19.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:19 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:19.617+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:19.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:19 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:19.617+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:19.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:19 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:19.705+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:19.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:19 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:19.705+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:19.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:19 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:19.705+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:19.995 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:48:19.995 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-09T08:48:19.995 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-09T08:48:19.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.6 2024-05-09T08:48:20.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:19 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:19.981+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:20.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:19 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:19.981+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:20.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:19 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:19.832+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:20.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:19 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:19.832+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:20.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:19 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:19.832+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:20.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:20 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:20.145+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:20.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:20 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:20.145+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:20.364 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:20 smithi028 systemd[1]: Stopping Ceph osd.6 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:48:20.681 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:20.612+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:20.682 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:20.612+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:20.682 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:20.612+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:20.682 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:20.365+0000 7f9cb7dc7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:48:20.682 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:20.365+0000 7f9cb7dc7700 -1 osd.6 143 *** Got signal Terminated *** 2024-05-09T08:48:20.682 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:20.365+0000 7f9cb7dc7700 -1 osd.6 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:48:20.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:20 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:20.410+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:20.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:20 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:20.410+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:20.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:20 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:20.410+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:21.182 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:20.687+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:21.182 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:20.687+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:21.182 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:20.687+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:21.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:21 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:21.022+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:21.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:21 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:21.022+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:21.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:20 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:20.853+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:21.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:20 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:20.853+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:21.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:20 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:20.853+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:21.444 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:21 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:21.115+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:21.444 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:21 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:21.115+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:21.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:21 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:21.444+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:21.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:21 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:21.444+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:21.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:21 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:21.444+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:21.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:21 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:21.444+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:21.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:21.628+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:21.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:21.628+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:21.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:21.628+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:21.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:21.728+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:21.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:21.728+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:21.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:21 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:21.728+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:22.350 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:22.023+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:22.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:22.023+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:22.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:22.023+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:22.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:21 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:21.853+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:22.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:21 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:21.853+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:22.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:21 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:21.853+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:22.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:21 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:21.853+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:22.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:22 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:22.090+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:22.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:22 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:22.090+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:22.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:22 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:22.090+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:22.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:22 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:22.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:22.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:22 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:22.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:22.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:22 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:22.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:22.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:22 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:22.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:22.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:22.638+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:22.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:22.638+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:22.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:22.638+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:22.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:22.688+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:22.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:22.688+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:22.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:22 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:22.688+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:23.350 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:22.980+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:23.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:22.980+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:23.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:22.980+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:23.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:22.852+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:23.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:22.852+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:23.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:22.852+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:23.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:22 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:22.852+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:23.505 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:23 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:23.123+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:23.505 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:23 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:23.123+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:23.505 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:23 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:23.123+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:23.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:23 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:23.506+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:23.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:23 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:23.506+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:23.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:23 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:23.506+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:23.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:23 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:23.506+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:23.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:23 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:23.610+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:23.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:23 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:23.610+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:23.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:23 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:23.610+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:23.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:23 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:23.610+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:23.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:23 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:23.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:23.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:23 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:23.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:23.933 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:23 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:23.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:23.933 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:23 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:23.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.017586+0000 front 2024-05-09T08:47:58.017622+0000 (oldest deadline 2024-05-09T08:48:22.717731+0000) 2024-05-09T08:48:24.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:23 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:23.814+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:24.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:23 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:23.814+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:24.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:23 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:23.815+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:24.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:23 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:23.815+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:24.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:24 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:24.021+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:24.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:24 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:24.021+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:24.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:24 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:24.021+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:24.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:24 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:24.082+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:24.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:24 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:24.082+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:24.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:24 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:24.082+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:24.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:24 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:24.506+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:24.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:24 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:24.506+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:24.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:24 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:24.506+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:24.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:24 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:24.506+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:24.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:24.579+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:24.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:24.579+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:24.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:24.579+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:24.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:24.579+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:24.933 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:24.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.009903+0000 front 2024-05-09T08:47:34.009937+0000 (oldest deadline 2024-05-09T08:47:58.010942+0000) 2024-05-09T08:48:24.933 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:24.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.712091+0000 front 2024-05-09T08:47:42.712321+0000 (oldest deadline 2024-05-09T08:48:03.712766+0000) 2024-05-09T08:48:24.933 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:24.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.516205+0000 front 2024-05-09T08:47:50.515893+0000 (oldest deadline 2024-05-09T08:48:13.415949+0000) 2024-05-09T08:48:24.933 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:24 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6[55626]: 2024-05-09T08:48:24.714+0000 7f9cb43bb700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.017586+0000 front 2024-05-09T08:47:58.017622+0000 (oldest deadline 2024-05-09T08:48:22.717731+0000) 2024-05-09T08:48:25.100 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:24 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:24.813+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:25.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:24 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:24.813+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:25.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:24 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:24.813+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:25.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:24 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:24.813+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:25.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:25 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:25.063+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:25.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:25 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:25.063+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:25.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:25 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:25.063+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:25.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:25 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:25.053+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:25.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:25 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:25.053+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:25.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:25 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:25.053+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:25.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:25 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:25.529+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:25.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:25 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:25.529+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:25.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:25 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:25.529+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:25.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:25 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:25.529+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:25.918 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:25 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:25.566+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:25.918 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:25 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:25.566+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:25.918 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:25 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:25.566+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:25.918 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:25 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:25.566+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:26.182 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:25 smithi028 podman[100298]: 2024-05-09 08:48:25.919281632 +0000 UTC m=+5.721942161 container died 37711b5ccc4dffa58ea6943b6b147709e7ddc0e2ebf99212b9407cad2c48a0d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-09T08:48:26.350 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:25 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:25.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:26.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:25 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:25.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:26.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:25 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:25.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:26.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:25 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:25.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:26.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:26.014+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:26.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:26.014+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:26.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:26.014+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:26.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:26 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:26.009+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:26.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:26 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:26.009+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:26.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:26 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:26.009+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:26.546 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:26 smithi028 podman[100298]: 2024-05-09 08:48:26.289787184 +0000 UTC m=+6.092447771 container cleanup 37711b5ccc4dffa58ea6943b6b147709e7ddc0e2ebf99212b9407cad2c48a0d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-09T08:48:26.547 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:26 smithi028 bash[100298]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6 2024-05-09T08:48:26.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:26 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:26.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:26.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:26 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:26.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:26.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:26 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:26.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:26.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:26 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:26.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:26.932 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:26 smithi028 podman[100314]: 2024-05-09 08:48:26.624859206 +0000 UTC m=+0.687083916 container remove 37711b5ccc4dffa58ea6943b6b147709e7ddc0e2ebf99212b9407cad2c48a0d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-09T08:48:26.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:26 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:26.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:26.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:26 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:26.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:26.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:26 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:26.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:26.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:26 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:26.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:27.285 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:26 smithi028 podman[100406]: 2024-05-09 08:48:26.864104752 +0000 UTC m=+0.018338836 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:27.285 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:26 smithi028 podman[100406]: 2024-05-09 08:48:26.96734982 +0000 UTC m=+0.121583917 container create 7d6dc49d8533cedebf3945046aec4c4aac517997c4710cf644d905cbd724e5ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:48:27.285 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:27 smithi028 podman[100406]: 2024-05-09 08:48:27.186491849 +0000 UTC m=+0.340725901 container init 7d6dc49d8533cedebf3945046aec4c4aac517997c4710cf644d905cbd724e5ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240507, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux ) 2024-05-09T08:48:27.285 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:27 smithi028 podman[100406]: 2024-05-09 08:48:27.190479511 +0000 UTC m=+0.344713568 container start 7d6dc49d8533cedebf3945046aec4c4aac517997c4710cf644d905cbd724e5ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-09T08:48:27.285 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:27 smithi028 podman[100406]: 2024-05-09 08:48:27.286495251 +0000 UTC m=+0.440729315 container attach 7d6dc49d8533cedebf3945046aec4c4aac517997c4710cf644d905cbd724e5ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-09T08:48:27.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:26.973+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:27.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:26.973+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:27.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:26.973+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:27.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:26.864+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:27.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:26.864+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:27.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:26.864+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:27.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:26 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:26.864+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:27.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:27 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:27.042+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:27.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:27 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:27.042+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:27.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:27 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:27.042+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:27.576 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 09 08:48:27 smithi028 podman[100406]: 2024-05-09 08:48:27.499004706 +0000 UTC m=+0.653238768 container died 7d6dc49d8533cedebf3945046aec4c4aac517997c4710cf644d905cbd724e5ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:48:27.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:27 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:27.519+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:27.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:27 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:27.519+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:27.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:27 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:27.519+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:27.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:27 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:27.519+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:27.931 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:27.576+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:27.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:27.576+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:27.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:27.576+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:27.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:27 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:27.576+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:28.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.6.service' 2024-05-09T08:48:28.133 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:48:28.133 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-09T08:48:28.133 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-09T08:48:28.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.7 2024-05-09T08:48:28.157 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:27 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:27.886+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:28.157 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:27 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:27.886+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:28.157 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:27 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:27.886+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:28.157 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:27 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:27.886+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:28.158 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:27 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:27.979+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:28.158 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:27 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:27.979+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:28.158 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:27 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:27.979+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:28.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:28 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:28.050+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:28.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:28 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:28.050+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:28.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:28 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:28.050+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:28.413 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:28 smithi165 systemd[1]: Stopping Ceph osd.7 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:48:28.850 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:28.413+0000 7f162266f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:48:28.851 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:28.413+0000 7f162266f700 -1 osd.7 143 *** Got signal Terminated *** 2024-05-09T08:48:28.851 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:28.413+0000 7f162266f700 -1 osd.7 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:48:28.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:28 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:28.495+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:28.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:28 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:28.495+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:28.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:28 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:28.495+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:28.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:28 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:28.495+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:28.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:28 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:28.567+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:28.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:28 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:28.567+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:28.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:28 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:28.567+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:28.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:28 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:28.567+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:29.350 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:28.960+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:29.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:28.960+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:29.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:28.960+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:29.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:28.887+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:29.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:28.887+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:29.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:28.887+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:29.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:28.887+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:29.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:29.020+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:29.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:29.020+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:29.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:29.020+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:29.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:29.524+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:29.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:29.524+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:29.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:29.524+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:29.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:29.524+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:29.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:29 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:29.529+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:29.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:29 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:29.529+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:29.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:29 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:29.529+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:29.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:29 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:29.529+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:30.350 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:29 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:29.939+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:30.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:29 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:29.939+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:30.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:29 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:29.939+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:30.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:29 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:29.862+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:30.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:29 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:29.862+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:30.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:29 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:29.862+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:30.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:29 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:29.862+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:30.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:29.994+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:30.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:29.994+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:30.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:29 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:29.994+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:30.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:30 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:30.514+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:30.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:30 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:30.514+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:30.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:30 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:30.514+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:30.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:30 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:30.514+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:30.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:30 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:30.574+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:30.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:30 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:30.574+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:30.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:30 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:30.574+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:30.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:30 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:30.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:31.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:30 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:30.981+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:31.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:30 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:30.981+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:31.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:30 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:30.982+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:31.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:30 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:30.872+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:31.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:30 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:30.872+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:31.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:30 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:30.872+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:31.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:30 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:30.872+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:31.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:30 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:30.968+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:31.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:30 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:30.968+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:31.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:30 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:30.968+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:31.859 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:31 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:31.512+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:31.859 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:31 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:31.512+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:31.859 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:31 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:31.513+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:31.859 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:31 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:31.513+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:31.859 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:31 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:31.513+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:31.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:31 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:31.610+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:31.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:31 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:31.610+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:31.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:31 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:31.610+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:31.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:31 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:31.610+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:32.100 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:32.023+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:32.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:32.023+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:32.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:32.023+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:32.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:31 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:31.838+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:32.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:31 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:31.838+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:32.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:31 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:31.838+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:32.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:31 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:31.838+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:32.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:31 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:31.938+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:32.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:31 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:31.938+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:32.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:31 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:31.938+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:32.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:31 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:31.938+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:32.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:32 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:32.483+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:32.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:32 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:32.483+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:32.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:32 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:32.483+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:32.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:32 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:32.483+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:32.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:32 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:32.483+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:32.931 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:32 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:32.625+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:32.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:32 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:32.625+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:32.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:32 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:32.625+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:32.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:32 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:32.625+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:33.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:33 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:33.032+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:42.924568+0000 front 2024-05-09T08:47:42.924635+0000 (oldest deadline 2024-05-09T08:48:08.125458+0000) 2024-05-09T08:48:33.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:33 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:33.032+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:48.125912+0000 front 2024-05-09T08:47:48.125932+0000 (oldest deadline 2024-05-09T08:48:12.226115+0000) 2024-05-09T08:48:33.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:33 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7[52350]: 2024-05-09T08:48:33.032+0000 7f161ec63700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:58.127572+0000 front 2024-05-09T08:47:58.127464+0000 (oldest deadline 2024-05-09T08:48:21.627564+0000) 2024-05-09T08:48:33.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:32.796+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:33.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:32.796+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:33.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:32.796+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:33.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:32.796+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:33.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:32.796+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:33.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:32 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:32.940+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:33.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:32 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:32.940+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:33.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:32 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:32.940+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:33.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:32 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:32.940+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:33.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:33 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:33.524+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:33.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:33 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:33.524+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:33.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:33 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:33.524+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:33.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:33 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:33.524+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:33.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:33 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:33.524+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:33.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:33 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:33.576+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:33.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:33 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:33.576+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:33.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:33 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:33.576+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:33.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:33 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:33.576+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:34.085 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:33 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:33.749+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:34.085 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:33 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:33.749+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:34.085 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:33 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:33.749+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:34.085 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:33 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:33.749+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:34.085 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:33 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:33.749+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:34.350 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:34 smithi165 podman[73543]: 2024-05-09 08:48:34.08667316 +0000 UTC m=+5.770827737 container died b656176f90c6a8cd933e0dd691dc968d11001705fa96ba4c70731d552b845e0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:48:34.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:33 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:33.900+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:34.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:33 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:33.900+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:34.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:33 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:33.900+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:34.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:33 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:33.900+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:34.735 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:34 smithi165 podman[73543]: 2024-05-09 08:48:34.446887611 +0000 UTC m=+6.131042188 container cleanup b656176f90c6a8cd933e0dd691dc968d11001705fa96ba4c70731d552b845e0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:48:34.735 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:34 smithi165 bash[73543]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7 2024-05-09T08:48:34.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:34 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:34.537+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:34.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:34 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:34.537+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:34.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:34 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:34.537+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:34.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:34 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:34.537+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:34.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:34 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:34.537+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:34.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:34 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:34.613+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:34.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:34 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:34.613+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:34.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:34 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:34.613+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:34.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:34 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:34.613+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:35.049 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:34 smithi165 podman[73560]: 2024-05-09 08:48:34.742171441 +0000 UTC m=+0.650696706 container remove b656176f90c6a8cd933e0dd691dc968d11001705fa96ba4c70731d552b845e0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-09T08:48:35.050 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:34 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:34.735+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:35.050 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:34 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:34.735+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:35.050 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:34 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:34.735+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:35.050 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:34 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:34.735+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:35.050 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:34 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:34.735+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:35.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:35 smithi165 podman[73647]: 2024-05-09 08:48:35.051378511 +0000 UTC m=+0.108594127 container create 13c94b97ded331485f7919f5fcb130dfb6d9639fe383772ef898f7edc7dc0f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:48:35.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:35 smithi165 podman[73647]: 2024-05-09 08:48:34.960622749 +0000 UTC m=+0.017838378 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:35.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:35 smithi165 podman[73647]: 2024-05-09 08:48:35.288143511 +0000 UTC m=+0.345359130 container init 13c94b97ded331485f7919f5fcb130dfb6d9639fe383772ef898f7edc7dc0f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507) 2024-05-09T08:48:35.351 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:35 smithi165 podman[73647]: 2024-05-09 08:48:35.291262579 +0000 UTC m=+0.348478196 container start 13c94b97ded331485f7919f5fcb130dfb6d9639fe383772ef898f7edc7dc0f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-09T08:48:35.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:34 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:34.943+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:35.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:34 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:34.943+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:35.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:34 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:34.943+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:35.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:34 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:34.943+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:35.772 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:35 smithi165 podman[73647]: 2024-05-09 08:48:35.395856225 +0000 UTC m=+0.453071863 container attach 13c94b97ded331485f7919f5fcb130dfb6d9639fe383772ef898f7edc7dc0f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240507, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc) 2024-05-09T08:48:35.772 INFO:journalctl@ceph.osd.7.smithi165.stdout:May 09 08:48:35 smithi165 podman[73647]: 2024-05-09 08:48:35.578466138 +0000 UTC m=+0.635681755 container died 13c94b97ded331485f7919f5fcb130dfb6d9639fe383772ef898f7edc7dc0f21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-09T08:48:35.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:35 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:35.517+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:35.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:35 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:35.517+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:35.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:35 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:35.517+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:35.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:35 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:35.517+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:35.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:35 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:35.517+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:35.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:35 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:35.608+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:35.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:35 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:35.608+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:35.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:35 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:35.608+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:35.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:35 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:35.608+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:35.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:35 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:35.608+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:36.100 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:35 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:35.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:36.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:35 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:35.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:36.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:35 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:35.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:36.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:35 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:35.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:36.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:35 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:35.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:36.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.7.service' 2024-05-09T08:48:36.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:35 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:35.944+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:36.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:35 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:35.944+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:36.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:35 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:35.944+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:36.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:35 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:35.944+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:36.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:48:36.363 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-09T08:48:36.363 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-09T08:48:36.363 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.8 2024-05-09T08:48:36.683 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:36 smithi184 systemd[1]: Stopping Ceph osd.8 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:48:36.683 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:36.467+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:36.683 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:36.467+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:36.683 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:36.467+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:36.683 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:36.467+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:36.684 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:36.467+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:36.684 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:36.684+0000 7f4748b6d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:48:36.684 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:36.684+0000 7f4748b6d700 -1 osd.8 143 *** Got signal Terminated *** 2024-05-09T08:48:36.684 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:36.684+0000 7f4748b6d700 -1 osd.8 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:48:36.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:36 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:36.574+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:36.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:36 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:36.574+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:36.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:36 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:36.574+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:36.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:36 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:36.574+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:36.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:36 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:36.574+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:37.100 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:36 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:36.804+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:37.100 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:36 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:36.804+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:37.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:36 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:36.804+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:37.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:36 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:36.804+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:37.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:36 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:36.804+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:37.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:36.957+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:37.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:36.957+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:37.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:36.957+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:37.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:36 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:36.957+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:37.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:37 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:37.477+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:37.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:37 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:37.477+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:37.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:37 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:37.477+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:37.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:37 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:37.477+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:37.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:37 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:37.477+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:37.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:37 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:37.543+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:37.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:37 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:37.543+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:37.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:37 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:37.543+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:37.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:37 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:37.543+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:37.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:37 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:37.543+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:38.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:37 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:37.813+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:38.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:37 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:37.813+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:38.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:37 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:37.813+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:38.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:37 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:37.813+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:38.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:37 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:37.813+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:38.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:37 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:37.953+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:38.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:37 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:37.953+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:38.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:37 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:37.953+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:38.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:37 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:37.953+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:38.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:38.465+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:38.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:38.465+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:38.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:38.465+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:38.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:38.465+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:38.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:38.465+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:38.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:38 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:38.512+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:38.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:38 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:38.512+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:38.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:38 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:38.512+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:38.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:38 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:38.512+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:38.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:38 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:38.512+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:38.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:38 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:38.512+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:39.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:38 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:38.763+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:39.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:38 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:38.763+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:39.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:38 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:38.763+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:39.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:38 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:38.763+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:39.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:38 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:38.763+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:39.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:38 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:38.763+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:39.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:38.999+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:39.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:38.999+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:39.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:38.999+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:39.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:38.999+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:39.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:38 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:38.999+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:39.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:39.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:39.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:39.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:39.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:39.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:39.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:39.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:39.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:39.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:39.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:39.490+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.953592+0000 front 2024-05-09T08:48:12.953567+0000 (oldest deadline 2024-05-09T08:48:38.853668+0000) 2024-05-09T08:48:39.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:39 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:39.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:39.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:39 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:39.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:39.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:39 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:39.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:39.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:39 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:39.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:39.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:39 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:39.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:39.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:39 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:39.546+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:40.100 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:39 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:39.762+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:40.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:39 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:39.762+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:40.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:39 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:39.762+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:40.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:39 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:39.762+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:40.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:39 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:39.762+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:40.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:39 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:39.762+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:40.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:39.996+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:40.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:39.996+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:40.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:39.996+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:40.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:39.996+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:40.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:39 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:39.996+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:40.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:40 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:40.448+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:40.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:40 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:40.448+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:40.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:40 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:40.448+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:40.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:40 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:40.448+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:40.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:40 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:40.448+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:40.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:40 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:40.448+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.953592+0000 front 2024-05-09T08:48:12.953567+0000 (oldest deadline 2024-05-09T08:48:38.853668+0000) 2024-05-09T08:48:40.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:40 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:40.557+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:40.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:40 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:40.557+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:40.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:40 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:40.557+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:40.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:40 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:40.557+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:40.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:40 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:40.557+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:40.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:40 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:40.557+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:41.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:40 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:40.737+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:41.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:40 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:40.737+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:41.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:40 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:40.737+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:41.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:40 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:40.737+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:41.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:40 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:40.737+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:41.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:40 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:40.737+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:41.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.024+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:41.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.024+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:41.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.024+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:41.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.024+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:41.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.024+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:41.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:41.399+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:33.446005+0000 front 2024-05-09T08:47:33.446164+0000 (oldest deadline 2024-05-09T08:47:56.945949+0000) 2024-05-09T08:48:41.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:41.399+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:40.948025+0000 front 2024-05-09T08:47:40.948045+0000 (oldest deadline 2024-05-09T08:48:06.247966+0000) 2024-05-09T08:48:41.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:41.399+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:50.349220+0000 front 2024-05-09T08:47:50.349244+0000 (oldest deadline 2024-05-09T08:48:16.149738+0000) 2024-05-09T08:48:41.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:41.399+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.850686+0000 front 2024-05-09T08:47:57.850800+0000 (oldest deadline 2024-05-09T08:48:20.750778+0000) 2024-05-09T08:48:41.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:41.399+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.352700+0000 front 2024-05-09T08:48:08.352841+0000 (oldest deadline 2024-05-09T08:48:30.652780+0000) 2024-05-09T08:48:41.857 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8[53233]: 2024-05-09T08:48:41.399+0000 7f4745161700 -1 osd.8 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.953592+0000 front 2024-05-09T08:48:12.953567+0000 (oldest deadline 2024-05-09T08:48:38.853668+0000) 2024-05-09T08:48:41.931 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:41 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:41.560+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:41.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:41 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:41.560+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:41.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:41 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:41.560+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:41.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:41 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:41.560+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:41.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:41 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:41.560+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:41.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:41 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:41.560+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:42.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:41 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:41.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:42.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:41 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:41.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:42.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:41 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:41.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:42.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:41 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:41.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:42.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:41 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:41.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:42.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:41 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:41.772+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:42.356 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:42 smithi184 podman[73448]: 2024-05-09 08:48:42.235873284 +0000 UTC m=+5.670682950 container died 03adce19dc4c97ed7018f32e94cea73289d13c6a54977ede0cd22b0de878db41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240507) 2024-05-09T08:48:42.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.994+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:42.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.994+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:42.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.994+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:42.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.994+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:42.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:41 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:41.994+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:42.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:42 smithi184 podman[73448]: 2024-05-09 08:48:42.608747614 +0000 UTC m=+6.043557289 container cleanup 03adce19dc4c97ed7018f32e94cea73289d13c6a54977ede0cd22b0de878db41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-09T08:48:42.856 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:42 smithi184 bash[73448]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8 2024-05-09T08:48:42.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:42.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:42.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:42.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:42.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:42.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:42.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:42.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:42.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:42.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:42.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:42 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:42.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:43.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:42 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:42.785+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:43.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:42 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:42.785+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:43.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:42 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:42.785+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:43.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:42 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:42.785+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:43.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:42 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:42.785+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:43.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:42 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:42.785+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:43.159 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:42 smithi184 podman[73465]: 2024-05-09 08:48:42.87664685 +0000 UTC m=+0.636931371 container remove 03adce19dc4c97ed7018f32e94cea73289d13c6a54977ede0cd22b0de878db41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-09T08:48:43.159 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:43 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:43.009+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:43.159 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:43 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:43.009+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:43.159 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:43 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:43.009+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:43.160 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:43 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:43.009+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:43.160 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:43 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:43.009+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:43.412 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:43 smithi184 podman[73559]: 2024-05-09 08:48:43.160869088 +0000 UTC m=+0.101534560 container create 150b5861dbb90ef61e10494054459bf6b0c89621243ad2071b3e0291300cac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:48:43.412 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:43 smithi184 podman[73559]: 2024-05-09 08:48:43.077234328 +0000 UTC m=+0.017899748 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:43.699 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:43 smithi184 podman[73559]: 2024-05-09 08:48:43.413680532 +0000 UTC m=+0.354345946 container init 150b5861dbb90ef61e10494054459bf6b0c89621243ad2071b3e0291300cac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8-deactivate, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-09T08:48:43.699 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:43 smithi184 podman[73559]: 2024-05-09 08:48:43.417247745 +0000 UTC m=+0.357913150 container start 150b5861dbb90ef61e10494054459bf6b0c89621243ad2071b3e0291300cac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-09T08:48:43.699 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:43 smithi184 podman[73559]: 2024-05-09 08:48:43.505177705 +0000 UTC m=+0.445843109 container attach 150b5861dbb90ef61e10494054459bf6b0c89621243ad2071b3e0291300cac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-09T08:48:43.933 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:43 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:43.540+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:43.934 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:43 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:43.540+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:43.934 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:43 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:43.540+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:43.934 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:43 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:43.540+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:43.934 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:43 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:43.540+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:43.934 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:43 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:43.540+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:44.017 INFO:journalctl@ceph.osd.8.smithi184.stdout:May 09 08:48:43 smithi184 podman[73559]: 2024-05-09 08:48:43.701253813 +0000 UTC m=+0.641919235 container died 150b5861dbb90ef61e10494054459bf6b0c89621243ad2071b3e0291300cac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-09T08:48:44.100 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:43 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:43.831+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:44.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:43 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:43.831+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:44.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:43 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:43.831+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:44.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:43 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:43.831+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:44.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:43 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:43.831+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:44.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:43 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:43.831+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:44.267 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:44 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:44.055+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:44.268 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:44 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:44.055+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:44.268 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:44 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:44.055+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:44.268 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:44 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:44.055+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:44.268 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:44 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:44.055+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:44.286 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.8.service' 2024-05-09T08:48:44.323 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:48:44.323 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-09T08:48:44.324 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-09T08:48:44.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.9 2024-05-09T08:48:44.682 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 systemd[1]: Stopping Ceph osd.9 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:48:44.683 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:44.588+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:44.683 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:44.588+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:44.683 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:44.588+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:44.683 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:44.588+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:44.683 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:44.588+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:44.683 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:44.588+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:45.100 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:44 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:44.792+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:45.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:44 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:44.792+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:45.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:44 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:44.792+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:45.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:44 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:44.792+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:45.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:44 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:44.792+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:45.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:44 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:44.792+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:45.182 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:44.784+0000 7f2f66dbf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:48:45.182 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:44.784+0000 7f2f66dbf700 -1 osd.9 143 *** Got signal Terminated *** 2024-05-09T08:48:45.182 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:44 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:44.784+0000 7f2f66dbf700 -1 osd.9 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:48:45.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.010+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:45.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.010+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:45.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.010+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:45.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.010+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:45.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.010+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:45.931 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:45.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:45.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:45.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:45.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:45.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:45.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:45.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:45.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:45.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:45.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:45 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:45.575+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:46.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:45 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:45.766+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:46.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:45 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:45.766+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:46.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:45 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:45.766+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:46.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:45 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:45.766+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:46.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:45 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:45.766+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:46.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:45 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:45.766+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:46.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.982+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:46.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.982+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:46.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.982+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:46.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.982+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:46.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:45 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:45.982+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:46.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:46 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:46.536+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:46.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:46 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:46.536+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:46.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:46 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:46.536+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:46.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:46 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:46.536+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:46.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:46 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:46.536+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:46.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:46 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:46.536+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:47.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:46 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:46.754+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:47.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:46 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:46.754+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:47.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:46 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:46.754+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:47.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:46 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:46.754+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:47.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:46 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:46.754+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:47.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:46 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:46.754+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:47.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:46 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:46.967+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:47.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:46 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:46.967+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:47.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:46 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:46.967+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:47.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:46 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:46.967+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:47.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:46 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:46.967+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:47.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:47 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:47.537+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:47.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:47 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:47.537+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:47.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:47 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:47.537+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:47.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:47 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:47.537+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:47.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:47 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:47.537+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:47.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:47 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:47.537+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:48.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:47 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:47.793+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:48.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:47 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:47.793+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:48.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:47 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:47.793+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:48.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:47 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:47.793+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:48.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:47 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:47.793+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:48.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:47 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:47.793+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:48.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:47 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:47.958+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:48.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:47 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:47.958+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:48.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:47 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:47.958+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:48.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:47 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:47.958+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:48.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:47 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:47.958+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:48.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:47 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:47.958+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:48.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:48.545+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:48.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:48.545+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:48.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:48.545+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:48.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:48.545+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:48.933 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:48.545+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:48.933 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:48 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:48.545+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:49.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:48 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:48.805+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:49.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:48 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:48.805+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:49.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:48 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:48.805+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:49.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:48 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:48.805+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:49.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:48 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:48.805+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:49.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:48 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:48.805+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:49.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:48 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:48.805+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:49.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:48 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:48.912+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:49.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:48 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:48.912+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:49.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:48 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:48.912+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:49.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:48 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:48.912+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:49.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:48 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:48.912+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:49.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:48 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:48.912+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:49.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:49 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:49.533+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:31.738899+0000 front 2024-05-09T08:47:31.738919+0000 (oldest deadline 2024-05-09T08:47:59.939793+0000) 2024-05-09T08:48:49.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:49 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:49.533+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:41.640972+0000 front 2024-05-09T08:47:41.640944+0000 (oldest deadline 2024-05-09T08:48:06.340778+0000) 2024-05-09T08:48:49.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:49 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:49.533+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.242415+0000 front 2024-05-09T08:47:49.242203+0000 (oldest deadline 2024-05-09T08:48:14.541996+0000) 2024-05-09T08:48:49.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:49 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:49.533+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:57.343834+0000 front 2024-05-09T08:47:57.343869+0000 (oldest deadline 2024-05-09T08:48:22.643926+0000) 2024-05-09T08:48:49.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:49 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:49.533+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.545226+0000 front 2024-05-09T08:48:08.545096+0000 (oldest deadline 2024-05-09T08:48:34.945796+0000) 2024-05-09T08:48:49.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:49 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9[62229]: 2024-05-09T08:48:49.533+0000 7f2f5f3ab700 -1 osd.9 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.946334+0000 front 2024-05-09T08:48:14.946361+0000 (oldest deadline 2024-05-09T08:48:38.346860+0000) 2024-05-09T08:48:50.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:49 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:49.810+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:50.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:49 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:49.810+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:50.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:49 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:49.810+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:50.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:49 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:49.810+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:50.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:49 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:49.810+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:50.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:49 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:49.810+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:50.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:49 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:49.810+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:50.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:49 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:49.885+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:50.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:49 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:49.885+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:50.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:49 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:49.885+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:50.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:49 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:49.885+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:50.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:49 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:49.885+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:50.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:49 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:49.885+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:50.682 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:50 smithi028 podman[100522]: 2024-05-09 08:48:50.392564978 +0000 UTC m=+5.861195274 container died b619ad9c4073b8fbb039c03cccf9ab030c44c2b12e5879e6727395ef14e4d345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-09T08:48:51.143 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:50 smithi028 podman[100522]: 2024-05-09 08:48:50.834348514 +0000 UTC m=+6.302978815 container cleanup b619ad9c4073b8fbb039c03cccf9ab030c44c2b12e5879e6727395ef14e4d345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc) 2024-05-09T08:48:51.143 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:50 smithi028 bash[100522]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9 2024-05-09T08:48:51.350 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:50 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:50.854+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:51.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:50 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:50.854+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:51.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:50 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:50.854+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:51.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:50 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:50.854+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:51.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:50 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:50.854+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:51.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:50 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:50.854+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:51.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:50 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:50.854+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:51.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:50 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:50.932+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:51.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:50 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:50.932+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:51.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:50 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:50.932+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:51.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:50 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:50.932+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:51.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:50 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:50.932+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:51.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:50 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:50.932+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:51.431 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:51 smithi028 podman[100539]: 2024-05-09 08:48:51.14418059 +0000 UTC m=+0.744220902 container remove b619ad9c4073b8fbb039c03cccf9ab030c44c2b12e5879e6727395ef14e4d345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:48:51.929 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:51 smithi028 podman[100630]: 2024-05-09 08:48:51.603338189 +0000 UTC m=+0.109485601 container create 95984af6a4e3f021029bc071988c18d7a10ede88048a0e625e18cd1406c833aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:48:51.929 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:51 smithi028 podman[100630]: 2024-05-09 08:48:51.511954137 +0000 UTC m=+0.018101562 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:51.929 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:51 smithi028 podman[100630]: 2024-05-09 08:48:51.930631542 +0000 UTC m=+0.436778952 container init 95984af6a4e3f021029bc071988c18d7a10ede88048a0e625e18cd1406c833aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:48:52.100 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:51 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:51.817+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:52.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:51 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:51.817+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:52.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:51 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:51.817+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:52.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:51 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:51.817+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:52.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:51 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:51.817+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:52.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:51 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:51.817+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:52.101 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:51 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:51.817+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:52.181 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:51 smithi028 podman[100630]: 2024-05-09 08:48:51.934510085 +0000 UTC m=+0.440657496 container start 95984af6a4e3f021029bc071988c18d7a10ede88048a0e625e18cd1406c833aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, RELEASE=squid-96805dc) 2024-05-09T08:48:52.182 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:52 smithi028 podman[100630]: 2024-05-09 08:48:52.013850188 +0000 UTC m=+0.519997607 container attach 95984af6a4e3f021029bc071988c18d7a10ede88048a0e625e18cd1406c833aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-09T08:48:52.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:51 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:51.898+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:52.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:51 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:51.898+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:52.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:51 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:51.898+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:52.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:51 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:51.898+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:52.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:51 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:51.898+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:52.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:51 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:51.898+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:52.558 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:52 smithi028 podman[100630]: 2024-05-09 08:48:52.256093394 +0000 UTC m=+0.762240802 container died 95984af6a4e3f021029bc071988c18d7a10ede88048a0e625e18cd1406c833aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:48:52.810 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 09 08:48:52 smithi028 podman[100657]: 2024-05-09 08:48:52.808856947 +0000 UTC m=+0.546666252 container remove 95984af6a4e3f021029bc071988c18d7a10ede88048a0e625e18cd1406c833aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-9-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240507) 2024-05-09T08:48:52.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.9.service' 2024-05-09T08:48:52.875 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:48:52.875 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-09T08:48:52.875 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-09T08:48:52.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.10 2024-05-09T08:48:53.176 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:52 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:52.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:52 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:52.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:52 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:52.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:52 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:52.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:52 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:52.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:52 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:52.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:52 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:52.860+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:52 smithi165 systemd[1]: Stopping Ceph osd.10 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.176+0000 7f95fd540700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.176+0000 7f95fd540700 -1 osd.10 143 *** Got signal Terminated *** 2024-05-09T08:48:53.177 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.176+0000 7f95fd540700 -1 osd.10 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:48:53.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:52 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:52.937+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:53.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:52 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:52.937+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:53.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:52 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:52.937+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:53.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:52 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:52.937+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:53.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:52 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:52.937+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:53.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:52 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:52.937+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:54.350 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.855+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:54.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.855+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:54.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.855+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:54.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.855+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:54.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.855+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:54.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.855+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:54.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:53 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:53.855+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:54.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:53 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:53.899+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:54.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:53 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:53.899+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:54.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:53 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:53.899+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:54.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:53 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:53.899+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:54.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:53 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:53.899+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:54.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:53 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:53.899+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:55.350 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:54 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:54.888+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:55.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:54 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:54.888+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:55.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:54 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:54.888+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:55.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:54 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:54.888+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:55.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:54 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:54.888+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:55.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:54 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:54.888+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:55.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:54 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:54.888+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:55.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:54 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:54.918+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:55.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:54 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:54.918+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:55.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:54 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:54.918+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:55.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:54 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:54.918+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:55.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:54 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:54.918+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:55.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:54 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:54.918+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:55.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:54 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:54.918+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:48:56.350 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:55 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:55.920+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:56.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:55 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:55.920+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:56.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:55 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:55.920+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:56.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:55 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:55.920+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:56.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:55 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:55.920+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:56.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:55 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:55.920+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:56.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:55 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:55.920+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:56.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:55 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:55.935+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:56.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:55 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:55.935+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:56.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:55 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:55.935+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:56.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:55 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:55.935+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:56.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:55 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:55.935+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:56.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:55 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:55.935+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:56.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:55 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:55.935+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:48:57.350 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:56 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:56.913+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:57.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:56 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:56.913+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:57.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:56 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:56.913+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:57.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:56 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:56.913+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:57.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:56 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:56.913+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:57.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:56 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:56.913+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:57.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:56 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:56.913+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:57.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:56 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:56.931+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:57.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:56 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:56.931+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:57.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:56 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:56.931+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:57.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:56 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:56.931+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:57.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:56 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:56.931+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:57.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:56 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:56.931+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:57.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:56 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:56.931+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:48:58.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:57 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:57.926+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:34.368232+0000 front 2024-05-09T08:47:34.368030+0000 (oldest deadline 2024-05-09T08:48:00.267929+0000) 2024-05-09T08:48:58.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:57 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:57.926+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6804 osd.1 since back 2024-05-09T08:47:43.069685+0000 front 2024-05-09T08:47:43.069758+0000 (oldest deadline 2024-05-09T08:48:05.369783+0000) 2024-05-09T08:48:58.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:57 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:57.926+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:49.470902+0000 front 2024-05-09T08:47:49.470771+0000 (oldest deadline 2024-05-09T08:48:14.170720+0000) 2024-05-09T08:48:58.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:57 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:57.926+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:55.271924+0000 front 2024-05-09T08:47:55.272067+0000 (oldest deadline 2024-05-09T08:48:21.171852+0000) 2024-05-09T08:48:58.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:57 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:57.926+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:08.673807+0000 front 2024-05-09T08:48:08.673838+0000 (oldest deadline 2024-05-09T08:48:32.173989+0000) 2024-05-09T08:48:58.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:57 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:57.926+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:12.174772+0000 front 2024-05-09T08:48:12.174634+0000 (oldest deadline 2024-05-09T08:48:37.974833+0000) 2024-05-09T08:48:58.352 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:57 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10[59363]: 2024-05-09T08:48:57.926+0000 7f95f5b2c700 -1 osd.10 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:24.976082+0000 front 2024-05-09T08:48:24.976108+0000 (oldest deadline 2024-05-09T08:48:47.876014+0000) 2024-05-09T08:48:58.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:57 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:57.889+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:58.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:57 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:57.889+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:58.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:57 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:57.889+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:58.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:57 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:57.889+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:58.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:57 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:57.889+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:58.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:57 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:57.889+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:58.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:57 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:57.889+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:48:58.893 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:58 smithi165 podman[73766]: 2024-05-09 08:48:58.618680197 +0000 UTC m=+5.553069119 container died 5ddfc3655ec8c5f074500ef3f42d29a14ddc9580c0f3fce7010058780e41054b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2) 2024-05-09T08:48:59.268 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:58 smithi165 podman[73766]: 2024-05-09 08:48:58.976959511 +0000 UTC m=+5.911348429 container cleanup 5ddfc3655ec8c5f074500ef3f42d29a14ddc9580c0f3fce7010058780e41054b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_BRANCH=HEAD) 2024-05-09T08:48:59.268 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:58 smithi165 bash[73766]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10 2024-05-09T08:48:59.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:58.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:48:59.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:58.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:48:59.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:58.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:48:59.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:58.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:48:59.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:58.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:48:59.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:58.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:48:59.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:58 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:58.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:48:59.591 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:59 smithi165 podman[73786]: 2024-05-09 08:48:59.269855016 +0000 UTC m=+0.644462223 container remove 5ddfc3655ec8c5f074500ef3f42d29a14ddc9580c0f3fce7010058780e41054b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:48:59.851 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:59 smithi165 podman[73881]: 2024-05-09 08:48:59.496359431 +0000 UTC m=+0.018082659 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:48:59.851 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:59 smithi165 podman[73881]: 2024-05-09 08:48:59.595553446 +0000 UTC m=+0.117276716 container create 7c8f6e3864ff0601e22e7df96481eb8f27d9713517159f927290fccc260098e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10-deactivate, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:48:59.851 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:59 smithi165 podman[73881]: 2024-05-09 08:48:59.789812507 +0000 UTC m=+0.311535731 container init 7c8f6e3864ff0601e22e7df96481eb8f27d9713517159f927290fccc260098e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-09T08:48:59.851 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:59 smithi165 podman[73881]: 2024-05-09 08:48:59.792902885 +0000 UTC m=+0.314626117 container start 7c8f6e3864ff0601e22e7df96481eb8f27d9713517159f927290fccc260098e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507, RELEASE=squid-96805dc) 2024-05-09T08:49:00.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:59.836+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:49:00.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:59.836+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:49:00.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:59.836+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:49:00.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:59.836+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:49:00.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:59.836+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:49:00.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:59.836+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:49:00.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:48:59 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:48:59.836+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:49:00.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:48:59 smithi165 podman[73881]: 2024-05-09 08:48:59.868608455 +0000 UTC m=+0.390331696 container attach 7c8f6e3864ff0601e22e7df96481eb8f27d9713517159f927290fccc260098e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:49:00.351 INFO:journalctl@ceph.osd.10.smithi165.stdout:May 09 08:49:00 smithi165 podman[73881]: 2024-05-09 08:49:00.102959241 +0000 UTC m=+0.624682464 container died 7c8f6e3864ff0601e22e7df96481eb8f27d9713517159f927290fccc260098e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-09T08:49:00.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.10.service' 2024-05-09T08:49:00.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:00.696 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-09T08:49:00.696 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-09T08:49:00.696 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.11 2024-05-09T08:49:00.995 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 systemd[1]: Stopping Ceph osd.11 for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:00.995 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.798+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:49:00.996 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.798+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:49:00.996 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.798+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:49:00.996 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.798+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:49:00.996 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.798+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:49:00.996 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.798+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:49:00.996 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.798+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:49:00.996 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.995+0000 7f23f11b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:00.996 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.995+0000 7f23f11b1700 -1 osd.11 143 *** Got signal Terminated *** 2024-05-09T08:49:00.996 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:00 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:00.995+0000 7f23f11b1700 -1 osd.11 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:49:02.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:01.815+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:49:02.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:01.815+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:49:02.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:01.815+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:49:02.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:01.815+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:49:02.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:01.815+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:49:02.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:01.815+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:49:02.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:01 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:01.815+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:49:03.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:02.855+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:49:03.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:02.855+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:49:03.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:02.855+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:49:03.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:02.855+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:49:03.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:02.855+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:49:03.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:02.855+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:49:03.107 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:02 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:02.855+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:49:04.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:03.857+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:49:04.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:03.857+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:49:04.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:03.857+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:49:04.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:03.857+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:49:04.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:03.857+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:49:04.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:03.857+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:49:04.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:03 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:03.857+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:49:05.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:04.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:49:05.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:04.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:49:05.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:04.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:49:05.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:04.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:49:05.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:04.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:49:05.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:04.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:49:05.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:04 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:04.879+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:49:06.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:05.861+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-09T08:47:32.969353+0000 front 2024-05-09T08:47:32.969297+0000 (oldest deadline 2024-05-09T08:47:58.269234+0000) 2024-05-09T08:49:06.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:05.861+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6804 osd.2 since back 2024-05-09T08:47:51.572361+0000 front 2024-05-09T08:47:51.572314+0000 (oldest deadline 2024-05-09T08:48:17.472492+0000) 2024-05-09T08:49:06.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:05.861+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-09T08:47:59.073866+0000 front 2024-05-09T08:47:59.074037+0000 (oldest deadline 2024-05-09T08:48:21.274496+0000) 2024-05-09T08:49:06.356 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:05.861+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6812 osd.4 since back 2024-05-09T08:48:07.175568+0000 front 2024-05-09T08:48:07.175645+0000 (oldest deadline 2024-05-09T08:48:31.275502+0000) 2024-05-09T08:49:06.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:05.861+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6812 osd.5 since back 2024-05-09T08:48:14.176521+0000 front 2024-05-09T08:48:14.176840+0000 (oldest deadline 2024-05-09T08:48:38.276602+0000) 2024-05-09T08:49:06.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:05.861+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-09T08:48:22.377814+0000 front 2024-05-09T08:48:22.377843+0000 (oldest deadline 2024-05-09T08:48:47.077735+0000) 2024-05-09T08:49:06.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:05.861+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.165:6820 osd.7 since back 2024-05-09T08:48:32.279224+0000 front 2024-05-09T08:48:32.279269+0000 (oldest deadline 2024-05-09T08:48:54.579104+0000) 2024-05-09T08:49:06.357 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:05 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11[60096]: 2024-05-09T08:49:05.861+0000 7f23e979d700 -1 osd.11 143 heartbeat_check: no reply from 172.21.15.184:6820 osd.8 since back 2024-05-09T08:48:39.279885+0000 front 2024-05-09T08:48:39.279903+0000 (oldest deadline 2024-05-09T08:49:05.179897+0000) 2024-05-09T08:49:06.846 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:06 smithi184 podman[73676]: 2024-05-09 08:49:06.579763155 +0000 UTC m=+5.687646651 container died 81f4d492a830c150757fb201f04b4804a7394cfcadde9c1587f753a28b65a466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-05-09T08:49:07.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:06 smithi184 podman[73676]: 2024-05-09 08:49:06.945969106 +0000 UTC m=+6.053852602 container cleanup 81f4d492a830c150757fb201f04b4804a7394cfcadde9c1587f753a28b65a466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:49:07.106 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:06 smithi184 bash[73676]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11 2024-05-09T08:49:07.554 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:07 smithi184 podman[73693]: 2024-05-09 08:49:07.247097344 +0000 UTC m=+0.660184488 container remove 81f4d492a830c150757fb201f04b4804a7394cfcadde9c1587f753a28b65a466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_CLEAN=True) 2024-05-09T08:49:07.856 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:07 smithi184 podman[73787]: 2024-05-09 08:49:07.556140597 +0000 UTC m=+0.108966368 container create f46ab9905260db05e252f163f0f1872eeacd867bb7eb13f7ae1b16c8cdc12be1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T08:49:07.857 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:07 smithi184 podman[73787]: 2024-05-09 08:49:07.465583266 +0000 UTC m=+0.018409051 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:49:08.239 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:07 smithi184 podman[73787]: 2024-05-09 08:49:07.950214405 +0000 UTC m=+0.503040176 container init f46ab9905260db05e252f163f0f1872eeacd867bb7eb13f7ae1b16c8cdc12be1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc) 2024-05-09T08:49:08.239 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:07 smithi184 podman[73787]: 2024-05-09 08:49:07.953351766 +0000 UTC m=+0.506177537 container start f46ab9905260db05e252f163f0f1872eeacd867bb7eb13f7ae1b16c8cdc12be1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:49:08.239 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:08 smithi184 podman[73787]: 2024-05-09 08:49:08.050039445 +0000 UTC m=+0.602865217 container attach f46ab9905260db05e252f163f0f1872eeacd867bb7eb13f7ae1b16c8cdc12be1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T08:49:08.240 INFO:journalctl@ceph.osd.11.smithi184.stdout:May 09 08:49:08 smithi184 podman[73787]: 2024-05-09 08:49:08.240596612 +0000 UTC m=+0.793422385 container died f46ab9905260db05e252f163f0f1872eeacd867bb7eb13f7ae1b16c8cdc12be1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240507) 2024-05-09T08:49:08.796 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@osd.11.service' 2024-05-09T08:49:08.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:08.834 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-09T08:49:08.834 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-09T08:49:08.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.a 2024-05-09T08:49:09.182 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:49:08 smithi028 systemd[1]: Stopping Ceph mds.a for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:09.682 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:49:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a[69053]: 2024-05-09T08:49:09.186+0000 7f6ddab14700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:09.682 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:49:09 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a[69053]: 2024-05-09T08:49:09.186+0000 7f6ddab14700 -1 mds.a *** got signal Terminated *** 2024-05-09T08:49:11.511 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:49:11 smithi028 podman[100753]: 2024-05-09 08:49:11.199209069 +0000 UTC m=+2.164625485 container died 9c62fc1c03aded0f1d12e49ff45a5b30126d7d9361b104789c09117eff835439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS) 2024-05-09T08:49:11.852 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:49:11 smithi028 podman[100753]: 2024-05-09 08:49:11.594842329 +0000 UTC m=+2.560258753 container cleanup 9c62fc1c03aded0f1d12e49ff45a5b30126d7d9361b104789c09117eff835439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:49:11.852 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 09 08:49:11 smithi028 bash[100753]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-a 2024-05-09T08:49:11.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.a.service' 2024-05-09T08:49:12.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:12.033 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-09T08:49:12.033 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-09T08:49:12.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.d 2024-05-09T08:49:12.364 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:49:12 smithi028 systemd[1]: Stopping Ceph mds.d for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:12.681 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:49:12 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[91682]: 2024-05-09T08:49:12.364+0000 7f56a2dfe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:12.682 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:49:12 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d[91682]: 2024-05-09T08:49:12.364+0000 7f56a2dfe700 -1 mds.d *** got signal Terminated *** 2024-05-09T08:49:15.033 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:49:14 smithi028 podman[100875]: 2024-05-09 08:49:14.740759233 +0000 UTC m=+2.480645249 container died 6f72f5305c15b165b06aac9c8b8f84d0a7c0f90b7dfc8c7f91c6b5863384d4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-05-09T08:49:15.381 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:49:15 smithi028 podman[100875]: 2024-05-09 08:49:15.131736999 +0000 UTC m=+2.871623022 container cleanup 6f72f5305c15b165b06aac9c8b8f84d0a7c0f90b7dfc8c7f91c6b5863384d4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True) 2024-05-09T08:49:15.381 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 09 08:49:15 smithi028 bash[100875]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-d 2024-05-09T08:49:15.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.d.service' 2024-05-09T08:49:15.542 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:15.542 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-09T08:49:15.542 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-09T08:49:15.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.g 2024-05-09T08:49:15.932 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:49:15 smithi028 systemd[1]: Stopping Ceph mds.g for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:15.932 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:49:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g[68693]: 2024-05-09T08:49:15.877+0000 7f9732521700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:15.932 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:49:15 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g[68693]: 2024-05-09T08:49:15.877+0000 7f9732521700 -1 mds.g *** got signal Terminated *** 2024-05-09T08:49:19.432 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:49:19 smithi028 podman[100998]: 2024-05-09 08:49:19.182236882 +0000 UTC m=+3.404094799 container died a6350a7cbf4a1d273a40f97d2ea3fee247bddf142e24d49b59e9adc8e10127d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-05-09T08:49:19.810 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:49:19 smithi028 podman[100998]: 2024-05-09 08:49:19.544313914 +0000 UTC m=+3.766171827 container cleanup a6350a7cbf4a1d273a40f97d2ea3fee247bddf142e24d49b59e9adc8e10127d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0) 2024-05-09T08:49:19.810 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 09 08:49:19 smithi028 bash[100998]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-g 2024-05-09T08:49:19.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.g.service' 2024-05-09T08:49:20.007 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:20.008 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-09T08:49:20.008 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-09T08:49:20.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.j 2024-05-09T08:49:20.380 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 09 08:49:20 smithi028 systemd[1]: Stopping Ceph mds.j for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:20.682 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 09 08:49:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-j[67742]: 2024-05-09T08:49:20.380+0000 7f06c477a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:20.682 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 09 08:49:20 smithi028 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-j[67742]: 2024-05-09T08:49:20.380+0000 7f06c477a700 -1 mds.j *** got signal Terminated *** 2024-05-09T08:49:23.874 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 09 08:49:23 smithi028 podman[101119]: 2024-05-09 08:49:23.588968269 +0000 UTC m=+3.352907155 container died aa97bd4f2d1f41ca6634141e2a37ee603d9c9c33f59a825a61ba07a5af1c18ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-j, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240507, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc) 2024-05-09T08:49:24.181 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 09 08:49:23 smithi028 podman[101119]: 2024-05-09 08:49:23.964016387 +0000 UTC m=+3.727955281 container cleanup aa97bd4f2d1f41ca6634141e2a37ee603d9c9c33f59a825a61ba07a5af1c18ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:49:24.182 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 09 08:49:23 smithi028 bash[101119]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-j 2024-05-09T08:49:24.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.j.service' 2024-05-09T08:49:24.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:24.444 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-09T08:49:24.444 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-09T08:49:24.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.b 2024-05-09T08:49:24.755 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:49:24 smithi165 systemd[1]: Stopping Ceph mds.b for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:25.100 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:49:24 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b[64615]: 2024-05-09T08:49:24.755+0000 7ff065185700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:25.101 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:49:24 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b[64615]: 2024-05-09T08:49:24.755+0000 7ff065185700 -1 mds.b *** got signal Terminated *** 2024-05-09T08:49:27.405 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:49:27 smithi165 podman[73986]: 2024-05-09 08:49:27.133717105 +0000 UTC m=+2.477319236 container died 55ee3e387e7a6563ba17ff278aac220fc807a1560a8ad123e3c1910f612cbe48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-96805dc) 2024-05-09T08:49:27.780 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:49:27 smithi165 podman[73986]: 2024-05-09 08:49:27.497197553 +0000 UTC m=+2.840799679 container cleanup 55ee3e387e7a6563ba17ff278aac220fc807a1560a8ad123e3c1910f612cbe48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-09T08:49:27.780 INFO:journalctl@ceph.mds.b.smithi165.stdout:May 09 08:49:27 smithi165 bash[73986]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-b 2024-05-09T08:49:27.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.b.service' 2024-05-09T08:49:27.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:27.951 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-09T08:49:27.951 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-09T08:49:27.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e 2024-05-09T08:49:28.350 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:49:28 smithi165 systemd[1]: Stopping Ceph mds.e for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:28.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:49:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[69112]: 2024-05-09T08:49:28.267+0000 7fb414f22700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:28.351 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:49:28 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e[69112]: 2024-05-09T08:49:28.267+0000 7fb414f22700 -1 mds.e *** got signal Terminated *** 2024-05-09T08:49:31.466 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:49:31 smithi165 podman[74118]: 2024-05-09 08:49:31.137788617 +0000 UTC m=+2.967488923 container died b378486507542c3c301b47ecc43ce3b094681f92c2cacf02d2d6cf03009ebcec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS) 2024-05-09T08:49:31.466 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:49:31 smithi165 podman[74118]: 2024-05-09 08:49:31.467734812 +0000 UTC m=+3.297435125 container cleanup b378486507542c3c301b47ecc43ce3b094681f92c2cacf02d2d6cf03009ebcec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-09T08:49:31.850 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:49:31 smithi165 bash[74118]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e 2024-05-09T08:49:31.850 INFO:journalctl@ceph.mds.e.smithi165.stdout:May 09 08:49:31 smithi165 podman[74133]: 2024-05-09 08:49:31.669178981 +0000 UTC m=+0.530342289 container remove b378486507542c3c301b47ecc43ce3b094681f92c2cacf02d2d6cf03009ebcec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-e, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=squid-96805dc) 2024-05-09T08:49:31.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.e.service' 2024-05-09T08:49:31.921 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:31.922 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-09T08:49:31.922 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-09T08:49:31.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.h 2024-05-09T08:49:32.228 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:32 smithi165 systemd[1]: Stopping Ceph mds.h for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:32.600 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:49:32.228+0000 7facb05c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:32.601 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:32 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h[64300]: 2024-05-09T08:49:32.228+0000 7facb05c1700 -1 mds.h *** got signal Terminated *** 2024-05-09T08:49:35.404 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:35 smithi165 podman[74241]: 2024-05-09 08:49:35.140440182 +0000 UTC m=+2.998204513 container died 65dfbf08fba2f7180771c8a821bf5f4e6bc3fffdbb9c40833c1d1d8cf6b0a38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD) 2024-05-09T08:49:35.687 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:35 smithi165 podman[74241]: 2024-05-09 08:49:35.479721929 +0000 UTC m=+3.337486247 container cleanup 65dfbf08fba2f7180771c8a821bf5f4e6bc3fffdbb9c40833c1d1d8cf6b0a38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc) 2024-05-09T08:49:35.688 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:35 smithi165 bash[74241]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h 2024-05-09T08:49:35.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.h.service' 2024-05-09T08:49:35.938 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:35 smithi165 podman[74255]: 2024-05-09 08:49:35.689263563 +0000 UTC m=+0.545868803 container remove 65dfbf08fba2f7180771c8a821bf5f4e6bc3fffdbb9c40833c1d1d8cf6b0a38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-h, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:49:35.938 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:35 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.h.service: Deactivated successfully. 2024-05-09T08:49:35.939 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:35 smithi165 systemd[1]: Stopped Ceph mds.h for 3891b212-0ddc-11ef-bc98-c7b262605968. 2024-05-09T08:49:35.939 INFO:journalctl@ceph.mds.h.smithi165.stdout:May 09 08:49:35 smithi165 systemd[1]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.h.service: Consumed 5min 8.244s CPU time. 2024-05-09T08:49:35.949 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:35.949 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-09T08:49:35.950 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-09T08:49:35.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.k 2024-05-09T08:49:36.350 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:49:36 smithi165 systemd[1]: Stopping Ceph mds.k for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:36.850 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:49:36 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k[63660]: 2024-05-09T08:49:36.390+0000 7f523f9e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:36.851 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:49:36 smithi165 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k[63660]: 2024-05-09T08:49:36.390+0000 7f523f9e9700 -1 mds.k *** got signal Terminated *** 2024-05-09T08:49:39.440 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:49:39 smithi165 podman[74362]: 2024-05-09 08:49:39.138054285 +0000 UTC m=+2.966445102 container died b45a185d7e34eded395dd00a51a82d988f4aa6877bfbad5169cc023479d8da13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-09T08:49:39.441 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:49:39 smithi165 podman[74362]: 2024-05-09 08:49:39.441872936 +0000 UTC m=+3.270263751 container cleanup b45a185d7e34eded395dd00a51a82d988f4aa6877bfbad5169cc023479d8da13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:49:39.691 INFO:journalctl@ceph.mds.k.smithi165.stdout:May 09 08:49:39 smithi165 bash[74362]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-k 2024-05-09T08:49:39.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.k.service' 2024-05-09T08:49:39.837 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:39.837 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-09T08:49:39.838 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-09T08:49:39.838 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.c 2024-05-09T08:49:40.352 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:49:39 smithi184 systemd[1]: Stopping Ceph mds.c for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:40.606 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:49:40 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:49:40.352+0000 7f0c8df86700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:40.606 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:49:40 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c[65070]: 2024-05-09T08:49:40.352+0000 7f0c8df86700 -1 mds.c *** got signal Terminated *** 2024-05-09T08:49:43.411 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:49:43 smithi184 podman[73892]: 2024-05-09 08:49:43.138904356 +0000 UTC m=+3.104016762 container died 16b205826f39c8598752120a1df1dc78987f74579ca206c84e797ed771ada3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-09T08:49:43.684 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:49:43 smithi184 podman[73892]: 2024-05-09 08:49:43.485275592 +0000 UTC m=+3.450387984 container cleanup 16b205826f39c8598752120a1df1dc78987f74579ca206c84e797ed771ada3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, ceph=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:49:43.685 INFO:journalctl@ceph.mds.c.smithi184.stdout:May 09 08:49:43 smithi184 bash[73892]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-c 2024-05-09T08:49:43.906 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.c.service' 2024-05-09T08:49:43.940 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:43.940 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-09T08:49:43.940 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-09T08:49:43.940 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.f 2024-05-09T08:49:44.313 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:49:44 smithi184 systemd[1]: Stopping Ceph mds.f for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:44.606 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:49:44 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:49:44.314+0000 7fc5b9010700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:44.606 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:49:44 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f[64769]: 2024-05-09T08:49:44.314+0000 7fc5b9010700 -1 mds.f *** got signal Terminated *** 2024-05-09T08:49:47.488 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:49:47 smithi184 podman[74025]: 2024-05-09 08:49:47.149057504 +0000 UTC m=+2.988611710 container died ac05c3efc91d257756575b8fc73362196f0995a10b670254834d0cbfe43f2742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:49:47.488 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:49:47 smithi184 podman[74025]: 2024-05-09 08:49:47.488922653 +0000 UTC m=+3.328476857 container cleanup ac05c3efc91d257756575b8fc73362196f0995a10b670254834d0cbfe43f2742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:49:47.856 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:49:47 smithi184 bash[74025]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f 2024-05-09T08:49:47.856 INFO:journalctl@ceph.mds.f.smithi184.stdout:May 09 08:49:47 smithi184 podman[74037]: 2024-05-09 08:49:47.665106005 +0000 UTC m=+0.517780858 container remove ac05c3efc91d257756575b8fc73362196f0995a10b670254834d0cbfe43f2742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, ceph=True, io.buildah.version=1.33.5) 2024-05-09T08:49:47.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.f.service' 2024-05-09T08:49:47.950 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:47.950 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-09T08:49:47.951 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-09T08:49:47.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.i 2024-05-09T08:49:48.274 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:49:48 smithi184 systemd[1]: Stopping Ceph mds.i for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:48.606 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:49:48 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:49:48.274+0000 7f94c1fd0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:48.606 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:49:48 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i[65393]: 2024-05-09T08:49:48.274+0000 7f94c1fd0700 -1 mds.i *** got signal Terminated *** 2024-05-09T08:49:51.356 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:49:51 smithi184 podman[74143]: 2024-05-09 08:49:51.16366023 +0000 UTC m=+2.990294173 container died 5514586d1cef6e2b9a420749fe873b47fdafda8b1c96cbb3e906a3d3f55ae89b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-09T08:49:51.783 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:49:51 smithi184 podman[74143]: 2024-05-09 08:49:51.533976442 +0000 UTC m=+3.360610393 container cleanup 5514586d1cef6e2b9a420749fe873b47fdafda8b1c96cbb3e906a3d3f55ae89b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-09T08:49:51.784 INFO:journalctl@ceph.mds.i.smithi184.stdout:May 09 08:49:51 smithi184 bash[74143]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-i 2024-05-09T08:49:51.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.i.service' 2024-05-09T08:49:51.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:51.937 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-09T08:49:51.937 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-09T08:49:51.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.l 2024-05-09T08:49:52.106 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:49:52 smithi184 systemd[1]: Stopping Ceph mds.l for 3891b212-0ddc-11ef-bc98-c7b262605968... 2024-05-09T08:49:52.606 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:49:52 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l[64458]: 2024-05-09T08:49:52.262+0000 7f4038b35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:52.606 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:49:52 smithi184 ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l[64458]: 2024-05-09T08:49:52.262+0000 7f4038b35700 -1 mds.l *** got signal Terminated *** 2024-05-09T08:49:55.420 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:49:55 smithi184 podman[74265]: 2024-05-09 08:49:55.141160935 +0000 UTC m=+2.987650580 container died e8df3acc9d5d430db04757cf259344a3f00a4995bc00f57baed7ff392edb930c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-09T08:49:55.778 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:49:55 smithi184 podman[74265]: 2024-05-09 08:49:55.50436326 +0000 UTC m=+3.350852905 container cleanup e8df3acc9d5d430db04757cf259344a3f00a4995bc00f57baed7ff392edb930c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-09T08:49:55.778 INFO:journalctl@ceph.mds.l.smithi184.stdout:May 09 08:49:55 smithi184 bash[74265]: ceph-3891b212-0ddc-11ef-bc98-c7b262605968-mds-l 2024-05-09T08:49:55.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3891b212-0ddc-11ef-bc98-c7b262605968@mds.l.service' 2024-05-09T08:49:55.959 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:55.959 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-09T08:49:55.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T08:49:56.162 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:49:58.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T08:49:58.979 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:50:01.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T08:50:01.813 INFO:teuthology.orchestra.run.smithi184.stdout:Deleting cluster with fsid: 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T08:50:04.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:04.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:04.282 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:50:04.308 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-09T08:50:04.310 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/remote/smithi028/crash 2024-05-09T08:50:04.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/crash -- . 2024-05-09T08:50:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T08:50:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-05-09T08:50:04.344 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/remote/smithi165/crash 2024-05-09T08:50:04.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/crash -- . 2024-05-09T08:50:04.383 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T08:50:04.383 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now 2024-05-09T08:50:04.385 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/remote/smithi184/crash 2024-05-09T08:50:04.386 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/crash -- . 2024-05-09T08:50:04.425 INFO:teuthology.orchestra.run.smithi184.stderr:tar: /var/lib/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T08:50:04.425 INFO:teuthology.orchestra.run.smithi184.stderr:tar: Error is not recoverable: exiting now 2024-05-09T08:50:04.426 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-09T08:50:04.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-09T08:50:04.514 INFO:tasks.cephadm:Compressing logs... 2024-05-09T08:50:04.515 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T08:50:04.517 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T08:50:04.519 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T08:50:04.558 INFO:teuthology.orchestra.run.smithi184.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-09T08:50:04.558 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T08:50:04.559 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log 2024-05-09T08:50:04.559 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.2.log 2024-05-09T08:50:04.559 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.f.log 2024-05-09T08:50:04.559 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T08:50:04.559 INFO:teuthology.orchestra.run.smithi184.stderr:gzip/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.f.log: -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.5.log 2024-05-09T08:50:04.559 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.c.log 2024-05-09T08:50:04.561 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mon.c.log 2024-05-09T08:50:04.561 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mgr.z.log 2024-05-09T08:50:04.561 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.c.log: /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.i.log 2024-05-09T08:50:04.562 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T08:50:04.562 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.11.log 2024-05-09T08:50:04.562 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.8.log 2024-05-09T08:50:04.562 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T08:50:04.562 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.l.log 2024-05-09T08:50:04.563 INFO:teuthology.orchestra.run.smithi165.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-09T08:50:04.563 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T08:50:04.563 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log 2024-05-09T08:50:04.567 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T08:50:04.567 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.7.log 2024-05-09T08:50:04.567 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.b.log 2024-05-09T08:50:04.567 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.4.log 2024-05-09T08:50:04.568 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.1.log 2024-05-09T08:50:04.568 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mgr.y.log 2024-05-09T08:50:04.568 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.h.log 2024-05-09T08:50:04.568 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.10.log 2024-05-09T08:50:04.568 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T08:50:04.568 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.e.log 2024-05-09T08:50:04.568 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T08:50:04.568 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mon.b.log 2024-05-09T08:50:04.568 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.k.log 2024-05-09T08:50:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T08:50:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:: No such file or directory 2024-05-09T08:50:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log 2024-05-09T08:50:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T08:50:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.g.log 2024-05-09T08:50:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log: /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.j.log 2024-05-09T08:50:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.3.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.0.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mgr.x.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mon.a.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.6.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.a.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.d.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.9.log 2024-05-09T08:50:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-09T08:50:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-09T08:50:04.576 INFO:teuthology.orchestra.run.smithi028.stderr: 90.7% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T08:50:04.599 INFO:teuthology.orchestra.run.smithi028.stderr: 84.1% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T08:50:04.673 INFO:teuthology.orchestra.run.smithi028.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T08:50:04.674 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mon.b.log: /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.k.log: 83.9% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T08:50:04.685 INFO:teuthology.orchestra.run.smithi028.stderr: 90.1% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T08:50:04.698 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.l.log: 83.8% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T08:50:04.772 INFO:teuthology.orchestra.run.smithi165.stderr: 92.5% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mgr.y.log.gz 2024-05-09T08:50:04.796 INFO:teuthology.orchestra.run.smithi184.stderr: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T08:50:04.799 INFO:teuthology.orchestra.run.smithi028.stderr: 94.4% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.j.log.gz 2024-05-09T08:50:04.802 INFO:teuthology.orchestra.run.smithi184.stderr: 90.3% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T08:50:04.810 INFO:teuthology.orchestra.run.smithi184.stderr: 92.6% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mgr.z.log.gz 2024-05-09T08:50:04.810 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T08:50:04.830 INFO:teuthology.orchestra.run.smithi165.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T08:50:04.848 INFO:teuthology.orchestra.run.smithi165.stderr: 90.8% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T08:50:04.852 INFO:teuthology.orchestra.run.smithi165.stderr: 90.3% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T08:50:04.910 INFO:teuthology.orchestra.run.smithi165.stderr: 91.0% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T08:50:04.961 INFO:teuthology.orchestra.run.smithi184.stderr: 91.0% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T08:50:05.091 INFO:teuthology.orchestra.run.smithi184.stderr: 90.7% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T08:50:09.257 INFO:teuthology.orchestra.run.smithi028.stderr: 90.5% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mgr.x.log.gz 2024-05-09T08:50:12.879 INFO:teuthology.orchestra.run.smithi028.stderr: 95.4% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.g.log.gz 2024-05-09T08:50:13.954 INFO:teuthology.orchestra.run.smithi165.stderr: 91.9% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mon.b.log.gz 2024-05-09T08:50:28.799 INFO:teuthology.orchestra.run.smithi184.stderr: 91.8% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mon.c.log.gz 2024-05-09T08:50:29.093 INFO:teuthology.orchestra.run.smithi028.stderr: 90.1% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mon.a.log.gz 2024-05-09T08:51:38.035 INFO:teuthology.orchestra.run.smithi165.stderr: 95.5% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.e.log.gz 2024-05-09T08:52:09.251 INFO:teuthology.orchestra.run.smithi028.stderr: 95.0% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.a.log.gz 2024-05-09T08:52:26.566 INFO:teuthology.orchestra.run.smithi028.stderr: 94.8% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.d.log.gz 2024-05-09T08:54:43.889 INFO:teuthology.orchestra.run.smithi165.stderr: 93.7% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.h.log.gz 2024-05-09T08:54:50.847 INFO:teuthology.orchestra.run.smithi165.stderr: 95.1% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.k.log.gz 2024-05-09T08:55:31.521 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.0.log.gz 2024-05-09T08:56:47.617 INFO:teuthology.orchestra.run.smithi184.stderr: 95.3% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.l.log.gz 2024-05-09T08:59:44.236 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.6.log.gz 2024-05-09T09:01:15.525 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.3.log.gz 2024-05-09T09:01:49.170 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.9.log.gz 2024-05-09T09:01:49.172 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-05-09T09:01:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:real 11m44.642s 2024-05-09T09:01:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:user 7m43.014s 2024-05-09T09:01:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m17.658s 2024-05-09T09:06:57.660 INFO:teuthology.orchestra.run.smithi165.stderr: 91.3% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.10.log.gz 2024-05-09T09:11:07.916 INFO:teuthology.orchestra.run.smithi165.stderr: 91.1% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.1.log.gz 2024-05-09T09:14:17.922 INFO:teuthology.orchestra.run.smithi165.stderr: 91.0% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.7.log.gz 2024-05-09T09:16:04.199 INFO:teuthology.orchestra.run.smithi184.stderr: 91.2% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.8.log.gz 2024-05-09T09:16:10.307 INFO:teuthology.orchestra.run.smithi165.stderr: 91.2% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.4.log.gz 2024-05-09T09:17:12.920 INFO:teuthology.orchestra.run.smithi165.stderr: 93.5% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.b.log.gz 2024-05-09T09:17:12.946 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-05-09T09:17:12.946 INFO:teuthology.orchestra.run.smithi165.stderr:real 27m8.412s 2024-05-09T09:17:12.947 INFO:teuthology.orchestra.run.smithi165.stderr:user 10m18.785s 2024-05-09T09:17:12.947 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m28.572s 2024-05-09T09:18:05.009 INFO:teuthology.orchestra.run.smithi184.stderr: 91.2% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.11.log.gz 2024-05-09T09:18:11.588 INFO:teuthology.orchestra.run.smithi184.stderr: 93.6% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.c.log.gz 2024-05-09T09:19:57.187 INFO:teuthology.orchestra.run.smithi184.stderr: 93.6% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.i.log.gz 2024-05-09T09:20:00.158 INFO:teuthology.orchestra.run.smithi184.stderr: 91.2% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.2.log.gz 2024-05-09T09:21:12.331 INFO:teuthology.orchestra.run.smithi184.stderr: 93.4% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-mds.f.log.gz 2024-05-09T09:21:31.229 INFO:teuthology.orchestra.run.smithi184.stderr: 91.0% -- replaced with /var/log/ceph/3891b212-0ddc-11ef-bc98-c7b262605968/ceph-osd.5.log.gz 2024-05-09T09:21:31.231 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-05-09T09:21:31.232 INFO:teuthology.orchestra.run.smithi184.stderr:real 31m26.698s 2024-05-09T09:21:31.232 INFO:teuthology.orchestra.run.smithi184.stderr:user 11m25.958s 2024-05-09T09:21:31.232 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m30.821s 2024-05-09T09:21:31.232 INFO:tasks.cephadm:Archiving logs... 2024-05-09T09:21:31.234 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/remote/smithi028/log 2024-05-09T09:21:31.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T09:22:50.150 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/remote/smithi165/log 2024-05-09T09:22:50.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T09:24:35.337 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/remote/smithi184/log 2024-05-09T09:24:35.339 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T09:26:30.663 INFO:tasks.cephadm:Removing cluster... 2024-05-09T09:26:30.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 --force 2024-05-09T09:26:30.883 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T09:26:31.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 --force 2024-05-09T09:26:32.459 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T09:26:33.404 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3891b212-0ddc-11ef-bc98-c7b262605968 --force 2024-05-09T09:26:33.878 INFO:teuthology.orchestra.run.smithi184.stdout:Deleting cluster with fsid: 3891b212-0ddc-11ef-bc98-c7b262605968 2024-05-09T09:26:34.828 INFO:tasks.cephadm:Removing cephadm ... 2024-05-09T09:26:34.829 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T09:26:34.849 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T09:26:34.869 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T09:26:34.884 INFO:tasks.cephadm:Teardown complete 2024-05-09T09:26:34.884 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-09T09:26:34.898 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-09T09:26:34.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T09:26:34.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T09:26:34.911 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T09:26:35.067 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-09T09:26:35.078 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-09T09:26:35.078 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:26:35.081 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:26:35.083 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:26:35.096 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:26:35.097 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:26:35.098 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:26:35.123 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:26:35.123 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-05-09T09:26:35.123 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 7 377 94 +409us[ +468us] +/- 44ms 2024-05-09T09:26:35.123 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -850us[ -850us] +/- 65ms 2024-05-09T09:26:35.123 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 100 +349us[ +407us] +/- 65ms 2024-05-09T09:26:35.123 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:26:35.125 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:26:35.125 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-05-09T09:26:35.125 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 35 +385us[ +385us] +/- 45ms 2024-05-09T09:26:35.125 INFO:teuthology.orchestra.run.smithi184.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -816us[ -788us] +/- 64ms 2024-05-09T09:26:35.125 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +362us[ +362us] +/- 66ms 2024-05-09T09:26:35.125 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:26:35.176 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:26:35.176 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-05-09T09:26:35.176 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 35 +377us[ +377us] +/- 45ms 2024-05-09T09:26:35.176 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -833us[ -832us] +/- 64ms 2024-05-09T09:26:35.176 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +340us[ +340us] +/- 66ms 2024-05-09T09:26:35.176 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:26:35.188 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-09T09:26:35.199 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-09T09:26:35.199 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-09T09:26:35.230 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T09:26:35.381 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T09:26:35.500 DEBUG:teuthology.orchestra.run.smithi184:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T09:26:35.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:26:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:35.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:26:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:35.661 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:26:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:35.689 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-09T09:26:35.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-09T09:26:35.747 INFO:teuthology.task.internal:Duration was 4709.842355 seconds 2024-05-09T09:26:35.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-09T09:26:35.794 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-09T09:26:35.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T09:26:35.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T09:26:35.800 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T09:26:35.853 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:26:35.854 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:26:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:26:36.196 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-09T09:26:36.196 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-05-09T09:26:36.196 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T09:26:36.231 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-05-09T09:26:36.231 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T09:26:36.257 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2024-05-09T09:26:36.258 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T09:26:36.291 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-09T09:26:36.291 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T09:26:36.293 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T09:26:36.300 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T09:26:36.359 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-09T09:26:36.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T09:26:36.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T09:26:36.375 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T09:26:36.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-09T09:26:36.816 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-09T09:26:36.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T09:26:36.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T09:26:36.883 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T09:26:36.918 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-09T09:26:36.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:36.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:36.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:36.970 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-05-09T09:26:36.977 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-05-09T09:26:36.991 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2024-05-09T09:26:37.017 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:37.039 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:37.053 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:37.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-09T09:26:37.081 INFO:teuthology.task.internal:Transferring archived files... 2024-05-09T09:26:37.082 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/remote/smithi028 2024-05-09T09:26:37.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T09:26:37.127 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/remote/smithi165 2024-05-09T09:26:37.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T09:26:37.164 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662/remote/smithi184 2024-05-09T09:26:37.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T09:26:37.200 INFO:teuthology.task.internal:Removing archive directory... 2024-05-09T09:26:37.201 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T09:26:37.203 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T09:26:37.205 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T09:26:37.252 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-09T09:26:37.266 INFO:teuthology.task.internal:Not uploading archives. 2024-05-09T09:26:37.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-09T09:26:37.291 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-09T09:26:37.291 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T09:26:37.294 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T09:26:37.297 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T09:26:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: 265566 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 09:26 /home/ubuntu/cephtest 2024-05-09T09:26:37.310 INFO:teuthology.orchestra.run.smithi165.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 09:26 /home/ubuntu/cephtest 2024-05-09T09:26:37.311 INFO:teuthology.orchestra.run.smithi184.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 09:26 /home/ubuntu/cephtest 2024-05-09T09:26:37.312 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-09T09:26:37.332 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-09T09:26:37.443 INFO:teuthology.nuke:Checking targets against current locks 2024-05-09T09:26:37.469 DEBUG:teuthology.nuke:shortname: smithi028 2024-05-09T09:26:37.470 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T09:26:37.492 DEBUG:teuthology.nuke:shortname: smithi165 2024-05-09T09:26:37.492 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T09:26:37.497 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 07:59:54.855068', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMHBIq3GGshNY65sHkAumqYoJbrsfwbEG/PpJI39TKdBB57T+2C8fuvrUiMY00zvBV0r+7ih4ruPVkXX/XM0GKg='} 2024-05-09T09:26:37.513 DEBUG:teuthology.nuke:shortname: smithi184 2024-05-09T09:26:37.513 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T09:26:37.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 07:59:54.856495', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLz4MksoomatCa3JC/zdLKZkqG/7hq+3vKxPXA7GPD3W6Dun7A2J9yM9Jw0rPExTwHSz7kQDpN3SB/4XhxHbr7k='} 2024-05-09T09:26:37.535 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698662', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 07:59:54.853376', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpjG8knIJPj/tgexreKisoANPHcYa8Xpu229akHay7Z5wPPa9xwlwivgLqePebWBYqHxev4I2pfLemqRCDMQys='} 2024-05-09T09:26:37.544 INFO:teuthology.orchestra.console.smithi028:Power off 2024-05-09T09:26:37.544 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T09:26:37.568 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-05-09T09:26:37.568 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T09:26:37.586 INFO:teuthology.orchestra.console.smithi165:Power off 2024-05-09T09:26:37.586 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T09:26:37.599 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-05-09T09:26:37.610 DEBUG:teuthology.orchestra.console.smithi165:power off output: Chassis Power Control: Down/Off 2024-05-09T09:26:37.610 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T09:26:37.630 INFO:teuthology.orchestra.console.smithi184:Power off 2024-05-09T09:26:37.630 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T09:26:37.640 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-05-09T09:26:37.656 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-05-09T09:26:37.656 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T09:26:37.682 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-05-09T09:26:41.600 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T09:26:41.640 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T09:26:41.683 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T09:26:41.712 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-05-09T09:26:41.752 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-05-09T09:26:41.793 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-05-09T09:26:45.717 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T09:26:45.752 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T09:26:45.793 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T09:26:45.831 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-05-09T09:26:45.863 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is off 2024-05-09T09:26:45.903 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-05-09T09:26:45.932 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-05-09T09:26:45.963 INFO:teuthology.orchestra.console.smithi165:Power off completed 2024-05-09T09:26:46.004 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-05-09T09:26:46.109 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} duration: 4709.842354774475 failure_reason: 'error during scrub thrashing: rank damage found: {''backtrace''}' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-05-09T09:26:46.109 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T09:26:46.196 INFO:teuthology.run:FAIL